Unmanned ship path planning method and system based on evolutionary reinforcement learning under the influence of wind wave and medium

Abstract

The application relates to the technical field of autonomous navigation of unmanned ships, and discloses an unmanned ship path planning method and system based on evolutionary reinforcement learning under the influence of wind and waves and a medium. The method comprises the following steps: acquiring the motion state of an unmanned ship and inputting the acquired motion state into a trained target strategy network to obtain a sailing decision action, combining wind and wave environment interference to perform dynamic state transition, and enabling the unmanned ship to sail from a starting position to a target position; and initializing a population with a strategy network tuple, performing evolutionary screening based on value network long-term return evaluation through crossover and mutation, and training a target strategy network. The application solves the problems of poor adaptability to dynamic wind and wave environments and lack of long-term sequence decision-making ability in the prior art.

Description

Technical Field

[0001] This invention belongs to the field of autonomous navigation technology for unmanned vessels, and specifically relates to a path planning method, system, and medium for unmanned vessels based on evolutionary reinforcement learning under the influence of wind and waves. Background Technology

[0002] With the continuous development of intelligent technologies, unmanned surface vessels (USVs) have demonstrated advantages such as autonomy, intelligence, high efficiency, and low risk in marine development engineering. Path planning, as one of the core technologies for autonomous navigation of USVs, directly affects navigation efficiency, safety, and application scope. However, existing technologies have the following problems: First, the static and idealized nature of environmental modeling does not adequately consider the coupling effects of dynamic and time-varying environments such as wind and waves in the real ocean. Second, there is a contradiction between the global optimization capability and real-time performance of the algorithm. In addition, existing methods rely on explicit mechanical formulas to model wind and wave environments, which is insufficient for adapting to highly dynamic and non-uniform environments such as changing wind directions and irregular waves.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] To address or at least mitigate one or more of the above problems, a method, system, and medium for unmanned surface vessel (USV) path planning based on evolutionary reinforcement learning under the influence of wind and waves are provided. The method comprises an application phase and a model training phase. In the application phase, a trained target policy network outputs navigation decisions in real time based on the USV's motion state. This is combined with a dynamics module incorporating wind and wave environmental interference to perform state transitions, enabling the USV to navigate along the planned path from its starting position to its target position. In the model training phase, the target policy network is trained through joint evolutionary optimization using a genetic algorithm and a deep deterministic policy gradient network.

[0005] To achieve the above objectives, according to a first aspect of the present invention, a path planning method for unmanned vessels based on evolutionary reinforcement learning under the influence of wind and waves is provided, comprising the following steps: S1. Obtain the motion state of the unmanned vessel at the current moment; S2. Input the motion state into the target policy network to obtain navigation decision actions; The policy network corresponding to the target policy network is trained through the following steps: The population is initialized using tuples of the policy network and the corresponding target policy network as population individuals; Perform crossover and mutation operations on the policy network in the population to generate new individuals and insert them into the population; Using the path planning task of an unmanned vessel navigating from its starting position to its target position under wind and wave interference as the evaluation scenario, the value network in the deep deterministic policy gradient network is used to complete the individual preparation through experience caching, network parameter updates and soft updates within a preset preparation round. Then, the Q value of each individual at each time step within the evaluation time is accumulated to obtain the cumulative evaluation value. Individuals that can complete the path planning task under wind and wave interference are selected to enter the next generation. Repeat the above training steps, and obtain the target policy network by soft updating the policy network of the best individual obtained at the end; S3. Based on the navigation decision action and the disturbance of wind and waves, perform dynamic state transition to obtain the motion state of the unmanned vessel at the next moment. Repeat steps S1-S3 to make the unmanned vessel travel along the planned path from the starting position to the target position.

[0006] To achieve the above objectives, according to a second aspect of the present invention, an unmanned surface vessel (USV) path planning system based on evolutionary reinforcement learning under the influence of wind and waves is provided, the USV path planning system comprising: The state acquisition module is used to acquire the motion state of the unmanned vessel at the current moment; The decision-making module is used to input the motion state into the trained target policy network to obtain navigation decision actions; The training module is used to train the target policy network. The genetic module is used to initialize the population using tuples from the policy network and the corresponding target policy network as individuals. Perform crossover and mutation operations on the policy network in the population to generate new individuals and insert them into the population; The Deep Deterministic Policy Gradient Network module is used to evaluate the path planning task of an unmanned vessel from its starting position to its target position. It utilizes the value network in the Deep Deterministic Policy Gradient Network to complete individual preparation through experience caching, network parameter updates, and soft updates within a preset preparation round. Then, it accumulates the Q-values ​​of each individual at each time step within the evaluation time to obtain the cumulative evaluation value. Individuals that can complete the path planning task under wind and wave interference are selected to enter the next generation. Repeat the above training steps, and obtain the target policy network by soft updating the policy network of the best individual obtained at the end; The dynamics module is used to perform dynamic state transitions based on the navigation decision actions and wind and wave environmental disturbances, so as to obtain the motion state of the unmanned vessel at the next moment. The control module is used to repeatedly trigger the state acquisition module, decision module and state transition module, so that the unmanned vessel can sail from the starting position to the target position along the planned path.

[0007] To achieve the above objectives, according to a third aspect of the present invention, a computer-readable storage medium is provided storing a computer program, which, when executed by a processor, is used to implement the unmanned vessel path planning method based on evolutionary reinforcement learning under the influence of wind and waves as described above.

[0008] By adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art: This invention proposes a joint optimization mechanism within a framework integrating a genetic algorithm module and a deep deterministic policy gradient network module. This mechanism uses the policy network population as evolving individuals and value network evaluation as fitness signals. The policy network directly outputs actions, adaptively learning effective decisions under wind and wave disturbances without relying on explicit environmental force modeling. This addresses the shortcomings of previous methods that relied on empirical formulas and simplified wave spectra for environmental modeling, and lacked adaptability to highly dynamic and non-uniform environments such as changing winds and irregular waves. At the decision evaluation level, a value network evaluation system based on long-term cumulative returns is introduced. The cumulative evaluation value used for individual evaluation... The target value network is recursively accumulated along the entire evaluation time series, which expands the optimization objective from the single-step evaluation function of the original method to a multi-step long-term return, solving the problems of the original method being based on single-step rolling optimization, lacking long-term sequence decision-making ability, and being unable to balance global path quality and real-time performance.

[0009] The specific embodiments of the present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0010] The accompanying drawings, as part of this invention, are provided to further illustrate the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention, but do not constitute an undue limitation thereof. Clearly, the drawings described below are merely some embodiments; those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0011] In the attached diagram: Figure 1 This is a flowchart illustrating the path planning method for unmanned vessels based on evolutionary reinforcement learning under the influence of wind and waves in this specific embodiment. Figure 2 This is a schematic diagram illustrating the interaction process between the genetic algorithm module, the deep deterministic policy gradient network module, and the dynamics module in this specific embodiment. Figure 3 This is a comparison chart of the path planning visualization results in this specific implementation method; Figure 4 This is a schematic diagram of the unmanned vessel path planning system based on evolutionary reinforcement learning under the influence of wind and waves in this specific embodiment. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments will be clearly and completely described below with reference to the accompanying drawings. The following embodiments are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0013] Please see Figure 1 and Figure 2 This invention provides a method for unmanned surface vessel path planning based on evolutionary reinforcement learning under the influence of wind and waves, comprising the following steps: S1. Obtain the motion state of the unmanned vessel at the current moment; S2. Input the motion state into the target policy network to obtain navigation decision actions; The policy network corresponding to the target policy network is trained through the following steps: The population is initialized using tuples of the policy network and the corresponding target policy network as population individuals; Perform crossover and mutation operations on the policy network in the population to generate new individuals and insert them into the population; Using the path planning task of an unmanned vessel navigating from its starting position to its target position under wind and wave interference as the evaluation scenario, the value network in the deep deterministic policy gradient network is used to complete the individual preparation through experience caching, network parameter updates and soft updates within a preset preparation round. Then, the Q value of each individual at each time step within the evaluation time is accumulated to obtain the cumulative evaluation value. Individuals that can complete the path planning task under wind and wave interference are selected to enter the next generation. Repeat the above training steps, and obtain the target policy network by soft updating the policy network of the best individual obtained at the end; S3. Based on the navigation decision action and the disturbance of wind and waves, perform dynamic state transition to obtain the motion state of the unmanned vessel at the next moment. Repeat steps S1-S3 to make the unmanned vessel travel along the planned path from the starting position to the target position.

[0014] It should be noted that, in this embodiment, the execution entity of the unmanned surface vessel (USV) path planning method based on evolutionary reinforcement learning under the influence of wind and waves is an USV path planning device based on evolutionary reinforcement learning under the influence of wind and waves. This device can be an electronic device, a component of an electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, vehicle-mounted electronic devices, wearable devices, etc., while non-mobile electronic devices can be servers and personal computers, etc., and this invention does not impose specific limitations. The following description uses a server as the execution entity as an example to illustrate the USV path planning method based on evolutionary reinforcement learning under the influence of wind and waves in this embodiment.

[0015] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0016] Model architecture: The model includes a genetic algorithm module, a deep deterministic policy gradient network module, and a dynamics module. First, the genetic algorithm module forms the overall framework for the model's data processing. It uses tuples from the policy network and the target policy network... As an individual in the population, it continuously performs crossover, mutation, evaluation, and selection operations after population initialization until the evaluation results meet the requirements, and finally outputs the target policy network with the optimal evaluation results. The deep deterministic policy gradient module is used for the evaluation and selection parts of the genetic algorithm, serving as the algorithm's output. This module uses a tuple of a value network and a target value network. Evaluate individuals within the population and guide the strategy network. With value network Update the module. The dynamics module is used to convert the discrete action sequence arranged in unit time series given by the policy network into a continuous dynamic process, that is, the state transition process of the unmanned vessel according to the action sequence, which directly affects the results of individual evaluation and the practical application of the policy network.

[0017] Genetic Algorithm Module: The genetic algorithm module serves as the framework of this method, maintaining tuples of the policy network and the target policy network. As an individual in the population, it performs crossover, mutation, evaluation, and selection operations, ultimately outputting the optimal target policy network. This network can directly guide the path planning of unmanned vessels.

[0018] Policy Network and target policy network The network structures are exactly the same, consisting of 5-dimensional state input, hidden layers, and 2-dimensional action output. Any state in the 5-dimensional state space can be described by a 5-tuple, as shown below: (4.1); Among the 5 elements It is the position of the unmanned vessel in the geodetic coordinate system; It is the bow heading angle (the angle between the bow and the x-axis); It is the speed of travel along the bow direction; It is the angular velocity of the head about the center of mass.

[0019] Any state in a two-dimensional action space can be described by a tuple, as shown below: (4.2); Two of the elements are the linear acceleration of the unmanned boat. angular acceleration .

[0020] During the operation of the policy network, it is required that in each unit of time... After it ends, based on the current state Given an action (Action), which will be continuously executed within the next unit of time, the unmanned surface vessel (USV) will be given corresponding linear and angular accelerations. The hidden layer structure of the policy network is determined based on the continuity of the task's state space and action space, with the following criteria: a multi-layer fully connected feedforward network is used, typically with 2 to 3 hidden layers, and the number of neurons in each layer decreasing progressively from input to output. Specifically, for the 5-dimensional state input and 2-dimensional continuous action output in this embodiment, a two-layer hidden layer structure is preferred, with 256 and 128 neurons respectively; if the complexity of the task environment increases (e.g., increased obstacle density or more dynamic wind and wave models), it can be expanded to three layers, with 512, 256, and 128 neurons respectively. This structure selection criterion is based on the following considerations: a sufficiently wide first layer (256 or 512) to extract state features, followed by progressively compressed layers to learn action representations; and through comparative experiments, the average cumulative reward and training convergence speed of the policy network in the verification scenario are used as indicators to select the structure with the best performance. General policy network structures in reinforcement learning can be applied, but the above-mentioned criteria should be followed. This embodiment uses a three-layer fully connected feedforward neural network.

[0021] Before starting the genetic algorithm module, several individuals are initialized in the population. Subsequent crossover and mutation operations only involve the policy network. The crossover operator is denoted as... The meaning is for the population The parameters in the policy networks of the two individuals are randomly exchanged. The randomized strategies primarily consider uniform crossover and simulated binary crossover. The selection criteria are as follows: If the policy network parameter size is small (e.g., less than 1000 hidden layer neurons) and it is necessary to maintain the local search ability of the offspring within the range of the parent's parameters, simulated binary crossover (SBX) is preferred. After crossover, the distribution range of the offspring parameters is similar to that of the parent, which is beneficial for algorithm convergence. If the parameter size is large or the population diversity is insufficient, uniform crossover is preferred to explore new parameter combinations with a higher probability and enhance population diversity. In practice, a small-scale pre-experiment can be conducted: run the genetic algorithm for 20-50 generations using candidate crossover operators, compare the rate of improvement in average fitness and the final convergence value, and select the one with better performance as the final crossover method. This is also the method adopted in this embodiment. Note that the obtained result is inserted into the population as a new individual and will not overwrite the original. Two individuals. The mutation operator targets each gene locus (a parameter in the network) of the individual. The definition is as follows: (4.3); in, Representing network parameters The result of the mutation It is a random perturbation function. To control the distribution of perturbations, different distributions can be used for different types of mutations. After crossover and mutation are completed, individuals in the population are evaluated and screened.

[0022] Deep deterministic policy gradient network module: This module employs a deep deterministic policy gradient network approach from reinforcement learning, using a tuple of a value network and a target value network. This system generates Q-values ​​to guide network updates and also uses these Q-values ​​to evaluate and screen individuals. The network structure is exactly the same, and the receiving state is identical. and actions The corresponding 7-dimensional input outputs a 1-dimensional real value. , representing state Make an action The value of.

[0023] Before starting the evaluation and screening, set the hyperparameters: preparation rounds. Evaluation time The evaluation time It consists of several units of time Composed of [various components]. In this embodiment, the preparation rounds are set to n = 5, and the evaluation time T = 1000 (i.e., composed of 1000 unit time steps, each unit time step corresponding to an actual duration of 0.1 seconds, so the total evaluation time is 100 seconds). The above values ​​can be adjusted according to the complexity of the task scenario. For example, in a larger water area or under stronger wind and wave interference, n can be appropriately increased to 10 and T to 2000 to ensure the sufficiency of the evaluation. In addition, a quadruple experience is defined. as follows: (4.4); in, It refers to the current state of an individual. That is, the action output by an individual's policy network based on its current state. That is According to the action The state after the transfer.

[0024] Define reward function as follows: (4.5a); (4.5b); (4.5c); (4.5d); (4.5e); in, It's a reward for getting closer to the goal. It's a distance traveled reward. It is a smoothness penalty. It is the distance between this decision and the goal. This represents the distance to the goal after this decision. This is the bow angle of the unmanned vessel prior to this decision. This refers to the bow angle of the unmanned vessel after the decision is made. Rewards indicating a terminated state.

[0025] If this decision does not lead to the termination of the mission, the corresponding reward will be: It consists of three parts. It is a reward for being close to the target, which takes effect when the distance to the target is less than 100 units, and the closer you are to the target, the greater the reward. It's a distance traveled reward. It is the distance between this decision and the goal. This represents the distance to the goal after this decision. A reward is given when the decision brings the person closer to the goal; otherwise, a penalty is imposed. It is a smoothness penalty. This is the bow angle of the unmanned vessel prior to this decision. This refers to the heading angle of the unmanned vessel after the decision is made.

[0026] In general, when the distance to the target is relatively far, by The system guides the unmanned vessel towards the target, but with small rewards and penalties, thus encouraging exploration of decision-making; while when the vessel is close to the target, the system... Guide the unmanned vessel to quickly approach the target; at the same time It is always active and has a high penalty value to prevent the unmanned vessel from making unreasonable turning decisions, thus reflecting realistic situations. If this decision leads to mission termination, reaching the target location, or colliding with an obstacle, the corresponding reward is... Successfully reaching the goal will earn a reward of 800 points, while failure will result in a penalty of -400 points.

[0027] Define an experience cache collection It is used to preserve experience.

[0028] Consider a specific individual The assessment, assuming the assessment time The corresponding unit time series is For any unit of time At the beginning, the individual's current state is considered as Actions are generated using individual policy networks. The corresponding reward will be given. And use the dynamics module to complete the state transition to obtain The experience gained from this decision will be cached. Repeat this process until every unit of time in the time series has been processed. Then, retrieve the experience cache from that individual. Several empirical data points are uniformly and randomly sampled, the number of which is determined by the hyperparameter. Determined. Using a certain empirical sample. For value network and target value network tuples and individuals Policy network and target policy network The update is performed. The process is as follows: First, use the target value network. and target policy network Estimate the value of the target state .

[0029] (4.6); in, This is a discount factor, typically set to 0.99 for continuous control tasks. Indicates the state In action The state after the down-transfer.

[0030] Then use value networks Estimate state value : (4.7); Repeat this process until all the experience from this sampling has been processed. Let's assume the batch size of the samples used for training is... In this embodiment, it is set to ( Next, we calculate the value network loss function. And update the value network. Among them... is a hyperparameter representing the learning rate, with an initial value of 0.01, and adaptive decay controlled by the optimizer.

[0031] (4.8); (4.9); in, This indicates the specific number of samples used in the training batch. Corresponding unit time , The loss function is the gradient of the value network.

[0032] Using all the gradients sampled earlier, and the updated value network, update the policy network using gradient ascent. (Policy network gradient ascent) The estimation is performed using the following formula: (4.10); (4.11); in, This indicates that the value network outputs information about actions. gradient, Represents an individual The policy network outputs the gradient with respect to its own parameters.

[0033] Next, we will discuss the target policy network. and target value network Perform a soft update: (4.12); (4.13); Among them, by hyperparameters To control the update level, all values ​​are set to 0.001.

[0034] This series of operations is considered a round of preparation before an individual's formal evaluation. When the number of preparation rounds reaches... Next, a formal evaluation and screening process will be conducted. All individuals will be given the same initial conditions. During the evaluation time Within, the individual's goal-policy network is utilized at each unit of time. Complete the decision-making process and utilize the target value network. Estimate the value of this decision in this state and accumulate it. Still assume the evaluation time includes... The cumulative assessed value per unit time is The calculation formula is shown below.

[0035] (4.14); Finally, based on the assessment value of each individual... Sort and filter the top results name( This is an adjustable hyperparameter, which can be set to the original number of individuals in the population, thus completing one operation of the genetic algorithm module. It checks whether the individual currently ranked first can complete the task within a specified time (e.g., the evaluation time). The algorithm provides a series of decisions to help the unmanned surface vessel successfully reach its target state from the initial state. If the target state is reached, the algorithm terminates, and the individual's target policy network is taken as the result. Output the result. Otherwise, the genetic algorithm module continues working.

[0036] Dynamics module: This module primarily completes tasks based on a given state. and actions The state transition is completed to obtain the new state. The operation is as follows. Due to the short unit time, even with external forces, it can be assumed that the ship's acceleration and angular acceleration are approximately constant within a time step, using uniform acceleration kinematics integrals. This part uniformly records the time step corresponding to each unit time as... The state transition process is as follows: (4.15a); (4.15b); (4.15c); (4.15d); (4.15e); (4.15f); in, For water resistance acceleration, This is the projection of the resultant acceleration of the wind and waves acting on the bow direction. and Right now The given linear acceleration and angular acceleration per unit time, It is the total linear acceleration. and These are the angular velocity and linear velocity at the beginning of a unit of time. and These are the angular velocity and linear velocity at the end of a unit of time; and The coordinates at the start time have been determined. and These are the coordinates of the end time; It is the heading angle at the initial moment. It is the heading angle at the end of the time. This represents the specific time of a time step; the entire system of equations describes how the current state... and actions Given the current state and its surrounding environment, give the next state. .

[0037] Time step At the end of the time, the new state of the unmanned ship is: ; for and The calculation method for these resistances is defined below, using Froude's resistance theory to divide the total ship resistance into two parts: frictional resistance and residual resistance, and simplifying it into a form proportional to the square of the velocity: (4.16); in, Indicates the total resistance of the ship. It is the density of water. It is the wetted surface area. It is the coefficient of frictional resistance (according to the ITTC-1957 formula). calculate, (For Captain Reynolds number) This is the roughness subsidy coefficient. For ship speed.

[0038] Obtain water resistance acceleration as follows: (4.17); in, For ship quality.

[0039] Next, we consider the effect of wind and calculate the relative wind speed. : (4.18); in, This is the absolute wind speed vector relative to the geodetic coordinate system. This refers to the ship's speed relative to the ground.

[0040] The wind power uses the following aerodynamic drag formula: (4.19); in, air density, The projected area of ​​the ship above the waterline. This is the wind resistance coefficient.

[0041] Dividing this force by the mass gives the wind acceleration. as follows: (4.20); Next, we consider wave drift force, based on linear wave theory and wave energy spectrum (e.g., JONSWAP or Pierson-Moskowitz spectrum), the second-order average wave drift force. Approximately: (4.21); in, For the sake of righteousness, the waves rise high. Represents gravitational acceleration. For the width of the boat, This is the wave drift force coefficient (obtained from wave charts), and the direction of the force is consistent with the direction of the main wave.

[0042] Let the wave angle be... Then the vector of this force in the fixed coordinate system as follows: (4.22); The wave drift force acceleration is: (4.23); This dynamic model assumes that the ship's lateral motion is strongly suppressed by hydrodynamics and rudder effect (no sideslip), and only the projection of environmental forces in the bow direction needs to be considered as a factor in acceleration and deceleration. Combined with the previously mentioned wind acceleration... Calculate the total acceleration of the environment Define the unit vector that determines the direction the ship's bow is. This allows us to calculate the projection of the total environmental acceleration onto the ship's direction of travel, i.e. : (4.24a); (4.24b); Through the above steps, the dynamics module can generate data at any initial time per unit unit. This is mapped to the new state at the end of each unit of time. .

[0043] In the collaborative design of the genetic algorithm module and the deep deterministic policy gradient network module, to address the problem that the original method relies on explicit mechanical formulas to model wind and wave environments and is difficult to adapt to strong dynamic disturbances such as changing winds and irregular waves, a proposal is made to use a reinforcement learning agent to implicitly master the dynamic characteristics of the environment through trial and error by interacting with the environment. The policy network directly takes the unmanned vessel's state as input and action commands as output, without embedding any empirical formulas or preset wave spectra in the decision-making process, thus avoiding inaccurate motion predictions caused by model simplification. At the same time, to address the problem that the original method's greedy decision-making using single-step rolling optimization lacks long-term sequence decision-making ability, a value network from the deep deterministic policy gradient network is introduced. By estimating long-term cumulative rewards through Bellman recursion, the evaluation of each action not only considers the current reward but also includes the value prediction of future multi-step states, overcoming the short-sightedness defect of the original method that only optimizes the single-step evaluation function. Furthermore, the crossover, mutation, and selection operations in the evolutionary strategy are applied to the policy network population. The population-level diversity and global search capability are used to iteratively optimize the reinforcement learning strategy. This allows the training process to not only make efficient local improvements with the guidance of the value network, but also to escape local optima through genetic operations. As a result, the path planning capability that balances global optima and real-time performance is obtained in scenarios where complex obstacles and dynamic wind and waves coexist.

[0044] Comparative experiment: Evaluation metrics: To verify the technical effect of the present invention, a simulation experiment was conducted, and the number of path steps (Step) was used as the evaluation metric to assess the path planning results.

[0045] Baseline selection: To demonstrate the effectiveness of the proposed method, a commonly used method in the field was selected for comparative experiments: Reinforcement Learning (RL) and Reinforcement Learning with Genetic Algorithm (RL_GA) methods, as detailed below: Reinforcement Learning: A Cutting-Edge Approach to Modeling Unmanned Vessel Path Planning as a Markov Decision Process. This approach employs a deep deterministic policy gradient network architecture, defining a 5-dimensional state space (including position, heading, and velocity) and a 2-dimensional continuous action space (including linear and angular acceleration). A value network estimates the long-term cumulative reward of state-action pairs, guiding the policy network's learning with a multi-level reward function encompassing target proximity, movement efficiency, motion smoothness, and the task's final state. This method eliminates the need for explicit empirical formulas for wind and wave environmental forces. Instead, through extensive interaction and trial-and-error with the environment, the policy network implicitly masters the motion patterns under complex sea conditions, outputting a continuous action sequence that balances global guidance and dynamic obstacle avoidance.

[0046] Genetic optimization of reinforcement learning models: A post-processing method for evolutionary optimization of pre-trained reinforcement learning policy networks. Multiple pre-trained policy network parameters are encoded as individuals in a population. Network weight fragments between different individuals are exchanged using a crossover operator, and random perturbations are applied to the weights using a mutation operator, generating new policies with diversity. Then, a value network is used to evaluate the fitness of each mutated individual, selecting high-performing policies to be retained for the next generation. This method performs local perturbation search near converged policies, aiming to escape local optima that may be trapped during reinforcement learning training, but the genetic operation is separated from the policy training process.

[0047] Detailed information on the experimental results is shown in Table 1: Table 1: Comparison of evaluation indicators between the present invention and existing methods

[0048] The results, as shown in Table 1, consistently demonstrate the lowest time and total distance traveled in all unmanned surface vessel (USV) path planning tasks. This proves the reliability of the proposed method in improving USV path planning performance and demonstrates its potential and advantages in practical applications. To further illustrate the experimental results, the results were also visualized as follows: Figure 3 Furthermore, the visualization results clearly show that the method proposed in this invention ( Figure 3 The path planned by (c) is closest to the ideal shortest path, with a smooth path, no invalid detours, and no oscillating backsliding; while the reinforcement learning method ( Figure 3 (a) Local jitter exists; the method of genetics for reinforcement learning models ( Figure 3 The excessive detour in b) further demonstrates the robustness of the proposed method for unmanned vessel path planning under the influence of wind and waves.

[0049] Based on the same inventive concept, this invention also provides an unmanned surface vessel (USV) path planning system based on evolutionary reinforcement learning under the influence of wind and waves, the USV path planning system comprising: The state acquisition module is used to acquire the motion state of the unmanned vessel at the current moment; The decision-making module is used to input the motion state into the trained target policy network to obtain navigation decision actions; The training module is used to train the target policy network. The genetic module is used to initialize the population using tuples from the policy network and the corresponding target policy network as individuals. Perform crossover and mutation operations on the policy network in the population to generate new individuals and insert them into the population; The Deep Deterministic Policy Gradient Network module is used to evaluate the path planning task of an unmanned vessel from its starting position to its target position. It utilizes the value network in the Deep Deterministic Policy Gradient Network to complete individual preparation through experience caching, network parameter updates, and soft updates within a preset preparation round. Then, it accumulates the Q-values ​​of each individual at each time step within the evaluation time to obtain the cumulative evaluation value. Individuals that can complete the path planning task under wind and wave interference are selected to enter the next generation. Repeat the above training steps, and obtain the target policy network by soft updating the policy network of the best individual obtained at the end; The dynamics module is used to perform dynamic state transitions based on the navigation decision actions and wind and wave environmental disturbances, so as to obtain the motion state of the unmanned vessel at the next moment. The control module is used to repeatedly trigger the state acquisition module, decision-making module, and dynamics module to enable the unmanned vessel to navigate along the planned path from the starting position to the target position.

[0050] Based on the same inventive concept, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the unmanned vessel path planning method based on evolutionary reinforcement learning under the influence of wind and waves as described above.

[0051] The program product of the present invention for implementing the above-described method may employ a portable compact disk read-only memory and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In the present invention, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0052] It should be noted that a computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0053] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-described technical content to create equivalent embodiments without departing from the scope of the present invention. The implementation schemes in the above embodiments can also be further combined or replaced. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A path planning method for unmanned surface vessels based on evolutionary reinforcement learning under the influence of wind and waves, characterized in that, Includes the following steps: S1. Obtain the motion state of the unmanned vessel at the current moment; S2. Input the motion state into the target policy network to obtain navigation decision actions; The target policy network is trained through the following steps: The population is initialized using tuples of the policy network and the corresponding target policy network as population individuals; Perform crossover and mutation operations on the policy network in the population to generate new individuals and insert them into the population; Using the path planning task of an unmanned vessel navigating from its starting position to its target position under wind and wave interference as the evaluation scenario, the value network in the deep deterministic policy gradient network is used to complete the individual preparation through experience caching, network parameter updates and soft updates within a preset preparation round. Then, the Q value of each individual at each time step within the evaluation time is accumulated to obtain the cumulative evaluation value. Individuals that can complete the path planning task under wind and wave interference are selected to enter the next generation. Repeat the above training steps, and obtain the target policy network by soft updating the policy network of the best individual obtained at the end; S3. Based on the navigation decision action and the disturbance of wind and waves, perform dynamic state transition to obtain the motion state of the unmanned vessel at the next moment. Repeat steps S1-S3 to make the unmanned vessel travel along the planned path from the starting position to the target position.

2. The method according to claim 1, characterized in that, The target strategy network takes the motion state of the unmanned vessel as input and navigation decision actions as output. The motion state of the unmanned vessel is a quintuple (x, y, ψ, v, ω), where x and y are the position coordinates of the unmanned vessel in the geodetic coordinate system, ψ is the heading angle, v is the forward velocity along the bow direction, and ω is the turning angular velocity. The navigation decision actions are a binary tuple (a, α), where a is the linear acceleration and α is the angular acceleration.

3. The method according to claim 1, characterized in that, The step of performing crossover and mutation operations on the policy network in the population to generate new individuals and insert them into the population includes: The policy networks of two individuals are selected from the population. Some parameters in the two networks are randomly swapped according to a preset crossover method. The new individuals generated after the parameter swap are inserted into the population, while the original two individuals are retained. Iterate through each parameter in the individual policy network, apply a random perturbation function independently to each parameter, and add the output value of the perturbation function to the original parameter to obtain the mutated new parameter.

4. The method according to claim 1, characterized in that, The value network in the deep deterministic policy gradient network completes individual preparation within a preset preparation round through experience caching, network parameter updates, and soft updates. Then, the Q-values ​​of each individual at each time step within the evaluation time are accumulated to obtain the cumulative evaluation value. Individuals capable of completing the path planning task under wind and wave interference are selected to enter the next generation; including: For each individual, at each unit of time within the evaluation time T, the individual's policy network generates a navigation decision action based on the current motion state, calculates the reward through the reward function, completes the state transition through the dynamics module, and caches the experience of this decision into the experience cache set. Experience is sampled from the experience cache set, the target Q-value is estimated using the target value network and the target policy network, the current Q-value is estimated using the value network, the value network loss function is calculated and the value network parameters are updated, the policy gradient is calculated using the updated current value network, and the policy network parameters are updated using gradient ascent; then the target value network and target policy network parameters are updated respectively using a soft update method; this step is repeated until the preset number of preparation rounds is reached; After completing the pre-set number of rounds of preparation, the updated target policy network and target value network are used to accumulate the value of the individual's decision sequence within the evaluation time T. The accumulated evaluation value is used as the individual's fitness for ranking and screening, and the individuals with the highest ranking are retained to enter the next generation of the population. Determine whether the top-ranked individual in the current population can reach the target state from the initial state within the evaluation time T. If yes, output the target policy network of that individual as the final optimization result; otherwise, return to the initial step and continue iterative optimization using the filtered population.

5. The method according to claim 4, characterized in that, The reward for each experience in the experience cache set is calculated in the following way: The reward function is composed of a weighted sum of a target proximity reward, a movement distance reward, and a smoothness penalty. The target proximity reward takes effect when the distance to the target is less than a preset threshold and increases as the distance decreases. The movement distance reward applies a positive or negative reward based on the change in the distance to the target before and after the decision. The smoothness penalty applies a penalty based on the absolute value of the change in the heading angle before and after the decision. When a decision does not lead to the termination of the task, the reward is the weighted sum of the results; when a decision leads to the success of the task, a positive experience reward is given, and when a decision leads to the failure of the task, a negative experience penalty is given.

6. The method according to claim 1, characterized in that, The process of performing dynamic state transition based on the navigation decision action and environmental disturbances such as wind and waves to obtain the motion state of the unmanned vessel at the next moment includes: Based on water resistance acceleration Wind acceleration and wave drift force acceleration Calculate the projection of the resultant acceleration of the wind and waves in the direction of the ship's bow. Furthermore, based on the linear acceleration during navigation decision-making actions... Obtain the total acceleration The position, velocity, heading angle, and turning angular velocity of the unmanned vessel at the next moment are calculated based on the uniform acceleration kinematic model.

7. A path planning system for unmanned surface vessels based on evolutionary reinforcement learning under the influence of wind and waves, characterized in that, The unmanned vessel path planning system includes: The state acquisition module is used to acquire the motion state of the unmanned vessel at the current moment; The decision-making module is used to input the motion state into the trained target policy network to obtain navigation decision actions; The training module is used to train the target policy network. The genetic module is used to initialize the population using tuples from the policy network and the corresponding target policy network as individuals. Perform crossover and mutation operations on the policy network in the population to generate new individuals and insert them into the population; The Deep Deterministic Policy Gradient Network module is used to evaluate the path planning task of an unmanned vessel navigating from its starting position to its target position. It utilizes the value network in the Deep Deterministic Policy Gradient Network to evaluate the long-term cumulative returns of the decision sequence of individuals within the evaluation time, and selects individuals that can complete the path planning task under wind and wave interference to enter the next generation. Repeat the above training steps until the optimal target policy network that can successfully complete the path planning task is obtained; The dynamics module is used to perform dynamic state transitions based on the navigation decision actions and wind and wave environmental disturbances, so as to obtain the motion state of the unmanned vessel at the next moment. The control module is used to repeatedly trigger the state acquisition module, decision-making module, and dynamics module to enable the unmanned vessel to navigate along the planned path from the starting position to the target position.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it is used to implement the unmanned vessel path planning method based on evolutionary reinforcement learning under the influence of wind and waves as described in any one of claims 1-6.

Images