An unmanned ship formation control optimization method based on a dual-channel reinforcement learning framework

Abstract

The application discloses an unmanned ship formation control optimization method based on a double-channel reinforcement learning framework, aiming at solving the problems of existing adaptive dynamic programming control parameter setting difficulty, traditional genetic algorithm easy to fall into local optimum and multi-dimensional error physical quantity dimension not unified. The application constructs an unmanned ship formation mathematical model and a bottom online control channel, and constructs an upper offline optimization channel; a dimensionless fitness function based on state energy ratio is designed; the upper channel utilizes a reinforcement learning intelligent agent to dynamically decide a genetic operator selection strategy according to a population evolution state to generate a control parameter, the bottom channel utilizes the parameter to perform simulation and feeds back a performance index to the upper channel to update a decision model, and the optimal control parameter is output through double-channel closed-loop iteration. The application improves control parameter optimization efficiency and precision, eliminates the interference of order of magnitude difference on optimization, and significantly enhances the dynamic response speed, steady-state tracking precision and adaptive capacity of the unmanned ship formation system.

Description

Technical Field

[0001] This invention belongs to the field of unmanned surface vessel (USV) swarm control technology, and particularly relates to an optimization method for USV formation control based on a dual-channel reinforcement learning framework. Background Technology

[0002] Multi-unmanned surface vessel (USV) cooperative formations are widely used in military reconnaissance, maritime search and rescue, and environmental monitoring, where control performance is crucial. Existing technologies, such as adaptive dynamic programming-based control methods, can handle nonlinear systems through online learning. However, this method has significant drawbacks: its control performance is highly dependent on the manual selection of controller gain and key algorithm parameters. In real-world complex sea conditions, improper parameter settings can easily lead to system oscillations or even instability, failing to meet high-precision control requirements. Due to the lack of efficient automated parameter tuning methods, current methods primarily rely on manual adjustments based on expert experience, which is labor-intensive and makes it difficult to achieve globally optimal performance.

[0003] To optimize controller parameters, existing technologies attempt to introduce genetic algorithms for offline optimization. However, traditional genetic algorithms typically employ fixed crossover and mutation probabilities, resulting in a lack of adaptability in their search strategies. When dealing with high-dimensional, nonlinear parameter optimization problems such as unmanned surface vessel (USV) formation control, this fixed strategy cannot be dynamically adjusted based on the population's evolutionary state. When the search process gets stuck in local optima or premature convergence occurs, the algorithm struggles to escape, leading to slow convergence and a final result that may not be globally optimal, severely impacting the efficiency and quality of parameter tuning.

[0004] Furthermore, existing methods for constructing optimization objectives typically use a linear weighted sum of position error and heading angle error as the fitness function. This presents serious problems in practical applications: the position error of an unmanned surface vessel (USV) is measured in meters and is relatively large, while the heading angle error is measured in radians and is relatively small. There is a significant difference in their physical dimensions and orders of magnitude. This difference makes the optimization algorithm prone to being dominated by the large position error during iteration, thus neglecting the optimization of the small heading angle error. To balance the two, the weighting coefficients need to be manually adjusted repeatedly. This process lacks theoretical guidance and is extremely difficult, often leading to inconsistent control performance and even system state degradation, making it impossible to achieve synchronous and high-precision convergence of position and attitude.

[0005] In summary, single control methods struggle to achieve parameter adaptability, conventional optimization algorithms face bottlenecks in search efficiency and global optimization capabilities, and traditional evaluation metrics are hampered by interference from differences in physical dimensions. Therefore, in the field of unmanned surface vessel (USV) formation control, there is an urgent need for an innovative solution capable of automatically, efficiently, and evenly tuning controller parameters and addressing the aforementioned challenges within a unified framework. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention proposes an unmanned surface vessel (USV) formation control optimization method based on a dual-channel reinforcement learning framework, thereby resolving the issues present in the prior art.

[0007] Firstly, to achieve the above objectives, this invention provides an unmanned surface vessel (USV) formation control optimization method based on a dual-channel reinforcement learning framework, comprising the following steps: Constructing a mathematical model and underlying online control channel for an unmanned surface vessel (USV) swarm system; Construct an upper-layer offline optimization channel, which includes a genetic algorithm based on reinforcement learning agent decision-making; Design a dimensionless fitness function based on the state energy ratio; S1. Using the reinforcement learning agent, dynamically decide the selection strategy of genetic operators based on the population evolution state to generate a new population with control parameters. S2. Input the control parameter population into the underlying online control channel for simulation, and calculate the performance index based on the dimensionless fitness function; S3. Calculate the reward value based on the feedback of the performance index, update the decision model of the reinforcement learning agent, and cyclically execute S1 and S2 based on the update result until the convergence condition is met and the optimal control parameters are output.

[0008] Optionally, constructing the underlying online control channel includes: adopting a cascaded dual-loop control architecture, wherein the dual-loop control architecture includes an outer loop kinematic virtual controller and an inner loop dynamic torque controller; both the outer loop kinematic virtual controller and the inner loop dynamic torque controller adopt a reinforcement learning structure that includes an identifier, an evaluator, and an actuator.

[0009] Optionally, designing a dimensionless fitness function based on the state-energy ratio includes: The actual trajectory energy and reference trajectory energy of the unmanned surface vessel (USV) formation system are calculated separately; the ratio of the actual trajectory energy to the reference trajectory energy is calculated to obtain the state energy ratio; the weighted sum of the deviations between the state energy ratio and the unit value is defined as the fitness function value.

[0010] Optionally, in S1, the reinforcement learning agent dynamically decides the selection strategy of genetic operators based on the population evolution state, including: defining the population evolution state as a progressive state and a stagnant state; defining a discrete action space containing different combinations of crossover probabilities and mutation probabilities; and the reinforcement learning agent selecting actions from the action space according to the exploration and utilization strategy to determine the genetic operator parameters used to generate the offspring population.

[0011] Optionally, the discrete action space includes a first action, a second action, and a third action; the first action is a combination of high crossover probability and low mutation probability; the second action is a combination of low crossover probability and high mutation probability; and the third action is a combination of medium crossover probability and low mutation probability.

[0012] Optionally, in S2, calculating the performance index based on the dimensionless fitness function includes: sequentially configuring each set of parameters in the control parameter population into the underlying online control channel; running a time-domain simulation of the unmanned surface vessel formation system; collecting the system's state response data; and calculating the fitness function value corresponding to each set of parameters based on the state response data.

[0013] Optionally, in S3, calculating the reward value based on the feedback of the performance index includes: comparing the optimal fitness function value of the current generation population with the optimal fitness function value of the previous generation population; if the current generation is better, a positive reward signal is generated and the population evolution state at the next moment is determined to be a progressive state; if the current generation is not better, a negative reward signal is generated and the population evolution state at the next moment is determined to be a stagnant state.

[0014] Optionally, in S3, updating the decision model of the reinforcement learning agent includes: using the reward signal and the population evolution state at the next time step, updating the decision value table of the reinforcement learning agent online through the Bellman equation.

[0015] In a second aspect, the present invention also provides a computer terminal device, comprising: One or more processors; A memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the unmanned surface vessel formation control optimization method based on the dual-channel reinforcement learning framework in the first aspect above.

[0016] Thirdly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps of the unmanned surface vessel formation control optimization method based on a dual-channel reinforcement learning framework in the first aspect described above.

[0017] Compared with the prior art, the present invention has the following advantages and technical effects: This invention provides an unmanned surface vessel (USV) formation control optimization method based on a dual-channel reinforcement learning framework. Employing a dual-channel learning framework combining low-level online control and high-level offline optimization, it effectively overcomes the shortcomings of traditional genetic algorithms, such as slow convergence speed and susceptibility to local optima, significantly improving the efficiency and accuracy of control parameter tuning. Simultaneously, the dimensionless fitness function based on state-energy ratio constructed in this invention completely eliminates the interference caused by the inconsistency in physical dimensions and orders of magnitude between position and heading errors by transforming multidimensional heterogeneous errors into energy ratio deviations, thus avoiding the control state degradation and divergence problems caused by traditional weighted sum methods. Ultimately, this invention enables the USV formation system to obtain globally optimal control gain, achieving high-precision tracking of the desired trajectory while maintaining rapid dynamic response, significantly enhancing the overall adaptive capability and robustness of the system. Attached Figure Description

[0018] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart illustrating an unmanned surface vessel formation control optimization method based on a dual-channel reinforcement learning framework, according to an embodiment of the present invention. Figure 2 The flowchart of the improved genetic algorithm based on Q-learning is shown in the embodiment of the present invention. Figure 3 This is a diagram showing the internal structure of the underlying ADP controller in an embodiment of the present invention. Figure 4 This is a graph showing the convergence performance of the fitness function in an embodiment of the present invention. Figure 5 This is a first-round tracking trajectory diagram according to an embodiment of the present invention; Figure 6 This is a convergence diagram of the first round of tracking error in an embodiment of the present invention; Figure 7 This is the twentieth round tracking trajectory diagram of an embodiment of the present invention; Figure 8 This is the convergence diagram of the tracking error in the twentieth round of this invention. Detailed Implementation

[0019] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0021] Example 1 like Figure 1 As shown, this embodiment provides an unmanned surface vessel (USV) formation control optimization method based on a dual-channel reinforcement learning framework, including: Constructing a mathematical model and underlying online control channel for an unmanned surface vessel (USV) swarm system; Construct an upper-layer offline optimization channel, which includes a genetic algorithm based on reinforcement learning agent decision-making; Design a dimensionless fitness function based on the state energy ratio; S1. Using the reinforcement learning agent, dynamically decide the selection strategy of genetic operators based on the population evolution state to generate a new population with control parameters. S2. Input the control parameter population into the underlying online control channel for simulation, and calculate the performance index based on the dimensionless fitness function; S3. Calculate the reward value based on the feedback of the performance index, update the decision model of the reinforcement learning agent, and cyclically execute S1 and S2 based on the update result until the convergence condition is met and the optimal control parameters are output.

[0022] Specifically, the implementation process of this embodiment includes: This embodiment employs an unmanned surface vessel (USV) formation control optimization strategy based on dual-channel reinforcement learning. It combines an improved genetic algorithm based on Q-learning with a low-level ADP controller to construct a dual-channel closed-loop system encompassing upper-level parameter optimization and low-level real-time control. This system aims to address the difficulties of traditional methods in controlling parameter tuning relying on manual experience and insufficient tracking accuracy in complex dynamic environments. The system leverages the powerful global search capability of the improved genetic algorithm, introducing a reinforcement mechanism based on Q-learning to accelerate population evolution and escape local optima, automatically searching for optimal parameters for the controller. Simultaneously, the low-level ADP controller utilizes a neural network to learn the system dynamics online, achieving high-precision tracking of the reference trajectory. Experiments were conducted involving multiple rounds of USV formation tracking simulations, comparing the algorithm's tracking performance in the initial stage with that after iterative optimization. The results show that the strategy proposed in this embodiment significantly reduces tracking error with increasing training rounds (e.g., from round 1 to round 20), enabling rapid convergence of the fitness function and achieving high-precision reproduction of the desired trajectory while maintaining formation stability.

[0023] like Figure 1As shown, this invention constructs a two-layer closed-loop control architecture comprising an upper-layer offline optimization channel and a lower-layer online control channel. In the upper-layer offline optimization channel, the Q-Learning agent maintains a Q-table, which is updated based on the current evolutionary state. The system dynamically guides the genetic algorithm population to select the optimal crossover and mutation operators through an operator selection strategy, thereby generating a high-quality candidate solution set. In the lower-level online control channel, an unmanned surface vessel (USV) formation model and an ADP controller are deployed. The ADP controller adopts a "Critic-Actor" structure, calculating control torque in real time based on formation state feedback to drive the USV's movement. The upper and lower layers form a closed loop through data interaction: the upper-level optimization channel transmits the evolved control gain and parameters to the lower layer as initial or updated parameters for the ADP controller; during operation, the lower-level control channel feeds back the evaluated energy ratio fitness to the upper layer as a basis for evaluating the current parameter quality, thus driving the next round of parameter evolution and strategy updates.

[0024] Furthermore, establish a mathematical model for unmanned surface vessel (USV) formations: In the Earth coordinate system, the first The kinematic and dynamic model of the underactuated unmanned surface vessel is as follows: ; in, Here are the position and heading angle vectors in the Earth coordinate system. For rotation matrix, The matrix represents the pitch, sway, and bow angular velocities in the ship's coordinate system. These are the reduced-order inertia matrix, the Coriolis centripetal matrix, and the damping matrix, respectively, and their specific expansion forms are as follows: ; Consider a group of USVs with a leader-follower structure, and the distance measured using the line-of-sight (LOS) method. and relative angle Defined as: ; The formation output vector is represented as The error dynamic equation is established as follows: ; Wherein, error vector , It is a virtual control signal. Includes unmodeled dynamics in relative motion.

[0025] Furthermore, the construction of the underlying online control channel includes: adopting a cascaded dual-loop control architecture, which includes an outer-loop kinematic virtual controller and an inner-loop dynamic torque controller; both the outer-loop kinematic virtual controller and the inner-loop dynamic torque controller adopt a reinforcement learning structure that includes an identifier, an evaluator, and an actuator.

[0026] Specifically, the implementation process of this embodiment includes: This embodiment employs a cascaded dual-loop control architecture, with the outer loop being a kinematic virtual controller and the inner loop being a dynamic torque controller. Both loops utilize an Identifier-Critic-Actor (ICA) reinforcement learning structure to achieve online optimal control.

[0027] For the kinematic loop (outer loop), design an outer loop controller to generate the desired virtual control signal. : ; in, This is the positive definite gain matrix of the kinematic controller; and These are the basis function vectors of the corresponding RBF neural network. , and Here are the network weights for Identifier, Critic, and Actor, respectively, and their update law is designed as follows: ; Similarly, for the dynamic loop (inner loop), an actual torque control law is designed. for: ; in, Here is the positive definite gain matrix of the kinematic controller. and Let be the basis function vector of the dynamic loop. , and Let be the weights of the Identifier, Critic, and Actor networks in the dynamic loop, respectively, and their update law be designed as follows: .

[0028] Furthermore, the design of a dimensionless fitness function based on state-energy ratio includes: The actual trajectory energy and reference trajectory energy of the unmanned surface vessel (USV) formation system are calculated separately; the ratio of the actual trajectory energy to the reference trajectory energy is calculated to obtain the state energy ratio; the weighted sum of the deviations between the state energy ratio and the unit value is defined as the fitness function value.

[0029] Furthermore, in S2, calculating the performance index based on the dimensionless fitness function includes: sequentially configuring each set of parameters in the control parameter population into the underlying online control channel; running a time-domain simulation of the unmanned surface vessel formation system; collecting the system's state response data; and calculating the fitness function value corresponding to each set of parameters based on the state response data.

[0030] Specifically, the implementation process of this embodiment includes: To address the issue that the physical dimensions of position error (meter-level, relatively large) and heading angle error (radian-level, relatively small) in unmanned surface vessel (USV) formations are not uniform, which makes it easy for the traditional weighted sum form of fitness function to neglect angle optimization, this embodiment designs an evaluation index based on "energy ratio".

[0031] First, a set of control parameters from the current population is input into the underlying ADP controller, within a time interval. The system undergoes time-domain simulation. The system's "trajectory energy" is defined as the integral of the state variable norm over the time domain. The actual trajectory energy and the reference trajectory energy are calculated separately. ; in, and They represent the simulation time domain respectively. Inside, the actual trajectory of the follower unmanned surface vessel relative to the relative distance and relative angle The actual trajectory energy integral term; and These represent the corresponding expected reference trajectories with respect to the expected distance. and expected perspective The reference trajectory energy integral term. The integral term reflects the cumulative state deviation and control cost of the system throughout the entire dynamic process, and is the basis for subsequent calculation of the dimensionless energy ratio. In the digital simulation implementation, the integral is approximated by accumulating discrete sampling points.

[0032] Based on this, in order to eliminate the influence of physical dimensions, the ratio of the actual energy to the reference energy is calculated: ; in To prevent a constant with a denominator of zero, this ratio should ideally approach 1 under ideal tracking conditions. Finally, the comprehensive fitness function is defined. This is a weighted sum of the deviations between the energy ratio and the unit value "1": ; in , This is the weighting coefficient. The smaller the value of this function, the closer the system response corresponding to the control parameters is to the desired trajectory, and the less affected it is by differences in magnitude.

[0033] The internal structure of the underlying ADP controller designed in this embodiment is as follows: Figure 3 As shown, this section illustrates the complete signal processing procedure from trajectory error to control torque. First, it receives information about the desired trajectory and the actual trajectory affected by environmental disturbances, then calculates the position and attitude errors. The error signal is input to the ADP core module, where the Critic network evaluates the current system performance and the Actor network generates the optimal virtual control law. The output signal is decomposed into surge velocity and yaw angle commands by the virtual control decoupling module. Subsequently, the signal passes through a first-order filter to obtain smooth derivative information, and the desired kinematic signal is calculated using a virtual control law. Finally, by combining the kinematic errors with the dynamic model, the actual control torque acting on the unmanned surface vessel's propulsion system was calculated. This enables closed-loop control.

[0034] Furthermore, in S1, the reinforcement learning agent dynamically decides the selection strategy of genetic operators based on the population evolution state, including: defining the population evolution state as a progressive state and a stagnant state; defining a discrete action space containing different combinations of crossover probabilities and mutation probabilities; and the reinforcement learning agent selecting actions from the action space according to the exploration and utilization strategy to determine the genetic operator parameters used to generate the offspring population.

[0035] Furthermore, the discrete action space includes a first action, a second action, and a third action; the first action is a combination of high crossover probability and low mutation probability; the second action is a combination of low crossover probability and high mutation probability; and the third action is a combination of medium crossover probability and low mutation probability.

[0036] Furthermore, in S3, calculating the reward value based on the feedback of the performance index includes: comparing the optimal fitness function value of the current generation population with the optimal fitness function value of the previous generation population; if the current generation is better, a positive reward signal is generated and the population evolution state at the next moment is determined to be a progressive state; if the current generation is not better, a negative reward signal is generated and the population evolution state at the next moment is determined to be a stagnant state.

[0037] Furthermore, in S3, updating the decision model of the reinforcement learning agent includes: using the reward signal and the population evolution state at the next time step, updating the decision value table of the reinforcement learning agent online through the Bellman equation.

[0038] Specifically, the implementation process of this embodiment includes: like Figure 2 As shown, this embodiment uses a Q-Learning reinforcement learning agent to dynamically intervene in the evolutionary process of GA, and constructs a closed-loop optimization strategy that includes "state perception - action selection - reward feedback".

[0039] First, to simplify calculations and improve convergence efficiency, the evolutionary state of the population is... The population is divided into two discrete states: "progressive state" and "stagnant state." "Progressive state" represents a significant decrease in the optimal fitness of the current generation compared to the previous generation (i.e., performance improvement), while "stagnant state" represents no improvement or only a slight improvement in the optimal fitness of the current generation, indicating that the population may be trapped in a local optimum. Next, the set of actions guided by the Q-learning agent is defined. The three actions are as follows: High crossover rate ) and low variation rate ( ), using the recombination of superior genes to accelerate population convergence; Low crossover rate ) and high variability ( By introducing strong random perturbations, the population can be helped to escape local extreme value traps. Medium crossover rate ( ) and low variation rate ( ), to maintain the normal evolutionary rhythm.

[0040] Before each generation of evolution begins, the Q-Learning agent first determines the current state... (Initially set to a stagnant state), utilizing The strategy involves looking up the Q-table to select an action. Based on the parameters of the selected action (… , The GA sequentially performs roulette wheel selection, single-point crossover, and random reset mutation operations to generate the offspring population. Subsequently, the offspring population is input into the underlying control loop for simulation evaluation to calculate individual fitness. An elite retention strategy is adopted, mixing and sorting parents and offspring, retaining the individuals with the highest fitness. Each individual becomes part of the next generation of the population.

[0041] Based on this, the global optimal fitness of the current generation population is calculated. and the optimal value of the previous generation Compare and generate reward signals And update the status: If This indicates a performance improvement, and a large positive reward is given. ), and the state at the next moment. It is determined to be in a "progressive state"; if This indicates performance stagnation and incurs a negative penalty. ), and the state at the next moment. It is determined to be in a "stagnant state".

[0042] Finally, the agent determines the reward based on the feedback. and the state in the next moment The Bellman equation is used to update the corresponding Q values ​​in the Q table online: ; in For learning rate, This is the discount factor. The exploration rate increases with iteration. The decay rate is gradually reduced by 0.95, causing the agent to gradually shift from initial random exploration to utilizing the optimal strategy.

[0043] The improved genetic algorithm based on Q-learning designed in this embodiment is as follows: Figure 2 As shown, this module is mainly responsible for offline optimization of controller parameters. After the algorithm starts, it initializes the population size, number of iterations, and initial values ​​of the Q-Table before entering the main loop. The Q-learning agent performs action selection: based on the current evolutionary state of the population, the agent selects an action from three operator strategies: "mutation only," "mixed mode," or "crossover only." Subsequently, the population performs the corresponding genetic operations according to the selected strategy and generates a new generation of individuals. To evaluate the quality of individuals, the algorithm calls the unmanned surface vessel model for simulation calculations, obtains tracking error and energy consumption data, and calculates the fitness value. Then, feedback updates are performed, using the fitness improvement as a reward signal to update the Q-Table, enabling the agent to learn a better operator selection strategy. Finally, the termination condition is checked. If the maximum number of iterations is met, the optimal parameters are output; otherwise, the algorithm returns to the action selection step to continue the loop.

[0044] like Figures 4 to 8 As shown, this embodiment verifies the convergence performance of the optimization algorithm and the evolution effect of the formation control strategy at different training stages through simulation experiments. First, Figure 4 The fitness convergence curve of the Q-GA algorithm is shown. As the number of generations increases, the fitness function value decreases in a stepwise manner, stabilizing in the 13th generation, indicating that the algorithm can effectively search for the optimal control parameters that minimize system energy and error costs. To visually demonstrate the difference in control before and after optimization, Figure 5 and Figure 6 This shows the tracking results during the first iteration (not fully optimized); it can be seen that in the initial stage, the follower's trajectory ( Figure 5 There is a significant positional deviation between the navigator and the tracker, and the tracking error curve ( Figure 6It exhibits significant oscillations and overshoot, making it difficult to achieve stable tracking in a short period. In contrast, Figure 7 and Figure 8 This shows the final tracking results after twenty rounds of iteration and optimization; at this point, the follower's trajectory ( Figure 7 The tracking error curve closely matched the navigator's expected trajectory, maintaining good formation; meanwhile, the tracking error curve ( Figure 8 The response converges to near zero and remains stable within a very short time. In summary, by comparing the experimental data from the first and twentieth rounds, it is demonstrated that the dual-channel optimization strategy proposed in this embodiment can significantly improve the dynamic response speed and steady-state accuracy of the controller, effectively realizing the collaborative control task of multiple unmanned surface vessel formations.

[0045] Example 2 In this embodiment, a computer terminal device is provided, including: One or more processors; A memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the above-described unmanned surface vessel formation control optimization method based on a dual-channel reinforcement learning framework.

[0046] In this embodiment, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the above-described unmanned surface vessel formation control optimization method based on a dual-channel reinforcement learning framework.

[0047] This invention provides an unmanned surface vessel (USV) formation control optimization method based on a dual-channel reinforcement learning framework. Employing a dual-channel learning framework combining low-level online control and high-level offline optimization, it effectively overcomes the shortcomings of traditional genetic algorithms, such as slow convergence speed and susceptibility to local optima, significantly improving the efficiency and accuracy of control parameter tuning. Simultaneously, the dimensionless fitness function based on state-energy ratio constructed in this invention completely eliminates the interference caused by the inconsistency in physical dimensions and orders of magnitude between position and heading errors by transforming multidimensional heterogeneous errors into energy ratio deviations, thus avoiding the control state degradation and divergence problems caused by traditional weighted sum methods. Ultimately, this invention enables the USV formation system to obtain globally optimal control gain, achieving high-precision tracking of the desired trajectory while maintaining rapid dynamic response, significantly enhancing the overall adaptive capability and robustness of the system.

[0048] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for optimizing unmanned surface vessel (USV) formation control based on a dual-channel reinforcement learning framework, characterized in that, Includes the following steps: Constructing a mathematical model and underlying online control channel for an unmanned surface vessel (USV) swarm system; Construct an upper-layer offline optimization channel, which includes a genetic algorithm based on reinforcement learning agent decision-making; Design a dimensionless fitness function based on the state energy ratio; S1. Using the reinforcement learning agent, dynamically decide the selection strategy of genetic operators based on the population evolution state to generate a new population with control parameters. S2. Input the control parameter population into the underlying online control channel for simulation, and calculate the performance index based on the dimensionless fitness function; S3. Calculate the reward value based on the feedback of the performance index, update the decision model of the reinforcement learning agent, and cyclically execute S1 and S2 based on the update result until the convergence condition is met and the optimal control parameters are output.

2. The method according to claim 1, characterized in that, The construction of the underlying online control channel includes: adopting a cascaded dual-loop control architecture, which includes an outer-loop kinematic virtual controller and an inner-loop dynamic torque controller; both the outer-loop kinematic virtual controller and the inner-loop dynamic torque controller adopt a reinforcement learning structure that includes an identifier, an evaluator, and an actuator.

3. The method according to claim 1, characterized in that, The design of a dimensionless fitness function based on state-energy ratio includes: The actual trajectory energy and reference trajectory energy of the unmanned surface vessel (USV) formation system are calculated separately; the ratio of the actual trajectory energy to the reference trajectory energy is calculated to obtain the state energy ratio; the weighted sum of the deviations between the state energy ratio and the unit value is defined as the fitness function value.

4. The method according to claim 1, characterized in that, In S1, the reinforcement learning agent dynamically decides the selection strategy of genetic operators based on the population evolution state, including: defining the population evolution state as a progressive state and a stagnant state; defining a discrete action space containing different combinations of crossover probabilities and mutation probabilities; and the reinforcement learning agent selecting actions from the action space according to the exploration and utilization strategy to determine the genetic operator parameters used to generate the offspring population.

5. The method according to claim 4, characterized in that, The discrete action space includes a first action, a second action, and a third action; the first action is a combination of high crossover probability and low mutation probability; the second action is a combination of low crossover probability and high mutation probability; and the third action is a combination of medium crossover probability and low mutation probability.

6. The method according to claim 3, characterized in that, In S2, calculating the performance index based on the dimensionless fitness function includes: sequentially configuring each set of parameters in the control parameter population into the underlying online control channel; running a time-domain simulation of the unmanned surface vessel formation system; collecting the system's state response data; and calculating the fitness function value corresponding to each set of parameters based on the state response data.

7. The method according to claim 1, characterized in that, In S3, calculating the reward value based on the feedback of the performance index includes: comparing the optimal fitness function value of the current generation population with the optimal fitness function value of the previous generation population; if the current generation is better, a positive reward signal is generated and the population evolution state at the next moment is determined to be a progressive state; if the current generation is not better, a negative reward signal is generated and the population evolution state at the next moment is determined to be a stagnant state.

8. The method according to claim 1, characterized in that, In S3, updating the decision model of the reinforcement learning agent includes: using the reward signal and the population evolution state at the next time step, updating the decision value table of the reinforcement learning agent online through the Bellman equation.

9. A computer terminal device, characterized in that, include: One or more processors; A memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors perform the steps of the method as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-8.

Images