Bionic fish leader-follower formation control method based on deep reinforcement learning

Abstract

The application relates to a kind of bionic fish leader-follower formation control method based on deep reinforcement learning, comprising: constructing competitive double Q network;From the preset simulation flow field environment, the state vector of each follower and leader in formation is obtained, and the state vector of current time step is input into competitive double Q network, cooperate with the action selection strategy of follower based on imitating leader behavior, the action of each follower in formation is selected;Competitive double Q network is based on the formation reward / punishment of each follower, the imitation learning loss function with attenuation coefficient is optimized, and the foregoing steps are repeated to obtain the trained network model;Online load network model, and obtain the state vector of each follower and leader in simulation flow field environment, based on state vector, the Q value of each action under the current state of each follower is evaluated;Each follower selects the action corresponding to the maximum Q value, maintains the formation configuration of leader and follower, and completes the formation control task.

Description

Technical Field

[0001] This invention relates to the field of formation path tracking control, and more particularly to a biomimetic fish leader-follower formation control method based on deep reinforcement learning. Background Technology

[0002] With the continuous development of unmanned systems, unmanned underwater vehicles (UUVs) have been widely used in various military operations and civilian missions, including hydrological surveying, current field mapping, military reconnaissance, and oceanographic research. Bionic robotic fish, as a special type of underwater robot, mimic the superb movement skills of fish in nature. Compared with traditional UUVs, they possess significant advantages in flexibility. Inspired by fish school behavior, the formation control of bionic fish is gradually becoming an emerging research topic, showing great practical potential in various complex tasks such as marine exploration and rescue.

[0003] Scholars both domestically and internationally have conducted extensive research on the formation control problem of biomimetic fish, but most existing methods rely on precise models of fish dynamics and fluid mechanics. Due to the complexity, time-varying nature, and nonlinearity of these models, they are often simplified during problem analysis. As an alternative approach, reinforcement learning methods can operate independently of environmental models and are applicable to decision-making and control problems in unknown environments. Therefore, using reinforcement learning to address these challenges has attracted increasing attention.

[0004] However, the biomimetic fish formation task scenario is quite complex. Followers influence each other and the flow field environment. The agent explores the state space extensively through trial and error. Directly applying existing deep reinforcement learning algorithms often requires huge time and computational costs, and it is even difficult to learn effective control strategies. Therefore, there is an urgent need to develop efficient deep reinforcement learning training techniques. Summary of the Invention

[0005] The purpose of this invention is to provide a biomimetic fish leader-follower formation control method based on deep reinforcement learning, which solves the problems of difficult modeling of complex dynamic flow field environments, time-consuming and inefficient formation control training.

[0006] To achieve the above-mentioned objectives, this invention provides a biomimetic fish leader-follower formation control method based on deep reinforcement learning, comprising:

[0007] S1. Construct a competitive dual-Q network that includes a main network and a target network;

[0008] S2. Obtain the state vectors of each follower and the leader in the formation from the preset simulated flow field environment, input the state vectors at the current time step into the competing double Q-network, and, in cooperation with the follower action selection strategy based on imitating the leader's behavior, select the actions of each follower in the formation;

[0009] S3. The competing double Q-network optimizes the imitation learning loss function with a decay coefficient based on the formation rewards / punishments of each follower, and repeat steps S2 to S3 to obtain a trained network model;

[0010] S4. Online load the network model, the network model obtains the state vectors of each follower and the leader in the preset simulated flow field environment, and evaluates the Q-values of each follower's various actions in the current state based on the state vectors;

[0011] S5. Each follower selects the action corresponding to the maximum Q-value, so as to maintain the formation configuration between the leader and the follower and complete the formation control task.

[0012] According to one aspect of the present invention, in step S2, in the step of obtaining the state vectors of each follower and the leader in the formation from the preset simulated flow field environment, the state vector includes seven elements, and the seven elements are respectively: the horizontal deviation and vertical deviation of the desired formation target position in the current follower's fish body coordinate system, the angles of attack of the leader and the follower in the global coordinate system, the historical actions executed by the follower in the past two time steps, and the odd / even flapping cycle flag bit to which the follower belongs at the current time step.

[0013] According to one aspect of the present invention, in step S2, in the step of inputting the state vectors at the current time step into the competing double Q-network, and, in cooperation with the follower action selection strategy based on imitating the leader's behavior, selecting the actions of each follower in the formation, it includes:

[0014] S21. Input the state vector into the competing double Q-network and output the Q-values of the corresponding various actions under the state vector;

[0015] S22. Generate a random number p within (0, 1);

[0016] S23. If the current number of training rounds for the competing double Q-network is less than the set threshold and p is greater than the exploration rate ε, then the follower selects and executes the action corresponding to the maximum Q-value; if 0.8ε < p < ε, then the follower randomly selects an action from the predefined action set and executes it, where the action set includes the 5 maximum swing amplitudes that the follower can adopt; if p < 0.8ε, then the follower directly executes the action of the leader at the current time step;

[0017] If the current number of training rounds for the competitive dual-Q network is greater than or equal to the set threshold, and p is greater than the exploration rate ε, then the follower selects and executes the action corresponding to the maximum Q value; if p < ε, then the follower randomly selects and executes an action from a predefined action set.

[0018] According to one aspect of the invention, in step S3, for each training round in which steps S2 to S3 are repeated, the exploration rate ε of the follower action selection strategy is decayed once by a preset decay rate δ.

[0019] According to one aspect of the present invention, in step S3, the competitive dual-Q network optimizes the imitation learning loss function with a decay coefficient based on the formation reward / penalty of each of the followers, and repeats steps S2 to S3 to obtain the trained network model, comprising:

[0020] S31. Each follower executes the action selected in step S2 to obtain the state vector and reward value for the next time step;

[0021] S32. Collect the historical experience of each follower interacting with the environment and store it in the experience pool; wherein, the historical experience is a quadruple (s,a,r,s′), where s is the state vector of a certain follower and the leader at the current time step, a is the action performed by the follower, i.e., the maximum amplitude of its tail flapping, r is the reward obtained by the follower for performing action a, and s′ is the new state vector formed by the follower and the leader together in the next time step after the follower performs action a;

[0022] S33. At each preset time step, random samples are taken from the experience pool, and the main network parameters of the main network in the competitive dual Q network are updated in batches based on the imitation learning loss function with a decay coefficient through the backpropagation algorithm. In addition, the target network parameters in the target network are softly updated, that is, the target network parameters are updated towards the main network parameters.

[0023] S34. When the number of training rounds of repeating steps S2 to S3 reaches a preset condition, the update of the main network parameters and the target network parameters is terminated, and the model parameters of the trained competitive dual-Q network are saved to obtain the network model.

[0024] According to one aspect of the invention, in step S3, the imitation learning loss function with attenuation coefficient consists of two parts: one part is the TD error L composed of the action value output by the competitive dual Q network and the TD target. Q The other part is the imitation learning term λL with a decay coefficient λ. I, where the decay coefficient λ decays at a decay rate μ in each training round.

[0025] According to one aspect of the invention, in step S3, each training round ends when the follower meets the following condition: the follower swims out of the defined domain; or, the total time steps of the follower's swimming exceed the maximum time steps specified for each round.

[0026] According to one aspect of the present invention, step S4, which involves loading the network model online, wherein the network model obtains the state vector of each follower and leader in a preset simulated flow field environment, and evaluating the Q-value of each action of each follower in the current state based on the state vector, includes:

[0027] S41. The network model obtains the state vector of each follower and leader in the current time step in the preset simulation flow field environment. The state vector contains seven elements, and the seven elements are: the horizontal and vertical deviations of the expected formation target position in the current follower's fish body coordinate system, the angle of attack of the leader and follower in the global coordinate system, the historical actions performed by the follower in the past two time steps, and the odd / even beat cycle flag of the follower in the current time step.

[0028] S42. Using the state vector obtained in step S41 as the input to the network model, and evaluating the Q value of each follower's action in the current state based on the Q function of the network model.

[0029] According to one aspect of the invention, the competitive dual-Q network includes: a state value function branch and an advantage function branch;

[0030] The state value function branch and the dominant function branch each contain four fully connected layers; wherein, the first fully connected layer of the state value function branch and the dominant function branch is a shared layer with 128 hidden nodes, the second fully connected layer of the state value function branch and the dominant function branch is a shared layer with 64 hidden nodes, the third fully connected layer of the state value function branch and the third fully connected layer of the dominant function branch each have 64 hidden nodes, and the fourth fully connected layer of the state value function branch and the fourth fully connected layer of the dominant function branch each have 1 and 5 hidden nodes, respectively.

[0031] According to one aspect of the present invention, the present invention is based on imitation learning and further optimizes the reinforcement learning method. Behavioral cloning technology is introduced in the early stage of training, which improves the exploration strategy of the follower agent and improves the training efficiency and convergence speed of the algorithm.

[0032] According to one aspect of the present invention, a highly efficient training method for biomimetic fish "leader-follower" based on deep reinforcement learning models biomimetic fish formation control as a Markov decision process and applies reinforcement learning (RL) to solve it. This reinforcement learning process does not rely on an environment model and does not require prior environmental knowledge, making it suitable for sequential decision-making in unknown environments. This invention applies deep reinforcement learning algorithms to solve the biomimetic fish formation control problem in dynamic flow fields, improving the formation adaptability of biomimetic robotic fish in complex flow environments.

[0033] According to one aspect of the present invention, the biomimetic fish "leader-follower" efficient training method based on deep reinforcement learning changes the traditional ε-greedy strategy. By combining it with an imitation learning loss function with a decay coefficient, the method intentionally guides the followers to imitate the leader's actions in the initial stage of training, increases the number of positive samples in the experience pool, reduces the blindness of the followers' exploration, and thus improves learning efficiency and performance. Attached Figure Description

[0034] Figure 1 This is a schematic diagram illustrating the steps of a leader-follower formation training method in a biomimetic fish according to an embodiment of the present invention;

[0035] Figure 2 This is a schematic diagram illustrating a formation control system for a leader-follower formation training method in a biomimetic fish according to an embodiment of the present invention.

[0036] Figure 3 This is a schematic diagram illustrating the structure of a competitive dual Q-network in a leader-follower formation training method for biomimetic fish according to an embodiment of the present invention.

[0037] Figure 4 This is a schematic diagram showing a comparison of the training stages of a leader-follower formation training method in a biomimetic fish according to an embodiment of the present invention with the learning curves of a traditional reinforcement learning method.

[0038] Figure 5 This is a schematic representation of the trajectory of a three-fish formation scenario and keyframes at different times in a biomimetic fish leader-follower formation training method according to an embodiment of the present invention. Detailed Implementation

[0039] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. The embodiments cannot be described in detail here, but the embodiments of the present invention are not limited to the following embodiments.

[0040] Combination Figure 1 and Figure 2As shown, according to one embodiment of the present invention, the leader-follower formation training method for biomimetic fish based on deep learning mainly includes two parts: an efficient training stage and an online execution stage. The efficient training stage obtains the state vectors of each follower and the leader fish from the simulated flow field environment, inputs the state vector of the current time step into a competitive dual-Q network, and, in conjunction with an action selection strategy based on imitation behavior, enables each follower to select actions in the flow field formation environment. The competitive dual-Q network optimizes the imitation learning loss with a decay coefficient through formation rewards (penalties) for each follower, enabling followers to efficiently explore the unknown environment in the early stages of reinforcement learning. The online execution stage loads the trained competitive dual-Q network. Based on the state of each follower obtained from the simulated environment, the competitive dual-Q network evaluates the Q-value of each action of each follower in the current state. Each follower selects the action corresponding to the maximum Q-value, thereby maintaining the formation configuration and completing the formation control task.

[0041] Based on the above process, this solution specifically includes the following steps:

[0042] S1. Construct a competitive dual-Q network that includes a main network and a target network;

[0043] S2. Obtain the state vector of each follower and leader in the formation from the preset simulation flow field environment, and input the state vector of the current time step into the competitive dual Q network. In addition, select the action of each follower in the formation in conjunction with the follower action selection strategy based on imitating the behavior of the leader.

[0044] S3. The competitive dual-Q network optimizes the imitation learning loss function with decay coefficient based on the formation reward / penalty of each follower, and repeats steps S2 to S3 to obtain the trained network model;

[0045] S4. Load the network model online. The network model obtains the state vector of each follower and leader in the preset simulation flow field environment. Based on the state vector, evaluate the Q value of each follower's actions in the current state.

[0046] S5. Each follower selects the action corresponding to the maximum Q value, thereby maintaining the formation configuration between the leader and followers and completing the formation control task.

[0047] According to one embodiment of the present invention, in step S1, the competitive dual-Q network comprising a master network and a target network is constructed using a D3QN network.

[0048] Combination Figure 1 and Figure 2As shown, according to one embodiment of the present invention, in step S2, the step of obtaining the state vector of each follower and leader in the formation from a preset simulated flow field environment, the state vector contains seven elements, and the seven elements are: the horizontal and vertical deviations of the desired formation target position in the current follower's body coordinate system, the angle of attack of the leader and followers in the global coordinate system, the historical actions performed by the followers in the past two time steps, and the odd / even beat cycle flag of the follower in the current time step. In the process of constructing the state vector, incorporating the historical actions performed by the followers in the past two time steps into the state representation can conveniently and effectively capture the historical impact of vortices generated by the follower's actions propagating in a complex flow field; incorporating the odd / even beat cycle flag of the follower in the current time step into the state representation can conveniently and effectively ensure the periodicity and a certain degree of symmetry of the follower's "tail-whipping" action.

[0049] The above settings provide an effective guarantee for ensuring the accurate maintenance of the state between the leader and followers of the bionic fish, improve the efficiency of formation control in fluid environments, and effectively reduce the difficulty of control.

[0050] According to one embodiment of the present invention, step S2, which involves inputting the state vector of the current time step into a competitive dual-Q network, and selecting the action of each follower in the formation in accordance with a follower action selection strategy based on imitating leader behavior, includes:

[0051] S21. Input the state vector into the competing dual Q network and output the Q value of each action corresponding to the state vector.

[0052] S22. Generate a random number p within the range (0,1); this can be achieved using the Python built-in function random.random().

[0053] S23. If the current number of training rounds for the competing double Q-network is less than the set threshold and p is greater than the exploration rate ε, then the follower selects and executes the action corresponding to the maximum Q value; if 0.8ε < p < ε, then the follower randomly selects an action from the predefined action set and executes it, where the action set contains 5 maximum swings that the follower can adopt, namely {0.10, 0.11, 0.12, 0.13, 0.14}; if p < 0.8ε, then the follower directly executes the action of the leader at the current time step; in this embodiment, in the constructed action combination, the maximum swing it contains is a specialized design for the biomimetic fish flow field environment, effectively different from the real-time swing, which is more conducive to ensuring the effective control of the formation in the biomimetic fish flow field environment. Specifically, the computational fluid dynamics (CFD) simulation environment used in this solution requires that the speed and even the acceleration of the fish body during movement are smooth and there are no sudden changes. If the "real-time swing" is directly used as the follower's action output, the CFD solver is likely to generate grid calculation errors due to sudden changes in the fish body speed / acceleration; for this reason, in this embodiment, the follower can still achieve actions such as turning, accelerating, and decelerating by changing the "maximum swing" at each time step, obtaining an effect close to modifying the "real-time swing", effectively ensuring the consistency of the actions.

[0054] If the current number of training rounds for the competing double Q-network is greater than or equal to the set threshold and p is greater than the exploration rate ε, then the follower selects and executes the action corresponding to the maximum Q value; if p < ε, then the follower randomly selects an action from the predefined action set and executes it, that is, at this time ε has decayed to a very small value, and at this time the ε-greedy strategy is executed, that is, the follower randomly selects an action from the action set with a probability of the exploration rate ε and executes it, and executes the action corresponding to the maximum Q value with a probability of 1 - ε.

[0055] According to an embodiment of the present invention, in step S3, each time steps S2 to S3 are repeatedly executed for one training round, the exploration rate ε of the follower's action selection strategy decays once at a preset decay rate δ.

[0056] According to an embodiment of the present invention, in step S3, in the step of the competing double Q-network optimizing the imitation learning loss function with a decay coefficient based on the formation rewards / punishments of each follower and repeatedly executing steps S2 to S3 to obtain the trained network model, it includes:

[0057] S31. Each follower selects and executes the action chosen in step S2 to obtain the state vector and reward value for the next time step. In this embodiment, each follower obtains its own and the environment's state information to form a state vector of length 7. The state vector is input into a competitive dual Q network to obtain the Q value corresponding to each action. Then, it selects and executes the action based on the action selection strategy chosen in step S2 to obtain the state vector and reward value for the next time step.

[0058] S32. Collect the historical experience of each follower interacting with the environment and store it in the experience pool; where the historical experience is a quadruple (s,a,r,s′), where s is the state vector of a follower and the leader at the current time step, a is the action performed by the follower, i.e. the maximum amplitude of its tail flapping, r is the reward obtained by the follower for performing action a, and s′ is the new state vector formed by the follower and the leader together in the next time step after the follower performs action a;

[0059] S33. At each preset time step, random samples are taken from the experience pool, and the main network parameters of the main network in the competitive double Q network are updated in batches based on the imitation learning loss function with decay coefficient through the backpropagation algorithm. In addition, the target network parameters in the target network are softly updated, that is, the target network parameters are updated towards the main network parameters.

[0060] S34. When the number of training rounds of repeating steps S2 to S3 reaches the preset condition, the update of the main network parameters and the target network parameters ends, and the model parameters of the trained competitive double Q network are saved to obtain the network model.

[0061] According to one embodiment of the present invention, in step S3, the imitation learning loss function with attenuation coefficient consists of two parts: one part is the TD error L composed of the action value output by the competing dual Q network and the TD target. Q The other part is the imitation learning term λL with a decay coefficient λ. I , where the decay coefficient λ decays at a decay rate μ in each training round.

[0062] According to one embodiment of the present invention, in step S3, the update process is carried out in rounds; during the update process, each training round starts from the initial state. When the follower meets any of the following conditions, the training round ends and the learning of the next training round begins again: 1) the follower swims out of the defined domain range; 2) the total time steps of the follower exceed the maximum time steps specified in each round.

[0063] like Figure 1As shown, according to one embodiment of the present invention, step S4, which involves loading the network model online, obtaining the state vector of each follower and leader in a preset simulated flow field environment, and evaluating the Q-value of each follower's actions in the current state based on the state vector, includes:

[0064] S41. The network model obtains the state vector of each follower and leader in the current time step in the preset simulation flow field environment. The state vector contains seven elements, which are: the horizontal and vertical deviations of the expected formation target position in the current follower's fish body coordinate system, the angle of attack of the leader and followers in the global coordinate system, the historical actions performed by the followers in the past two time steps, and the odd / even beat cycle flag of the followers in the current time step.

[0065] S42. Using the state vector obtained in step S41 as the input to the network model, and evaluating the Q value of each follower's actions in the current state based on the Q function of the network model.

[0066] According to one embodiment of the present invention, in step S5, each follower selects the action corresponding to the maximum Q value and executes the action, thereby generating the state vector of the next time step. Then, steps S4 to S5 are repeated to maintain the formation configuration between the leader and the followers until the leader swims its pre-specified trajectory and completes the formation control task.

[0067] like Figure 3 As shown, according to one embodiment of the present invention, the competitive dual-Q network includes: a state value function branch and an advantage function branch;

[0068] The state-value function branch and the dominant function branch each contain four fully connected layers. The first fully connected layer of the state-value function branch and the dominant function branch is a shared layer with 128 hidden nodes. The second fully connected layer of the state-value function branch and the dominant function branch is a shared layer with 64 hidden nodes. The third fully connected layer of the state-value function branch and the third fully connected layer of the dominant function branch each have 64 hidden nodes. The fourth fully connected layer of the state-value function branch and the fourth fully connected layer of the dominant function branch each have 1 and 5 hidden nodes, respectively. The competitive dual-Q network includes a main network and a target network. The two networks have the same structure but different parameters; therefore, both the main network and the target network contain the same four fully connected layer structure described above.

[0069] To further illustrate this solution, examples are provided in conjunction with the accompanying drawings.

[0070] Example 1

[0071] Combination Figure 1 and Figure 2As shown, a preset simulation flow field environment is set: the leader swims out in a circular trajectory. The maximum swing amplitude of the two followers is controlled so that they are located two units to the upper right and lower right of the leader, respectively (i.e., the centroid Δx = 2, Δy = ±2).

[0072] The offline training phase is described below:

[0073] S1. Construct a competitive dual-Q network consisting of a main network and a target network; wherein the competitive dual-Q network is constructed based on a deep reinforcement learning framework.

[0074] S2. Obtain the state vectors of each follower and leader in the formation from the preset simulated flow field environment, and input the state vector of the current time step into the competitive dual Q network. Then, in conjunction with a follower action selection strategy based on imitating the leader's behavior, select the action of each follower in the formation; specifically including:

[0075] The state vectors (Δx, Δy, θ) of each follower and leader are obtained from the preset simulated flow field environment. l θ f a t-2 a t-1 The state vector has 7 elements, including the horizontal and vertical deviations Δx and Δy of the desired target position in the follower's body coordinate system, and the angle of attack θ between the leader and the follower in the global coordinate system. l The historical action a performed by the follower in the past two time steps, along with θf. t-2 With a t-1 The flag indicating the odd / even beat cycle of the follower at the current time step (flag = 1 if the current time step is odd; flag = 2 if the current time step is even).

[0076] Action selection is based on a follower action selection strategy that imitates leader behavior; a 7-length state vector is input into a competitive dual-Q network, which outputs the Q-value of each action; for example... Figure 3 As shown, in this embodiment, the competitive dual-Q network includes a state value function branch and a dominance function branch; both the state value function branch and the dominance function branch contain four fully connected layers (excluding the input layer and the output layer); the number of hidden nodes in the first fully connected layer (shared) of both branches is 128, the number of hidden nodes in the second fully connected layer (shared) of both branches is 64, and the number of hidden nodes in the third fully connected layer (separate) of both branches is 64. Except for the output layer, all branches use the ReLU activation function.

[0077] Furthermore, the number of hidden nodes in the last layer of the state value function branch and the advantage function branch are 1 and 5 respectively, representing the value function of the current state and the advantage functions of 5 candidate actions in the current state. The output of the network is the Q value of each candidate action in the current state, and its calculation formula is:

[0078]

[0079] Among them, V(s) is the value function of the current state output by the state value function branch; A(s, a) is the Q value of each candidate action in the current state output by the advantage function branch; |A| represents the size of the candidate action set. In this embodiment, |A| = 5;

[0080] Furthermore, generate a random number p that follows a uniform distribution within (0, 1);

[0081] Furthermore, if the current training episode number is less than the set threshold and p is greater than the exploration rate ε, the follower selects and executes the action corresponding to the maximum Q value; if 0.8ε < p < ε, the follower randomly selects an action from the action set and executes it. The action set is the 5 maximum swing amplitudes that the follower can take; if p < 0.8ε, the follower directly executes the action of the leader at the current time step;

[0082] If the current training episode number is greater than or equal to the set threshold, at this time ε has decayed to a very small value, and at this time the ε-greedy strategy is executed, that is, the follower randomly selects an action from the action set with a probability of ε and executes it, and executes the action corresponding to the maximum Q value with a probability of 1 - ε;

[0083] S3. The competitive double Q network optimizes the imitation learning loss function with a decay coefficient based on the formation rewards / punishments of each follower, and repeats steps S2 to S3 to obtain a trained network model; where, for each repetition of steps S2 to S3 for one training episode, the exploration rate ε of the follower's action selection strategy decays once at a decay rate of δ; where, in each training episode, ε ← min(0.05, 0.995ε);

[0084] In this embodiment, it specifically includes:

[0085] Establish a data buffer D with a capacity of N as an experience pool for saving historical experiences, which is initially empty. In this embodiment, N = 10000.

[0086] Furthermore, each follower selects and executes the action chosen in step S2 to obtain the state vector and reward value for the next time step. Each follower obtains its own and the environment's state information to form a state vector of length 7. The state vector is input into the competitive dual Q network to obtain the Q value corresponding to each action. Then, it selects and executes the action based on the action selection strategy chosen in step S2 to obtain the state vector and reward value for the next time step.

[0087] Furthermore, historical experiences of each follower's interaction with the environment are collected and stored in an experience pool. Each historical experience is a quadruple (s, a, r, s′), where s is the state vector of a follower and the leader at the current time step, a is the action performed by the follower (i.e., the maximum amplitude of its tail flapping), r is the reward gained by the follower for performing action a, and s′ is the new state vector formed by the follower and the leader at the next time step after performing action a.

[0088]

[0089] In this implementation, if the amount of data in the experience pool exceeds the maximum capacity of the experience pool, the latest experience data is used to replace the oldest experience data.

[0090] Furthermore, every C time steps, the Adam algorithm is applied to update the main network parameters in batches according to the loss function. The imitation learning loss function L with a decay coefficient is defined as follows:

[0091]

[0092] in,

[0093]

[0094] Where: N b Indicates the batch size, 0≤γ≤1 represents the discount factor, and cross_entrophy(A f A l ) represents the N extracted. b In the sample, the action distribution A of the followers f Distribution of Leader's Actions A l The cross-entropy measures the closeness between two distributions, and λ represents the weight of the cross-entropy loss. In this embodiment, the learning rate of the main network is set to 0.001, and N... b =128, γ=0.99, decays from 1 to 0 at a decay rate of 0.985.

[0095] Simultaneously, the target network parameters are softly updated, that is, the target network parameters ω are slowly updated towards the main network parameters ω. - :

[0096] ω - ←τω+(1-τ)ω - (5)

[0097] Where τ is the soft update rate. In this embodiment, C = 4, τ = 0.01.

[0098] Furthermore, when the number of training rounds in steps S2 to S3 reaches a preset condition, the update of the main network parameters and the target network parameters ends, and the model parameters of the trained competitive dual-Q network are saved to obtain the network model. The update process is performed on a round-by-round basis. During the update process, each training round starts from the initial state. The training round ends and the next training round begins when the follower meets any of the following conditions: 1) the follower swims out of the domain boundary; 2) the total time steps of the follower exceed the maximum time steps specified for each round. In this embodiment, the maximum time steps specified for each round are 200. Figure 4 As shown, in this embodiment, a total of 1200 training rounds were conducted.

[0099] The online execution phase is described below:

[0100] The network model is loaded online; in this case, the pre-trained network model is loaded after a competitive dual Q network is pre-constructed.

[0101] Furthermore, the network model obtains the state vector of each follower and leader in the current time step in the preset simulated flow field environment. The state vector contains seven elements, which are: the horizontal and vertical deviations of the expected formation target position in the current follower's fish body coordinate system, the angle of attack of the leader and followers in the global coordinate system, the historical actions performed by the followers in the past two time steps, and the odd / even beat cycle flag of the followers in the current time step.

[0102] Furthermore, the state vector obtained in step S41 is used as the input to the network model, and the Q value of each follower's action in the current state is evaluated based on the Q function of the network model.

[0103] S5. Each follower selects the action corresponding to the maximum Q value and executes the action, thereby generating the state vector for the next time step. Steps S4 to S5 are then repeated to maintain the formation configuration between the leader and followers until the leader completes its pre-assigned trajectory, thus completing the formation control task. In this embodiment, the trajectory diagram and keyframes of the three-fish formation are as follows: Figure 5 As shown.

[0104] The above description is merely an example of a specific solution of the present invention. For any devices and structures not described in detail herein, it should be understood that they are implemented using common devices and methods already available in the art.

[0105] The above description is merely one embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A biomimetic fish leader-follower formation control method based on deep reinforcement learning, including: S1. Construct a competitive dual-Q network that includes a main network and a target network; S2. Obtain the state vector of each follower and leader in the formation from the preset simulated flow field environment, and input the state vector of the current time step into the competitive dual Q network. In addition, in conjunction with the follower action selection strategy based on imitating the leader's behavior, select the action of each follower in the formation. The state vector contains seven elements, and the seven elements are: the horizontal and vertical deviations of the expected formation target position in the current follower's fish body coordinate system, the angle of attack of the leader and follower in the global coordinate system, the historical actions performed by the follower in the past two time steps, and the odd / even beat cycle flag of the follower in the current time step. S3. The competitive dual-Q network optimizes the imitation learning loss function with a decay coefficient based on the formation reward / penalty for each of the followers, and repeats steps S2 to S3 to obtain the trained network model; wherein, it includes: S31. Each follower performs the action selected in step S2 to obtain the state vector and reward value for the next time step; S32. Collect the historical experience of each follower interacting with the environment and store it in the experience pool; wherein, the historical experience is a quadruple. ,in, s Let be the state vector of a certain follower and the leader at the current time step. a The action performed by the follower is the maximum amplitude of its tail flapping. r Perform actions for the followers a The rewards obtained s' Perform actions for the followers a Then, in the next time step, a new state vector is formed together with the leader; S33. At each preset time step, random samples are taken from the experience pool, and the main network parameters of the main network in the competitive dual Q network are updated in batches based on the imitation learning loss function with a decay coefficient through the backpropagation algorithm. In addition, the target network parameters in the target network are softly updated, that is, the target network parameters are updated towards the main network parameters. S34. When the number of training rounds of repeating steps S2 to S3 reaches a preset condition, the update of the main network parameters and the target network parameters is terminated, and the model parameters of the trained competitive double Q network are saved to obtain the network model. S4. Load the network model online. The network model obtains the state vector of each follower and leader in the preset simulation flow field environment. Based on the state vector, evaluate the Q value of each action of each follower in the current state. S5. Each follower selects the action corresponding to the maximum Q value, thereby maintaining the formation configuration between the leader and the followers and completing the formation control task.

2. The biomimetic fish leader-follower formation control method according to claim 1, characterized in that, In step S2, the step of inputting the state vector of the current time step into the competitive dual-Q network, and selecting the action of each follower in the formation in conjunction with the follower action selection strategy based on imitating leader behavior, includes: S21. Input the state vector into the competitive dual Q network and output the Q value of each action corresponding to the state vector; S22. Generate random numbers within the range (0,1). p ; S23. If the current number of training rounds for the competitive dual-Q network is less than a set threshold, and p Greater than the exploration rate ε Then the follower selects and executes the maximum. Q The action corresponding to the value; if 0.8ε < p < ε The follower then randomly selects and executes an action from a predefined set of actions, wherein the set of actions contains five maximum swing amplitudes that the follower can employ; if p <0.8 ε Then the follower directly executes the leader's action at the current time step; If the current competition is dual Q The number of training epochs for the network is greater than or equal to the set threshold, and p Greater than the exploration rate ε Then the follower selects and executes the maximum. Q The action corresponding to the value; if p < ε Then the follower randomly selects an action from a predefined set of actions and executes it.

3. The biomimetic fish leader-follower formation control method according to claim 2, characterized in that, In step S3, after each training round of repeating steps S2 to S3, the exploration rate of the follower action selection strategy is increased. ε It decays once at a preset decay rate δ.

4. The biomimetic fish leader-follower formation control method according to claim 3, characterized in that, In step S3, the imitation learning loss function with attenuation coefficient consists of two parts: one part is the TD error composed of the action value output by the competitive dual-Q network and the TD target. The other part is the imitation learning term with a decay coefficient λ. , where the decay coefficient λ decays at a decay rate μ in each training round.

5. The biomimetic fish leader-follower formation control method according to claim 4, characterized in that, In step S3, each training round ends when the follower meets the following conditions: the follower swims out of the defined domain; or the total time steps of the follower's swimming exceed the maximum time steps specified for each round.

6. The biomimetic fish leader-follower formation control method according to claim 5, characterized in that, In step S4, the network model is loaded online, and the network model obtains the state vector of each follower and leader in a preset simulated flow field environment. The step of evaluating the Q value of each action of each follower in the current state based on the state vector includes: S41. The network model obtains the state vector of each follower and leader in the current time step in the preset simulation flow field environment. The state vector contains seven elements, and the seven elements are: the horizontal and vertical deviations of the expected formation target position in the current follower's fish body coordinate system, the angle of attack of the leader and follower in the global coordinate system, the historical actions performed by the follower in the past two time steps, and the odd / even beat cycle flag of the follower in the current time step. S42. Using the state vector obtained in step S41 as the input to the network model, and evaluating the Q value of each follower's action in the current state based on the Q function of the network model.

7. The biomimetic fish leader-follower formation control method according to claim 6, characterized in that, The competitive dual-Q network includes: a state value function branch and a dominance function branch; The state value function branch and the dominant function branch each contain four fully connected layers; wherein, the first fully connected layer of the state value function branch and the dominant function branch is a shared layer with 128 hidden nodes, the second fully connected layer of the state value function branch and the dominant function branch is a shared layer with 64 hidden nodes, the third fully connected layer of the state value function branch and the third fully connected layer of the dominant function branch each have 64 hidden nodes, and the fourth fully connected layer of the state value function branch and the fourth fully connected layer of the dominant function branch each have 1 and 5 hidden nodes, respectively.

Images