Reinforcement learning training loop method and device for biomimetic robotic fish navigation task

Abstract

The application relates to a reinforcement learning training loop method and device for a bionic robotic fish navigation task, wherein the method comprises the following steps: constructing a reinforcement learning training platform based on a fixed-point navigation task of a bionic robotic fish; establishing a kinematics model of the bionic robotic fish in a fluid to determine a numerical simulation environment; sending an instruction to the bionic robotic fish in the numerical simulation environment based on the reinforcement learning training platform, and collecting at least one of the attitude, speed and acceleration of the bionic robotic fish in the numerical simulation environment in real time to complete numerical simulation training; sending an instruction to the bionic robotic fish in a real experiment to control the swimming behavior of the bionic robotic fish, and synchronously receiving the position coordinates of the bionic robotic fish calculated by a binocular positioning system to perform real experiment training. Therefore, the problems that a related technology cannot accurately reflect the motion state of a robotic fish in a real experiment, is prone to causing a very high calculation cost, and cannot provide training data required by reinforcement learning within an acceptable time are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of bionic fish, and particularly relates to a reinforcement learning training loop method and device for a navigation task of a bionic robotic fish. BACKGROUND

[0002] In recent years, underwater micro robots have attracted extensive attention, and bionic robotic fish are particularly concerned due to their high energy efficiency and low noise. However, current bionic robotic fish can only move in a fixed mode or need to be manually controlled. In underwater detection and reconnaissance tasks, the flow field environment is usually complex, and the single movement mode or manual control of the robotic fish often cannot cope with real task scenarios. The reinforcement learning method can automatically train the robotic fish to complete a specific task, and has become a powerful tool for studying the swimming problem of the robotic fish. Reinforcement learning training requires a large amount of trajectory data of the robotic fish moving in the flow, which can generally be obtained from numerical simulation or field experiments. However, numerical simulation is limited by computing power and numerical accuracy, and the data obtained often deviates from the real environment, and the strategy obtained by training is difficult to be directly migrated to the experiment, and obtaining data through experiments will have a high cost.

[0003] In related technologies, the movement data of the robotic fish is generally obtained in a numerical simulation environment and trained. For example, a robotic fish model required for numerical simulation is established, trajectory, attitude, motion state and other data of the robotic fish are obtained through numerical simulation, and are synchronized for a reinforcement learning training algorithm. The motion strategy given by the algorithm is applied to the simulation environment, new motion data is provided, and then is continuously used for training. The motion strategy after training is deployed on the real robotic fish and tested in the field.

[0004] However, in related technologies, the motion strategy trained in the simulation environment is often difficult to be directly applied to the real experiment, and the simulation environment is limited by computing power and model accuracy, and cannot accurately reflect the motion state of the robotic fish in the real experiment, for example, complex nonlinear flow will interfere with the swimming speed and swimming direction of the robotic fish. The interference is usually difficult to be accurately described by the model, and the accurate solution of the influence of the flow on the robotic fish by using fluid-structure coupling numerical simulation will result in extremely high computing cost, and cannot provide the training data required by the reinforcement learning within an acceptable time, which needs to be improved. SUMMARY

[0005] The application provides a reinforcement learning training loop method and device for a navigation task of a biomimetic fish, to solve the problem that in the prior art, data obtained based on numerical simulation is used for reinforcement learning training, so that the swimming strategy obtained through training is difficult to be directly deployed in experiments, the simulation environment is limited by computing power and model accuracy, cannot accurately reflect the motion state of the biomimetic fish in a real experiment, and is prone to causing extremely high computing cost and being unable to provide training data required by reinforcement learning within an acceptable time.

[0006] The first aspect of the application provides a reinforcement learning training loop method for a navigation task of a biomimetic fish, comprising the following steps: constructing a reinforcement learning training platform based on a point navigation task of the biomimetic fish; establishing a kinematic model of the biomimetic fish in a fluid to determine a numerical simulation environment; sending an instruction to the biomimetic fish in the numerical simulation environment based on the reinforcement learning training platform, to control the swimming behavior of the biomimetic fish, and collecting at least one of the attitude, speed and acceleration of the biomimetic fish in the numerical simulation environment in real time, and synchronously obtaining the position coordinates of the biomimetic fish from the numerical simulation environment, to complete numerical simulation training; sending an instruction to the biomimetic fish in an actual experiment, to control the swimming behavior of the biomimetic fish, and collecting at least one of the attitude, speed and acceleration of the biomimetic fish sent through a wireless signal in real time, and synchronously receiving the position coordinates of the biomimetic fish calculated by a binocular positioning system, to perform actual experiment training.

[0007] Optionally, in an embodiment of the application, the construction of the reinforcement learning training platform comprises: taking the sensor signal and the position signal of the biomimetic fish as the state of reinforcement learning, and taking the turning and forward motion mode of the biomimetic fish as the selectable action, training through a preset proximal policy optimization algorithm, to obtain the reinforcement learning training platform.

[0008] Optionally, in an embodiment of the application, the loss function of the preset proximal policy optimization algorithm is:

[0009]

[0010] wherein θ and θ ′ respectively represent a network for parameter updating and a network for action sampling, s t and a t respectively represent the state signal measured by the biomimetic fish and the control action performed by the biomimetic fish, is a advantage function.

[0011] Optionally, in an embodiment of the present application, the establishing the kinematic model of the biomimetic fish in the fluid, determining the numerical simulation environment comprises: establishing at least one motion mode of the biomimetic fish, and establishing the kinematic model according to at least one parameter of the swimming speed, the pitch angle speed and the yaw angle speed of the biomimetic fish in a specific flow in all motion modes, to simulate the generated trajectory number, and determine the numerical simulation environment.

[0012] Optionally, in an embodiment of the present application, the training in the field experiment comprises: in the field experiment, calculating the coordinates of the biomimetic fish in a three-dimensional space coordinate system to obtain the actual position of the biomimetic fish, and synchronously obtaining the measured signal data of the biomimetic fish, and finally training by using the reinforcement learning platform until the biomimetic fish can complete the designated task in the field experiment.

[0013] The second aspect embodiment of the present application provides a reinforcement learning training loop device for a biomimetic fish navigation task, comprising: a construction module configured to construct a reinforcement learning training platform based on a point navigation task of a biomimetic fish; an establishment module configured to establish a kinematic model of the biomimetic fish in a fluid, and determine a numerical simulation environment; a first training module configured to send instructions to the biomimetic fish in the numerical simulation environment based on the reinforcement learning training platform, to control the swimming behavior of the biomimetic fish, and to collect at least one data of the attitude, speed and acceleration of the biomimetic fish in the numerical simulation environment in real time, and to synchronously obtain the position coordinates of the biomimetic fish from the numerical simulation environment, to complete the numerical simulation training; and a second training module configured to send instructions to the biomimetic fish in a field experiment, to control the swimming behavior of the biomimetic fish, and to collect at least one data of the attitude, speed and acceleration of the biomimetic fish sent by a wireless signal in real time, and to synchronously receive the position coordinates of the biomimetic fish calculated by a binocular positioning system, to perform the training in the field experiment.

[0014] Optionally, in an embodiment of the present application, the construction module comprises: a first training unit configured to take the sensor signal and the position signal of the biomimetic fish as the state of reinforcement learning, and take the turning and forward motion mode of the biomimetic fish as the selectable action, and train by using a preset proximal policy optimization algorithm to obtain the reinforcement learning training platform.

[0015] Optionally, in an embodiment of the present application, the loss function of the preset proximal policy optimization algorithm is:

[0016]

[0017] wherein θ and θ ′respectively represent a network for parameter updating and a network for action sampling, s t and a t respectively represent a state signal measured by the biomimetic fish and a control action executed by the biomimetic fish, is an advantage function.

[0018] Optionally, in an embodiment of the present application, the establishing module comprises an establishing unit configured to establish at least one motion mode of the biomimetic fish, establish the kinematic model according to at least one parameter of the swimming speed, the pitch angle speed and the yaw angle speed of the biomimetic fish in a specific flow in all motion modes, determine the number of generated trajectories, and determine the numerical simulation environment.

[0019] Optionally, in an embodiment of the present application, the second training module comprises a second training unit configured to calculate the coordinates of the biomimetic fish in a three-dimensional space coordinate system in the field experiment to obtain the actual position of the biomimetic fish, synchronously obtain the measured signal data of the biomimetic fish, and finally train by using the reinforcement learning platform until the biomimetic fish can complete the designated task in the field experiment.

[0020] The third aspect embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the reinforcement learning training loop method for the biomimetic fish navigation task as described in the above embodiments.

[0021] The fourth aspect embodiment of the present application provides a computer readable storage medium, which stores a computer program executable by a processor to implement the reinforcement learning training loop method for the biomimetic fish navigation task as described above.

[0022] The embodiments of the present application can combine numerical simulation and experimental data, use numerical simulation to obtain a large amount of coarse-grained data for reinforcement learning pre-training, then obtain a small amount of high-quality data in an experimental environment for fine-tuning training, and finally enable the reinforcement learning strategy to be applied to real fish in the experimental environment, so that the user can obtain a robust swimming strategy through the reinforcement learning method, overcoming the problem that the reinforcement learning strategy obtained based on numerical simulation simulation is difficult to apply to the actual scene, and avoiding the high cost of training based on experimental data. Therefore, the problems in the related art that the motion strategy trained in the simulation environment is often difficult to be directly applied to the real experiment, the simulation environment is limited by the computing power and the model precision, cannot accurately reflect the motion state of the fish in the real experiment, and is prone to high computing cost, cannot provide the training data required by the reinforcement learning within an acceptable time, etc. are solved.

[0023] Additional aspects and advantages of the present application will be made apparent from the following description, which, when taken in conjunction with the accompanying drawings, whores to explain the present application. BRIEF DESCRIPTION OF DRAWINGS

[0024] The above and / or additional aspects and advantages of the present application will become apparent and be made clear to those skilled in the art from the following description, taken in conjunction with the accompanying drawings, illustrating by way of example the best mode contemplated for carrying out the present application.

[0025] Figure 1 A flowchart of a reinforcement learning training loop method for a bionic robotic fish navigation task according to an embodiment of the present application;

[0026] Figure 2 A PPO algorithm network structure diagram of a reinforcement learning training loop method for a bionic robotic fish navigation task according to an embodiment of the present application;

[0027] Figure 3 A reinforcement learning platform diagram of a reinforcement learning training loop method for a bionic robotic fish navigation task according to an embodiment of the present application;

[0028] Figure 4 A binocular vision positioning diagram of a reinforcement learning training loop method for a bionic robotic fish navigation task according to an embodiment of the present application;

[0029] Figure 5 A robotic fish navigation task reinforcement learning training loop diagram of a reinforcement learning training loop method for a bionic robotic fish navigation task according to an embodiment of the present application;

[0030] Figure 6 A structural diagram of a reinforcement learning training loop device for a bionic robotic fish navigation task according to an embodiment of the present application;

[0031] Figure 7 A structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0032] Embodiments of the present application are described in detail below with reference to the attached drawings, which show by way of example, embodiments in which like reference numerals refer to like elements or elements having the same or similar function throughout. The embodiments described below are examples intended to explain the present application, and are not to be understood as limiting the present application.

[0033] The method and device for reinforcement learning training loop of the navigation task of the biomimetic fish robot are described below with reference to the accompanying drawings. In the related art mentioned in the background, the motion strategy trained in the simulation environment is often difficult to be directly applied to the real experiment. The simulation environment is limited by the computing power and model accuracy, and cannot accurately reflect the motion state of the robot in the real experiment, and is prone to high computing cost, which cannot provide the training data required by reinforcement learning within an acceptable time. The present application provides a reinforcement learning training loop method for the navigation task of the biomimetic fish robot. In the method, numerical simulation and experimental data can be combined to obtain a large amount of coarse-grained data for reinforcement learning pre-training, and then a small amount of high-quality data is obtained in the experimental environment for fine-tuning training, so that the reinforcement learning strategy can be applied to the real robot in the experimental environment. Thus, the user can obtain a robust swimming strategy through the reinforcement learning method, overcoming the problem that the reinforcement learning strategy obtained based on numerical simulation simulation is difficult to be applied to the actual scene. Also, the high cost caused by training based on experimental data is avoided. Thus, the problems in the related art, such as the motion strategy trained in the simulation environment being difficult to be directly applied to the real experiment, the simulation environment being limited by the computing power and model accuracy, and being unable to accurately reflect the motion state of the robot in the real experiment, and being prone to high computing cost, which cannot provide the training data required by reinforcement learning within an acceptable time, are solved.

[0034] Specifically, Figure 1 A flowchart of the reinforcement learning training loop method for the navigation task of the biomimetic fish robot provided by the present application is shown.

[0035] As Figure 1 shown, the reinforcement learning training loop method for the navigation task of the biomimetic fish robot includes the following steps:

[0036] In step S101, a reinforcement learning training platform is constructed based on the fixed-point navigation task of the biomimetic fish robot.

[0037] In actual execution, the present application can construct a reinforcement learning training platform based on the fixed-point navigation task of the biomimetic fish robot, thereby facilitating reinforcement learning pre-training, training the robot to autonomously adjust the swimming strategy in a complex flow field to achieve stable and efficient directional motion or stay in a specific area, and finally enabling the reinforcement learning strategy to be applied to the real robot in the experimental environment.

[0038] Optionally, in an embodiment of the present application, constructing the reinforcement learning training platform includes: taking the sensor signals and position signals of the biomimetic fish robot as the state of reinforcement learning, and taking the motion modes of turning and advancing of the biomimetic fish robot as the selectable actions, training through a preset proximal strategy optimization algorithm to obtain the reinforcement learning training platform.

[0039] It is understood that reinforcement learning in the embodiments of this application is an unsupervised machine learning method that can use the interaction data between an agent such as a robotic fish and the environment to train a decision-maker to give the agent the best action in different states. Reinforcement learning relies on the real-time interaction between the robotic fish and the environment.

[0040] As one possible implementation, embodiments of this application can use sensor signals from the biomimetic robotic fish, as well as position signals provided by an external positioning system, as the state s for reinforcement learning. n And the turning and forward movement patterns of the bionic robotic fish are selected as actions. n Based on the fixed-point navigation task of the biomimetic robotic fish, the reward function of reinforcement learning is set as the distance between the current time step and the previous time step. The distance between the biomimetic robotic fish and the target position can be expressed as: r n =d n+1 -d n .

[0041] This application embodiment can be trained using a pre-defined proximal policy optimization algorithm (PPO2) to obtain a reinforcement learning training platform. PPO (Proximal Policy Optimization) is a deep reinforcement learning algorithm that can handle cases where both states and actions are continuous values. Specifically, the PPO algorithm involves two deep neural networks: an action network (Actor) and a critique network (Critic).

[0042] like Figure 2 As shown, the action network takes the current state as input and outputs the probability distribution of the action. This method uses a Gaussian distribution for all actions; therefore, when the action is a d... a When dealing with 2D vectors, the output of the action network is a 2D vector. a The vector is 3D, including the mean and variance of each dimension of the action. In this embodiment, the mean and variance are used for sampling when selecting a specific action. The input to the evaluation network is also the current state, and the output is the value function (scalar) of the current state.

[0043] In this embodiment, the sensor signals and position signals of the biomimetic robotic fish can be used as the state for reinforcement learning, and trained by a preset proximal strategy optimization algorithm. This ensures that the robotic fish can be further trained to autonomously adjust its swimming strategy in complex flow fields, so as to achieve stable and efficient directional movement or stay in a specific area.

[0044] It should be noted that the reinforcement learning problem is designed for the fixed-point navigation task of the robotic fish, and the PPO reinforcement learning algorithm is used in the embodiment of the application. In fact, other mature, reliable and suitable reinforcement learning algorithms can replace the PPO algorithm, such as DDPG, D3QN, Q-Learning, etc.

[0045] Optionally, in an embodiment of the application, the loss function of the preset proximal policy optimization algorithm is:

[0046]

[0047] wherein θ and θ ′ respectively represent the network for parameter updating and the network for action sampling, s t and a t respectively represent the state signal measured by the bionic robotic fish and the control action performed by the bionic robotic fish, is the advantage function.

[0048] In actual execution, the loss function of the preset proximal policy optimization algorithm can be used in the embodiment of the application to:

[0049]

[0050] improve the accuracy of calculation, wherein the generalized advantage estimation can be used for calculation in the embodiment of the application.

[0051] Further, after designing the reinforcement learning problem, a reinforcement learning training platform can be written to enable the reinforcement learning strategy to be trained using experience fragments generated in numerical simulation or experiments. As shown in the reinforcement learning training platform diagram Figure 3 , the embodiment of the application can build a numerical simulation environment, save experience fragments for playback caching, and the PPO algorithm involves two deep neural networks, namely the action network (Actor) and the evaluation network (Critic). The evaluation network combined with the target network can obtain the loss function of the PPO algorithm, and the evaluation network is updated after gradient descent.

[0052] In step S102, a kinematic model of the bionic robotic fish in the fluid is established, and a numerical simulation environment is determined.

[0053] In actual execution, the embodiment of the application can establish a kinematic model of the bionic robotic fish in the fluid to determine the numerical simulation environment, thereby ensuring that the reinforcement learning strategy can be applied to the real robotic fish in the experimental environment.

[0054] Optionally, in an embodiment of the present application, a kinematic model of the biomimetic robotic fish in the fluid is established, and a numerical simulation environment is determined, including: establishing at least one motion mode of the biomimetic robotic fish, and establishing a kinematic model according to at least one parameter of the swimming speed, the pitch angle speed and the yaw angle speed of the biomimetic robotic fish in a specific flow in all motion modes, to simulate the number of generated trajectories, and determine the numerical simulation environment.

[0055] Specifically, in order to pre-train in the numerical simulation environment, the embodiment of the present application needs to establish a kinematic model of the biomimetic robotic fish in the fluid. First, at least one motion mode of the biomimetic robotic fish is established, including but not limited to straight line motion, left turn at different speeds, right turn, etc.; then, the kinematic model of the biomimetic robotic fish is established by measuring the parameters such as swimming speed, pitch angle speed and yaw angle speed of the robotic fish in a specific flow in all motion modes through experiments, and the number of trajectories generated by numerical simulation is used to determine the numerical simulation environment, thereby taking advantage of a large amount of data provided by numerical simulation and high-quality data provided by experiments to make up for the defects of the reinforcement learning strategy that is difficult to land.

[0056] The embodiment of the present application can use the following biomimetic robotic fish model:

[0057] v = + v s n,

[0058]

[0059] wherein v and n are the translational speed and forward direction of the robotic fish, ω s is the rotational angular speed of the robotic fish, and u is the local fluid speed of the robotic fish.

[0060] Next, the embodiment of the present application can use computational fluid dynamics related methods to perform numerical simulation based on the obtained kinematic model of the robotic fish, to obtain the motion trajectory data of the robotic fish in the flow.

[0061] In step S103, based on the reinforcement learning training platform, an instruction is sent to the biomimetic robotic fish in the numerical simulation environment to control the swimming behavior of the biomimetic robotic fish, and at least one data of the attitude, speed and acceleration of the biomimetic robotic fish in the numerical simulation environment is collected in real time, and the position coordinates of the biomimetic robotic fish are synchronously obtained from the numerical simulation environment, to complete the numerical simulation training.

[0062] In actual execution, the embodiment of the application can send instructions to the bionic robotic fish in the numerical simulation environment based on the reinforcement learning training platform, control the swimming behavior of the bionic robotic fish by using the numerical simulation software interface, and collect the parameters of the bionic robotic fish in the numerical simulation environment in real time, including but not limited to attitude, position, speed, acceleration and the like, so that the training can be performed in the numerical simulation environment, and ultimately the numerical simulation training can be completed.

[0063] In step S104, instructions are sent to the bionic robotic fish in the field experiment to control the swimming behavior of the bionic robotic fish, and at least one of the attitude, speed and acceleration data sent by the bionic robotic fish through wireless signals is collected in real time, and the position coordinates of the bionic robotic fish calculated by the binocular positioning system are received synchronously to perform field experiment training.

[0064] As a possible implementation manner, the embodiment of the application can obtain the required training data through wireless communication and visual positioning system. Specifically, the embodiment of the application can send instructions to the bionic robotic fish in the field experiment to control the swimming behavior of the bionic robotic fish, so as to obtain the required training data through the measured signals and external positioning system, and use the Qt program framework to write the control system to send instructions to the bionic robotic fish to control the swimming behavior of the bionic robotic fish. Specifically, the selectable swimming behavior of the bionic robotic fish is divided into six types, namely, stopping, large left turning, small left turning, straight forward, small right turning and large right turning. The swimming behavior is determined in advance by hardware programming. After the control system in the upper computer sends a specific instruction to the bionic robotic fish, the bionic robotic fish will execute the swimming behavior specified by the instruction until the next instruction is received.

[0065] Meanwhile, the control system has a signal receiving function, which can collect the attitude, speed, acceleration and the like of the bionic robotic fish sent through wireless signals in real time, and synchronously receive the position coordinates of the bionic robotic fish calculated by the binocular positioning system. Based on the real-time signal data, the control system uses the reinforcement learning algorithm to infer and obtain the optimal control action, and sends it to the bionic robotic fish in real time. In addition, the upper computer establishes a database to store the data of the bionic robotic fish to support the real-time monitoring and control of the bionic robotic fish by the upper computer.

[0066] The embodiment of the application can further obtain the required training data through the measured signals and external positioning system, so as to ensure that the reinforcement learning strategy can be applied to the real bionic robotic fish in the experimental environment after the above-mentioned reinforcement learning pre-training is performed by using a large amount of coarse-grained data obtained by numerical simulation, and fine-tuning training is performed by using a small amount of high-quality data obtained by field experiment.

[0067] Optionally, in an embodiment of the present application, to conduct field experiment training, including: in the field experiment, calculating the coordinates of the biomimetic robotic fish in the three-dimensional space coordinate system to obtain the actual position of the biomimetic robotic fish, and synchronously obtaining the measured signal data of the biomimetic robotic fish, using the reinforcement learning platform to conduct final training until the biomimetic robotic fish can complete the designated task in the field experiment.

[0068] It can be understood that the external positioning of the biomimetic robotic fish in the embodiment of the present application is based on a binocular positioning algorithm, which can use two fixed cameras at different positions to identify the target of the robotic fish, ensuring that in the field experiment, the coordinates of the biomimetic robotic fish in the three-dimensional space coordinate system are calculated, thereby helping to obtain the actual position of the biomimetic robotic fish until the training is completed.

[0069] In actual execution process, as shown in Figure 4 , the binocular positioning algorithm in the embodiment of the present application can be divided into three steps:

[0070] (1) obtaining the images of the two cameras at the current time;

[0071] (2) processing the images to identify the coordinates (x (i) ,y (i) ) of the center position of the robotic fish in the images, wherein i represents the i-th camera;

[0072] (3) through the camera coordinate transformation relationship in the following steps, the three-dimensional coordinates (x w ,y w ,z w ) of the center position of the robotic fish in the world coordinate system are solved.

[0073]

[0074]

[0075] wherein z ci ,f (i) are the internal parameter coefficients of the camera, R 33 (i) and T 31 (i) are the rotation and translation matrices of the camera, respectively.

[0076] After obtaining the three-dimensional position of the biomimetic fish in the water area, the embodiment of the application can send and store the relevant information to the control system, thereby providing a feasible and complete reinforcement learning training loop, so that the user can obtain a robust swimming strategy through the reinforcement learning method, and good results can also be obtained in the experimental environment. Combining the advantages of numerical simulation and experiment, the problem that the reinforcement learning strategy obtained based on numerical simulation simulation is difficult to apply to the actual scene is overcome, and the high cost caused by training based on experimental data is also avoided.

[0077] It should be noted that the binocular vision positioning algorithm is used to obtain the three-dimensional spatial coordinates of the fish. In fact, according to specific conditions, other positioning algorithms such as infrared positioning, ultrasonic positioning, laser positioning, GPS positioning, etc. can also be used.

[0078] Specifically, as shown in Figure 5 The working principle of the reinforcement learning training loop method of the biomimetic fish navigation task in the embodiment of the application is described in detail with a specific embodiment.

[0079] As shown in Figure 5 The real-time motion state of the biomimetic fish can be sent to the host computer, and the control system in the host computer can send control information to the biomimetic fish. The position information of the biomimetic fish is measured in real time by the binocular vision positioning system and uploaded to the host computer.

[0080] Further, the host computer in the embodiment of the application can perform reinforcement learning on the state information through the reinforcement learning decision maker, and send the decision information to the host computer; the embodiment of the application can train the experience fragments generated in numerical simulation or experiment to obtain the reinforcement learning strategy.

[0081] The reinforcement learning training loop method for biomimetic robotic fish navigation tasks proposed in this application combines numerical simulation and experimental data. Numerical simulation yields a large amount of coarse-grained data for reinforcement learning pre-training, while a small amount of high-quality data is obtained in the experimental environment for fine-tuning training. Ultimately, the reinforcement learning strategy can be applied to a real robotic fish in the experimental environment, enabling users to obtain robust swimming strategies through reinforcement learning. This overcomes the problem that reinforcement learning strategies obtained entirely from numerical simulation are difficult to apply to real-world scenarios and avoids the high costs associated with training solely based on experimental data. Therefore, this solves the problem in related technologies where training only focuses on the joint motion parameters of the robotic fish (e.g., tail fin swing parameters) rather than the specific robotic fish task (e.g., the fixed-point navigation problem considered in this solution). Furthermore, the simulation environment is limited by computational power and model accuracy, failing to accurately reflect the motion state of the robotic fish in real experiments, and easily leading to extremely high computational costs, making it impossible to provide the training data required for reinforcement learning within an acceptable timeframe.

[0082] Next, referring to the accompanying drawings, a reinforcement learning training loop device for a biomimetic robotic fish navigation task proposed according to an embodiment of this application is described.

[0083] Figure 6 This is a schematic diagram of the reinforcement learning training loop device for the navigation task of the biomimetic robotic fish according to an embodiment of this application.

[0084] like Figure 6 As shown, the reinforcement learning training loop device 10 for the biomimetic robotic fish navigation task includes: a construction module 100, an establishment module 200, a first training module 300, and a second training module 400.

[0085] Specifically, module 100 is constructed for building a reinforcement learning training platform for point-to-point navigation tasks based on biomimetic robotic fish.

[0086] Module 200 is established to create a kinematic model of the biomimetic robotic fish in fluid and to determine the numerical simulation environment.

[0087] The first training module 300 is used to send instructions to the bionic robotic fish in the numerical simulation environment based on the reinforcement learning training platform to control the swimming behavior of the bionic robotic fish, and to collect at least one of the attitude, velocity and acceleration of the bionic robotic fish in the numerical simulation environment in real time, and to synchronously obtain the position coordinates of the bionic robotic fish from the numerical simulation environment in order to complete the numerical simulation training.

[0088] The second training module 400 is configured to send instructions to the bionic robotic fish in the field experiment to control the swimming behavior of the bionic robotic fish, and to collect at least one of the posture, speed and acceleration of the bionic robotic fish sent by the bionic robotic fish through wireless signals in real time, and to synchronously receive the position coordinates of the bionic robotic fish calculated by the binocular positioning system to perform field experiment training.

[0089] Optionally, in an embodiment of the present application, the construction module 100 comprises a first training unit.

[0090] The first training unit is configured to take the sensor signal and the position signal of the bionic robotic fish as the state of reinforcement learning, and take the turning and forward movement mode of the bionic robotic fish as the selectable action, and train the bionic robotic fish through a preset proximal policy optimization algorithm to obtain a reinforcement learning training platform.

[0091] Optionally, in an embodiment of the present application, the loss function of the preset proximal policy optimization algorithm is:

[0092]

[0093] wherein θ and θ ′ respectively represent a network for parameter updating and a network for action sampling, s t and a t respectively represent a state signal measured by the bionic robotic fish and a control action performed by the bionic robotic fish, is an advantage function.

[0094] Optionally, in an embodiment of the present application, the establishment module 200 comprises an establishment unit.

[0095] The establishment unit is configured to establish at least one movement mode of the bionic robotic fish, to establish a kinematic model according to at least one parameter of the swimming speed, the pitch angle speed and the yaw angle speed of the bionic robotic fish in a specific flow, to simulate the number of generated trajectories, and to determine a numerical simulation environment.

[0096] Optionally, in an embodiment of the present application, the second training module comprises a second training unit.

[0097] The second training unit is configured to calculate the coordinates of the bionic robotic fish in a three-dimensional space coordinate system in the field experiment to obtain the actual position of the bionic robotic fish, to synchronously obtain the measured signal data of the bionic robotic fish, to perform final training by using the reinforcement learning platform, and to enable the bionic robotic fish to complete a designated task in the field experiment.

[0098] It should be noted that the foregoing explanation of the embodiment of the reinforcement learning training loop method for the navigation task of the biomimetic fish is also applicable to the embodiment of the reinforcement learning training loop device for the navigation task of the biomimetic fish, which will not be repeated here.

[0099] The reinforcement learning training loop device for the navigation task of the biomimetic fish according to the embodiment of the present application can combine numerical simulation and experimental data, use numerical simulation to obtain a large amount of coarse-grained data for pre-training of reinforcement learning, and then obtain a small amount of high-quality data in an experimental environment for fine-tuning training, so that the reinforcement learning strategy can be applied to real fish in the experimental environment, thereby enabling the user to obtain a robust swimming strategy through the reinforcement learning method, overcoming the problem that the reinforcement learning strategy obtained based on numerical simulation simulation is difficult to apply to the actual scene, and avoiding the high cost of training based on experimental data. Thus, the problem in the related art that the movement strategy trained only in the simulation environment is often difficult to be directly applied to the real experiment, the simulation environment is limited by the computing power and the model accuracy, and cannot accurately reflect the movement state of the fish in the real experiment, and is prone to result in high computing cost and cannot provide the training data required by the reinforcement learning within an acceptable time is solved.

[0100] Figure 7 The structure schematic diagram of the electronic device provided by the embodiment of the present application is provided. The electronic device can include:

[0101] The memory 701, the processor 702, and the computer program stored in the memory 701 and executable on the processor 702.

[0102] The processor 702 implements the reinforcement learning training loop method for the navigation task of the biomimetic fish provided in the above embodiments when executing the program.

[0103] Further, the electronic device further includes:

[0104] The communication interface 703 is used for communication between the memory 701 and the processor 702.

[0105] The memory 701 is used to store the computer program executable on the processor 702.

[0106] The memory 701 can include a high-speed RAM memory, and can also include a non-volatile memory, for example, at least one disk memory.

[0107] If the memory 701, the processor 702 and the communication interface 703 are implemented independently, the communication interface 703, the memory 701 and the processor 702 can be connected with each other through a bus and complete communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 7 Only one thick line is used to represent the bus in the figure, but it does not mean that there is only one bus or only one type of bus.

[0108] Optionally, in a specific implementation, if the memory 701, the processor 702 and the communication interface 703 are integrated on a chip, the memory 701, the processor 702 and the communication interface 703 can complete communication between each other through an internal interface.

[0109] The processor 702 can be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the present application.

[0110] The embodiment also provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the above-mentioned method for training a reinforcement learning loop for a navigation task of a bionic robotic fish.

[0111] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0112] Moreover, the terms "first", "second", "third", etc. are used herein only to describe different steps or categories of steps. Thus, the use of the term "first" does not imply that different steps must be in a time sequence. Nor is it implied that a "first" step must precede a "second" step, that a "second" step, etc. must follow a "first" step, etc. Furthermore, when a process or method is described herein with several steps or several categories of steps, it should be understood that these are merely illustrative of the steps that can be employed in the process or method. Not all of the steps can be required, and in some cases, additional steps can be employed. The order of the steps can be varied, and some of the steps can be performed simultaneously. The steps can be performed in an order different than that described herein. The steps can be performed in any order, unless otherwise specified.

[0113] Any process or method described in flowcharts or otherwise described herein can be understood as representing code modules, segments, or portions of code that include one or more executable instructions for performing specific logic functions (or steps) or portions thereof, and the various embodiments of the application can include additional or fewer steps performing the same or equivalent functions. In some embodiments, the order of steps can be varied, and / or some steps can be performed simultaneously, unless otherwise specifically noted. The various embodiments of the application can be embodied in a number of different forms, all of which have been contemplated to be within the scope of the applicable patent princi¬ ples described herein.

[0114] Logic and / or steps represented in flowcharts or otherwise described herein, for example, can be embodied in computer-readable media, which can be any available media that can be accessed by a general purpose or special purpose computing system, device, or apparatus to execute instructions stored in the media. By way of example, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a general purpose or special purpose computing system, device, or apparatus. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or combination thereof) to a general purpose or special purpose computing system, device, or apparatus, the computer-readable media used for this purpose can be any available media that can be accessed by this system, device, or apparatus. Combinations of the above should also be included within the scope of the computer-readable media.

[0115] It should be understood that portions of the application can be realized with a combination of hardware, software, firmware, or their combination. In the above-described embodiments, the N steps or methods can be realized with software or firmware stored in a memory and executed by a suitable instruction execution system. As in another embodiment, if realized with hardware, any one or their combination of the following technologies known in the art can be used: discrete logic circuit with logic gate circuit for implementing logic functions on data signals, application specific integrated circuit with suitable combination logic gate circuit, programmable gate array (PGA), field programmable gate array (FPGA), etc.

[0116] Those skilled in the art of the present technology can understand that all or part of the steps carried out by the above-mentioned embodiments can be completed by a program instructing the relevant hardware, and the program can be stored in a computer readable storage medium. When the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0117] In addition, each functional unit in each embodiment of the present application can be integrated into one processing module, or each unit can be physically present separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be realized in the form of hardware or in the form of a software function module. The integrated module, if realized in the form of a software function module and sold or used as an independent product, can also be stored in a computer readable storage medium.

[0118] The above-mentioned storage medium can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above-mentioned embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-mentioned embodiments within the scope of the present application.

Claims

1. A method for reinforcement learning training loop of a biomimetic robotic fish navigating a task, characterized in that, The method comprises the following steps: Based on the point navigation task of the biomimetic fish robot, a reinforcement learning training platform is constructed, wherein the reinforcement learning training platform is constructed by taking the sensor signals and position signals of the biomimetic fish robot as the states of reinforcement learning, taking the turning and forward movement modes of the biomimetic fish robot as selectable actions, and training through a preset proximal policy optimization algorithm to obtain the reinforcement learning training platform; The loss function of the preset proximal policy optimization algorithm is: wherein, and respectively represent a network for performing parameter updating and a network for performing action sampling, and respectively represent a state signal measured by the bionic robotic fish and a control action performed by the bionic robotic fish, is an advantage function; A kinematic model of the biomimetic fish robot in fluid is established to determine a numerical simulation environment; Based on the reinforcement learning training platform, instructions are sent to the biomimetic fish robot in the numerical simulation environment to control the swimming behavior of the biomimetic fish robot, at least one of the attitude, speed and acceleration of the biomimetic fish robot in the numerical simulation environment is collected in real time, and the position coordinates of the biomimetic fish robot are synchronously obtained from the numerical simulation environment to complete numerical simulation training; Instructions are sent to the biomimetic fish robot in the real experiment to control the swimming behavior of the biomimetic fish robot, at least one of the attitude, speed and acceleration of the biomimetic fish robot sent through wireless signals is collected in real time, and the position coordinates of the biomimetic fish robot calculated by the binocular positioning system are synchronously received to perform real experiment training, wherein the real experiment training comprises calculating the coordinates of the biomimetic fish robot in a three-dimensional space coordinate system in the real experiment to obtain the actual position of the biomimetic fish robot, synchronously obtaining the measured signal data of the biomimetic fish robot, and finally training by using the reinforcement learning training platform until the biomimetic fish robot can complete the specified task in the real experiment.

2. The method of claim 1, wherein, The establishment of the kinematic model of the biomimetic fish robot in fluid to determine the numerical simulation environment comprises: At least one movement mode of the biomimetic fish robot is established, at least one parameter of the swimming speed, pitch angle speed and yaw angle speed of the biomimetic fish robot in a specific flow is measured, the kinematic model is established to simulate the generated trajectory number, and the numerical simulation environment is determined.

3. A reinforcement learning training loop device for a biomimetic robotic fish navigation task, characterized in that, Comprises: A construction module is configured to construct a reinforcement learning training platform based on the point navigation task of the biomimetic fish robot, and the construction module comprises a first training unit configured to take the sensor signals and position signals of the biomimetic fish robot as the states of reinforcement learning, take the turning and forward movement modes of the biomimetic fish robot as selectable actions, train through a preset proximal policy optimization algorithm, and obtain the reinforcement learning training platform; The loss function of the preset proximal policy optimization algorithm is: wherein, and respectively represent a network for performing parameter updating and a network for performing action sampling, and respectively represent a state signal measured by the bionic fish and a control action performed by the bionic fish, is an advantage function; An establishment module is configured to establish a kinematic model of the biomimetic fish robot in fluid and determine a numerical simulation environment. The first training module is configured to send instructions to the bionic robotic fish in the numerical simulation environment based on the reinforcement learning training platform to control the swimming behavior of the bionic robotic fish, and to collect at least one of the posture, speed and acceleration of the bionic robotic fish in the numerical simulation environment in real time and synchronously acquire the position coordinates of the bionic robotic fish from the numerical simulation environment to complete the numerical simulation training. The second training module is configured to send instructions to the bionic robotic fish in the field experiment to control the swimming behavior of the bionic robotic fish, and to collect at least one of the posture, speed and acceleration of the bionic robotic fish sent through wireless signals in real time and synchronously receive the position coordinates of the bionic robotic fish calculated by the binocular positioning system to perform the field experiment training. The second training module includes a second training unit configured to calculate the coordinates of the bionic robotic fish in a three-dimensional space coordinate system in the field experiment to obtain the actual position of the bionic robotic fish, synchronously acquire the measured signal data of the bionic robotic fish, and perform final training by using the reinforcement learning training platform until the bionic robotic fish can complete the designated task in the field experiment.

4. The apparatus of claim 3, wherein, The establishing module includes: An establishing unit configured to establish at least one motion mode of the bionic robotic fish, establish the kinematic model according to at least one parameter of the swimming speed, the pitch angle speed and the yaw angle speed of the bionic robotic fish in all motion modes in a specific flow, and determine the numerical simulation environment according to the number of generated trajectories.

5. An electronic device, comprising: The computer program is stored in the memory and executable on the processor, and the processor executes the program to implement the reinforcement learning training loop method for the navigation task of the bionic robotic fish according to any one of claims 1-2. The program is executed by the processor to implement the reinforcement learning training loop method for the navigation task of the bionic robotic fish according to any one of claims 1-2.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that, ​

Images