Three-dimensional snakelike robot tracking control method based on state sparse perception

Abstract

The invention discloses a three-dimensional snakelike robot tracking control method based on state sparse perception, and belongs to the technical field of robot control, and the method comprises the steps: constructing a strategy network based on deep reinforcement learning; environment state parameters are obtained, and potential actions of the three-dimensional snake-shaped robot are generated according to the environment state parameters and the strategy network; generating a predicted action sequence by using the potential actions; generating a motion gait sequence of the three-dimensional snakelike robot based on the predicted action sequence; and controlling the three-dimensional snakelike robot to sequentially execute gaits in the motion gait sequence to adjust the advancing direction. According to the three-dimensional snake-shaped robot tracking control method based on state sparse perception, potential representation of an action sequence is introduced, a control strategy optimizes potential actions to predict dense movement gaits, and therefore the training efficiency and the movement performance are obviously improved.

Description

Technical Field

[0001] The invention relates to the technical field of robot control, and in particular to a three-dimensional snake-like robot tracking control method based on state sparse perception. Background Art

[0002] As a bionic multi-module robot, snake robots are widely used in exploration, reconnaissance, search and rescue and other fields. Their high maneuverability and strong adaptability make their motion control the focus of many researchers.

[0003] The three-dimensional snake robot consists of alternating yaw joints and pitch joints. This unique mechanism enables it to perform a variety of three-dimensional gaits, such as sinus lift gait, crawling gait, track gait, and side-winding gait. Deep reinforcement learning (DRL) is an optimization technique widely used in robot control and has achieved remarkable results. Autonomous training of motion control using the DRL algorithm can avoid the tedious modeling and parameter adjustment process; in addition, the learned control strategy usually shows excellent generalization ability. Therefore, many outstanding studies have used DRL to optimize the control strategy of snake robots. However, the current motion control methods using DRL to optimize snake robots focus on controlling planar gaits. According to research, there are still few applications of DRL in the motion control of three-dimensional snake robots.

[0004] In addition, in the motion control task of this paper, state perception is sparse and can only be updated periodically, which is significantly different from other typical tasks. Sparse state perception is a common problem in robot control, and its causes are diverse. For example, in complex environments, occlusion by obstacles, processor computing power limitations, communication interruptions, insufficient energy, etc., can all lead to incomplete state perception. In this case, the control strategy cannot obtain the current state in time to update the motion gait, resulting in motion interruption or even robot falls.

[0005] Although the problem of sparse state perception is common in robot control, existing studies have shown that this problem has not been fully studied and solved in the field of robots. For example, the Chinese invention patent with publication number CN113063416A proposes a robot posture fusion method based on adaptive parameter complementary filtering, which improves the posture estimation accuracy and robustness of the robot during high-speed motion by dynamically adjusting the fusion weights of the accelerometer and the magnetometer. However, this method is mainly aimed at planar motion control, and has not yet solved the motion control problem of a three-dimensional snake-like robot under sparse state perception. The Chinese invention patent with publication number CN116442246B proposes a posture balance control method applied to a robot, which improves the continuity of the robot's posture balance control by frame splitting the robot's motion model. However, this method is mainly aimed at the balance control of bipedal or multi-legged robots, and has not yet solved the motion control problem of a three-dimensional snake-like robot under sparse state perception.

[0006] Therefore, how to use the DRL algorithm to effectively adjust the 3D gait to complete the motion control task under sparse environmental state perception is a challenging task.

[0007] In order to solve the above problems, it is urgent to design a new algorithm that can predict the dense three-dimensional snake-like robot's movements based on sparse states, so as to realize the random point tracking control task of its tracked gait. Summary of the invention

[0008] The present invention aims to solve at least one of the technical problems in the related art to a certain extent. To this end, the present invention provides a tracking control method for a three-dimensional snake-like robot based on sparse state perception, which can enable a wheelless three-dimensional snake-like robot with high redundancy to maintain dense and continuous motion under sparse state perception and successfully complete the point tracking task by introducing a potential representation of action sequences.

[0009] To achieve the above objectives, in a first aspect, the present application provides a three-dimensional snake-like robot tracking control method based on state sparse perception, comprising: Build a policy network based on deep reinforcement learning; Acquiring environmental state parameters, and generating potential actions of the three-dimensional snake-like robot according to the environmental state parameters and the strategy network; generating a predicted action sequence using the potential actions; generating a motion gait sequence of a three-dimensional snake-like robot based on the predicted motion sequence; The three-dimensional snake-like robot is controlled to sequentially execute the gaits in the motion gait sequence to adjust the forward direction.

[0010] Preferably, controlling the three-dimensional snake-like robot to sequentially execute the gaits in the motion gait sequence to adjust the forward direction comprises: After the three-dimensional snake-like robot has completed all gaits in the motion gait sequence, it obtains a corresponding gait sequence reward, and re-executes the steps of obtaining environmental state parameters and generating potential actions of the three-dimensional snake-like robot based on the strategy network.

[0011] Preferably, the environmental state parameters include external state parameters and internal state parameters, the external state parameters are used to characterize the relative position relationship between the three-dimensional snake-like robot and the target point, and the internal state parameters are used to characterize the operating state parameters of the three-dimensional snake-like robot itself.

[0012] Preferably, the environmental state parameters include external state parameters and internal state parameters; The external state parameters include the distance and relative angle between the head of the three-dimensional snake-like robot and the target point, and the distance between the head position of the three-dimensional snake-like robot and the straight line connecting the previous target point and the current target point, which are used to characterize the relative position relationship between the three-dimensional snake-like robot and the target point; The internal state parameters include the turning curvature of the track gait in the previous time step, and the actual joint angles in the previous time step and the current time step, which are used to characterize the operating state parameters of the three-dimensional snake-like robot itself.

[0013] Preferably, the generating of the predicted action sequence using the potential action comprises: transforming the potential action of the current step k into Decoded into an action sequence containing H actions, Among them, the potential actions of the 3D snake robot , where z is the latent variable and AOA is the amplitude of the conditional action oscillation of z.

[0014] Preferably, generating a motion gait sequence of a three-dimensional snake-like robot based on the predicted motion sequence comprises: The turning curvature used to modify the movement direction of the track gait is selected as a decoding action by a parameterized gait generator to obtain a gait parameter sequence; The mapping between gait parameter sequences and latent variables is established through a conditional variational autoencoder.

[0015] Preferably, the training of the conditional variational autoencoder includes: establishing a dynamic transition predictor for mapping latent variables to dynamic transitions, facilitating mapping between gait parameter sequences having similar effects on dynamic transitions and similar latent variables.

[0016] Preferably, the training of the conditional variational autoencoder includes energy-saving excitation for the decoded gait parameter sequence to reduce the variance of the reconstructed gait parameter sequence.

[0017] In a second aspect, an embodiment of the present application provides an electronic device, comprising one or more processors and a memory, the memory being used to store code instructions, and the one or more processors being used to run the code instructions to execute the method described in the first aspect or any possible implementation of the first aspect.

[0018] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program or instructions are stored. When the computer program or instructions are run on a computer, the computer executes the method described in the first aspect or any possible implementation of the first aspect.

[0019] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a three-dimensional snake-like robot tracking control method based on sparse state perception. By introducing a potential representation of action sequences, the highly redundant wheelless three-dimensional snake-like robot can maintain dense and continuous motion under sparse state perception and successfully complete the point tracking task.

[0020] Meanwhile, in order to learn a compact latent variable model, three mechanisms are proposed to ensure its efficient training, semantic smoothness, and motion energy efficiency.

[0021] Other features and advantages of the present application will be described in the subsequent description, and in part will become apparent from the description, or it may be understood through the implementation of the present application that the objects and other advantages of the present application can be realized and obtained through the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A flowchart of a three-dimensional snake-like robot tracking control method based on state sparse perception according to the present invention; Figure 2 This is a connection block diagram of a point tracking task system in a three-dimensional snake-like robot tracking control method based on state sparse perception according to the present invention; Figure 3 This is an experimental effect diagram of a point tracking task in a three-dimensional snake-like robot tracking control method based on state sparse perception of the present invention; Figure 4 A schematic diagram of the robustness of a three-dimensional snake-like robot tracking control method based on state sparse perception in a state sparse perception environment to the single module length of the three-dimensional snake-like robot; Figure 5 A schematic diagram of the robustness of a three-dimensional snake-like robot tracking control method based on state sparse perception to a ground environment in a state sparse perception environment according to the present invention; Figure 6This is a schematic diagram of the generalizability of a three-dimensional snake-like robot tracking control method based on state sparse perception in the present invention for different robot starting points, initial postures and target points in a state sparse perception environment. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] like Figure 1 As shown, the first embodiment provided by the present invention is a three-dimensional snake-like robot tracking control method based on state sparse perception, comprising: Build a policy network based on deep reinforcement learning; Obtaining environmental state parameters, and generating potential actions of the 3D snake-like robot based on the environmental state parameters and the strategy network; Generate predicted action sequences using latent actions; Generate motion gait sequences of 3D snake-like robots based on predicted motion sequences; The three-dimensional snake-like robot is controlled to sequentially execute the gaits in the motion gait sequence to adjust the forward direction.

[0025] By introducing the potential representation of action sequences, the present invention can enable a highly redundant wheelless three-dimensional snake-like robot to maintain dense and continuous motion under sparse state perception and successfully complete the point tracking task.

[0026] Specifically, the environmental state parameters include external state parameters and internal state parameters. The external state parameters are used to characterize the relative position relationship between the 3D snake-like robot and the target point, including the distance and relative angle between the head of the 3D snake-like robot and the target point, and the distance between the head position of the 3D snake-like robot and the straight line connecting the previous target point and the current target point. The internal state parameters are used to characterize the running state parameters of the 3D snake-like robot itself, including the turning curvature of the track gait in the previous time step, and the actual joint angles in the previous time step and the current time step.

[0027] Considering that the environment state cannot be updated in real time, this scheme designs a sparse state to dense action potential control method based on representation reinforcement learning method. This framework combines potential representation optimization and model-free reinforcement learning. This framework can predict dense motion gaits under sparse state perception, thereby ensuring effective motion performance.

[0028] The specific implementation scheme of this embodiment is as follows: The 3D snake robot consists of 25 modules, each with a length and mass of 0.08m and 264g, respectively. Each module is connected to the next module through joints, with pitch and yaw joints alternating. Dynamixel-XH540 motors are used as actuators for the joints. A motion capture system is used to track a reflective ball mounted on the robot head, thereby updating the head position of the learning strategy. The learned point following strategy runs on a computer. The commands for the joint angles and the actual state of the robot are transmitted between the computer and the actual robot via USB to RS485 conversion. The state of the environment is updated every 20 control time steps, with a control frequency of 20 Hz.

[0029] like Figure 2 As shown in Figure 1, during the strategy optimization process, at the current time step k, the point tracking strategy takes the environmental state parameters captured by the motion capture system as input and sends joint instructions to the 3D snake robot. The 3D snake robot returns the robot state after executing the instructions. For a snake robot with n joints, the environmental state parameters are represented by the state vector s k To express, The external state of the snake robot and internal state Composition. External state Including the distance between the robot head and the target point and relative angle , and the distance between the robot head and the straight line connecting the previous target point and the current target point , to indicate whether the 3D snake robot moves directly to the target point. Internal state The turning curvature of the track gait at the previous time step and the actual joint angles at the previous and current time steps , composition.

[0030] Based on the state of the environment , the policy network generates appropriate potential actions (i.e. latent space representation) to generate the continuous motion gait of the snake robot. Specifically, the latent action By latent variables and The conditional action oscillation amplitude ( ). To achieve dense control, we first use the learned latent representation to transform Decoded into an action sequence containing H actions, represented as .

[0031] Through the action sequence , a novel parametric gait generation method is designed to modulate the tracked gait to generate a satisfactory motion gait. The parametric gait generation method selects the parameters of the tracked gait as The robot is then driven forward by the action in the track gait, which is then generated using the gait parameter sequence. In order to effectively adjust the track gait to reach a given target point, the selection of gait parameters is crucial. The target shape of the track gait is determined by four gait parameters: , , and Determine, where w is the width of the crawler gait, h is the height of the crawler gait; d is the distance between the two arc segments in the crawler gait, and c is the curvature of the crawler gait. It is used to adjust different characteristics of the gait. This scheme selects the curvature c used to modify the movement direction of the crawler gait as the decoding action. Thus, the action sequence is defined as a sequence of gait parameters , which is described as follows:

[0032] In addition, the other three gait parameters are predetermined and kept constant. The various gait parameters in , which can generate the movement gait of the snake-like robot Next, the snake robot performs the gait sequence The gait in the figure is used to adjust the direction of progress.

[0033] Snake robot completes gait sequence After that, the environment changes to a new state And provide corresponding rewards , which is defined as the sum of rewards for each locomotion gait performed by the robot, is explicitly defined as:

[0034] in, Represents the reward obtained by the robot after executing the i-th motion gait. In order to encourage the snake robot to reach the target point quickly and accurately, the reward The construction is as follows:

[0035]

[0036] First element is the arrival time reward, which aims to encourage the robot to reach the target point as quickly as possible. , is the distance between the robot head and the target point before and after the robot performs the i-th motion gait, represents the execution time of the i-th motion gait, is a positive weight parameter. The second element The accuracy reward is to reward the robot for reaching the target point accurately. is less than a given threshold (e.g. 0.02m), then is assigned a relatively large positive value, otherwise, .

[0037] By repeating the above process, trajectories can be continuously collected. Used for training point tracking control strategies. Relying on the above-mentioned sparse state to dense action potential control method framework based on representational reinforcement learning method, dense motion gaits can be generated from sparse updated states, thereby facilitating the robot to accurately navigate to the target point.

[0038] Preferably, generating a motion gait sequence of a three-dimensional snake-like robot based on a predicted motion sequence comprises: The turning curvature used to modify the movement direction of the track gait is selected as a decoding action by a parameterized gait generator to obtain a gait parameter sequence; The mapping between gait parameter sequences and latent variables is established through a conditional variational autoencoder.

[0039] On the basis of the above, in order to Predicting gait parameter sequences , a compact and expressive latent representation is needed. This scheme introduces three specific mechanisms, namely action-oscillation-amplitude ( ) conditions, dynamic transition prediction, and energy-saving incentives to obtain a satisfactory latent variable representation through self-supervised training.

[0040] First, this design uses a conditional variational autoencoder to map the original gait parameter sequence space to the latent variable space. For the encoder input, in addition to the environmental state parameters, the action-oscillation-amplitude ( ) as an additional condition of the latent variable model, which explicitly helps the latent variable model capture the oscillation characteristics of the gait parameter sequence. This condition effectively enhances the understanding of the latent variable model of the gait parameter sequence and further improves its training efficiency.

[0041] Gait parameter sequence of Represents its relative amplitude oscillation characteristics. It is defined as the normalized mean of the amplitude changes between adjacent gait parameters in the sequence, limited to the interval [0, 1], and the calculation formula is:

[0042] in, is the maximum amplitude change between adjacent gait parameters. , latent variables Encoding gait parameter sequence , with status and As a condition, the encoding and decoding process is as follows:

[0043]

[0044] in represents the gait parameter sequence predicted by the decoder, and the encoder is parameterized by The deep neural network representation of Parameterization. As shown in the decoder, the latent variables can be decoded by and status and The conditions to derive Therefore, in the above DRL optimization process, in order to make the potential action Z predictable , introduced The optimization goal of the conditional variational autoencoder is to minimize the following confidence lower bound, including reconstruction and regularization losses:

[0045] Among them, B is the data buffer, and the first item represents the generated gait parameter sequence and truth value The reconstruction loss between and standard normal distribution The Kullback-Leibler (KL) divergence of is the weight parameter.

[0046] Thus, the training of the above conditional variational autoencoder successfully establishes a mapping between the gait parameter sequence and the latent variables.

[0047] Preferably, generating a motion gait sequence of a three-dimensional snake-like robot based on a predicted action sequence further comprises: establishing a dynamic transformation predictor for mapping latent variables to dynamic transformations, facilitating mapping between gait parameter sequences having similar effects on dynamic transformations and similar latent variables

[0048] Specifically, based on the training of the above-mentioned conditional variational autoencoder, a mapping between gait parameter sequences and latent variables is successfully established. Two sets of numerically similar gait parameter sequences can be mapped to similar latent variables. However, these two sequences may have significantly different effects on the dynamic transitions of the environment. In order to enhance the semantic smoothness of the latent representation and further improve the training efficiency, a dynamic transition predictor is introduced to be trained together with the conditional variational autoencoder, which explicitly reflects how the latent variables affect the dynamic transitions of the environment. The dynamic transition predictor maps the latent variables to the dynamic transitions, thereby facilitating the mapping between gait parameter sequences and similar latent variables that have similar effects on the dynamic transitions.

[0049] More precisely, the dynamic transition predictor consists of the state transitions and action conversion The composition is as follows:

[0050]

[0051] in represents the change between adjacent environmental states, while Represents a sequence of gait parameters By dynamically switching the predictor, and The predictions are as follows:

[0052]

[0053] in , With decoder Share the same neural network but have different heads. The training goal is to minimize the dynamic transformation of predictions The mean square error between it and its true value is explicitly expressed as:

[0054] Preferably, the training of the conditional variational autoencoder includes an energy-saving excitation for the decoded gait parameter sequence to reduce the variance of the reconstructed gait parameter sequence.

[0055] Specifically, based on the above design, if there is excessive oscillation in the gait parameters within the gait parameter sequence, it will cause violent swings in the movement direction of the snake robot. This not only leads to unnecessary energy consumption, but also poses challenges to the stable transfer from simulation to reality, and may even damage the physical snake robot. Noting this problem, energy-saving incentives for decoding gait parameter sequences are also incorporated into the training of the conditional variational autoencoder, which makes the resulting motion gait smooth and easy to transfer to the actual snake robot. Specifically, the optimization goal is to reduce the reconstruction of the gait parameter sequence The variance of is expressed as follows:

[0056] In summary, the overall optimization loss of the latent representation Build it as follows:

[0057] in and are positive parameters, which can be used to adjust the weights of these three items.

[0058] In summary, the present invention designs an efficient sparse state to dense action latent control method based on representational reinforcement learning for the point tracking task of snake-like robots. The framework can predict dense motion gaits under sparse state perception, thereby ensuring effective motion performance. In addition, the present invention adopts self-supervised training to establish mappings between multiple action spaces and latent spaces. Three mechanisms are proposed to ensure efficient training, semantic smoothness and energy efficiency of latent variable models.

[0059] An embodiment of the present application provides an electronic device, including: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to implement the following functions: by introducing a potential representation of an action sequence, a highly redundant wheelless three-dimensional snake-like robot can maintain dense and continuous motion under sparse state perception and successfully complete a point tracking task.

[0060] The embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are run on an electronic device, the electronic device can realize the following functions: by introducing a potential representation of an action sequence, a highly redundant wheelless three-dimensional snake-like robot can maintain dense and continuous motion under sparse state perception, and successfully complete the point tracking task. The method described in the above embodiment can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. If implemented in software, the function can be stored as one or more instructions or codes on a computer-readable medium or transmitted on a computer-readable medium. The computer-readable medium may include computer storage media and communication media, and may also include any medium that can transfer a computer program from one place to another. The storage medium may be any target medium that can be accessed by a computer.

[0061] The testing process is as follows: Test 1: If Figure 3 As shown, it is tested whether the above three-dimensional snake robot can effectively complete the point navigation task in a state sparse perception environment.

[0062] According to the design in the invention, it is tested whether it can control the three-dimensional snake-like robot to reach the specified target point with high precision. The experimental setting is as follows: The starting point is (0.0, 0.0) m. The robot's initial direction is forward, directly aligned with the target point at (2.15, 0.0) m. The experimental scene is as follows Figure 3 As shown in (a), the movement trajectory of the snake-like robot head under the control of the present application solution is shown in the attached Figure 3 (b) As shown. It can be seen that the snake robot flexibly adjusts the movement direction and successfully reaches the target point with an accuracy of 0.010m. Based on the experimental results, it can be concluded that in the case of sparse state perception, the application scheme can ensure satisfactory tracking accuracy.

[0063] Test 2: If Figure 4 As shown, the robustness of the present application scheme to the ground environment and the length of a single module of the robot in a state-sparse perception environment is tested.

[0064] To further verify the robustness of the proposed design, two different sets of experiments were carried out under the interference of different module lengths and ground friction.

[0065] First, the module length of the snake robot Since the module length plays a crucial role in the track configuration fitting and gait generation process, the robustness test of different module lengths was carried out. For this purpose, point tracking experiments were conducted on another customized 3D snake robot with 25 modules, each with a length of 0.095m. Figure 4 (a). The robot's starting point, target point, and initial direction are the same as in the previous test. The movement trajectory of the robot head is shown in the attached figure. Figure 4 (b) As shown in the figure, the snake robot always maintains the correct direction of movement and eventually reaches the target point with an error of 0.008m. This shows that no matter how the length of the robot module changes, the proposed algorithm can successfully drive the robot to complete the point tracking task with sparse state perception.

[0066] Test 3: If Figure 5 As shown, in addition, in practical applications, 3D snake robots usually need to traverse different terrains. In the previous verification, in order to ensure the safety of the robot, the point tracking experiment was performed on a soft pad that provided greater friction. In order to evaluate the robustness of the algorithm to ground friction, a point tracking experiment was performed on a smooth tile floor with less friction. The selection of the starting point and target point as well as the initial robot direction is also the same as in the first test. The experimental setup is shown in the attached figure. Figure 5 (a) shows the point tracking results. Figure 5 (b) shows that the snake robot successfully reaches the target point with a final error of 0.012m. By comparing the trajectories, it can be seen that the robot's motion trajectories on two different surfaces are similar. This shows that the proposed algorithm is insensitive to changing ground conditions. In addition, it is able to adjust the motion gait in time according to the actual ground state to ensure good control performance.

[0067] Test 4: If Figure 6 As shown, the solution of the present application is generalizable to different starting points, initial postures and target points of robots in a state-sparse perception environment.

[0068] In order to evaluate the generalization of the present invention under sparse state perception conditions, target point tracking experiments were conducted for different starting points, target points, and initial robot positions. The actual scene and the corresponding motion trajectory of the robot head are shown in the attached figure. Figure 6 In case 1, in order to further enhance the randomness, a new type of snake-like robot with a single module length of 0.095m is used for tracking experiments. Figure 6 (a) With Figure 6 (b) Set different starting points, initial directions and target points for the robot. Figure 6 (c) It can be seen that under the control of the proposed algorithm, the robot quickly changes its direction of movement and reaches the target point accurately, with the final distance deviation being only 0.006m. In case 2, the snake robot needs to track the point behind it. Figure 6 (d) It can be seen that during the tracking process, the snake robot successfully turned backwards and finally reached the target point with an error of 0.010m. Based on the tracking performance of these two experiments, it can be inferred that this method effectively adapts to any initial position and orientation of the robot and can be generalized to track any target point.

[0069] In summary, the present invention designs a motion control method for a three-dimensional snake-like robot based on representational reinforcement learning, which is used for dense gait control under sparse environmental state perception conditions, ensuring that the snake-like robot can reach the target point efficiently and accurately, and the present application scheme has satisfactory robustness and generalizability to different environments and robot changes and uncertainties. Therefore, it has high practical application value.

[0070] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0071] Those skilled in the art should understand that the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code. Among them, the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM for short), electrically erasable programmable read-only memory (EEpROM for short), erasable programmable read-only memory (EpROM for short), programmable read-only memory (pROM for short), read-only memory (ROM for short), magnetic memory, flash memory, magnetic disk or optical disk. These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which is implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0072] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The specific parameters of the embodiments in the present invention are not limited thereto, such as the number of modules can be adjusted according to actual needs. The device embodiments described above are merely schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

Claims

1. A three-dimensional snake-like robot tracking control method based on state sparse perception, characterized in that: include: Build a policy network based on deep reinforcement learning; Acquiring environmental state parameters, and generating potential actions of the three-dimensional snake-like robot according to the environmental state parameters and the strategy network; generating a predicted action sequence using the potential actions; generating a motion gait sequence of a three-dimensional snake-like robot based on the predicted motion sequence; The three-dimensional snake-like robot is controlled to sequentially execute the gaits in the motion gait sequence to adjust the forward direction.

2. A three-dimensional snake-like robot tracking control method based on state sparse perception according to claim 1, characterized in that: The controlling of the three-dimensional snake-like robot to sequentially execute the gait adjustment direction in the motion gait sequence comprises: After the three-dimensional snake-like robot has completed all gaits in the motion gait sequence, it obtains a corresponding gait sequence reward, and re-executes the steps of obtaining environmental state parameters and generating potential actions of the three-dimensional snake-like robot based on the strategy network.

3. A three-dimensional snake-like robot tracking control method based on state sparse perception according to claim 2, characterized in that: The environmental state parameters include external state parameters and internal state parameters; The external state parameters include the distance and relative angle between the head of the three-dimensional snake-like robot and the target point, and the distance between the head position of the three-dimensional snake-like robot and the straight line connecting the previous target point and the current target point, which are used to characterize the relative position relationship between the three-dimensional snake-like robot and the target point; The internal state parameters include the turning curvature of the track gait in the previous time step, and the actual joint angles in the previous time step and the current time step, which are used to characterize the operating state parameters of the three-dimensional snake-like robot itself.

4. A three-dimensional snake-like robot tracking control method based on state sparse perception according to claim 3, characterized in that: The generating of the predicted action sequence by using the potential action comprises: using a decoder to convert the potential action of the current step k Decoded into an action sequence containing H actions, Among them, the potential actions of the 3D snake robot , where z is the latent variable and AOA is the amplitude of the conditional action oscillation of z.

5. A three-dimensional snake-like robot tracking control method based on state sparse perception according to claim 4, characterized in that: The generating of the motion gait sequence of the three-dimensional snake-like robot based on the predicted motion sequence comprises: The turning curvature used to modify the movement direction of the track gait is selected as a decoding action by a parameterized gait generator to obtain a gait parameter sequence; The mapping between gait parameter sequences and latent variables is established through a conditional variational autoencoder.

6. A three-dimensional snake-like robot tracking control method based on state sparse perception according to claim 5, characterized in that: The training of the conditional variational autoencoder includes: establishing a dynamic transition predictor for mapping latent variables to dynamic transitions, and promoting mapping between gait parameter sequences and similar latent variables that have similar effects on dynamic transitions.

7. A three-dimensional snake-like robot tracking control method based on state sparse perception according to claim 6, characterized in that: The training of the conditional variational autoencoder includes an energy-saving excitation for the decoded gait parameter sequence to reduce the variance of the reconstructed gait parameter sequence.

8. An electronic device, characterized in that: The electronic device comprises: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions so that the electronic device executes the method described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 7.

Images