Multi-mode end-to-end reinforcement learning four-foot robot dog reconnaissance and control method for street battle

Abstract

The invention discloses a multi-modal end-to-end reinforcement learning four-footed robot dog reconnaissance and control method for an alley battle, and the method comprises the following steps: (1) generating a global path through an A * algorithm, and carrying out the real-time planning through combining with a dynamic window algorithm, thereby obtaining a local path; (2) inputting multi-modal perception data (local path, point cloud information, environment depth map, motion state information and environment privilege information), training a teacher strategy by combining a parallel near-end strategy optimization algorithm of a variational auto-encoder, and generating a corresponding action instruction; (3) inputting multi-modal data with noise, minimizing the loss of the teacher strategy and updating the student strategy in combination with a belief encoder; and (4) deploying the student strategy to a robot dog, inputting path information, street battle environment information and motion state information in real time, and generating an action instruction. According to the invention, multi-modal data and end-to-end reinforcement learning are fused to realize efficient sensing of the robot dog to the street battle environment, and the method is suitable for robot dog reconnaissance and control tasks in the street battle scene.

Description

TECHNICAL FIELD

[0001] The present application relates to a multi-modal end-to-end reinforcement learning quadruped robot reconnaissance and control method for alley warfare, and in particular to the fields of multi-modal reinforcement learning and robot control BACKGROUND

[0002] With the continuous evolution of modern warfare patterns, the demand for reconnaissance and control in urban warfare scenarios, especially in alley warfare scenarios, is increasingly urgent. Traditional manual reconnaissance methods have low efficiency, high safety risks and other problems in complex environments, and are difficult to meet real-time and precision requirements. Quadruped robots, with their excellent terrain adaptability and flexible motion performance, have become an important equipment for executing reconnaissance and assisting combat tasks in alley warfare environments. However, the autonomous control capability and real-time decision-making capability of quadruped robots in complex environments still have a lot of room for improvement in the prior art.

[0003] Traditional reinforcement learning methods still have many problems in the application of quadruped robots in alley warfare. First, the strategy of offline training with fixed scene data cannot adapt to dynamic and complex alley warfare environments, and the model generalization capability is insufficient. Second, in terms of multi-modal information processing, existing methods cannot effectively fuse multi-source data such as vision and laser radar, and cannot fully utilize the complementary characteristics of multi-modal data, resulting in low accuracy and efficiency of perception and decision-making. Third, for the complexity and dynamic changes in continuous time (CT) tasks in alley warfare environments, existing reinforcement learning methods still have limitations in real-time performance and convergence. Especially in the case of multi-modal information fusion, the instability of the model and the difficulty of training are significantly increased, which seriously restricts its application in real scenarios. Therefore, it is of great theoretical significance and application value to study a multi-modal end-to-end reinforcement learning quadruped robot reconnaissance and control method for alley warfare scenarios. SUMMARY

[0004] To solve the above problems, the present application provides a quadruped robot reconnaissance and control method based on multi-modal end-to-end reinforcement learning, which aims to solve the key technical problems of high difficulty in multi-modal information processing, fast dynamic changes and high real-time requirements of current quadruped robots in complex alley warfare environments. The present application fuses multi-modal information data such as vision and laser radar, integrates processing of multi-source sensor data such as vision and laser radar, combines the autonomous learning capability of deep reinforcement learning models, innovatively introduces a dynamic constraint mechanism and a multi-modal data joint optimization strategy, and fully integrates the characteristics of real alley warfare scenarios, overcoming the problems of model instability and poor adaptability in dynamic environments of existing methods. The present application can significantly improve the perception accuracy and control efficiency of quadruped robots, has the characteristics of low computational cost and fast training speed, and can realize efficient reconnaissance and precise control in complex alley warfare environments, and has important theoretical research value and practical application prospect.

[0005] Technical solution:

[0006] 1. A multi-modal end-to-end reinforcement learning four-legged robot reconnaissance and control method for tunnel warfare, characterized by the following steps:

[0007] Step 1, path planning process: the path planning process includes global path and local path, the global path uses the search algorithm A* based on environment cost to generate path point sequence from the starting point to the target point, providing reference target point for local path, the local path S1 is planned in real time through dynamic window (DWA) algorithm, the input information includes surrounding environment point cloud information S2 obtained by laser radar and environment depth map S3 obtained by depth camera, forming path planning set for constructing local environment map and reference point of global path, the output of local path is used to guide the robot to bypass obstacles and dynamically adjust to maintain consistency with the global path, when the global path is blocked, the global path can be re-planned; if the local path is affected by dynamic obstacles, the path is optimized in real time;

[0008] Step 2, training teacher strategy, for efficiently analyzing complex task requirements and providing ideal behavior paradigm for student strategy, first construct privileged teacher strategy network π teacher with parameters θ teacher and value network V teacher with parameters ω teacher , and define regularized reward function R = r linear_tracking + r angular_tracking + r joint_motion + r joint_torques + r action_rate + r collisions + r destination + r obstacle_avoidance + loss teacher_reconstruction . Build tunnel warfare task scene in Genesis simulation environment, the scene is configured with laser radar and depth camera; collect input information: including path planning set four-legged robot posture and motion state information S4 obtained by inertial measurement unit (IMU) and environment semantic privileged information S5, constitute teacher perception data set S teacher = (S1, S2, S3, S4, S5), based on the teacher perception data set, output action instruction through privileged teacher strategy network, including: 12 joint torque control information A1 = (τ1, τ2, …, τ 12 ) and 12 joint steering angle control information A2 = (θ1, θ2, …, θ 12 ), form action information A = (A1, A2), deploy proximal policy optimization (PPO) algorithm through parallelization framework, teacher strategy network π teacherValue Network V teacher Interacting within multiple independently running Genesis environments, diverse trajectory data is collected to improve the generalization ability of the policy. Privileged information and state features are extracted using a multilayer perceptron, and combined with depth maps and point cloud data, then fused using a variational autoencoder (VAE) module to generate high-quality action sequences. Trajectory data is stored in the teacher trajectory buffer pool B. teacher This is used for subsequent policy optimization. The PPO algorithm updates the policy and value network based on trajectory data, optimizing the objective function through pruning loss, entropy regularization, and temporal difference method. Simultaneously, the VAE module utilizes the reconstruction loss of the latent space. teacher_reconstruction Provide reward adjustments to the strategy to make the teacher's strategy output more stable, robust and generalizable;

[0009] Step 3: Train student strategies and construct a system with parameters θ. student Student Strategy Network π student The input is a local path S1 generated in real time by the path planning module and a noisy lidar point cloud. Noisy environmental depth map The student perception dataset consists of the quadruped robot dog's posture and motion state information S4. The student policy network outputs joint movement commands (A1, A2) consistent with the teacher's policy. During training, a Belief decoder and Belief encoder module are introduced to model and encode beliefs in the input after privileged information removal, enhancing the network's understanding and response to local paths. Simultaneously, a supervised learning framework is used, employing the teacher's policy π. teacher As a guide, minimize the student policy π. student Teacher Strategies π teakcher The strategy gap loss L between them policy At the same time, the reconstruction loss L is introduced. reconstruction The output of the Belief decoder is decoded and compared with the teacher privilege information to optimize the decoder performance and obtain the total loss L. total =L policy +L reconstruction By jointly optimizing the policy gap loss and reconstruction loss, the student policy network parameters θ are updated. student ;

[0010] Step 4, Deployment and Execution: Deploy the trained student policy network π student Deployed to the robot dog control terminal, input data is collected in real time, including local path S1, point cloud information S2, environmental depth map S3, and quadruped robot dog posture and motion state information S4. Based on the input data (S1, S2, S3, S4), action commands (τ1, τ2, ..., τ4) for controlling the quadruped robot dog's joints are generated in real time through a student policy network.12 ) and (θ1, θ2, …, θ 12 ) to execute reconnaissance and street fighting task instructions. During the execution of the reconnaissance and street fighting tasks, the quadruped robot dog feeds back dynamic change information in real time through sensors and adjusts the strategy output in real time combined with the local path to ensure the accuracy and environmental adaptability of the action. The path planning uses the positioning information of the Beidou positioning, inertial measurement unit (IMU) and laser radar data fusion to update the current position of the robot dog and the distance from the reconnaissance point in real time, and dynamically optimizes the local path.

[0011] 2. The multi-modal end-to-end reinforcement learning quadruped robot dog reconnaissance and control method for street fighting, characterized in that the hardware device is an absolute shadow X20 in the cloud, and the hardware includes:

[0012] (1) Joint: contains a joint structure with 12 degrees of freedom, and each leg is composed of 3 independent driving joints for reconnaissance in complex terrain and dangerous environment such as street fighting;

[0013] (2) High-precision laser radar: used to collect three-dimensional point cloud information around the quadruped robot dog, including the position and shape of obstacles in street fighting;

[0014] (3) Depth camera: used to obtain depth map information in the street fighting scene;

[0015] (4) Inertial measurement unit (IMU): used to obtain attitude and acceleration information during the movement of the robot dog;

[0016] (5) Beidou satellite positioning module: used to obtain real-time position information of the quadruped robot dog and provide distance data between the reconnaissance point;

[0017] (6) Communication module: used for real-time transmission of data between the robot dog and the control center;

[0018] (7) High-performance embedded computing unit: used for localized, path planning and strategy network inference calculation.

[0019] 3. The multi-modal end-to-end reinforcement learning quadruped robot dog reconnaissance and control method for street fighting, characterized in that the regularized reward function R includes linear velocity tracking reward r linear_tracking , angular velocity tracking reward r angular_tracking , joint motion penalty r joint_motion , joint torque penalty r joint_torques , action rate penalty r action_rate , collision penalty r collisions , destination advancement reward r destination , obstacle avoidance reward r obstacle_avoidance , and reconstruction loss loss teacher_reconstructionThe reward sub-items include:

[0020]

[0021] r joint_torques =-||τ j || 2 ,

[0022]

[0023] r collisolins =-n collisolin ,

[0024] r destination =-||p goal -p dog || 2 ,

[0025]

[0026] loss teacher_reconstruction =-||F reconstruction -F origin || 2 -D KL (q(h t ||Input)||p(h t ))

[0027] in, V represents the desired linear velocity (components in the x and y planes). b,xy This represents the robot dog's actual linear velocity (components in the x and y planes). Let ω represent the desired angular velocity (rotational velocity about the z-axis). b,z This represents the robot dog's actual angular velocity (rotational speed around the z-axis). This represents joint acceleration (the second derivative of the joint angle). This represents the joint velocity (the first derivative of the joint angle). q represents the desired joint angle. j τ represents the actual joint angle of the robot dog. j n represents the actual torque of the robot dog's joints. collision p represents the number of collisions. goal p represents the location of the target point. dog p represents the current position of the robot dog. o Indicates the location of the obstacle, r safe F represents the safe distance threshold (the minimum safe distance between the robot dog and obstacles). reconstruction F represents the reconstructed features generated by the model. origin h represents the original feature. tlatent variables representing the VAE encoder output;

[0028] 4. The lane war-oriented multi-modal end-to-end reinforcement learning quadruped robot reconnaissance and control method, wherein the path planning process mainly comprises the following steps:

[0029] (1) global path planning is performed by using an A* algorithm, the input is a lane war starting point and an ending point (a reconnaissance point), and a series of path points are output as a global reference path, in the process, the A* algorithm performs path search by combining a path cost and a heuristic estimation function: f(n)=g(n)+h(n)+c(n), wherein f(n) is a total cost of a node n, g(n) is an actual path cost from a starting point to the current node n, h(n) is a heuristic estimation cost from the current node to a target node, and c(n) is an environment-based cost (obstacle density and threat area in a lane war scene), and the heuristic function h(n) uses a Euclidean distance to estimate a distance of the node to the target point:

[0030] (2) local path planning is performed by using a dynamic window (DWA) algorithm, the input includes a local path S1, a local environment map generated from laser radar point cloud information S2 and depth map information S3, the DWA algorithm calculates an optimal local path to avoid dynamic obstacles and maintain consistency with the global path: Score(v, ω)=G(v, ω)+O(v, ω)+V(v, ω), wherein G(v, ω) is a target cost, indicating a distance between a trajectory end point and a target point; O(v, ω) is an obstacle cost, indicating a distance between a trajectory and a nearest obstacle; and V(v, ω) is a speed cost, indicating a speed of the trajectory, constraint conditions of the DWA include a speed constraint and a stopping distance constraint, ensuring that the robot dog can safely and quickly bypass obstacles in the lane war and approach the target;

[0031] (3) the robot dog can bypass obstacles in the lane war while ensuring stability of finally reaching the target point by continuously adjusting and optimizing the local path, steps 1 and 2 are repeated until the robot dog successfully reaches the target point and avoids all obstacles.

[0032] 5. The lane war-oriented multi-modal end-to-end reinforcement learning quadruped robot reconnaissance and control method, wherein the privileged teacher policy network π teacher and the value network V teacher training process and the policy network parameter θ teacher and the value network parameter ω teacher updating process comprise the following steps:

[0033] (1) the PPO algorithm is deployed through a parallel framework, and the teacher policy network π is trained in parallel in multiple Genesis environment instances.teacher and value network V teacher In each interaction process, the quadruped robot interacts with the environment to obtain state information S t , action information A t , reward value R t , and next state information S t+1 S t is input into a multi-layer perception (MLP) for feature fusion to obtain a fused feature vector F t_origin , which is then input into the value network V teacher to obtain an estimated value V t S t+1 is also processed by the MLP and input into the value network V teacher to obtain V t+1 The time difference error is calculated: δ t = R t + γV t+1 - V t The value network parameters are updated using the formula ;

[0034] (2) In the training process, the fused state feature F t_origin is encoded into latent features by the VAE module to generate hidden variables h t h t is input into the decoder to reconstruct the feature F t_reconstruction The feature reconstruction loss is calculated: loss teacher_reconstruction = -||F reconstruction - F origin || 2 - D KL (q(h t |Input)||p(h t )) , and the reconstruction loss is included in the calculation of the total reward value;

[0035] (3) According to the policy gradient theorem, the teacher policy network parameters are optimized, and the action value is calculated by the Bellman equation The policy network parameter θ teacher is updated according to the formula teacher = θ teacher + Δθ teacher ;

[0036] (4) The trajectory data generated by the teacher policy network is stored in the teacher trajectory buffer B student Repeat steps 1 to 3 until the policy network can efficiently output the torque and steering angle of the 12 joints of the quadruped robot.

[0037] 6. The lane warfare-oriented multi-modal end-to-end reinforcement learning quadruped robot dog reconnaissance and control method, characterized in that the student policy network parameters θ are updated by jointly optimizing the policy gap loss and the reconstruction loss. student comprising the following steps:

[0038] (1) defining the policy gap loss minimizing the student policy π by supervised learning student and the teacher policy π teacher ;

[0039] (2) introducing a Belief encoder and a Belief decoder module, the Belief encoder compresses the perception features into a low-dimensional belief representation space to capture the hidden state information of the environment, and the Belief decoder reconstructs the original perception features from the belief representation (hidden features), thereby ensuring that the belief representation can truly reflect the state of the lane warfare environment, using the Belief encoder to model and encode the multi-modal input after removing the privileged information, and defining the reconstruction loss where E is the belief encoder and D is the belief decoder.

[0040] (3) constructing the objective function J(θ student ) = L policy + L reconstruction , optimizing the objective function J(θ student ) by gradient descent, and updating the student policy network parameters;

[0041] (4) repeating steps 1 to 3 until the student policy network can generate efficient action instructions without privileged information.

[0042] 7. The lane warfare-oriented multi-modal end-to-end reinforcement learning quadruped robot dog reconnaissance and control method, characterized in that the student policy network π student is deployed to the robot dog end, which means that the local path S1, point cloud information S2, environment depth map S3, and quadruped robot pose and motion state information S4 are received in real time at a frequency of 50Hz, the input data is processed by the student Belief encoder and multi-layer perception (MLP), and torque instructions (τ1, τ2, …, τ 12 ) and angle instructions (θ1, θ2, …, θ 12) to execute reconnaissance and urban combat tasks. The Belief encoder is used to integrate sensor data and generate urban combat environment beliefs, and the MLPs further process this information to optimize the control policy. During execution, the robot dog dynamically perceives changes in the urban combat environment through sensor feedback and adjusts the policy output in real time in combination with the local path, ensuring the accuracy and environmental adaptability of the reconnaissance action. The path planning uses Beidou positioning, IMU and laser radar data fusion positioning information to update the current position of the robot dog and the distance to the reconnaissance point in real time, dynamically optimizing the local path to ensure the efficiency and reliability of the urban combat task execution.

[0043] Advantages: Compared with the prior art, the present application has the following advantages:

[0044] (1) The present application uses a multi-modal end-to-end reinforcement learning method, combining global path planning and local path planning (A* algorithm and DWA algorithm), which has better training ability, generalization ability and ability to cope with complex urban combat scenes.

[0045] (2) The teacher network training of the present application introduces a variational autoencoder VAE module, which can extract deep features from multi-modal perception information and generate a unified feature vector with higher semantic expression ability, helping the teacher policy network to efficiently analyze complex task requirements and generate fine control.

[0046] (3) The student network training of the present application introduces a Belief encoding and decoding module, which can model and belief encode multi-modal inputs after removing privileged information, enhancing the network's understanding and response ability to local paths and improving the robustness of multi-modal perception and environmental understanding.

[0047] (4) The present application can dynamically adjust the action instructions of the quadruped robot dog by receiving sensor data (laser radar point cloud, depth map, IMU data, etc.) in real time and combining local path planning, ensuring its accuracy and adaptability in the urban combat environment. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 Structure diagram of the multi-modal end-to-end reinforcement learning quadruped robot dog reconnaissance and control method for urban combat of the present application

[0049] Figure 2 Flowchart of the multi-modal end-to-end reinforcement learning quadruped robot dog reconnaissance and control method for urban combat of the present application

[0050] Figure 3 Strategy architecture diagram of the multi-modal end-to-end reinforcement learning quadruped robot dog reconnaissance and control method for urban combat of the present application DETAILED DESCRIPTION

[0051] The present application will be further illustrated below in conjunction with specific embodiments, which should be understood as merely illustrative of the present application and not used to limit the scope of the present application, and various modifications of the present application by those skilled in the art after reading the present application all fall within the scope defined by the claims attached hereto.

[0052] The present application is a multi-modal end-to-end reinforcement learning four-legged robot reconnaissance and control method for street fighting, the research object is a four-legged robot, and the method adopted is multi-modal end-to-end reinforcement learning.

[0053] Figure 1 As shown, the structure diagram of the multi-modal end-to-end reinforcement learning four-legged robot reconnaissance and control method for street fighting of the present application is composed of multiple key modules, which cooperatively complete the reconnaissance and control tasks. The Beidou satellite positioning module provides accurate position, laying the foundation for global path planning; the laser radar and depth camera generate point cloud and depth information, enhancing the environmental perception ability and helping to identify dynamic and static obstacles; the communication module ensures information interaction with the remote control center or other robots, realizing intelligence sharing; the inertial measurement unit (IMU) monitors the attitude and motion state, ensuring stability in complex terrain; the motion controller and perception controller are respectively responsible for four-legged motion and multi-sensor data fusion, guiding path planning and decision-making; the thigh, shank and knee joint motion mechanisms support flexible action, coping with narrow space and complex terrain; the lithium battery provides long endurance capability. Under the synergistic action of these modules, the robot dog can accurately navigate, bypass obstacles, and optimize reconnaissance and action strategies through reinforcement learning to efficiently complete the task target in street fighting.

[0054] Figure 2 As shown, the whole process of model training of the multi-modal end-to-end reinforcement learning four-legged robot reconnaissance and control method for street fighting of the present application contains four steps:

[0055] In step S201, the teacher strategy aims to efficiently analyze complex task requirements and provide an ideal behavior paradigm for the student strategy. By constructing the teacher strategy network π teacher and the value network V teacher , combined with multiple reward functions (such as tracking, obstacle avoidance, collision penalty, etc.), the four-legged robot is trained in the Genesis simulation environment. The input data includes the path planning set, IMU attitude and motion state information, and environmental semantic privilege information, forming the perception data set S teacher . The teacher strategy outputs the torque control and steering angle control information A=(A1,A2) of 12 joints. During the training process, the PPO algorithm is used to optimize the strategy and value network, and the latent features of depth map and point cloud data are extracted through the variational autoencoder (VAE) module to generate high-quality action sequences, which are stored in the teacher trajectory buffer B teacherTrajectory data is used to clip loss, entropy regularization, and time-difference optimization objective function, while reconstructing loss loss teacher_reconstruction Enhance the stability, robustness and generalization ability of the strategy, so as to provide a standardized behavior reference for student strategy training.

[0056] In step S202, student strategy training aims to achieve efficient decision-making behavior consistent with the teacher strategy under the condition of removing privileged information. By constructing a student strategy network π student , the student perception dataset S student consisting of input noisy multi-modal perception dataset is outputted as joint action instruction A=(A1, A2) consistent with the teacher strategy. The Belief encoder and decoder modules are introduced to model and encode the input after removing the privileged information, enhancing the student strategy's understanding and adaptability to the local path. A supervised learning framework is adopted to minimize the strategy gap loss L policy between the student strategy and the teacher strategy. At the same time, the input is decoded by the Belief decoder and compared with the teacher's privileged information, introducing the reconstruction loss L reconstruction to optimize the decoder performance. Finally, the student strategy network is continuously updated by jointly optimizing the strategy gap loss and the reconstruction loss.

[0057] In step S203, the path planning module generates an optimal reference path from the starting point P start =(x start ,y start ) to the target point P goal =(x goal ,y goal ) using the A* algorithm-based global path planning module. The A* algorithm searches for the path through the total cost function f(n)=g(n)+h(n)+c(n), where g(n) is the actual path cost from the starting point to the current node n, calculated as the cumulative path length: where k is the number of nodes passed by the current path. h(n) is the heuristic estimated cost from the current node to the target node, calculated using the Euclidean distance: c(n) is the environment-based cost used to introduce additional constraints or preferences, such as avoiding dangerous areas. The A* algorithm selects the optimal path by calculating the total cost function f(n) until the shortest path from the starting point to the target point is found, outputting the global path S global =[P1, P2, …, P n ], where each P i =(x i ,y i) is the path point. The global path provides a reference for subsequent local path planning. The local path planning uses a dynamic window algorithm (DWA) to avoid dynamic obstacles and approach the target point. DWA calculates the trajectory score under different speeds and angular velocities through the following scoring function: Score(v, omega) = G(v, omega) + O(v, omega) + V(v, omega), where: G(v, omega) is the target cost, representing the distance between the trajectory endpoint and the local target point; O(v, omega) is the obstacle cost, representing the minimum distance between the trajectory and the nearest obstacle, where d min d is the minimum distance between the trajectory and the obstacle, d safe is the safety distance; V(v, omega) = alpha v is the speed cost, representing the speed of the trajectory, where alpha is a weight coefficient. The constraint conditions of DWA include: speed constraint: v min <= v <= v max , omega min <= omega <= omega max ; stopping distance constraint: ensure that the robot dog can stop safely to avoid colliding with obstacles, and the constraint condition is: v 2 / (2a max ) <= d min , where a max is the maximum deceleration. DWA generates the optimal local path by searching in the range of candidate speed and angular velocity to maximize the scoring function Score(v, omega).

[0058] In step S204, the trained student policy network pi student and the path planning module are deployed to the robot dog end in the deployment stage, and the environment multi-modal information and the local path are input in real time. The student policy network directly generates the action instructions (tau1, tau2, …, tau 12 ) and (theta1, theta2, …, theta 12 ) of the control joint according to the input data. When the robot dog performs reconnaissance and street battle tasks, it dynamically perceives the environmental changes through sensor feedback and adjusts the policy output in real time in combination with the local path.

[0059] Figure 3 As shown in the figure, the present application is a multi-modal end-to-end reinforcement learning four-legged robot dog reconnaissance and control method flowchart and strategy framework diagram for street battles:

[0060] In step S301, the continuous time system x(t) = f(x(t), a(t)) is defined in the four-legged robot dog simulation environment Genesis: where x(t) is the state and a(t) is the action. Based on the dynamic programming principle, the target value function of the teacher policy network is defined as Q(x, a): Assuming that the Q function satisfies the Bellman equation and satisfies the following partial differential form: The goal of the teacher policy is to find the optimal action by optimizing the objective function Using the Actor-Critic framework, define the policy network (Actor) π teacher and the value network (Critic) V teacher . By interactive sampling, collect trajectory data of the teacher policy in the simulation environment, calculate the time difference error (TD-error) to update the value network: δ t = R t + γV teacher (x t+1 )- V teacher (x t ). When updating the policy, use PPO clipping objective function to limit the update amplitude, and introduce policy entropy regularization term to maintain exploration ability. The specific optimization objective is: where is the advantage function. To improve the stability of training, use the target network π target and V target , respectively, to delay the synchronization of policy network and value network parameters, to prevent instability caused by rapid updates during training. The target network parameter update rule is: θ target ← τθ + (1-τ)θ target , ω target ← τω + (1-τ)ω target , where τ is the soft update parameter. Process the input depth map and point cloud data through the VAE module to extract key features, and combine them with environmental perception information to generate higher quality action sequences. The reconstruction loss of VAE loss teacher_reconstruction = -||F reconstruction -F origin || 2 -D KL (q(h t ||Input)||p(h t )) is included in the reward function to optimize the policy performance. The adjusted reward function is as follows: R = r linear_tracking + r angluar_tracking + r joint_motion + r joint_torpues + r action_rate + r collisions + r destination + r obstacle_avoidance + loss teacher_reconstruction , to improve the stability, robustness and generalization ability of the teacher policy to complex tasks. Collect trajectory data during the execution of the teacher policy and store it in the trajectory buffer B teacherSpecifically, this includes: LiDAR point cloud information, depth camera information, local path, IMU perception information, and environmental privileged information; the action sequence A = (A1, A2), containing torque control information for 12 joints A1 = (τ1, τ2, ..., τ...). 12 The steering angle control information A2 = (θ1, θ2, ..., θ) and 12 joints. 12 These data are then used to train and optimize student models, improving their generalization ability.

[0061] In step S302, the input to the student policy network is the student-perceived dataset S from the teacher buffer pool after removing environmental privilege information. student : Where S1 represents the local path generated in real time by the path planning module. This represents noisy lidar point cloud data. S4 represents the noisy environmental depth map, and S5 represents the pose and motion state information of the quadruped robot dog. A Belief decoder and Belief encoder module are introduced to model and encode beliefs from the input after privileged information removal, enhancing the student network's understanding and adaptation to local paths. In the encoding stage: the Belief encoder f... enc Student perception data S student Encoded Beliefs: z belief =f enc (S student ), where z belief Belief encoding represents the latent features of the local environment. During the decoding phase, a Belief decoder is used. dec For z belief Decode the data to generate a reconstruction prediction of the privileged information. The decoder performance will be optimized by comparing the information with teacher privilege information. During the training of the student policy network, the following loss function is used to jointly optimize the model: Policy Gap Loss L. policy Using teacher strategies π within a supervised learning framework teacher As a guide, minimize the student policy π. student Teacher Strategies π teacher Strategy gap loss between: Reconstruction loss L reconstruction : Output of the Belief decoder Information on teacher privileges S teacher Compare and optimize decoder performance: This constitutes the total loss L total The total loss function is a weighted sum of the policy gap loss and the reconstruction loss, used to jointly optimize the student policy network: L total =Lpolicy +L reconstruction . The total loss L total is used to update the student policy network parameters θ student : where α is the learning rate. The data sampling and training process of the entire student network is as follows: buffer data sampling: sampling trajectory data (S teacher , A) from the teacher trajectory buffer B student , ensuring that the input data does not contain privileged information; forward propagation: input the perception data S student to the student policy network to generate action instructions A student ; loss calculation: calculate the total loss L policy according to L reconstruction and L total ; parameter update: optimize the student policy network based on the total loss.

[0062] In step S303, the trained student policy π student is combined with the path planning module to provide navigation capabilities when performing tasks. The global path S global provides the overall navigation direction for the student policy, while the local path planning dynamically adjusts the target point through DWA to avoid dynamic obstacles. The student policy π student generates joint action instructions A = (A1, A2) for the quadruped robot according to the input local target point P local = (x local , y local ) in the global path, laser point cloud information S2, depth map information S3, and the motion state S4 of the robot dog, where A1 = (τ1, τ2, …, τ 12 ) is the torque control information, and A2 = (θ1, θ2, …, θ 12 ) is the steering angle control information. Finally, the student policy is continuously adjusted through feedback to ensure that the robot dog can efficiently avoid obstacles and successfully complete the reconnaissance task in a complex lane warfare environment.

Claims

1. A multi-modal end-to-end reinforcement learning quadruped robot reconnaissance and control method for alley war, characterized in that, Comprising the following steps: Step 1, path planning process: the path planning process includes two parts of global path and local path, the global path uses the search algorithm A* based on the environment cost to generate the path point sequence from the starting point to the target point, to provide the reference target point for the local path, the local path S1 is planned in real time through the dynamic window (DWA) algorithm, the input information includes the surrounding environment point cloud information S2 obtained by the laser radar and the environment depth map S3 obtained by the depth camera, to form the path planning set The local environment map is used for constructing the reference point of the global path, and the output of the local path is used for guiding the robot to bypass the obstacles and dynamically adjusting to maintain the consistency with the global path, and when the global path is blocked, the global path can be re-planned; If the local path is affected by dynamic obstacles, optimize the path in real time; Step 2, train the teacher policy for efficient resolution of complex task requirements and to provide an ideal behavior paradigm for the student policy, first construct the parameterized privileged teacher policy network p with parameters teacher teacher with parameters teacher teacher and define the regularized reward function R = r linear_tracking + r angular_tracking + r joint_motion + r joint_torques + r action_rate + r collisions + r destination + r obstacle_avoidance + loss teacher_reconstruction . Build a tunnel warfare task scenario in the Genesis simulation environment, which is configured with a laser radar and a depth camera;​​ ， Collect input information: Including, path planning set The four-legged robot dog posture and motion state information S4 and environment semantic privilege information S5 obtained by the inertial measurement unit (IMU) constitute the teacher perception data set S teacher =(S1, S2, S3, S4, S5), based on the teacher perception data set, outputting action instructions through the privileged teacher policy network, including: 12 joint torque control information A1=(τ1, τ2,…, τ 12 ) and 12 joint steering angle control information A2=(θ1, θ2,…, θ 12 ), the action information amount A=(A1, A2) is composed, the proximal policy optimization PPO algorithm is deployed through the parallelization framework, the teacher policy network π teacher and the value network V teacher Interact in multiple independently running Genesis environments, collect diverse trajectory data to improve the generalization ability of the policy. Privilege information and state features are extracted through a multi-layer perceptron, and combined with depth map and point cloud data, the features are processed through a variational autoencoder VAE module to generate high-quality action sequences. The trajectory data is stored in the teacher trajectory buffer B teacher , used for subsequent policy optimization. The PPO algorithm updates the policy and value network based on the trajectory data, and optimizes the objective function through clipping loss, entropy regularization and time difference method. At the same time, the VAE module uses the reconstruction loss loss teacher_reconstruction of the latent space to provide reward correction for the policy, making the teacher policy output more stable, robust and with generalization ability; Step 3, train student policy, construct parameter θ student Student policy network π student Input: local path S1 generated by path planning module in real time, laser radar point cloud with noise Environment depth map with noise And quadruped robot posture and motion state information S4, constitute student perception data set The output of student policy network is joint action instruction (A1, A2) consistent with teacher policy, in the training process, introduce Belief decoder and Belief encoder module, model and belief code the input after removing privileged information, enhance the network's understanding and response ability to local path, at the same time, through the supervision learning framework, use teacher policy π teacher As guidance, minimize the strategy gap loss L student Between student policy π teacher And teacher policy π policy , At the same time, introduce reconstruction loss L reconstruction , Decode the output of Belief decoder and compare with teacher privileged information, optimize the decoder performance, get total loss L total = L policy + L reconstruction , Through the joint optimization of strategy gap loss and reconstruction loss, update student policy network parameter θ stdent ; Step 4, deployment and execution: the trained student policy network π student is deployed to the robot dog control terminal, real-time input data including local path S1, point cloud information S2, environment depth map S3, and four-legged robot dog posture and motion state information S4 are collected, based on the input data (S1, S2, S3, S4), the student policy network is used to generate control instructions (τ1, τ2, …, τ 12 ) and (θ1, θ2, …, θ 12 ) of the joints of the four-legged robot dog in real time, to execute reconnaissance and street battle task instructions. During the execution of the reconnaissance and street battle task, the four-legged robot dog feeds back dynamic change information in real time through the sensor, and adjusts the policy output in real time combined with the local path, to ensure the accuracy and environmental adaptability of the action. The path planning uses the positioning information of Beidou positioning, inertial measurement unit IMU and laser radar data fusion to update the current position of the robot dog and the distance to the reconnaissance point in real time, and dynamically optimizes the local path.

2. The lane warfare-oriented multi-modal end-to-end reinforcement learning quadruped robot dog reconnaissance and control method according to claim 1, characterized in that, The hardware device is the absolute shadow X20 in the cloud, and the hardware includes: (1) Joints: The joint structure contains 12 degrees of freedom, and each leg is composed of 3 independent driving joints for reconnaissance in complex terrain and dangerous environment such as street fighting; (2) High-precision laser radar: used to collect three-dimensional point cloud information around the quadruped robot dog, including the position and shape of obstacles in street fighting; (3) Depth camera: used to obtain depth map information in the street fighting scene; (4) Inertial measurement unit (IMU): used to obtain attitude and acceleration information during the movement of the robot dog; (5) Beidou satellite positioning module: used to obtain the position information of the quadruped robot dog in real time, and provide distance data between the reconnaissance point; (6) Communication module: used for real-time transmission of data between the robot dog and the control center; (7) High-performance embedded computing unit: used for localization, path planning and strategy network inference calculation.

3. The lane warfare-oriented multi-modal end-to-end reinforcement learning quadruped robot dog reconnaissance and control method according to claim 1, characterized in that, The regularized reward function R includes a linear velocity tracking reward r linear_tracking , an angular velocity tracking reward r angular_tracking , a joint movement penalty r joint_motion , a joint torque penalty r joint_torques , a motion rate penalty r action_rate , a collision penalty r collisions , a reward for advancing toward the destination r destination , a reward for avoiding obstacles r obstacle_avoidance , and a reconstruction loss loss teacher_reconstruction , wherein the reward sub-items include: r joint_torques = - ||τ j || 2 , r collisolins ＝-n collisolin , r destination = -||p goal -p dog || 2 , loss teacher_reconstruction = -||F reconstruction -F origin | 2 -D KL (q(h t ||Input)||p(h t )) where, denotes the desired linear velocity (component on the x and y plane), v b,xy denotes the actual linear velocity of the robot dog (component on the x and y plane), denotes the desired angular velocity (rotational velocity around the z axis), ω b,z denotes the actual angular velocity of the robot dog (rotational velocity around the z axis), denotes the joint acceleration (second derivative of the joint angle), denotes the joint velocity (first derivative of the joint angle), denotes the desired joint angle, q j denotes the actual joint angle of the robot dog, τ j denotes the actual torque of the robot dog joint, n collision denotes the number of collisions, p goal denotes the position of the target point, p dog denotes the current position of the robot dog, p o denotes the obstacle position, r safe denotes the safety distance threshold (minimum safe distance between the robot dog and the obstacle), F reconstruction denotes the reconstructed feature generated by the model, F origin denotes the original feature, h t denotes the latent variable output by the VAE encoder.

4. The lane warfare-oriented multi-modal end-to-end reinforcement learning quadruped robot dog reconnaissance and control method according to claim 1, characterized in that, The path planning process mainly includes the following steps: Step 1: Global path planning is performed by A* algorithm, the input is the starting point and the end point (the reconnaissance point) of the alley war, and the output is a series of path points as the global reference path. In this process, the A* algorithm searches the path by combining the path cost and the heuristic estimation function: f(n) = g(n) + h(n) + c(n), where f(n) is the total cost of node n, g(n) is the actual path cost from the starting point to the current node n, h(n) is the heuristic estimation cost from the current node to the target node, and c(n) is the cost based on the environment (obstacle density and threat area in the alley war scene). The heuristic function j(n) uses the Euclidean distance to estimate the distance from the node to the target point: Step 2: Use DWA algorithm for local path planning, input includes local path S1, laser radar point cloud information S2 and depth map information S3 generated local environment map, DWA algorithm calculates an optimal local path to avoid dynamic obstacles and maintain consistency with the global path: Score(v,ω)=G(v,ω)+O(v,ω)+V(v,ω), Wherein, G(v,ω) is the target cost, indicating the distance between the trajectory end point and the target point; O(v,ω) is the obstacle cost, indicating the distance between the trajectory and the nearest obstacle; V(v,ω) is the speed cost, indicating the speed of the trajectory, the constraint conditions of DWA include speed constraint and stopping distance constraint, to ensure that the robot dog can safely and quickly bypass the obstacles in the street fighting and approach the target; Step 3: Through continuous adjustment and optimization of the local path, the robot dog can bypass obstacles in street fighting while ensuring the stability of finally reaching the target point, repeat step 1 and step 2 until the robot dog successfully reaches the target point and avoids all obstacles.

5. The lane warfare-oriented multi-modal end-to-end reinforcement learning quadruped robot dog reconnaissance and control method according to claim 1, characterized in that, The privileged teacher policy network π teacher with the value network V teacher The training process of the value network V teacher and the update process of the policy network parameter θ teacher and the value network parameter ω Step 1: Deploy the PPO algorithm through a parallel framework to train the teacher policy network π in parallel in multiple Genesis environment instances teacher and the value network V teacher In each interaction process, the quadruped robot interacts with the environment to obtain state information S t , action information A t+1 , reward value R t , and next state information S t_origin S teacher is input into a multi-layer perceptron (MLP) for feature fusion to obtain a fused feature vector F t , which is then input into the value network V t+1 to obtain an estimated value V teacher S t+1 is also processed by the MLP and input into the value network V t to obtain V t The time difference error δ t+1 = R t + γV t_origin - V t is calculated, and the value network parameters are updated using the formula ; Step 2: During the training process, the fused state feature F t_origin latent feature encoding is performed by the VAE module to generate hidden variable h t , and h t is passed into the decoder for reconstruction to obtain the reconstructed feature F t_reconstruction , and the feature reconstruction loss: loss teacher_reconstruction is calculated: = -||F reconstruction -F origin || 2 -D KL (q(h t |Input)||p(h t )), and the reconstruction loss is included in the calculation of the total reward value; Step 3: Optimize the teacher policy network parameters according to the policy gradient theorem, calculate the action value through the Bellman equation According to the formula Update the policy network parameters θ teacher = θ teacher + Δθ teacher ; Step 4: Store the trajectory data generated by the teacher policy network in the teacher trajectory buffer pool B teacher Steps 1 to 3 are repeated until the policy network can efficiently output the torques of the 12 joints and steering angles of the quadruped robot.

6. The lane warfare-oriented multi-modal end-to-end reinforcement learning quadruped robot dog reconnaissance and control method according to claim 1, characterized in that, The student policy network parameter θ is updated by co-optimizing the strategy gap loss and the reconstruction loss student comprising the steps of: Step 1: Define policy gap loss Minimize child policy π through supervised learning student from teacher policy π teacher between teacher policy π Step 2: Introduce the Belief encoder and Belief decoder modules, the Belief encoder compresses the perception features into a low-dimensional belief representation space for capturing the implicit state information of the environment, and the Belief decoder reconstructs the original perception features from the belief representation (implicit features), thereby ensuring that the belief representation can truly reflect the state of the urban combat environment, using the Belief encoder to model and belief encode the multi-modal input after removing the privileged information, and defining a reconstruction loss where E is the belief encoder and D is the belief decoder; Step 3: construct the objective function J(θ student ) = L policy + L reconstruction , optimize the objective function J(θ student ) by gradient descent, update the student policy network parameters; Step 4: Repeat steps 1 to 3 until the student strategy network can generate efficient action instructions without privileged information.

7. The lane warfare-oriented multi-modal end-to-end reinforcement learning quadruped robot dog reconnaissance and control method according to claim 1, characterized in that, The student policy network π student Deployed to the robot dog, it means that through real-time reception of local path S1, point cloud information S2, environment depth map S3, and four-legged robot posture and motion state information S4 at a frequency of 50 Hz, the input data is processed through the student Belief encoder and multi-layer perception (MLPs) to generate torque instructions (τ1, τ2, …, τ 12 ) and angle instructions (θ1, θ2, …, θ 12 ) for controlling the robot dog's 12 joints, and the robot dog executes reconnaissance and street fighting tasks. The Belief encoder is used to integrate sensor data and generate a street fighting environment belief, and the MLPs further process this information to optimize the control strategy. During execution, the robot dog dynamically perceives changes in the street fighting environment through sensor feedback and adjusts the strategy output in real time in combination with the local path to ensure the accuracy and environmental adaptability of the reconnaissance action. Path planning uses Beidou positioning, IMU, and laser radar data fusion positioning information to update the robot dog's current position and distance from the reconnaissance point in real time, dynamically optimizing the local path to ensure the efficiency and reliability of the street fighting task execution.

Images