Robot control method and device, computer device and storage medium

Abstract

This application relates to a robot control method, apparatus, computer device, and storage medium. The method includes: a robot end collecting robot perception data and transmitting the robot perception data to a cockpit end via a communication link; the cockpit end acquiring preprocessed robot perception data via the communication link and transmitting a virtual reality scene to a virtual reality headset via a virtual reality host; the cockpit end controlling the cockpit robotic arm based on the robotic arm's acceleration and displacement using a cockpit controller; the cockpit end collecting the operator's head posture data via the virtual reality headset and generating robot head control commands based on the head posture data and gimbal angle data; and the robot end controlling the six-degree-of-freedom gimbal to execute the actions corresponding to the robot head control commands. This method can improve the control accuracy and stability of the robot.

Description

Technical Field

[0001] This application relates to the field of robotics, and in particular to a robot control method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the development of robotics, wheeled and tracked robots have been widely used. Traditional robots are controlled via teach pendants and control boxes, using joysticks to move the robot chassis, or buttons or joysticks to control joint movement or Cartesian spatial motion of the robotic arm, resulting in low operational efficiency. Traditional robots observe through cameras mounted on the end effector of the robotic arm and the body; however, the limited field of view makes it difficult to judge the distance relationship with the object being manipulated, leading to a low success rate. Traditional robots typically have only one robotic arm and cannot perform complex operations. Therefore, improving the control accuracy and stability of robots, as well as enhancing the immersive interactive experience for operators, are problems that need to be solved. Summary of the Invention

[0003] Therefore, it is necessary to provide a robot control method, device, computer equipment, and storage medium that can improve the control accuracy and stability of robots, as well as enhance the immersive interactive experience of operators when operating robots, in order to address the aforementioned technical problems.

[0004] In a first aspect, this application provides a robot control method, which is executed by a robot control system, the robot control system including a robot end and a cockpit end, and the robot control method including:

[0005] The robot collects robot perception data and sends the robot perception data to the cockpit via a communication link; the robot perception data includes environmental visual data, gimbal angle data of the six-degree-of-freedom gimbal, and force data of the robot arm; the communication link includes a time-sensitive network wired link and a wireless link; the state of the robot arm includes the position, speed, and torque of the robot arm;

[0006] The cockpit acquires preprocessed robot perception data through a communication link, and determines a virtual reality scene based on the environmental visual data through a virtual reality host, and sends the virtual reality scene to a virtual reality headset so that the operator can obtain a visual experience from the robot's perspective based on the virtual reality headset.

[0007] The cockpit controller determines the acceleration and displacement of the robotic arm based on the force data, and controls the cockpit robotic arm based on the acceleration and displacement of the robotic arm, so that the operator can obtain a tactile sense of presence of the bi-armed quadruped robot based on the cockpit robotic arm.

[0008] The cockpit collects the operator's head posture data through a virtual reality headset, and generates robot head control commands based on the head posture data and gimbal angle data through the virtual reality host, and sends the robot head control commands to the robot end through a communication link;

[0009] The robot controls the six-degree-of-freedom gimbal to execute the actions corresponding to the head control commands based on the robot head control commands.

[0010] In one embodiment, the robot end collects robot perception data and transmits the robot perception data to the cockpit end via a communication link, including:

[0011] The robot acquires environmental visual data through a binocular camera, determines the gimbal angle data of the quadruped robot by using the joint position sensing data fed back by the joint position sensors of the quadruped robot, and acquires force data through sensors on the manipulator arm. The environmental visual data, the gimbal angle data and the force data are used as robot perception data.

[0012] The robot sends the robot's perception data to the cockpit via a communication link.

[0013] In one embodiment, the robot control method further includes:

[0014] The cockpit acquires the operator's operation signals and encodes them into standardized instructions via a virtual reality host. The standardized instructions are then sent to the robot via a communication link. The operator inputs the operation signals by controlling the robotic arm, finger-type control hand, foot rudder, physical joystick, and / or physical buttons on the cockpit, or by manually jogging the robotic arm via a teach pendant, or by pre-inputting the operation signals through programming control.

[0015] The robot receives the operation signal through the communication link and controls the robot arm, six-degree-of-freedom gimbal and robot chassis to execute the work task corresponding to the operation signal.

[0016] In one embodiment, the cockpit acquires preprocessed robot perception data via a communication link, determines a virtual reality scene based on the environmental visual data using a virtual reality host, and sends the virtual reality scene to a virtual reality headset, including:

[0017] The cockpit acquires robot perception data via a communication link, and performs distortion correction and noise reduction on the environmental visual data through a virtual reality host to determine the target visual data; the target visual data includes left-eye visual data and right-eye visual data.

[0018] The cockpit employs a stereo matching algorithm to determine the disparity value based on the target visual data and generate a depth map based on the disparity value.

[0019] The cockpit renders a 3D scene based on the depth map and the gimbal angle data, determines the virtual reality scene, and sends the virtual reality scene to the virtual reality headset.

[0020] In one embodiment, the cockpit acquires the operator's head posture data via a virtual reality headset, and the virtual reality host generates robot head control commands based on the head posture data and gimbal angle data. These robot head control commands are then sent to the robot via a communication link, including:

[0021] The cockpit collects the operator's head posture data through a virtual reality headset, and then uses the virtual reality host to perform coordinate unification processing on the head posture data and gimbal angle data to determine standardized head data and standardized gimbal data.

[0022] The cockpit uses a virtual reality host to determine the viewing angle deviation between the operator's head view and the gimbal view of the six-degree-of-freedom gimbal, based on the standardized head data and standardized gimbal data.

[0023] The cockpit uses a virtual reality host to generate gimbal control commands based on the viewing angle deviation, and then sends the gimbal control commands to the robot via a communication link.

[0024] In one embodiment, after the robot end controls the six-degree-of-freedom gimbal to execute the action corresponding to the head control command according to the robot head control command, it further includes:

[0025] The robot collects the updated attitude data of the six-degree-of-freedom gimbal and sends the updated attitude data to the cockpit via the communication link.

[0026] In one embodiment, the robot control method described above further includes:

[0027] After receiving the folding command from the operator, the robot uses LiDAR and binocular cameras to scan the environment around the robot and generate a 3D obstacle map.

[0028] Based on the three-dimensional obstacle map, the robot controls the robotic arm and the six-degree-of-freedom gimbal to fold in the opposite direction according to the unfolding path, so as to lock the robotic arm and the six-degree-of-freedom gimbal to the storage position on the back of the robot.

[0029] Secondly, this application also provides a robot control device, the robot control device comprising:

[0030] A perception data acquisition module, deployed on the robot, is used to collect robot perception data and send the robot perception data to the cockpit via a communication link. The robot perception data includes environmental visual data, gimbal angle data of the six-degree-of-freedom gimbal, and force data of the robot arm. The communication link includes a time-sensitive network wired link and a wireless link.

[0031] The virtual reality scene determination module is deployed in the cockpit and is used to acquire preprocessed robot perception data through a communication link, determine the virtual reality scene based on the environmental visual data through the virtual reality host, and send the virtual reality scene to the virtual reality headset so that the operator can obtain the robot's perspective visual experience based on the virtual reality headset.

[0032] The robotic arm control module, deployed at the cockpit end, is used to determine the robotic arm acceleration and displacement based on the force data through the cockpit controller, and control the cockpit robotic arm at the cockpit end based on the robotic arm acceleration and displacement, so that the operator can obtain the tactile presence of the bi-armed quadruped robot based on the cockpit robotic arm.

[0033] The head control command determination module is deployed in the cockpit and is used to collect the operator's head posture data through the virtual reality headset, generate robot head control commands based on the head posture data and gimbal angle data through the virtual reality host, and send the robot head control commands to the robot end through the communication link.

[0034] The gimbal control module, deployed on the robot, is used to control the six-degree-of-freedom gimbal to execute the actions corresponding to the head control commands according to the robot head control commands.

[0035] Thirdly, this application also provides a computer device, including a memory and a processor. The memory stores a computer program that, when the computer device is a robot end, performs the following steps: collecting robot perception data and sending the robot perception data to the cockpit end via a communication link; the robot perception data includes environmental visual data, gimbal angle data of a six-degree-of-freedom gimbal, and force data of the robot arm; the communication link includes a time-sensitive network wired link and a wireless link; and controlling the six-degree-of-freedom gimbal to execute the action corresponding to the head control command according to the robot head control command sent from the cockpit end.

[0036] When the computer device is a cockpit, the following steps are performed: Preprocessed robot perception data is acquired via a communication link, and a virtual reality scene is determined based on the environmental visual data using a virtual reality host. This virtual reality scene is then sent to a virtual reality headset, allowing the operator to obtain a visual experience from the robot's perspective using the headset. The cockpit controller determines the robotic arm's acceleration and displacement based on the force data, and controls the cockpit robotic arm based on these accelerations and displacements, allowing the operator to experience tactile presence from the bi-armed quadruped robot. The operator's head posture data is collected via the virtual reality headset, and the virtual reality host generates robot head control commands based on the head posture data and gimbal angle data. These commands are then sent to the robot via a communication link.

[0037] Fourthly, this application also provides a robot control system, which includes a robot end and a cockpit end.

[0038] The aforementioned robot control method, device, computer equipment, and storage medium involve the robot end collecting robot perception data and transmitting it to the cockpit end via a communication link. The robot perception data includes environmental visual data, gimbal angle data of a six-degree-of-freedom gimbal, and force data of the robot's robotic arm. The communication link includes a time-sensitive network wired link and a wireless link. The cockpit end acquires the preprocessed robot perception data via the communication link, determines the virtual reality scene based on the environmental visual data using a virtual reality host, and sends the virtual reality scene to a virtual reality headset, enabling the operator to obtain the robot's perspective view based on the virtual reality headset. Experience: The cockpit controller determines the robotic arm's acceleration and displacement based on force data, and controls the cockpit robotic arm based on these accelerations and displacements, allowing the operator to experience the tactile presence of a bi-armed quadruped robot. The cockpit also collects the operator's head posture data via a virtual reality headset, and the virtual reality host generates robot head control commands based on the head posture data and gimbal angle data, sending these commands to the robot via a communication link. The robot then controls the six-degree-of-freedom gimbal to execute the corresponding head control actions. This solution addresses the problem of traditional robots using cameras mounted on the end of the robotic arm and the body, where limited viewing angles make it difficult to determine the distance relationship with the manipulated object, resulting in low success rates. In the above solution, the cockpit acquires pre-processed perception data from the robot via a communication link. On one hand, the virtual reality host constructs a virtual reality scene based on the environmental visual data and sends it to the VR headset, allowing the operator to experience the robot's perspective. On the other hand, the controller controls the cockpit's robotic arm based on force data, providing the operator with a tactile sense of presence. The VR headset collects the operator's head posture data, and the virtual reality host combines this with gimbal angle data to generate control commands and send them to the robot, driving the 6DoF gimbal to complete the corresponding actions. This improves the robot's control accuracy and stability, as well as enhancing the operator's immersive interactive experience when operating the robot. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating a robot control method in one embodiment;

[0040] Figure 2 This is a schematic diagram of a bi-armed quadruped robot in one embodiment;

[0041] Figure 3 This is a schematic diagram of a force feedback VR cockpit in one embodiment;

[0042] Figure 4 This is a flowchart illustrating the robot control method in another embodiment;

[0043] Figure 5 This is a schematic diagram of a bi-armed quadruped robot in another embodiment;

[0044] Figure 6 This is a schematic diagram of a bi-armed quadruped robot in another embodiment;

[0045] Figure 7 This is a structural block diagram of a robot control device in one embodiment;

[0046] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0048] In one embodiment, such as Figure 1 As shown, a robot control method is provided. The robot control method is executed by a robot control system, which includes a robot end and a cockpit end. The method includes the following steps:

[0049] S110: The robot collects robot perception data and sends the robot perception data to the cockpit via a communication link.

[0050] Robot perception data includes environmental visual data, gimbal angle data of the six-degree-of-freedom gimbal, and force data of the robot arm; communication links include time-sensitive network wired links and wireless links; robot arm status includes robot arm position, robot arm speed, and robot arm torque.

[0051] The robot to be controlled is a bi-armed quadruped robot, such as... Figure 2As shown. The robot end includes a robot controller, communication unit, switch, 6DoF (six degrees of freedom) gimbal, two robotic arms, two robotic hands, and a binocular camera. Environmental visual data is a set of images and spatial information collected by the robot end through visual sensors to characterize the features of its external environment. Gimbal angle data is a set of parameters used to accurately characterize the real-time spatial position and angle state of the gimbal. Force data covers the magnitude of force and torque values ​​of each joint of the robotic arm, as well as the direction of contact force and magnitude of contact torque when the end effector contacts an object. Time-Sensitive Network (TSN) wired link refers to a communication technology that ensures low latency and high determinism in data transmission. Wireless links can use Wi-Fi 7 or 5G URLLC technology, supporting mobile scenarios where robots work while moving. Latency can be controlled in the millisecond to tens of milliseconds, and bandwidth is high. It is mainly used to transmit environmental visual data and chassis movement commands, adapting to the needs of robots operating in dynamic environments. The dual-redundant link design improves communication reliability. When one link fails, if the wired link is cut off or the wireless signal weakens, the other link will automatically and seamlessly switch over without interrupting data transmission, ensuring that the operation can continue.

[0052] It should be noted that the hardware structure of the bi-armed quadruped robot includes a mobility and manipulation platform, a robot control system, and a robot communication unit. The mobility and manipulation platform includes the robot chassis, the robot's two robotic arms, a 6DoF gimbal, various sensors, and controllers. A 6DoF gimbal specifically refers to a single device with six degrees of freedom (DOF): three translational and three rotational. This means it has forward / backward, left / right, up / down translation, and pitch, yaw, and roll rotation. The robot control system, centered on the robot controller, connects the robotic arms, manipulator, 6DoF gimbal, and binocular cameras. It incorporates connection failure protection logic and supports teach pendant integration for manual control of the robotic arms. The robot communication unit, paired with a switch, establishes an information exchange channel with the control center. It utilizes a time-sensitive network (TSN) wired link and a dual-redundant wireless link, prioritizing data transmission via the TSN wired link. In the event of a TSN wired link failure, it switches to the dual-redundant wireless link to ensure reliable and low-latency data transmission. The system employs a dual-link architecture with both wired and wireless communication, ensuring continuous operation even in complex environments. Furthermore, the robot is equipped with an onboard fixed camera, utilizing a dual-vision solution of binocular and onboard fixed cameras to ensure that environmental images can still be acquired even if a single camera fails.

[0053] The cockpit includes interactive devices, a cockpit control system, and a cockpit communication unit. The interactive devices include two cockpit robotic arms with zero-force drag and force feedback capabilities, finger-type control hands, a VR (Virtual Reality) headset, rudders, physical joysticks, a VR host, a physical button panel, and an external teach pendant interface. The finger-type control hands are a basic two-finger type and support expansion. The VR headset integrates a 6DoF gimbal for spatial positioning, used only to capture the operator's head posture and display the virtual reality scene; it does not integrate control functions. The rudders are three-degree-of-freedom flight rudders; the operator can send control signals to control the chassis movement of the bi-armed quadruped robot. When the gimbal's range of motion exceeds the operator's normal head reach, the operator can send control signals via the physical joystick to adjust the 6DoF gimbal's viewpoint on the robot. The physical button panel includes an emergency stop button and a mode switch button. The cockpit control system, centered on the cockpit controller, connects to the interactive devices and processes operating commands and feedback signals. The VR host is used to render scenes in real time based on environmental visual data and transmit the rendered images to the VR headset, thereby constructing an immersive visual environment. The cockpit communication unit, in conjunction with the switch on the cockpit side, establishes a data transmission link with the robot side, relying on protocols such as TSN, Wi-Fi 7, and 5G to ensure low-latency command interaction and high-bandwidth video streaming. The cockpit side is integrated into the force feedback VR cockpit, which, like... Figure 3 As shown.

[0054] For example, the robot collects robot perception data and sends the robot perception data to the cockpit via a communication link, including:

[0055] The robot acquires environmental visual data through a binocular camera and determines the gimbal angle data of the quadruped robot through the joint position sensing data fed back by the joint position sensors of the quadruped robot. It also acquires force data through sensors on the manipulator arm. The environmental visual data, gimbal angle data, and force data are used as the robot's perception data. The robot sends the robot's perception data to the cockpit through a communication link.

[0056] It should be noted that force / torque sensors are pre-installed at the end of the robotic arm or at key joints. When the robotic arm performs operations such as grasping, twisting, and pressing, the sensors will capture force data in real time.

[0057] Specifically, the robot synchronously acquires environmental visual data via binocular cameras, collects gimbal angle data via the attitude sensor of the 6DoF gimbal, and collects force / torque data of the quadruped robot via force / torque sensors. The robot transmits this perceived data back to the cockpit via a communication link. The gimbal angle data, after kinematic analysis, yields pitch, yaw, roll, and translational positions in three-dimensional space, accurately reflecting the real-time attitude of the gimbal and its onboard binocular cameras. Force / torque sensors on the manipulator arms collect force data. This force data reflects the contact force when the quadruped robot grasps an object and the torque experienced by the joints during operation, directly impacting operational accuracy and safety.

[0058] In the above scheme, the binocular camera is responsible for stably acquiring global or local environmental visual information and follows the movement of the gimbal; the manipulator arm sensors are close to the end of the workpiece, which can collect core mechanical data such as contact force and torque in real time, reducing signal transmission loss. This ensures the accuracy of the acquisition of various sensing data while reducing data redundancy.

[0059] For example, the cockpit acquires the operator's operation signals, encodes the operation signals into standardized instructions through the virtual reality host, and sends the standardized instructions to the robot through a communication link; the robot acquires the operation signals through the communication link and controls the robot arm, six-degree-of-freedom gimbal, and robot chassis to execute the work tasks corresponding to the operation signals.

[0060] The operator inputs the operation signal by controlling the robotic arm at the cockpit end, the finger-type control hand, the foot rudder, the physical joystick and / or physical buttons, or by manually jogging the robotic arm through the teach pendant to input the operation signal, or by pre-inputting the operation signal through programming control.

[0061] For example, when the operator's arm exerts force, the sensors on the cockpit robotic arm can obtain the magnitude and direction of the force exerted by the operator's arm, so that the cockpit robotic arm moves in the direction of the force exerted by the operator's arm based on the magnitude of the force exerted by the operator's arm.

[0062] Specifically, the operator inputs operation signals by controlling the robotic arm, finger-type control hand, foot rudder, physical joystick, and / or physical buttons, or, in an emergency, manually jogs the robotic arm via a teach pendant to input operation signals, or pre-inputs preset operation signals through programming control. After receiving the operator's operation signals, the cockpit encodes them into standardized instructions that the robot can recognize, and sends these standardized instructions to the robot via a communication link. The robot can then receive the operation signals via the TSN wired link, identify them, determine the control signals for the robotic arm, six-DOF gimbal, and robot chassis, and control these components to execute the corresponding tasks based on the control signals.

[0063] The above solution provides diverse input methods for operation signals, adapting to different operational scenarios and emergency situations: During routine operations, operators can achieve precise control through the robotic arm, foot rudder, switches, and / or buttons in the cockpit; in complex operational scenarios, finger-type control hands and physical joysticks can be used to improve motion control precision; in emergency scenarios, manual jogging via the teach pendant is supported to avoid operation interruption due to main equipment failure; it also supports programmed preset operation signals to achieve unmanned automatic execution of standardized tasks. The combination of multiple input methods satisfies both the flexibility of manual remote operation and the convenience of automated operation. After receiving standardized instructions, the robot can parse the standardized instructions and generate corresponding control signals for the robotic arm, 6DoF gimbal, and quadruped chassis, realizing coordinated linkage of multiple devices.

[0064] S120: The cockpit acquires preprocessed robot perception data through a communication link, and determines the virtual reality scene based on the environmental visual data through the virtual reality host, and sends the virtual reality scene to the virtual reality headset so that the operator can obtain the robot's visual experience based on the virtual reality headset.

[0065] The virtual reality (VR) scene refers to an immersive 3D virtual environment constructed in a VR headset within the cockpit, based on environmental visual data and gimbal angle data collected by the robot. This environment is mapped 1:1 to the robot's actual working environment. The VR scene uses environmental visual data collected by the robot's binocular cameras as a foundation. A depth map is generated through a stereo matching algorithm, transforming the 2D image into a 3D point cloud with spatial depth and perspective. Combined with real-time attitude data from the 6DoF gimbal, the camera point cloud is aligned with real-world coordinates, completely replicating the robot's real working environment. Operators can intuitively see the robot arm's real-time movements and the position of the work object within the scene. Combined with the tactile feedback from the force feedback arm, operators can accurately complete tasks such as grasping, assembly, and inspection.

[0066] For example, the cockpit acquires preprocessed robot perception data via a communication link, determines the virtual reality scene based on environmental visual data using a virtual reality host, and sends the virtual reality scene to the virtual reality headset, including:

[0067] S1201: The cockpit acquires robot perception data through a communication link, and uses a virtual reality host to perform distortion correction and noise reduction on the environmental visual data to determine the target visual data.

[0068] The target visual data includes left-eye visual data and right-eye visual data;

[0069] Specifically, the environmental visual data is obtained by simultaneously acquiring two planar images of the same environment from the binocular stereo camera on the 6DoF gimbal of the robot. That is, the environmental visual data is the raw visual data with natural parallax. The VR host performs distortion correction, noise reduction and illumination enhancement on the environmental visual data to determine the target visual data, thereby eliminating lens errors and environmental interference, and ensuring pixel alignment and clear details in the left and right images.

[0070] S1202, the cockpit end uses a stereo matching algorithm to determine the disparity value based on the target visual data and generate a depth map based on the disparity value.

[0071] Specifically, using a stereo matching algorithm, the spatial distance corresponding to each pixel is calculated based on the pixel disparity between the left-eye visual data and the right-eye visual data, i.e., the disparity value. Based on the spatial distance, a depth map containing three-dimensional coordinate information is generated.

[0072] S1203: The cockpit renders a 3D scene based on the depth map and gimbal angle data, determines the virtual reality scene, and sends the virtual reality scene to the virtual reality headset.

[0073] Specifically, the coordinate system of the depth map is converted to a global reference coordinate system with the optical center of the binocular camera as the origin. Simultaneously, the real-time attitude data of the 6DoF gimbal is matched with the operator's head posture to ensure that the VR screen's perspective is synchronized with the operator's head orientation. The image containing depth information is converted into a split-screen display for the left and right eyes, conforming to human stereoscopic vision, and then output to the VR headset. The VR headset displays the corresponding images for each eye, and through the parallax fusion effect of the human eye, the operator perceives an immersive virtual reality scene with spatial depth and perspective.

[0074] For example, another method for operators to perceive an immersive virtual reality scene with spatial depth and distance levels can be: the cockpit terminal uses a virtual reality host to adjust the center distance of the left and right eye images, scaling factor, and pixel offset, etc., according to the driver's interpupillary distance and subjective evaluation parameters, so that the driver's left and right eye images are close to the actual viewing effect of the naked eye, so that the operator can generate a stereoscopic visual sense.

[0075] The above solution, based on pixel parallax of target visual data, calculates spatial distance and generates a depth map through a stereo matching algorithm. Its core advantage lies in transforming two-dimensional planar images into spatial data with three-dimensional coordinate information. The depth map accurately restores the near and far layers and object outlines of the robot's working environment, giving the virtual reality scene the realistic attributes of physical space. By synchronously adjusting the camera posture to align with the driver's viewing angle, it avoids problems such as missing point clouds and holes caused by object occlusion in the presented 3D scene due to differences in viewing angles.

[0076] S130, the cockpit controller determines the acceleration and displacement of the robotic arm based on force data, and controls the cockpit robotic arm based on the acceleration and displacement of the robotic arm, so that the operator can obtain a tactile sense of presence of the bi-armed quadruped robot based on the cockpit robotic arm.

[0077] Among them, tactile presence refers to the mechanical tactile experience that the operator obtains through the force feedback robotic arm at the cockpit end, which is completely matched to the operation of the remote bi-arm quadruped robot. It is one of the core perception dimensions for realizing immersive teleoperation.

[0078] Specifically, the cockpit controller receives force data transmitted back from the robot and calculates the acceleration and displacement parameters of the robot arm based on the robot arm's dynamics model. Then, based on these parameters, it drives the force feedback robot arm in the cockpit to complete the corresponding motion. This allows the operator to accurately perceive the mechanical state of the remote bi-armed quadruped robot when interacting with the work object through the tactile changes of the cockpit robot arm, thus obtaining a highly realistic tactile sense of presence.

[0079] For example, the cockpit robotic arm measures the operating force applied by the driver through force sensors at its end effector or joints. Based on a compliant control law algorithm, the operating force is converted into spatial displacement of the cockpit robotic arm's end effector, causing the end effector to move in the direction of the driver's operating force. The movements of the cockpit robotic arm are synchronously transmitted to the robot end, causing the robot end robotic arm to perform the same movements. When the robot end robotic arm touches an object in the environment during its movement, the force sensors on the robot end robotic arm can detect the magnitude and direction of the environmental contact force. The environmental contact force is transmitted to the cockpit end via a link, and together with the driver's operating force, it is processed by a bidirectional force feedback algorithm to convert the contact force into the motion output of the cockpit robotic arm.

[0080] For example, the driver applies a downward force of 10N to the robotic arm at the cockpit end, which moves downward with an acceleration of 0.1m / s². When the robotic arm at the robot end touches the object being manipulated, the force sensor at the robot end generates an upward counterforce of 8N. At this time, according to the two-way force feedback algorithm, the acceleration of both the robotic arm at the robot end and the robotic arm at the cockpit end decreases to 0.02m / s². When the driver applies the same 10N force, the acceleration obtained is significantly reduced, and the operator feels an 8N counterforce, resulting in a noticeable sense of resistance.

[0081] It should be noted that the force feedback error of the control robotic arm is kept within an extremely small range, accurately reproducing the tactile sensations of interaction between the operating end and the environment, such as the perception of object stiffness and elasticity. The control of the robotic arm supports three modes: master-slave control for remote operation, manual inching via the teach pendant, and automatic operation of preset programs. The control mode of the robotic arm can be flexibly switched according to the scenario. For example, in case of master-slave control failure, emergency operation can be performed via the teach pendant.

[0082] S140. The cockpit end collects the operator's head posture data through a virtual reality headset, and generates robot head control commands based on the head posture data and gimbal angle data through the virtual reality host, and sends the robot head control commands to the robot end through a communication link.

[0083] Specifically, the virtual reality headset in the cockpit collects the operator's head posture data in real time. The virtual reality host then matches and calibrates this data with the 6DoF gimbal angle data transmitted from the robot, generating corresponding robot head control commands. These commands are then sent to the robot via a communication link, driving the 6DoF gimbal to synchronize the binocular camera with the operator's head movements and adjust its viewing angle. This ensures real-time synchronization between the virtual reality scene's perspective and the robot's camera perspective, providing the operator with an immersive observation experience. When sending robot head control commands to the robot via the communication link, priority is given to transmitting commands via a TSN wired link to ensure low-latency transmission.

[0084] For example, the cockpit collects the operator's head posture data through a virtual reality headset, and the virtual reality host generates robot head control commands based on the head posture data and gimbal angle data, and sends the robot head control commands to the robot via a communication link, including:

[0085] The cockpit collects the operator's head posture data through a virtual reality headset, and then processes the head posture data and gimbal angle data using a virtual reality host to unify the coordinates and determine standardized head data and standardized gimbal data. Based on the standardized head data and standardized gimbal data, the cockpit uses the virtual reality host to determine the viewing angle deviation between the operator's head viewpoint and the gimbal viewpoint of the six-degree-of-freedom gimbal. Based on the viewing angle deviation, the cockpit uses the virtual reality host to generate gimbal control commands and sends the gimbal control commands to the robot via a communication link.

[0086] Specifically, the cockpit uses a virtual reality headset to collect the operator's head posture data in real time. Then, the virtual reality host performs coordinate unification processing on this head posture data and the gimbal angle data returned by the robot to obtain standardized head data and standardized gimbal data. Based on this standardized head data and standardized gimbal data, the viewing angle deviation between the operator's head view and the current view of the six-degree-of-freedom gimbal is calculated. Finally, the corresponding gimbal control command is generated based on the deviation and sent to the robot through the communication link to drive the six-degree-of-freedom gimbal to adjust its angle, thereby achieving precise synchronization between the operator's head view and the gimbal view.

[0087] For example, a method for calculating the viewpoint deviation between the operator's head viewpoint and the current viewpoint of the six-DOF gimbal based on standardized head data and standardized gimbal data can be as follows: The method for calculating the viewpoint deviation based on standardized head data and standardized gimbal data is as follows: Under a unified coordinate system, extract the three-dimensional attitude parameters of the operator's head, including pitch angle, yaw angle, and roll angle, and extract the real-time three-dimensional attitude parameters of the six-DOF gimbal. By performing difference calculations on the angle parameters, obtain the angle deviation values ​​between the head viewpoint and the gimbal viewpoint in the three rotational degrees of freedom. At the same time, combine the difference correction of the translational degree of freedom parameters, and finally output the complete viewpoint deviation data.

[0088] It should be noted that the end-to-end response latency of the VR headset's head posture data to the robot's 6DoF gimbal is at the low millisecond level, enabling natural synchronization between the operator's viewpoint and head movement. The 6DoF gimbal's viewpoint can be achieved through either VR headset tracking or manual control with a physical joystick, with these two control methods serving as backups for each other. Manual control with the physical joystick can be used for large-range continuous angle adjustments, while VR headset tracking supports natural viewpoint switching.

[0089] The above solution calculates the angle difference between the head view and the gimbal view based on standardized data, which can quantify the degree of real-time deviation between the two. The resulting gimbal control commands have clear adjustment directions and amplitudes, which can drive the gimbal to accurately compensate for the view deviation and ensure that the view corresponding to the virtual reality scene presented by the VR headset is precisely synchronized with the operator's head movements.

[0090] For example, after the robot controls the six-degree-of-freedom gimbal to execute the action corresponding to the head control command according to the robot head control command, it also includes:

[0091] The robot collects updated attitude data from the six-degree-of-freedom gimbal and sends the updated attitude data to the cockpit via a communication link.

[0092] The above solution involves the robot controlling a six-DOF gimbal to execute actions corresponding to the head control commands. The robot then sends updated posture data to the cockpit via a communication link. The virtual reality host in the cockpit receives this updated posture data, uses it as new gimbal angle data, and recalculates the coordinates and deviations against the operator's head posture data collected by the virtual reality headset. This generates a new round of gimbal control commands, ensuring precise synchronization between the operator's head movements and the gimbal's perspective. This avoids perspective misalignment caused by gimbal execution errors or environmental interference, guaranteeing the stability of the immersive observation experience.

[0093] S150: The robot controls the six-degree-of-freedom gimbal to execute the actions corresponding to the head control commands according to the robot head control commands.

[0094] In the aforementioned robot control method, the robot end collects robot perception data and sends it to the cockpit end via a communication link. The robot perception data includes environmental visual data, gimbal angle data of the six-degree-of-freedom gimbal, and force data of the robot arm. The communication link includes a time-sensitive network wired link and a wireless link. The cockpit end obtains the pre-processed robot perception data through the communication link and, through the virtual reality host, determines the virtual reality scene based on the environmental visual data, and sends the virtual reality scene to the virtual reality headset, allowing the operator to obtain a visual experience from the robot's perspective based on the virtual reality headset. The cockpit end... The cockpit controller determines the acceleration and displacement of the robotic arm based on force data, and controls the cockpit-side robotic arm based on the acceleration and displacement, allowing the operator to experience tactile presence from the cockpit-side robotic arm. The cockpit collects the operator's head posture data via a virtual reality headset, and the virtual reality host generates robot head control commands based on the head posture data and gimbal angle data, sending these commands to the robot via a communication link. The robot then controls the six-degree-of-freedom gimbal to execute the corresponding head control actions. This solution solves the problem of traditional robots using cameras mounted on the end of the robotic arm and the body, where limited viewing angles make it difficult to determine the distance relationship with the manipulated object, resulting in low operation success rates. In the above solution, the cockpit acquires pre-processed perception data from the robot via a communication link. On one hand, the virtual reality host constructs a virtual reality scene based on the environmental visual data and sends it to the VR headset, allowing the operator to experience the robot's perspective. On the other hand, the controller controls the cockpit's robotic arm based on force data, providing the operator with a tactile sense of presence. The VR headset collects the operator's head posture data, and the virtual reality host combines this with gimbal angle data to generate control commands and send them to the robot, driving the 6DoF gimbal to complete the corresponding actions. This improves the robot's control accuracy and stability, as well as enhancing the operator's immersive interactive experience when operating the robot.

[0095] For example, such as Figure 4 As shown, based on the above embodiments, the robot control method further includes:

[0096] S210 After receiving the folding command from the operator, the robot uses LiDAR and binocular cameras to scan the environment around the robot, determine the environmental information, and generate a three-dimensional obstacle map based on the environmental information.

[0097] The diameter of the 3D obstacle map can be set according to actual needs, for example, it can be 1 meter.

[0098] For example, a bi-armed quadruped robot with its robotic arm and 6DoF gimbal in an extended state, such as... Figure 5 As shown.

[0099] Specifically, after the bi-armed quadruped robot completes its task, the remote stop button on the throttle valve disconnects the master-slave control link and disables the force feedback function. A folding command is triggered, and the robot controller initiates folding obstacle avoidance planning. The LiDAR and binocular camera work together to scan the surrounding environment, generating a 3D obstacle map within a 1-meter radius. The throttle valve is an integrated operation control unit that combines core function control buttons; it is the core interactive platform for switching between operation modes and triggering key commands throughout the entire process.

[0100] S220: Based on a 3D obstacle map, the robot controls the robotic arm and the six-degree-of-freedom gimbal to fold in the opposite direction according to the unfolding path, so as to lock the robotic arm and the six-degree-of-freedom gimbal to the storage position on the back of the robot.

[0101] Specifically, based on a 3D obstacle map, the robot controls its robotic arm and 6DoF gimbal to fold in reverse along the original unfolding path. During this process, the robot's current pose is compared with the obstacle map in real time. If the distance between the robotic arm or 6DoF gimbal and an obstacle is detected to be less than a preset collision risk threshold, the folding action is immediately paused, and a new detour trajectory is planned to avoid collisions. Finally, the robotic arm and gimbal lock into the storage position on the robot's back, allowing the robot's center of gravity to refocus on the center of its four-legged chassis, reducing drag during subsequent movement and improving stability. The collision risk threshold can be set according to actual needs, for example, it could be 5 centimeters.

[0102] In addition, the robot can be controlled by foot rudders to move along a preset evacuation route and leave the work area. If necessary, it can be switched to the automatic return mode of the bi-armed quadruped robot. The bi-armed quadruped robot will autonomously return to the initial deployment point along the path it came from. After reaching the safe area, the operator will press the sleep button to control the robot chassis to complete the crouching action and enter a low-power standby state.

[0103] For example, a bi-armed quadruped robot with its robotic arm and 6DoF gimbal in a folded state, such as... Figure 6 As shown.

[0104] The above solution generates a 3D obstacle map of a specified range through collaborative scanning of LiDAR and binocular cameras, providing accurate environmental perception for the folding action of the robotic arm and 6DoF gimbal. Combined with a pause and replanning mechanism triggered by a collision risk threshold, it can effectively avoid scraping and collisions with surrounding obstacles during the folding process, ensuring the structural safety of the equipment.

[0105] For example, based on the above embodiments, the robot control method includes:

[0106] During the deployment and initialization of the robot control system, the cockpit and the robot establish a two-way communication connection via a TS wired link or a Wi-Fi 7 / 5G wireless link, and the robot control system automatically completes equipment self-checks. These self-checks include verifying the status of the force feedback robotic arm, VR headset, binocular camera, and robotic arm joints. The operator performs initialization operations via a simulated flight throttle valve. Initialization operations include: triggering a stand-up command, causing the quadrupedal robot's chassis to stand up from a dormant position using a preset gait, automatically adjusting its center of gravity to a stable position; and rotating the throttle valve knob to select a walking gait, for example, the pacing mode is suitable for flat surfaces, while the climbing mode is suitable for rugged terrain. The robot control system then enters a standby state, at which point the dual arms and VR gimbal are in a folded, stowed position.

[0107] The robot synchronously acquires environmental visual data via binocular cameras, collects gimbal angle data via the attitude sensor of the 6DoF gimbal, and collects force / torque data of the bi-armed quadruped robot via force / torque sensors. The robot transmits this perceived data back to the cockpit via a communication link. The gimbal angle data, after kinematic analysis, yields pitch, yaw, roll, and translational positions in three-dimensional space, accurately reflecting the real-time attitude of the gimbal and its onboard binocular cameras. Force / torque sensors on the manipulator arms collect force data. This force data reflects the contact force when the bi-armed quadruped robot grasps objects and the torque experienced by the joints during operation, directly impacting operational accuracy and safety.

[0108] The operator inputs operation signals by controlling the robotic arm, finger-type control hand, foot rudder, physical joystick, and / or physical buttons, or, in an emergency, manually jogs the robotic arm via a teach pendant to input operation signals, or pre-inputs preset operation signals through programmable control. After receiving the operator's operation signals, the cockpit encodes them into standardized instructions that the robot can recognize, and sends these standardized instructions to the robot via a communication link. The robot receives the operation signals via the TSN wired link, identifies them, determines the control signals for the robotic arm, six-DOF gimbal, and robot chassis, and then controls these components to execute the corresponding tasks based on the control signals.

[0109] Environmental visual data is obtained by simultaneously acquiring two planar images of the same environment from the left and right sides using a binocular stereo camera on a 6DoF gimbal on the robot. This means the environmental visual data is raw visual data with natural parallax. The VR host performs distortion correction, noise reduction, and illumination enhancement on the environmental visual data to determine the target visual data, thereby eliminating lens errors and environmental interference and ensuring pixel alignment and detail clarity in the left and right images. Using a stereo matching algorithm, the spatial distance corresponding to each pixel is calculated based on the pixel parallax value between the left and right eye visual data, generating a depth map containing 3D coordinate information. The coordinate system of the depth map is aligned to a global reference coordinate system with the optical center of the binocular camera as the origin. Simultaneously, the real-time posture data of the 6DoF gimbal is matched with the operator's head posture to ensure that the VR screen viewpoint is synchronized with the operator's head orientation. The image with depth information is converted into a split-screen display for the left and right eyes, conforming to human stereoscopic vision, and then output to the VR headset. The VR headset displays the corresponding images for each eye separately. Through the parallax fusion effect of the human eye, the operator perceives an immersive virtual reality scene with spatial depth and perspective.

[0110] The cockpit controller receives force data from the robot and calculates the acceleration and displacement parameters of the robot arm using the robot arm dynamics model. Based on these parameters, it drives the force feedback robot arm in the cockpit to complete the corresponding motion. This allows the operator to accurately perceive the mechanical state of the remote bi-arm quadruped robot when interacting with the work object through the tactile changes of the cockpit robot arm, thus obtaining a highly realistic tactile sense of presence.

[0111] The cockpit collects the operator's head posture data via a virtual reality headset. The virtual reality host then performs coordinate unification processing on the head posture data and gimbal angle data to determine standardized head and gimbal data. Based on this standardized data, the cockpit uses the virtual reality host to determine the viewing angle deviation between the operator's head viewpoint and the six-degrees-of-freedom (6DOF) gimbal viewpoint. The cockpit then generates gimbal control commands based on this deviation and sends them to the robot via a communication link. The robot collects updated posture data from the 6DOF gimbal and sends it back to the cockpit via the communication link. The cockpit's virtual reality host receives this updated posture data, uses it as new gimbal angle data, and performs coordinate unification and deviation calculation again with the operator's head posture data collected by the headset to generate a new round of gimbal control commands, ensuring that the operator's head movements and the gimbal viewpoint remain precisely synchronized. The robot then controls the 6DOF gimbal to execute the actions corresponding to the robot's head control commands.

[0112] After the bi-armed quadruped robot completes its task, the remote stop button on the throttle valve is pressed, disconnecting the master-slave control link and disabling the force feedback function. A folding command is triggered, and the robot controller initiates folding obstacle avoidance planning. The LiDAR and binocular camera work together to scan the surrounding environment, generating a 3D obstacle map within a 1-meter radius. Based on this 3D obstacle map, the robot controls its robotic arm and 6DoF gimbal to fold in the reverse direction along the original unfolding path. During this process, the robot's current pose is compared with the obstacle map in real time. If the distance between the robotic arm or 6DoF gimbal and an obstacle is detected to be less than a preset collision risk threshold, the folding action is immediately paused, and a new detour trajectory is planned to avoid collisions. Finally, the robotic arm and gimbal lock into the storage position on the robot's back, allowing the robot's center of gravity to refocus on the quadruped chassis, reducing drag during subsequent movement and improving stability.

[0113] Based on the above embodiments, a foot rudder device is used to realize the motion control of the robot chassis of the bi-armed quadruped robot. The mapping relationship between its mechanical structure and control logic is as follows: the minimum value of the two pedals of the foot rudder is used as the forward speed command of the chassis; when switching to reverse mode via the gear shift key on the throttle valve, the pedal depth is mapped to the reverse speed. The difference in the pedal depth is used as the steering angular velocity command; a positive difference indicates leftward steering, and a negative difference indicates rightward steering. Simultaneously, the bi-armed quadruped robot integrates a LiDAR for environmental perception assistance, scanning obstacles in a range of 0.5-5m in front in real time. When an obstacle is detected, automatic deceleration or pre-stop operation is performed. For example, when an obstacle greater than or equal to 1m and less than 2m is detected, automatic deceleration is performed; when an obstacle greater than 0.5m and less than 1m is detected, a pre-stop operation is performed. For obstacles with a height greater than 30cm, the system activates a local path planning algorithm to automatically generate a detour line based on the obstacle contour, maintaining the directional trend of the foot rudder commands during detour to avoid conflict between automatic obstacle avoidance and the operator's intentions. During the walking phase, the robotic arm and six-DOF gimbal remain in a retracted position. Folding the arms and retracting them to the center of the back concentrates the overall center of gravity in the central area of ​​the four-legged support surface, reducing inertia fluctuations during walking and improving gait stability. The retracted posture reduces passive swaying of the robotic arm joints during chassis vibrations, minimizing additional torque loss and preventing fatigue damage to the linkage structure due to high-frequency vibrations. The six-DOF gimbal robotic arm precisely stops and locks in the preset retracted position, and rigid fixation reduces elastic deformation of the multi-link structure during walking, thus reducing image jitter in the binocular camera. View control includes a linked mode and a manual mode. In linked mode, changes in the VR headset's head posture drive the gimbal to rotate synchronously, enabling first-person perspective observation. In manual mode, the yaw and pitch angles of the gimbal are independently controlled via a 2D joystick integrated with a throttle valve, suitable for precise fixed-point observation. The gimbal yaw angle is ±180°, and the pitch angle is -60° to 90°. An onboard fixed camera synchronously captures panoramic images, which are displayed in real time on the cockpit auxiliary display screen as a backup for VR vision. After the bi-armed quadruped robot moves to the target work area, the operator sends a command through the throttle valve station key. The quadruped chassis adjusts the posture of the four limbs to fix the body in a horizontal and stable state. After triggering the deployment command, the robot controller starts the automatic path planning algorithm. The robotic arm unfolds from the back storage position along a preset safety trajectory. The end effector moves to the initial work posture and performs obstacle avoidance checks to ensure that there are no collisions during the robotic arm deployment process. The VR gimbal returns from 2-axis mode to 6DoF full degree of freedom. The binocular camera is adjusted to a horizontal viewing angle. After the deployment is completed, the system sends a ready signal, and the throttle valve indicator light turns green.During the teleoperation phase, the operation mode is entered via the "teleoperation start button" on the throttle valve. At this point, the system activates a full-function control closed loop, with the master-slave control robotic arm and the robot manipulator forming a 1:1 motion mapping. The opening and closing movements of the finger-type control hand directly drive the end effector, enabling precise operations such as object grasping and assembly in conjunction with the force feedback system. The VR headset's 6DoF spatial positioning and 6DoF gimbal achieve full-degree-of-freedom synchronization, and the binocular camera image, rendered by the VR host, provides an immersive first-person perspective. The robot chassis auxiliary control includes three control strategies: motion mode, stationary mode, and height adjustment. In motion mode, the operator continues to control the robot's overall translation and rotation via foot rudders to adapt to fine-tuning needs in the work position; the system automatically reduces foot rudder sensitivity to minimize body sway caused by excessive manipulation. In stationary mode, the foot rudder directional values ​​are mapped to chassis heading, pitch, or roll fine-tuning commands, enabling precise adjustments to the robot's attitude. Height adjustment refers to rotating the throttle valve knob to control the standing height of the four-legged chassis, adapting to different work height requirements. The standing height can be adjusted within ±200mm.

[0114] In the above robot control method, the robot end collects robot perception data and sends the robot perception data to the cockpit end via a communication link. The robot perception data includes environmental visual data, gimbal angle data of the six-degree-of-freedom gimbal, and force data of the robot arm. The communication link includes a time-sensitive network wired link and a wireless link. The cockpit end obtains the preprocessed robot perception data through the communication link, determines the virtual reality scene based on the environmental visual data through the virtual reality host, and sends the virtual reality scene to the virtual reality headset so that the operator can obtain the robot's perspective visual experience based on the virtual reality headset. The cockpit controller determines the robotic arm's acceleration and displacement based on force data, and controls the cockpit-side robotic arm based on these accelerations and displacements, allowing the operator to experience tactile presence from the cockpit-side robotic arm. The cockpit collects the operator's head posture data via a virtual reality headset, and the virtual reality host generates robot head control commands based on the head posture data and gimbal angle data, sending these commands to the robot via a communication link. The robot then controls the six-degree-of-freedom gimbal to execute the corresponding head control actions. This solution addresses the problem of traditional robots using cameras mounted on the end of the robotic arm and the body, where limited viewing angles make it difficult to determine distance relationships with the manipulated object, resulting in low success rates. In the above solution, the cockpit acquires pre-processed perception data from the robot via a communication link. On one hand, the virtual reality host constructs a virtual reality scene based on the environmental visual data and sends it to the VR headset, allowing the operator to experience the robot's perspective. On the other hand, the controller controls the cockpit's robotic arm based on force data, providing the operator with a tactile sense of presence. The VR headset collects the operator's head posture data, and the virtual reality host combines this with gimbal angle data to generate control commands and send them to the robot, driving the 6DoF gimbal to complete the corresponding actions. This improves the robot's control accuracy and stability, as well as enhancing the operator's immersive interactive experience when operating the robot.

[0115] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0116] Based on the same inventive concept, this application also provides a robot control device for implementing the robot control method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more robot control device embodiments provided below can be found in the limitations of the robot control method described above, and will not be repeated here.

[0117] In one embodiment, such as Figure 7 As shown, a robot control device is provided, including: a perception data acquisition module 701, a virtual reality scene determination module 702, a robotic arm control module 703, a head control command determination module 704, and a gimbal control module 705, wherein:

[0118] The perception data acquisition module 701 is deployed on the robot and is used to collect robot perception data and send the robot perception data to the cockpit via a communication link. The robot perception data includes environmental visual data, gimbal angle data of the six-degree-of-freedom gimbal, and force data of the robot arm. The communication link includes a time-sensitive network wired link and a wireless link.

[0119] The virtual reality scene determination module 702 is deployed in the cockpit and is used to acquire preprocessed robot perception data through a communication link, determine the virtual reality scene based on the environmental visual data through the virtual reality host, and send the virtual reality scene to the virtual reality headset so that the operator can obtain the robot's perspective visual experience based on the virtual reality headset.

[0120] The robotic arm control module 703 is deployed at the cockpit end. It is used to determine the acceleration and displacement of the robotic arm based on the force data through the cockpit controller, and control the cockpit robotic arm at the cockpit end based on the acceleration and displacement of the robotic arm, so that the operator can obtain the tactile presence of the bi-arm quadruped robot based on the cockpit robotic arm.

[0121] The head control command determination module 704 is deployed in the cockpit and is used to collect the operator's head posture data through the virtual reality headset, generate robot head control commands based on the head posture data and gimbal angle data through the virtual reality host, and send the robot head control commands to the robot end through the communication link.

[0122] The gimbal control module 705 is deployed on the robot and is used to control the six-degree-of-freedom gimbal to perform the actions corresponding to the head control commands according to the robot head control commands.

[0123] For example, the sensing data acquisition module 701 is specifically used for:

[0124] The robot acquires environmental visual data using a binocular camera, determines the gimbal angle data of the quadruped robot by using joint position sensing data fed back by the joint position sensors, and collects force data by sensors on the manipulator arm. The environmental visual data, the gimbal angle data, and the force data are used as the robot's perception data. The state of the robot arm includes the position, speed, and torque of the robot arm.

[0125] The robot's perception data is sent to the cockpit via a communication link.

[0126] Furthermore, the aforementioned robot control device also includes:

[0127] An operation signal processing module, deployed in the cockpit, is used to acquire the operator's operation signals, encode the operation signals into standardized instructions via a virtual reality host, and send the standardized instructions to the robot via a communication link. The operator inputs the operation signals by controlling the robotic arm, finger-type control hand, foot rudder, physical joystick and / or physical buttons in the cockpit, or by manually jogging the robotic arm via a teach pendant, or by pre-inputting the operation signals through programming control.

[0128] The task execution module, deployed on the robot, is used to acquire the operation signal through the communication link and control the robot arm, six-degree-of-freedom gimbal and robot chassis to execute the work task corresponding to the operation signal.

[0129] Furthermore, the virtual reality scene determination module 702 is specifically used for:

[0130] Robot perception data is acquired via a communication link, and the environmental visual data is subjected to distortion correction and noise reduction processing by a virtual reality host to determine the target visual data; the target visual data includes left-eye visual data and right-eye visual data.

[0131] Using a virtual reality host and a stereo matching algorithm, the disparity value is determined based on the target visual data, and a depth map is generated based on the disparity value.

[0132] The virtual reality host performs 3D scene rendering based on the depth map and the gimbal angle data to determine the virtual reality scene, and then sends the virtual reality scene to the virtual reality headset.

[0133] For example, the head control command determination module 704 is specifically used for:

[0134] The operator's head posture data is collected by a virtual reality headset, and the head posture data and gimbal angle data are processed by a virtual reality host to unify the coordinates and determine standardized head data and standardized gimbal data.

[0135] Using a virtual reality host, the viewing angle deviation between the operator's head view and the gimbal view of the six-degree-of-freedom gimbal is determined based on the standardized head data and standardized gimbal data.

[0136] The virtual reality host generates gimbal control commands based on the viewing angle deviation and sends the gimbal control commands to the robot via a communication link.

[0137] For example, the PTZ control module 705 is also specifically used for:

[0138] The updated attitude data of the six-degree-of-freedom gimbal is collected and sent to the cockpit via the communication link.

[0139] For example, the above-mentioned robot control device further includes:

[0140] The folding instruction execution module, deployed on the robot, is used to scan the robot's surrounding environment in collaboration with LiDAR and binocular cameras after receiving a folding instruction from the operator, and generate a 3D obstacle map.

[0141] Based on the three-dimensional obstacle map, the robot arm and the six-degree-of-freedom gimbal are controlled to fold in the opposite direction according to the unfolding path, so as to lock the robot arm and the six-degree-of-freedom gimbal to the storage position on the back of the robot.

[0142] Each module in the aforementioned robot control device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0143] In one embodiment, a computing device is provided, including a memory and a processor. The memory stores a computer program that, when the computing device is a robot end, performs the following steps: collecting robot perception data and sending the robot perception data to a cockpit end via a communication link; the robot perception data includes environmental visual data, gimbal angle data of a six-degree-of-freedom gimbal, and force data of a robot arm; the state of the robot arm includes the position, speed, and torque of the robot arm; the communication link includes a time-sensitive network wired link and a wireless link; and controlling the six-degree-of-freedom gimbal to execute the action corresponding to the robot head control command according to the robot head control command sent from the cockpit end.

[0144] When the computer device is a cockpit, the following steps are performed: Preprocessed robot perception data is acquired via a communication link, and a virtual reality scene is determined based on the environmental visual data using a virtual reality host. This virtual reality scene is then sent to a virtual reality headset, allowing the operator to obtain a visual experience from the robot's perspective using the headset. The cockpit controller determines the robotic arm's acceleration and displacement based on the force data, and controls the cockpit robotic arm based on these accelerations and displacements, allowing the operator to experience tactile presence from the bi-armed quadruped robot. The operator's head posture data is collected via the virtual reality headset, and the virtual reality host generates robot head control commands based on the head posture data and gimbal angle data. These commands are then sent to the robot via a communication link.

[0145] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program that, when the computer device is a robot end, performs the following steps: collecting robot perception data and transmitting the robot perception data to a cockpit end via a communication link; the robot perception data includes environmental visual data, gimbal angle data of a six-degree-of-freedom gimbal, and force data of a robot arm; the state of the robot arm includes the position, speed, and torque of the robot arm; the communication link includes a time-sensitive network wired link and a wireless link; and controlling the six-degree-of-freedom gimbal to execute the action corresponding to the robot head control command according to the robot head control command sent from the cockpit end.

[0146] When the computer device is a cockpit, the following steps are performed: Preprocessed robot perception data is acquired via a communication link, and a virtual reality scene is determined based on the environmental visual data using a virtual reality host. This virtual reality scene is then sent to a virtual reality headset, allowing the operator to obtain a visual experience from the robot's perspective using the headset. The cockpit controller determines the robotic arm's acceleration and displacement based on the force data, and controls the cockpit robotic arm based on these accelerations and displacements, allowing the operator to experience tactile presence from the bi-armed quadruped robot. The operator's head posture data is collected via the virtual reality headset, and the virtual reality host generates robot head control commands based on the head posture data and gimbal angle data. These commands are then sent to the robot via a communication link.

[0147] For example, in one embodiment, the computer device described above may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a robot control method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0148] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0149] In one embodiment, a robot control system is provided, the robot control system comprising a robot end and a cockpit end, wherein:

[0150] The robot collects robot perception data and sends it to the cockpit via a communication link. The robot then controls the six-degree-of-freedom gimbal to execute the actions corresponding to the head control commands according to the robot head control commands. The robot perception data includes environmental visual data, gimbal angle data of the six-degree-of-freedom gimbal, and force data of the robot arm. The communication link includes a time-sensitive network wired link and a wireless link.

[0151] The cockpit acquires preprocessed robot perception data via a communication link, and determines a virtual reality scene based on the environmental visual data using a virtual reality host, then sends the virtual reality scene to the virtual reality headset. The cockpit controller determines the robotic arm acceleration and displacement based on the force data, and controls the cockpit robotic arm based on the acceleration and displacement. The virtual reality headset collects the operator's head posture data, and the virtual reality host generates robot head control commands based on the head posture data and gimbal angle data, then sends these commands to the robot via the communication link.

[0152] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0153] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0154] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0155] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A robot control method, characterized in that, The robot control method is executed by a robot control system, which includes a robot end and a cockpit end. The robot control method includes: The robot collects robot perception data and sends it to the cockpit via a communication link. The robot perception data includes environmental visual data, gimbal angle data of a six-degree-of-freedom gimbal, and state and force data of the robot arm. The communication link includes a time-sensitive network wired link and a wireless link. The state of the robot arm includes the position, speed, and torque of the robot arm. The cockpit acquires preprocessed robot perception data through a communication link, and determines a virtual reality scene based on the environmental visual data through a virtual reality host, and sends the virtual reality scene to a virtual reality headset so that the operator can obtain a visual experience from the robot's perspective based on the virtual reality headset. The cockpit controller determines the acceleration and displacement of the robotic arm based on the force data, and controls the cockpit robotic arm based on the acceleration and displacement of the robotic arm, so that the operator can obtain a tactile sense of presence of the bi-armed quadruped robot based on the cockpit robotic arm. The virtual display host at the cockpit end collects the operator's head posture data through the virtual reality headset, sends it to the cockpit controller, and generates robot head control commands based on the head posture data and gimbal angle data. The robot head control commands are then sent to the robot end through the communication link. The robot controls the six-degree-of-freedom gimbal to execute the actions corresponding to the head control commands based on the robot head control commands.

2. The method according to claim 1, characterized in that, The robot end collects robot perception data and transmits the robot perception data to the cockpit end via a communication link, including: The robot acquires environmental visual data through a binocular camera, determines the gimbal angle data of the quadruped robot by using the joint position sensing data fed back by the joint position sensors of the quadruped robot, and acquires force data through sensors on the manipulator arm. The environmental visual data, the gimbal angle data and the force data are used as robot perception data. The robot sends the robot's perception data to the cockpit via a communication link.

3. The method according to claim 1, characterized in that, Also includes: The cockpit acquires the operator's operation signals and encodes them into standardized instructions via a virtual reality host. The standardized instructions are then sent to the robot via a communication link. The operator inputs the operation signals by controlling the robotic arm, finger-type control hand, foot rudder, physical joystick, and / or physical buttons on the cockpit, or by manually jogging the robotic arm via a teach pendant, or by pre-inputting the operation signals through programming control. The robot receives the operation signal through the communication link and controls the robot arm, six-degree-of-freedom gimbal and robot chassis to execute the work task corresponding to the operation signal.

4. The method according to claim 1, characterized in that, The cockpit acquires preprocessed robot perception data via a communication link, determines a virtual reality scene based on the environmental visual data using a virtual reality host, and sends the virtual reality scene to the virtual reality headset, including: The cockpit acquires robot perception data via a communication link, and performs distortion correction and interpupillary distance matching on the environmental visual data through a virtual reality host to determine the target visual data; the target visual data includes left-eye visual data and right-eye visual data. The cockpit employs a stereo matching algorithm to determine the disparity value based on the target visual data and generate a depth map based on the disparity value. The cockpit renders a 3D scene based on the depth map and the gimbal angle data, determines the virtual reality scene, and sends the virtual reality scene to the virtual reality headset.

5. The method according to claim 1, characterized in that, The cockpit collects the operator's head posture data via a virtual reality headset, and the cockpit controller generates robot head control commands based on the head posture data and gimbal angle data. These commands are then sent to the robot via a communication link, including: The cockpit collects the operator's head posture data through a virtual reality headset, and then uses the virtual reality host to perform coordinate unification processing on the head posture data and gimbal angle data to determine standardized head data and standardized gimbal data. The cockpit uses a virtual reality host to determine the viewing angle deviation between the operator's head view and the gimbal view of the six-degree-of-freedom gimbal, based on the standardized head data and standardized gimbal data. The cockpit uses a virtual reality host to generate gimbal control commands based on the viewing angle deviation, and then sends the gimbal control commands to the robot via a communication link.

6. The method according to claim 1, characterized in that, After the robot terminal controls the six-degree-of-freedom gimbal to execute the action corresponding to the head control command according to the robot head control command, it also includes: The robot calculates the updated attitude data of the six-degree-of-freedom gimbal and sends the updated attitude data to the cockpit via the communication link.

7. The method according to claim 1, characterized in that, Also includes: After receiving the folding command from the operator, the robot uses LiDAR and binocular cameras to scan the environment around the robot and generate a 3D obstacle map. Based on the three-dimensional obstacle map, the robot controls the robotic arm and the six-degree-of-freedom gimbal to fold in the opposite direction according to the unfolding path, so as to lock the robotic arm and the six-degree-of-freedom gimbal to the storage position on the back of the robot.

8. A robot control device, characterized in that, The robot control device includes: A perception data acquisition module, deployed on the robot, is used to collect robot perception data and send the robot perception data to the cockpit via a communication link. The robot perception data includes environmental visual data, gimbal angle data of the six-degree-of-freedom gimbal, and force data of the robot arm. The communication link includes a time-sensitive network wired link and a wireless link. The virtual reality scene determination module is deployed in the cockpit and is used to acquire preprocessed robot perception data through a communication link, determine the virtual reality scene based on the environmental visual data through the virtual reality host, and send the virtual reality scene to the virtual reality headset so that the operator can obtain the robot's perspective visual experience based on the virtual reality headset. The robotic arm control module, deployed at the cockpit end, is used to determine the robotic arm acceleration and displacement based on the force data through the cockpit controller, and control the cockpit robotic arm at the cockpit end based on the robotic arm acceleration and displacement, so that the operator can obtain the tactile presence of the bi-armed quadruped robot based on the cockpit robotic arm. The head control command determination module is deployed in the cockpit. It is used to collect the operator's head posture data through the virtual reality headset and send it to the cockpit controller. The cockpit controller generates robot head control commands based on the head posture data and gimbal angle data, and sends the robot head control commands to the robot through the communication link. The gimbal control module, deployed on the robot, is used to control the six-degree-of-freedom gimbal to execute the actions corresponding to the head control commands according to the robot head control commands.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the computer device is a robot end, the following steps are performed: Collect robot perception data and send the robot perception data to the cockpit end via a communication link; the robot perception data includes environmental visual data, gimbal angle data of the six-degree-of-freedom gimbal, and state and force data of the robot arm; the communication link includes, but is not limited to, time-sensitive network wired links and wireless links; control the six-degree-of-freedom gimbal to execute the actions corresponding to the head control commands according to the robot head control commands sent from the cockpit end; When the computer device is a cockpit, the following steps are performed: Preprocessed robot perception data is acquired via a communication link, and a virtual reality scene is determined based on the environmental visual data using a virtual reality host. This virtual reality scene is then sent to a virtual reality headset, allowing the operator to obtain a visual experience from the robot's perspective using the headset. The cockpit controller determines the robotic arm's acceleration and displacement based on the force data, and controls the cockpit robotic arm based on these accelerations and displacements, allowing the operator to experience tactile presence from the bi-armed quadruped robot. The operator's head posture data is collected via the virtual reality headset, and the virtual reality host generates robot head control commands based on the head posture data and gimbal angle data. These commands are then sent to the robot via a communication link.

10. A robot control system, characterized in that, The robot control system includes the robot end and the cockpit end as described in claim 1.

Images