Robot remote control system and method based on augmented reality and visual feedback

Abstract

The invention discloses a robot remote control system based on augmented reality and visual feedback, and the system comprises a visual collection module of a sensing layer which obtains an environment image in real time through a built-in camera; a rocker control module and a video stream transmission module of the augmented reality control layer realize a rocker control function and a video stream transmission function; a WebSocket client module and a WebSocket server module of the communication relay layer are used for transferring and transmitting instructions and data; and the robot execution control layer comprises a chassis control module and a motion execution module, and the chassis control module is operated and managed by the master control computing platform Intel NUC and is used for completing execution of a motion instruction. The invention further discloses a robot remote control method based on augmented reality and visual feedback. The invention belongs to the technical field of augmented reality technology and humanoid robot remote control, and solves the problems that in the prior art, an operation interface of a humanoid robot is not visual, visual feedback delay is serious, and network adaptability is insufficient.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of augmented reality and remote control of humanoid robots, and relates to a robot remote control system based on augmented reality and visual feedback, and a robot remote control method based on augmented reality and visual feedback. BACKGROUND

[0002] In recent years, artificial intelligence, humanoid robot technology and human-computer interaction technology have developed rapidly, and humanoid robots have been widely used in service industry, medical care, emergency rescue, security inspection and other fields. However, the current remote humanoid robot control system still has obvious defects: 1) the operation mode mainly uses two-dimensional screen and traditional keyboard and mouse interaction, which is difficult to provide sufficient spatial immersion, and the operator cannot intuitively and efficiently perceive the real environment of the humanoid robot. 2) The traditional visual feedback system has high delay and unsmooth picture transmission, which greatly reduces the control accuracy, stability and safety. 3) The existing video stream transmission method usually uses fixed code rate, which lacks dynamic adaptability and cannot effectively balance the image quality and real-time performance in complex network environment.

[0003] The development of augmented reality (AR) technology brings new opportunities for remote control of humanoid robots. AR technology can superimpose virtual information on real scenes, improve the spatial perception ability of operators, and enhance the immersive interaction experience. However, the integration of AR in the field of remote control of humanoid robots is still insufficient, and there are problems such as high video stream delay, poor network adaptability, and slow response of control instructions.

[0004] With the development and maturity of augmented reality technology, how to effectively integrate AR technology with remote control technology of humanoid robots to realize a remote operation system with better immersion, efficiency and stability has become an important technical problem to be solved. Therefore, it is urgent to develop an integrated system with efficient visual feedback and adaptive control strategy to meet the practical needs of remote control of humanoid robots in complex environments. SUMMARY

[0005] The present application provides a robot remote control system based on augmented reality and visual feedback, which solves the problems of non-intuitive operation, serious visual feedback delay and insufficient network adaptability of humanoid robots in the prior art.

[0006] Another object of the present application is to provide a robot remote control method based on augmented reality and visual feedback, which solves the problem that the remote operation control of humanoid robots in the prior art has certain gaps in immersion, real-time performance and stability.

[0007] The technical scheme adopted by the present application is a robot remote control system based on augmented reality and visual feedback, which comprises four levels of structures, namely a visual acquisition module located in the perception layer, which uses a built-in camera carried by a humanoid robot to obtain environmental images in real time; a joystick control module and a video stream transmission module located in the augmented reality control layer, which are responsible for realizing the joystick control function and the video stream transmission function; a WebSocket client module and a WebSocket server module located in the communication relay layer, which are used for the transfer of instructions and data; and a chassis control module and a motion execution module located in the robot execution control layer, wherein the chassis control module is run and managed by a main control computing platform Intel NUC, and the motion execution module is composed of the leg mechanism of the humanoid robot and is used to complete the execution of motion instructions.

[0008] Another technical scheme adopted by the present application is a robot remote control method based on augmented reality and visual feedback, which is implemented according to the following steps: Step 1: build a system hardware platform and complete functional integration; Step 2: collection and processing of user operation and spatial instructions; Step 3: standardize operation data and complete network encapsulation; Step 4: use WebSocket link for data transmission and timely abnormal recovery; Step 5: the main control computing platform Intel NUC implements instruction analysis, and the humanoid robot completes the bottom layer execution; Step 6: continuously and stably transmit visual acquisition information and publish ROS data stream; Step 7: adopt a dynamic transcoding rate mechanism to generate adaptive video stream; Step 8: multi-terminal visual feedback and full-link fault-tolerant closed-loop control.

[0009] The present application has the beneficial effects of improving the fusion and innovation of artificial intelligence, humanoid robot technology and human-computer interaction technology, including the following aspects: 1) greatly improving the spatial control intuitiveness, enhancing the spatial perception and immersive operation experience of users in the AR environment. 2) The visual feedback delay is significantly reduced, and the end-to-end delay is reduced by more than 30% on average, effectively improving the operation real-time performance. 3) An innovative dynamic video transcoding mechanism is adopted, so that the system can still provide stable and smooth visual feedback under complex network conditions, greatly enhancing the network adaptability. 4) Multi-terminal compatibility and fault-tolerant mechanism are realized, which significantly improves the stability, reliability and practical application value of the system. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 is the overall architecture diagram of the system of the present application; Figure 2 is the control flowchart of the method of the present application; Figure 3 is a mapping relationship diagram of handle control and humanoid robot action in the method of the application, which shows the mapping rule of XR (Extended Reality) handle direction input and humanoid robot motion instruction; Figure 4 is a multi-terminal synchronous display effect diagram of the humanoid robot perspective video stream in the method of the application, which shows the ability of the real-time video picture received by the humanoid robot terminal and the PC terminal at the same time under the VR perspective. DETAILED DESCRIPTION

[0011] The application will be described in detail below in combination with the drawings and specific embodiments.

[0012] Reference Figure 1 , the application is based on a robot remote control system based on augmented reality and visual feedback, which includes four levels of hardware structure, namely a visual acquisition module located in the perception layer, which uses the built-in camera carried by the humanoid robot Roban to obtain real-time environment images; a parallel working joystick control module and a video stream transmission module located in the augmented reality control layer, which respectively use the Unity 3D platform and the XR plug-in system (OpenXR, XR Interaction Toolkit, etc.), and are respectively responsible for realizing the joystick control function and the video stream transmission function; an interconnected WebSocket client module and a WebSocket server module located in the communication relay layer, the main control computing platform Intel NUC is responsible for implementing the function of the WebSocket server module, the PC terminal is responsible for implementing the function of the WebSocket client module, and both are deployed in the same network environment as the Meta Quest 3 and the Roban, and are used for the transfer of instructions and data; a parallel working chassis control module and a motion execution module located in the robot execution control layer, the running platform of the chassis control module is ROS Noetic, the chassis control module is run and managed by the built-in main control computing platform Intel NUC of the Roban, and the motion execution module is composed of the leg mechanism of the Roban body, and is used for completing the execution of the motion instruction.

[0013] The built-in camera on the humanoid robot body of the visual acquisition module located in the perception layer supports automatic exposure and focusing, and is suitable for complex lighting environments.

[0014] The terminal device selected by the enhanced reality control layer responsible for the joystick control module and the video stream transmission module is a head-mounted device, has remote sensing function, and is integrated with Unity3D and XR interaction function. The XR interaction is a spatial interaction based on the Meta Quest 3 enhanced reality device and a handle operation collection function, is realized by an XR plug-in system of Unity 3D, and is used for capturing the head-mounted device position, handle action, button input and the like of a user.

[0015] On the basis of the above-mentioned visual acquisition module and the video stream transmission module, the application innovatively proposes a dynamic transcoding rate mechanism (for a specific description of the mechanism, see step 7). The running process of the dynamic transcoding rate mechanism is preset in an Intel NUC main control computing platform of the humanoid robot, the platform serves as a core processing unit of the system and undertakes the task of visual data analysis and encoding scheduling. The dynamic transcoding rate mechanism is based on an inter-frame difference and edge detection algorithm, combines real-time bandwidth detection, and automatically adjusts the video encoding code rate, frame rate and GOP interval.

[0016] The application is implemented according to the following steps. Step 1: building a system hardware platform and completing function integration, The system hardware mainly includes: a head-mounted Meta Quest 3 responsible for a joystick control module and a video stream transmission module in an enhanced reality control layer, a humanoid robot Roban responsible for a chassis control module and a motion execution module in a robot execution control layer, a main control computing platform Intel NUC responsible for a WebSocket server module in a communication relay layer and a PC end responsible for a WebSocket client module, and a Roban built-in camera responsible for a visual acquisition module in a perception layer.

[0017] The enhanced reality head-mounted Meta Quest 3 is mainly used for collecting spatial interaction of an operator and remote rod instruction input; the humanoid robot Roban as an execution terminal can respond to control instructions to complete multi-degree-of-freedom motion and operation; the main control computing platform Intel NUC runs an Ubuntu operating system and a ROS (Robot Operating System) framework and is used for instruction analysis, data processing and communication management; the PC end as a WebSocket client realizes network bridging of front-end control and back-end execution; the Roban built-in camera is used for real-time acquisition of a first perspective picture of a work site and provides a data source for visual feedback.

[0018] The functions and configurations of the main hardware in the system are shown in Table 1.

[0019] Table 1, main hardware settings in the system

[0020] The above hardware devices realize high-speed interconnection through wired or wireless mode, work cooperatively, and constitute the physical basis of the remote control system of the application. Through reasonable construction and efficient integration of the above hardware platform, reliable hardware support is provided for subsequent space interaction, remote control, visual acquisition and multi-terminal visual feedback function realization.

[0021] Referring to Figure 2 It is the complete process of the method of the application from user input, instruction transmission, humanoid robot action execution to visual feedback, which is described in detail below.

[0022] Step 2: Collection and processing of user operation and spatial instructions, This step 3 realizes high-precision real-time perception and processing of the operator's space interaction action, and provides basic guarantee for accurate generation of subsequent remote control instructions.

[0023] Firstly, the user connects the augmented reality head-mounted Meta Quest 3 (including a handle and an XR head-mounted device) to the PC end responsible for implementing the WebSocket service module in the same network through a USB wired connection mode (or Link mode) to realize data streaming; then, the device pairing and connection are completed by using the Meta Quest Link software, so that the XR head-mounted device and the handle action data can be stably and high-speed transmitted to the PC end; In terms of software configuration, the system uses the Unity 2022.3 LTS development platform, enables the OpenXR plug-in through the XR PluginManagement module, and checks the Oculus Touch ControllerProfile in the OpenXR settings to realize automatic identification and signal mapping of the controller in the augmented reality head-mounted Meta Quest 3; subsequently, the XR Interaction Toolkit toolkit is introduced, and the XR Origin structure is built in the Unity scene, including a left-hand controller sub-object, a right-hand controller sub-object and a head controller sub-object. Under the left-hand controller sub-object, add the XR Controller (Action-based) component and bind it with the input action asset; by creating an Action named "moveVector" in the input action asset (type Value, control type Vector2), and binding it to the Primary 2D Axis of the XR controller, real-time collection of left-hand joystick two-dimensional input is achieved, which can be used to generate forward, backward, left turn, right turn, and other operation instructions; the project associates this Action with business logic through scripts, which can continuously obtain and determine the vector data of left-hand joystick input during running, and can output to the Unity console in real time for debugging and verification. The right-hand controller sub-object is similar to the left-hand controller sub-object, and the XRController (Action-based) component is also added to the right-hand controller sub-object and bound to the corresponding input action asset (Input Action Asset); the right-hand joystick can be used to control the direction of the humanoid robot or other interactive actions, and the specific operations include binding the Primary 2D Axis of the right-hand controller, real-time collection of two-dimensional input data of the right-hand joystick, and execution of forward, backward, left turn, right turn, and other control instructions.

[0024] The head sub-object is responsible for tracking the user's head pose and line-of-sight direction, providing spatial perception support for augmented reality interaction; the head sub-object is bound to the left-hand controller sub-object and the right-hand controller sub-object through the XR plugin, so that the user's head movement and controller action can be synchronized and reflected in the virtual environment in real time, improving the immersion and accuracy of the interaction.

[0025] At the same time, the Unity application in the augmented reality headset Meta Quest 3 also continuously monitors and collects multi-modal input information such as user spatial pose, line-of-sight direction, and function keys; All raw input data is converted into spatial coordinates, rotation angles, two-dimensional vectors, and discrete key signals through the bottom interface of the XR plugin system in real time; to ensure operation accuracy, the system performs debouncing filtering, rate normalization, and dead zone processing on the input signal to avoid false touch and small jitter affecting subsequent humanoid robot control; all standardized operation intentions and input data are integrated by custom event processing scripts (such as JoystickReader, ActionListener, etc.), unified into internal message structures, and stored in the local command queue, ready for the next step of data standardization, network encapsulation, and sending.

[0026] Step 3: Standardize the operation data and complete network encapsulation, The pre-processed and normalized spatial input data will be packaged into JSON format control instruction data packets in real time at the Unity front end; each control instruction data packet contains instruction type (such as movement, steering, function activation, etc.), direction vector, speed amplitude, user identification, timestamp and other information, ensuring that the instruction can be uniquely and accurately identified in the remote system; To improve the robustness of network transmission, the system will digitally sign and sequentially number the control instruction data packets to support backend verification and prevent instruction disorder; the packaged JSON control instructions are handed over to the WebSocket client module for subsequent network transmission.

[0027] Step 4: Use WebSocket link for data transmission and timely exception recovery, The WebSocket client integrated on the Unity side (based on websocket-sharp.dll and other libraries) will automatically connect to the specified IP and port of the Intel NUC main control computing platform of the humanoid robot after system startup, realizing low-latency bidirectional transmission of control instructions; whenever there is new joystick input or operation action information, the WebSocket client will immediately push the packaged control instruction data packet (JSON format) to the WebSocket server; during communication, the WebSocket link has mechanisms such as automatic reconnection, link heartbeat detection, timeout disconnection, and data packet retransmission; when network fluctuations, disconnections or packet loss are detected, the system will automatically trigger a reconnection and prompt the user of the current communication status through the UI interface; To further reduce latency and bandwidth consumption, the WebSocket channel uses a throttling strategy, actively pushing data only when input changes, and automatically sleeping when idle, maximizing system real-time performance and network adaptability.

[0028] Step 5: The main control computing platform Intel NUC implements instruction analysis, and the humanoid robot completes the bottom layer execution, This step innovatively integrates functions such as speed threshold limit, dynamic acceleration protection, minimum action threshold, instruction timeout automatic stop, and configurable human-computer interaction mapping table, together forming a multi-layer safety and redundancy protection mechanism (specific details in step 8), ensuring the safety, stability and adaptability of humanoid robot motion control, thereby significantly improving the application ability of the remote control system in industrial environments.

[0029] The ROS (Robot Operating System) master node runs on the master control platform Intel NUC, responsible for real-time receiving and analyzing JSON format control instructions transmitted through the WebSocket link; the server node "rosbridge_server" in the system continuously listens to the WebSocket port, converts the received control instruction message into ROS standard message (Twist), and publishes it to the / cmd_vel control topic; the ROS master node calls the differential drive algorithm according to the instruction type, dynamically calculates the motor control target speed and direction of the humanoid robot left and right legs; the generated motion command is sent to the chassis motor control board and servo unit through the serial port or CAN bus, driving the humanoid robot to realize forward, backward, left and right turning and multi-degree-of-freedom upper limb action composite behavior.

[0030] As shown in Figure 3, the XR handle in each direction and the button input are all mapped to the corresponding action of the humanoid robot, for example, the left handle is used for upper limb action control (such as raising hands, squatting), and the right handle is used for marching and turning control. The mapping relationship can be flexibly defined in the system action configuration file to support different humanoid robot platforms and action requirements.

[0031] Multi-layer security and redundancy protection mechanism is continuously effective in the whole process of instruction analysis and execution: speed threshold and acceleration limit prevent sudden impact; minimum action threshold avoids false touch and slight jitter; timeout automatic parking prevents out of control caused by link exception; the configurability of human-computer interaction mapping table ensures that the control strategy can be quickly adjusted according to the task scene. These measures and steps 3, step 4 generated and transmitted control instructions cooperate with each other to form a stable and reliable remote operation closed loop.

[0032] Step 6: continuous and stable transmission of visual acquisition information, publishing ROS data stream, While the motion instruction resolution and issuance are completed in the main control platform Intel NUC of the humanoid robot, the vision acquisition module cooperates to continuously provide the remote operator with on-site visual feedback. The built-in camera carried by the humanoid robot Roban has automatic exposure and automatic focusing functions, and can stably collect 30 frames per second of high-quality images under various lighting environments; the built-in camera is connected to the main control platform Intel NUC of the humanoid robot through a UVC interface, and is managed by a / dev / video0 device node under a Linux system; the vision acquisition module calls a ROS usb_cam function package, and after the node is started, the environment picture is continuously collected according to the set parameters (such as 640*480 resolution, 30fps frame rate, MJPEG format), and the collected raw image stream is encapsulated into a ROS standard sensor_msgs / Image message, which contains image resolution, encoding format, pixel data, timestamp and frame number, and is published to a unified image topic (such as / usb_cam / image_raw).

[0033] In order to ensure stable transmission of collected data in a complex network environment, the vision acquisition module is pre-configured with a local cache and a frame rate control mechanism. The local cache is used to store image frames at the collection end for a short time, to maintain the continuity and integrity of frame data when the network is instantaneously dithered, delayed or blocked, and to avoid picture interruption or tearing caused by data loss. The frame rate control mechanism limits the output frame rate to a preset value (such as 30fps), to prevent the processing end and the transmission link from being overloaded due to burst frames in high-performance mode, thereby causing delay and lag.

[0034] The local cache and the frame rate control mechanism ensure that the image data source entering the subsequent dynamic transcoding rate mechanism and the adaptive video stream generation link has the characteristics of time sequence integrity and constant rate, providing a stable basis for video stream transcoding, distribution and remote interaction.

[0035] Step 7: using a dynamic transcoding rate mechanism to generate an adaptive video stream, A web_video_server node (i.e. a video stream forwarding function node in ROS) based on ROS Noetic is deployed in the main control platform Intel NUC of the humanoid robot, for receiving the standard image message published by the vision acquisition module and performing real-time video encoding and distribution.

[0036] The application innovatively introduces a dynamic transcoding rate mechanism, which dynamically adjusts video encoding parameters based on the dual feedback of image content complexity analysis and network state detection, to realize adaptive video stream generation and efficient network transmission, thereby ensuring continuous, clear and low-delay visual feedback in a remote operation scene. The specific process is as follows: 7.1) Before each frame image is encoded, the picture complexity is judged by inter-frame difference, edge detection and other algorithms, and it is automatically identified whether the current scene is in a static, low change or high dynamic state. At the same time, the network detection function integrated in the video stream transmission module (based on WebRTC protocol) collects the network running state in real time, including uplink bandwidth, packet loss rate, jitter and end-to-end delay, etc. Parameters provide reference for subsequent dynamic adjustment; 7.2) In the Intel NUC main control platform, the video encoder (such as MJPEG or H.264) dynamically adjusts the code rate, frame rate and GOP interval (the frame number interval between key frames in video encoding) according to the combined results of content complexity and network state; for example, when the picture is static or the bandwidth is limited, the system automatically reduces the code rate from 3500kbps to 1500kbps and increases the GOP interval to reduce bandwidth occupation and prevent stuttering; when the scene dynamic increases or the bandwidth is sufficient, the system automatically restores high code rate and high frame rate to realize high-quality video backhaul; 7.3) The video stream after dynamic transcoding adopts MJPEG or H.264 encoding format, and is published through the web_video_server node with HTTP streaming interface, supporting concurrent access of public network and internal network multi-terminal (such as Unity terminal, Web terminal, mobile terminal).

[0037] From this, through the organic combination of step 6 local buffer and frame rate control mechanism (Local Buffer&Frame Rate Control) and step 7 dynamic transcoding rate mechanism, the whole link stability optimization from the image acquisition source to the video distribution end is realized, which significantly reduces the delay and packet loss rate in the remote operation scene, and guarantees the continuous, clear and low delay visual feedback experience of multi-terminal.

[0038] Step 8: Multi-terminal visual feedback and full-link fault-tolerant closed-loop control, The video stream after dynamic transcoding is pushed to the remote client through the HTTP port, supporting concurrent access of multiple platforms including Unity terminal, Web terminal, mobile terminal, etc.; in the Unity terminal, the first-person view picture of the humanoid robot is directly embedded into the augmented reality operation interface through RawImage control or custom rendering module, so that the operator can obtain synchronous immersive visual feedback in the XR scene; in the Web terminal and mobile terminal, the video stream address can be directly loaded through the browser to realize remote monitoring, team collaboration and multi-scene application.

[0039] To ensure the stability and high availability of the system in complex network environment, multiple layers of security and redundancy protection mechanisms are pre-installed in the video stream transmission module of the augmented reality control layer and the WebSocket client module of the communication relay layer, which are described as follows: 1) Automatic disconnection detection and reconnection: Both the WebSocket client module and the video stream transmission module have link state monitoring functions. Once the network is disconnected, the reconnection logic will be triggered immediately, and the data transmission will continue automatically after the connection is restored, avoiding interruptions in remote control; 2) Link log monitoring: The system integrates log collection functions in the above two modules, which can record the link running status in real time, including connection duration, disconnection events, packet loss rate, retransmission conditions, and other key parameters, and form link log files for operators or system managers for post-diagnosis and maintenance; 3) End-to-end delay statistics: By embedding timestamps in data packets, the full link delay from image acquisition to terminal display is calculated to ensure that the visual feedback and control instructions received by the operator are within the delay range, supporting dynamic tuning and performance optimization of the system.

[0040] In summary, the system and method of the present application ensure that even in complex situations such as network fluctuations, signal interruptions, or bandwidth mutations, the link can still be automatically detected and quickly restored, restoring the continuous transmission of control instructions and video streams within seconds, thereby minimizing operational interruptions and ensuring the continuity and stability of the remote control closed loop, and achieving real-time interaction and efficient fault-tolerant closed loop between multiple terminals.

[0041] Example 1 Unity3D platform and Meta Quest 3 controller interaction implementation (user operation and spatial instruction collection).

[0042] Experimental object or use condition: Meta Quest 3 is selected, and the development environment is Unity 2022.3 LTS. The experimental site is the laboratory host, which is connected to the PC end through USB wired (Link mode) to realize high-frame-rate, low-latency streaming communication. The Beta version of Meta Quest Link software is used for device pairing and function configuration.

[0043] Configuration and operation mode adopted: First, install and configure the Meta Quest Link software to realize wired connection between Meta Quest 3 and the PC end, ensuring stable transmission of controller action data. Then, configure the XR plugin and XR Origin structure in Unity to automatically recognize the Quest 3 controller and collect two-dimensional input from the left-hand remote rod in real time; the system continuously reads input data through scripts and outputs the results to the Unity console for debugging. Finally, these inputs are converted to JSON format and sent to the ROS system through WebSocket for further analysis and execution.

[0044] Parameter settings or component combination: Unity version: 2022.3 LTS, XR support: OpenXR plugin + Oculus Touch Controller Profile, Controller input: Primary 2D Axis (left-hand remote stick), input type Vector2, Communication method: USB wired connection (Link mode), Input action response: Unity Input System event mechanism.

[0045] Usage time: The system runs continuously for 2 hours for input testing and remote control experience.

[0046] Effects and conclusions: This embodiment 1 realizes high-precision input reading and interaction of the handle controller in Meta Quest 3 on the Unity 3D platform. The left-hand remote stick input can be converted into a two-dimensional vector in real time and accurately, and the user's operation direction can be successfully distinguished through script logic. The overall interaction of the system is smooth and low in delay, and the control instructions can be successfully transmitted to the humanoid robot backend through WebSocket communication, supporting real-time remote control of the humanoid robot with multiple degrees of freedom.

[0047] Embodiment 2 Realize remote control data transmission of WebSocket client module.

[0048] Experimental object or use condition: The Meta Quest 3 and Unity 2022.3 LTS platform are used as the front-end control environment, and the Intel NUC host computer running the ROS Noetic system is used as the back-end. The experimental network is a campus Wi-Fi with a bandwidth of 4~6Mbps, and the network environment includes occasional fluctuations and signal shielding.

[0049] Adopted structure configuration and running mode: In the Unity environment of the system front end, the WebSocket client module based on websocket-sharp.dll is integrated to realize low-delay and stable transmission of Quest 3 XR handle input data. The user operates the XR remote stick, and the input data is collected and formatted by the Unity XR system, then packaged into JSON format instructions by the WebSocket client module, and sent to the WebSocket server node of the back-end host computing platform Intel NUC.

[0050] The WebSocket client module and the WebSocket server module in the system undertake the following core functions: Establish and maintain real-time long connection between control terminal and execution terminal to ensure network channel availability. Structurally encapsulate operation data such as remote lever input and convert it into control messages that can be parsed by the backend. Monitor connection status, handle abnormal disconnection and automatic reconnection to ensure link closed loop.

[0051] Parameter settings or component combination methods: Communication protocol: WebSocket (long connection mode), Client library: websocket-sharp.dll (C#), Server address example: ws: / / 192.168.1.100:9090, Control message format (JSON example): { "command": "move", "direction": "forward", "speed": 0.5}, Where "command" is the instruction type, "direction" is the direction, and "speed" is the speed amplitude (0-1).

[0052] Message sending strategy: Only send new instructions when there is a significant change in the direction or amplitude of the remote lever, to avoid frequent invalid transmission.

[0053] Connection management measures: automatic reconnection, timeout detection (prompt if not connected for 10 seconds), and log output status feedback.

[0054] Usage time: continuous operation for 3 hours, including multiple manual network disconnection, reconnection and high-frequency operation simulation.

[0055] Effect and conclusion: The WebSocket client module can stably establish a long connection with the server, and the front-end remote lever control data is transmitted to the main control computing platform Intel NUC in real time and reliably. In the case of high-frequency operation on the control terminal or short-term network fluctuations, the system can automatically reconnect, and users will only receive a status prompt in the case of extreme network disconnection. Through input change detection and transmission throttling, the communication delay is controlled within 100ms on average, and there is no control lag caused by instruction loss or redundancy.

[0056] Example 3 Main control computing platform instruction parsing and underlying motion execution test.

[0057] Experimental object or usage conditions: Roban medium humanoid robot is selected, the main control computing platform is Intel NUC carrying ROS Noetic system, the control end sends motion instructions through Unity3D front end and handle in Meta Quest 3, local area network environment, network delay is less than 50 ms.

[0058] The structural configuration and operation mode adopted are as follows: The embodiment focuses on the instruction analysis and bottom layer execution link of the main control computing platform of the humanoid robot. The system sends the remote lever input from the Unity end through the WebSocket, the main control platform Intel NUC is used as the chassis control module, receives the standard JSON control instruction and analyzes it into the geometry_msgs / Twist type message, and then pushes it to the / cmd_vel control topic through the ROS message publishing mechanism.

[0059] The chassis control module is responsible for encapsulating the instruction specification and publishing the Twist message to the chassis interface of the humanoid robot at a fixed frequency, realizing the real-time issuance and control of the movement command. The motion execution module listens to the / cmd_vel topic, adjusts the PWM or speed of the servo motor of the left and right leg mechanism according to the Twist message content, and drives the humanoid robot to complete physical actions such as forward movement, backward movement, left turn and right turn.

[0060] Parameter setting or component combination method: ROS message type: geometry_msgs / Twist, only linear.x and angular.z components are used.

[0061] The instruction analysis rules are shown in the following table 2.

[0062] Table 2, instruction analysis rules

[0063] Frequency limit: the maximum frequency of control message publishing is set to 20 Hz; if there is no new instruction within 3 seconds, a stop command is automatically sent.

[0064] Safety threshold: the minimum speed threshold is set to 0.05 m / s, and the overspeed protection clipping is 0.6 m / s at most.

[0065] Use time: continuous operation for 1 hour, covering complex operations such as multiple rounds of forward movement, turning, stopping, etc.

[0066] Effect and conclusion: The actual measurement shows that the chassis control module can stably parse the control instructions received by the WebSocket end and adapt them as ROS executable messages. The motion execution module has a response delay of 100-200 ms. The humanoid robot runs smoothly and accurately turns. All safety redundancy strategies (such as timeout stop, speed filtering, and overrun protection) can effectively play a role. In multiple simulation network fluctuation and abnormal instruction scenarios, the humanoid robot does not have problems such as misoperation and loss of control.

[0067] It can be seen that embodiment 3 verifies the high real-time performance and high reliability of the instruction analysis of the master control platform and the underlying motion execution, providing a solid technical support for remote space operation.

[0068] Embodiment 4 Dynamic transcoding rate mechanism and visual feedback test under multi-network environment.

[0069] Experimental object or use condition: In the laboratory environment, the Roban humanoid robot, Meta Quest 3, and PC are connected online, and the data stream and XR space interaction between PC and Meta Quest 3 are realized through USB Link. The test network condition is a controllable wireless network environment with a bandwidth of 2-6 Mbps. The test covers various application scenarios such as remote teaching demonstration, dangerous environment monitoring, and intelligent security.

[0070] The structure configuration and operation mode adopted are as follows: The method of the present application adopts XR handle to remotely control the multi-degree-of-freedom motion of the humanoid robot. The instructions are transmitted in real time to the humanoid robot body through the WebSocket channel. The master control platform is equipped with a dynamic transcoding rate mechanism. The system simultaneously returns the first perspective video stream collected by the built-in camera of the humanoid robot, and dynamically adjusts the video encoding parameters according to the network environment.

[0071] During the test, VNC and SSH tools are used to remotely maintain and configure the master control platform Intel NUC and the humanoid robot body, ensuring the efficient operation of the system.

[0072] Parameter setting or component combination method: During the test, the video encoding adopts a dynamic transcoding rate mechanism, the code rate adjustment interval is 1500-3500 kbps, the frame rate is automatically adjusted to 15-30 fps, and the GOP interval is adaptively adjusted according to the network condition; the comparison group adopts a fixed code rate of 3500 kbps, and other parameters remain the same.

[0073] Time of use: The system runs continuously for 3 hours under the conditions of 2 Mbps, 4 Mbps, and 6 Mbps bandwidth, respectively, to complete the remote operation and visual feedback experiment.

[0074] Effects and conclusions: The dynamic transcoding rate mechanism can automatically optimize parameters according to network conditions to realize smooth video streaming. In a bandwidth environment of 2-6 Mbps, the average video delay is controlled within 150 ms, which is significantly lower than the fixed code rate scheme. The specific test data is shown in Table 3.

[0075] Table 3, comparison of experimental results

[0076] From the results in Table 3, it can be seen that the dynamic transcoding rate mechanism effectively reduces the delay and risk of freezing in different network and operation scenarios, while the image quality changes very little. In actual use, the user feedback system responds quickly, the picture is smooth, and the remote operation immersion is strong. Conclusion: The method of the present application can realize efficient remote operation and stable visual feedback in various network environments, and the performance is significantly better than that of the traditional fixed code rate scheme, and has good practical application prospect.

[0077] Example 5 Multi-terminal visual synchronization of remote operation scenarios.

[0078] Experimental object or use condition: Roban humanoid robots and Meta Quest 3 augmented reality headsets were selected for remote operation demonstration in a laboratory environment. The network environment is a normal 4G hotspot with a bandwidth of 2 Mbps.

[0079] Configuration and operation mode adopted: According to the method of the present application, the operator remotely issues control instructions through the XR headset, and the humanoid robot receives and executes the actions. The built-in camera on the head of the humanoid robot captures the first perspective picture, which is encoded by the dynamic transcoding rate mechanism of the main control computing platform, and then the video stream is synchronously returned through the web_video_server node in real time, and multiple terminals (such as PC terminal, Web terminal, etc.) are supported for synchronous display.

[0080] As shown in Figure 4 , the system can realize simultaneous display of the video stream under the perspective of the humanoid robot body in different clients (Roban terminal, PC terminal and VR terminal), ensuring that the operator and other observers can obtain visual information in real time.

[0081] Parameter setting or component combination: Video encoding code rate dynamic adjustment range 1000-2200kbps, frame rate automatic adjustment 15-25fps, GOP interval maximum setting 100 frames.

[0082] Use time: continuous operation for 2 hours.

[0083] Effect and conclusion: Under low bandwidth conditions, the system can automatically adjust the code rate and frame rate according to the network state, realize the synchronous display of video stream among multiple terminals, and the picture does not appear to be interrupted, and the delay is controlled at about 230ms. Whether in the Roban client or the PC web page, the real-time first-person view picture of the humanoid robot can be clearly and smoothly received, and the operator's remote operation experience is natural and the visual information is complete.

[0084] It can be seen that the structure and method of the application can realize multi-terminal video stream synchronization of remote operation under limited bandwidth, guarantee the stability and efficiency of operation, and are suitable for various remote control and team collaboration scenes.

[0085] Example 6 Standard remote teaching scene.

[0086] Experimental object or use condition: In the intelligent humanoid robot laboratory of a certain university, a Roban medium-sized humanoid robot is selected, equipped with Meta Quest 3 augmented reality head-mounted display and Intel NUC host computer, the environment is 5G Wi-Fi, and the bandwidth is 4Mbps.

[0087] The adopted structure configuration and operation mode are as follows: The Meta Quest 3 is used as the host end, the operator issues spatial remote lever instructions through the XR handle, the video acquisition module uses the built-in camera of the Roban humanoid robot, the host computing platform runs the ROS Noetic system and web_video_server, and the remote teaching software is embedded in the Unity end.

[0088] Parameter setting or component combination: the initial setting of the video encoding code rate is 3500kbps, the frame rate is 30fps, and the GOP interval is 60 frames. After dynamic transcoding is enabled, the code rate range is 1500~3500kbps, and the minimum frame rate is 15fps.

[0089] Use time: continuous operation for 4 hours.

[0090] Effect and conclusion: In the process of teacher-student interaction and remote demonstration, the system automatically adjusts the video code rate and frame rate according to the picture dynamics and network conditions, the vision is smooth, the delay is stable within 180ms, the picture quality does not decrease obviously under still picture, and the remote interaction does not stutter. Conclusion: The method of the application is completely suitable for the scene of remote teaching in colleges and universities, and guarantees high immersion and stable visual feedback.

Claims

1. A robot remote control system based on augmented reality and visual feedback, characterized by: It includes a four-level structure, namely the visual acquisition module located in the perception layer, which uses the built-in camera of the humanoid robot to obtain environmental images in real time; the joystick control module and video stream transmission module located in the augmented reality control layer, which are responsible for realizing the joystick control function and video stream transmission function; the WebSocket client module and WebSocket server module located in the communication relay layer, which are used for the transfer and transmission of instructions and data; the chassis control module and motion execution module located in the robot execution control layer. The chassis control module is run and managed by the main control computing platform Intel NUC, and the motion execution module is composed of the leg mechanism of the humanoid robot and is used to complete the execution of motion instructions.

2. The robot remote control system based on augmented reality and visual feedback according to claim 1, characterized in that: The visual acquisition module is implemented through the built-in camera of Roban, the joystick control module and the video stream transmission module are implemented through Meta Quest 3, the operating platform of the chassis control module is ROS Noetic, and the main control computing platform Intel NUC runs the Ubuntu operating system and ROS framework.

3. A robot remote control method based on augmented reality and visual feedback, characterized in that: Follow these steps to implement: Step 1: Build the system hardware platform and complete function integration; Step 2: Collection and processing of user operations and spatial instructions; Step 3: Standardize the operation data and complete the network encapsulation; Step 4: Use the WebSocket link for data transmission and timely recovery from abnormalities; Step 5: The main control computing platform Intel NUC implements instruction parsing, and the humanoid robot completes the underlying execution; Step 6: Continuously and stably transmit visual acquisition information and publish ROS data streams; Step 7: Use dynamic transcoding rate mechanism to generate adaptive video stream; Step 8: Multi-terminal visual feedback and full-link fault-tolerant closed-loop control.

4. The robot remote control method based on augmented reality and visual feedback according to claim 1, characterized in that: In step 2, the specific process is: First, the user connects the Meta Quest 3 augmented reality headset to a PC running the WebSocket server module on the same network via a USB cable to stream data. Then, the Meta Quest Link software is used to pair and connect the devices, ensuring stable and high-speed transmission of XR headset and controller motion data to the PC. In terms of software configuration, the system uses the Unity 2022.3 LTS development platform, enables the OpenXR plug-in through the XR Plugin Management module, and checks the Oculus Touch Controller Profile in the OpenXR settings to achieve automatic recognition and signal mapping of the controllers in the augmented reality headset Meta Quest 3; then, the XR Interaction Toolkit toolkit is introduced to build the XR Origin structure in the Unity scene, including the left-hand controller sub-object, the right-hand controller sub-object, and the head controller sub-object.

5. The robot remote control method based on augmented reality and visual feedback according to claim 4, characterized in that: The left-hand controller sub-object adds an XR Controller component and binds it to the input action configuration file. By creating an Action named "moveVector" in the input action configuration file and binding it to the XR controller's Primary 2D Axis, real-time acquisition of the left-hand joystick's two-dimensional input is achieved, used to generate forward, backward, left turn, and right turn commands. The project uses a script to associate this Action with the business logic, continuously acquiring and identifying the vector data input by the left-hand joystick during runtime, and outputting it to the Unity console in real time for debugging and verification. Add the XR Controller component to the right controller child object and bind it to the corresponding input action profile; The right joystick is used to control the direction of movement or other interactive actions of the humanoid robot. The two-dimensional input data of the right joystick is collected in real time to execute forward, backward, left turn, and right turn control commands. The head sub-object is responsible for tracking the user's head posture and gaze direction, providing spatial perception support for augmented reality interaction.

6. The robot remote control method based on augmented reality and visual feedback according to claim 1, characterized in that: In step 3, the specific process is: After pre-processing and standardization, the spatial input data will be packaged into JSON-formatted control command packets in real time on the Unity front-end. Each control command packet contains multiple pieces of information such as command type, direction vector, velocity amplitude, user ID, and timestamp. The control instruction data packets are digitally signed and sequentially numbered to support back-end verification and prevent instruction confusion; the packaged JSON control instructions are handed over to the WebSocket client module for subsequent network transmission.

7. The robot remote control method based on augmented reality and visual feedback according to claim 1, characterized in that: In step 4, the specific process is: The WebSocket client integrated into Unity automatically connects to the designated IP and port of the humanoid robot's main control computing platform, the Intel NUC, after system startup, enabling low-latency, two-way transmission of control commands. Whenever there's new joystick input or an action, the WebSocket client immediately pushes the encapsulated control command data packet to the server. During the communication process, the WebSocket link has automatic reconnection, link heartbeat detection, timeout disconnection, and data packet retransmission. When network fluctuations, disconnection, or packet loss are detected, the system automatically triggers reconnection and notifies the user of the current communication status through the UI. The WebSocket channel adopts a throttling strategy, actively pushing data only when the input changes, and automatically sleeping when idle.

8. The robot remote control method based on augmented reality and visual feedback according to claim 1, characterized in that: In step 5, the specific process is: Multi-layer safety and redundancy protection mechanisms remain in effect throughout the entire command parsing and execution process: speed thresholds and acceleration limits prevent sudden impacts; minimum action thresholds avoid false touches and minor jitters; automatic parking upon timeout prevents loss of control caused by link anomalies; and the configurability of the human-computer interaction mapping table ensures that the control strategy can be quickly adjusted according to the mission scenario.

9. The robot remote control method based on augmented reality and visual feedback according to claim 1, characterized in that: In step 6, the specific process is: A local cache and frame rate control mechanism is adopted. The local cache is used for short-term storage of image frames on the acquisition end, maintaining the continuity and integrity of frame data when the network is jittering, delayed or blocked, and avoiding screen interruption or tearing caused by data loss. The frame rate control mechanism limits the output frame rate to a preset value to prevent delays and freezes caused by excessive load on the processing end and transmission link due to frame explosion in high-performance mode.

10. The robot remote control method based on augmented reality and visual feedback according to claim 1, characterized in that: In step 7, the specific process is: A dynamic transcoding rate mechanism is introduced. This mechanism dynamically adjusts video encoding parameters based on dual feedback from image content complexity analysis and network status detection. The specific process is as follows: 7.1) Before encoding each frame, the system first determines the image complexity and automatically identifies whether the scene is static, low-change, or highly dynamic. Furthermore, the network detection function integrated into the video streaming module collects network status in real time, providing a reference for subsequent dynamic adjustments. 7.2) In the Intel NUC, the video encoder dynamically adjusts the bitrate, frame rate, and GOP interval based on a combination of content complexity and network conditions. When the scene changes dynamically or bandwidth becomes available, the system automatically restores high bitrate and frame rate to achieve high-quality video transmission. 7.3) The dynamically transcoded video stream is published via the HTTP streaming interface through the web_video_server node, supporting concurrent access by multiple terminals on the public network and the intranet.

Images