Robot remote operation control method and system based on VR head-mounted display, and medium

Abstract

The application discloses a robot remote operation control method and system based on a VR head-mounted display, and a medium, and relates to the technical fields of rehabilitation aids and robots.The method comprises the following steps: collecting the hand, fingertip and head pose in real time through the VR head-mounted display; judging whether the VR control mode is activated according to the average distance of the fingertips of the five fingers; after the VR control mode is activated, mapping the hand pose into double-arm movement and mapping the head pose into wheelchair movement; synchronously collecting human demonstration intention data and machine execution state data to form a training data set.The application adopts a double-arm cooperative design to meet the demand of double-hand operation, realizes high-precision real-time mapping control of hand movements through the VR head-mounted display, uses the five fingers to be stretched out to enable logic to guarantee operation safety, and synchronously collects the demonstration and execution data to provide a high-quality data basis for the general intelligent development of wheelchair mechanical arms.

Description

Technical Field

[0001] This invention relates to the field of rehabilitation assistive devices and robotics, and particularly to a method, system, and medium for remote operation and control of robots based on VR headsets. Background Technology

[0002] Currently, some products have combined wheelchairs with single-arm robotic arms, and the related control methods have also diversified. The wheelchair robotic arm control schemes disclosed in existing technologies mainly include the following types:

[0003] The robotic arm is controlled to move to the target position joint by joint or degree of freedom by using joysticks or buttons to perform grasping operations. This method offers high control precision but is inefficient, often requiring multiple complex steps to complete a simple grasping action, making it difficult to meet the real-time requirements of everyday use cases.

[0004] By collecting electroencephalogram (EEG), electromyogram (EMG), or eye-tracking signals, and using pattern recognition algorithms to interpret the user's intentions, the robotic arm can be controlled to move. While this type of solution possesses certain intelligent features, its accuracy is generally low due to limitations in the signal-to-noise ratio of biosignals and individual differences. In practical applications, it is highly prone to misoperation, posing significant safety risks to users.

[0005] This approach involves capturing environmental images via a camera, using object detection algorithms to identify the object to be manipulated, and automatically planning the robotic arm's trajectory. However, this method is highly dependent on algorithms, and its success rate drops significantly in scenarios with changing lighting, complex backgrounds, or object occlusion. Furthermore, it lacks the flexibility of human-computer interaction and struggles to adapt to diverse operational needs.

[0006] Furthermore, most existing wheelchair robotic arm solutions are single-arm structures, capable of only simple pushing, pulling, and grasping movements. However, many everyday tasks, such as opening bottles, wringing towels, carrying large objects, and assembling with both hands, require the coordination of both hands. The applicability of single-arm systems is extremely limited and cannot meet the actual needs of users.

[0007] A comprehensive analysis of existing technologies reveals the following main technical problems:

[0008] Most existing wheelchair robotic arm systems are single-arm designs, which cannot complete complex daily life tasks that require the coordination of both hands, resulting in narrow application scenarios and difficulty in meeting the diverse needs of users.

[0009] While joystick control offers high precision, it is inefficient and cumbersome to operate; while biosignal and image recognition control possesses intelligent features, its accuracy is low. It is difficult to achieve both simultaneously, resulting in a poor user experience.

[0010] Control methods based on recognition algorithms are prone to misoperation due to insufficient recognition accuracy, which may lead to the robotic arm accidentally hitting people or damaging objects, posing a significant safety risk to users.

[0011] In recent years, general-purpose robot models have developed rapidly, and a large number of high-quality training datasets have emerged in the fields of bi-arm and quadruped robots, driving the intelligent upgrading of related technologies. However, the field of wheelchair robotic arms remains in the traditional control stage, lacking high-quality operational data acquisition methods, which makes it difficult to introduce general-purpose large models to achieve intelligent breakthroughs, thus limiting the upper limit of development. Summary of the Invention

[0012] To achieve the above-mentioned objectives and other advantages of the present invention, the first objective of the present invention is to provide a remote operation control method for a robot based on a VR headset, comprising the following steps:

[0013] The VR headset worn on the user's head collects the user's hand pose data, fingertip pose data, and head pose data in real time.

[0014] The relative distance between the five fingertips is calculated based on the fingertip pose data, and it is determined whether the average distance between all fingertips is less than a preset threshold. If so, the VR control mode is activated.

[0015] During the activation of the VR control mode, the hand pose data is calculated into the target pose of the dual robotic arm end effectors, and the dual robotic arms are controlled to follow the user's hand movements in real time; at the same time, the head pose data is calculated into the speed and direction of the wheelchair body, and the wheelchair body is controlled to follow the user's head movements.

[0016] During the activation of the VR control mode, the first type of data and the second type of data are collected and time-stamped and aligned. The first type of data is the original pose data of the human hand and head collected by the VR headset, and the second type of data is the actual execution status data fed back by the dual robotic arm actuators during the follow-up movement.

[0017] The first type of data and the second type of data are combined and packaged to form a training dataset that contains a one-to-one correspondence between human teaching intentions and machine execution states, and then stored or uploaded.

[0018] Furthermore, the step of determining whether the average distance between all fingertips is less than a preset threshold includes:

[0019] Obtain the spatial coordinates of the five fingertips relative to a fixed reference coordinate system, as captured by the VR headset;

[0020] Calculate the spatial distance between adjacent fingertips. A total of five distance values ​​are calculated, including the fingertips of the thumb and index finger, the fingertips of the index finger and middle finger, the fingertips of the middle finger and ring finger, the fingertips of the ring finger and little finger, and the fingertips of the little finger and thumb.

[0021] Calculate the arithmetic mean of the five distance values;

[0022] When the arithmetic mean is less than a preset distance threshold, it is determined that the user is in a clenched fist or fist-like state, and the VR control mode is activated; otherwise, it is determined to be in a non-control state, and motion mapping is stopped.

[0023] Furthermore, the step of converting the hand pose data into the target pose of the dual-arm end effector includes:

[0024] Obtain the homogeneous transformation matrix of the odometer relative to the fixed reference coordinate system;

[0025] The homogeneous transformation matrix of the wrist and fingers relative to a fixed reference coordinate system, obtained through the VR headset, is converted into a local homogeneous transformation matrix relative to the wheelchair body odometer coordinate system.

[0026] Calculate the pose increment between the current time step and the previous time step;

[0027] The actual pose of the current end effector of the robotic arm is obtained, and the pose increment is superimposed with the actual pose of the current end effector to obtain the target pose of the robotic arm at the next moment.

[0028] The joint control target is solved by inverse kinematics, which drives the joint movement of the robotic arm.

[0029] Furthermore, when controlling the dual robotic arms to follow the user's hand movements in real time, a compliant control step is also included:

[0030] When the robotic arm joint motor is set to impedance control mode, the joint torque output satisfies the formula:

[0031]

[0032] in, This is the final torque exerted by the motor. and The current position and speed of the motor. and For the target position and velocity, To compensate for the bias caused by gravity and friction. and These are the stiffness coefficient and the damping coefficient, respectively.

[0033] The stiffness coefficient and damping coefficient It is configured to make the robotic arm compliant when in contact with objects, avoiding rigid collisions.

[0034] Furthermore, when controlling the dual robotic arms to move in real time following the user's hand movements, a gripper compliance control step is also included:

[0035] The target opening and closing angle of the end gripper is determined by calculating the spatial distance between the tip of the thumb and the tip of the index finger captured by the VR headset.

[0036] A maximum torque threshold is set on the gripper motor driver. When the gripper cannot reach the target position due to contact with the object during its movement, the motor maintains a constant torque mode instead of forcibly executing the position control mode.

[0037] Furthermore, the second type of data includes one or more of the following: the actual angle values ​​of each joint of the dual robotic arms, joint speed, joint current, actual opening and closing distance of the end gripper, gripping force feedback value, real-time image frames collected by the end vision sensor of the robotic arm, odometer information of the wheelchair body, and actual linear and angular velocities of the wheelchair body.

[0038] Furthermore, after the synchronous acquisition and timestamp alignment of the first and second types of data, a data annotation step is also included:

[0039] The VR headset's virtual environment interface receives semantic tag selections from the user via eye contact or gestures for the current operation, and inserts the selected semantic tags into the data stream for the corresponding time period.

[0040] And / or, through the voice recognition interface of the VR headset, the user's verbal commands are received in real time and converted into text semantic tags, which are then inserted into the data stream for the corresponding time period.

[0041] Furthermore, after the synchronous acquisition and timestamp alignment of the first and second types of data, a data evaluation step is also included:

[0042] The final hand pose in the first type of data is compared with the final robotic arm pose in the second type of data in real time, and the error between the two is calculated.

[0043] When the error exceeds a preset threshold, or when the vision sensor at the end of the robotic arm detects that the target object slips or is lost after being gripped, the data for the corresponding time period is marked as a failure case and removed from the standard training set and stored in the failure case library.

[0044] Furthermore, the steps of controlling the wheelchair body to move in accordance with the user's head movements include:

[0045] Obtain the homogeneous transformation matrix of the head relative to the wheelchair body odometer coordinate system;

[0046] Extract the pitch and yaw angles;

[0047] Calculate the pitch angle increment and yaw angle increment between the current time and the previous time.

[0048] The target linear velocity of the wheelchair body is obtained by multiplying the pitch angle increment by the first control coefficient, and the target angular velocity of the wheelchair body is obtained by multiplying the yaw angle increment by the second control coefficient.

[0049] A second objective of this invention is to provide a robot remote operation control system based on a VR headset, comprising:

[0050] The wheelchair body includes a chassis, a seat, and a support member for connecting the chassis and the seat;

[0051] The dual-arm actuator includes two lightweight multi-axis robotic arms, which are symmetrically mounted on the front sides of the seat. The end effector of each robotic arm integrates a visual sensor for sensing environmental information.

[0052] VR headsets are worn on the user's head and have built-in sensors for real-time capture of the user's hand, fingertips and head spatial posture.

[0053] The main control system is communicatively connected to the drive motors of the VR headset, the dual robotic arm actuators, and the wheelchair body, and is configured to execute the above-described method.

[0054] Furthermore, the chassis of the wheelchair body adopts a layout combining differential drive wheels and omnidirectional wheels, with the differential drive wheels positioned at the front, so that the system rotation center is located in the middle of the chassis.

[0055] Furthermore, the dual robotic arm actuator is fixed to the support member via an adjustable mounting adapter, which is provided with multiple height-adjustable mounting holes to accommodate the operating height requirements of different users.

[0056] Furthermore, the main control system includes a data recording module, which is automatically triggered during VR control mode enablement. The data recording module serializes and packages the collected data using the ROS Bag format, and the naming rules of the data packets include user identifier, action name, and date and time information.

[0057] A third object of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0058] Compared with the prior art, the beneficial effects of the present invention are:

[0059] This invention utilizes a VR headset to directly collect the user's hand pose data, fingertip pose data, and head pose data. The hand pose data is then calculated into the target pose of the dual robotic arm end effectors, and the head pose data is calculated into the wheelchair's motion speed and direction. This technical solution eliminates the cumbersome joint-by-joint operation of existing joystick control methods and avoids the complex algorithm analysis required for biosignal or image recognition control. Users can simultaneously control the dual robotic arms and the wheelchair through natural hand and head movements, achieving real-time mapping control of hand movements that aligns with human intuition. Compared to existing technologies, this invention offers a more intuitive and efficient control method, significantly reducing operation time and improving task execution efficiency.

[0060] This invention calculates the relative distance between the five fingertips based on fingertip pose data and determines whether the average distance between all fingertips is less than a preset threshold, using this as the enabling condition for activating the VR control mode. This technical solution solves the problem of misoperation caused by low recognition accuracy or unconscious user actions in existing technologies. When the user does not actively make a fist or close their fist gesture, the system does not respond to any motion mapping, and the robotic arm and wheelchair remain stationary; the control function is only activated when the user explicitly makes an enabling gesture. This enabling logic accurately distinguishes between intentional operation and unintentional actions, fundamentally avoiding the safety risks caused by misoperation.

[0061] This invention synchronously collects and timestamps two types of data during VR control mode activation: the first type is the original pose data of the human hand and head collected by the VR headset, i.e., the human teaching intention; the second type is the actual execution state data fed back by the dual robotic arm actuators during follow-up movements, i.e., the machine execution state. These two types of data are combined and packaged to form a one-to-one corresponding training dataset. This technical solution solves the technical bottleneck of existing wheelchair robotic arm systems lacking high-quality data acquisition methods and hindering their development towards general-purpose intelligence. Compared with existing technologies, this invention is not only an operation control terminal but also a high-quality data acquisition platform, providing the necessary data foundation for the subsequent introduction of general-purpose large models and the realization of autonomous intelligence in wheelchair robotic arms.

[0062] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it according to the contents of the specification, the preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings. Specific embodiments of the present invention are given in detail below with reference to the accompanying drawings. Attached Figure Description

[0063] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0064] Figure 1 This is a structural diagram of a robot remote operation control system based on a VR headset.

[0065] Figure 2 This is an electrical block diagram of a robot remote operation control system based on a VR headset.

[0066] Figure 3 A diagram showing the relationship between various coordinate systems in the Apple Vision Pro system;

[0067] Figure 4 This is a flowchart of the program logic for the Apple Vision Pro client.

[0068] Figure 5 This is a flowchart of the main control program for the wheelchair.

[0069] Figure 6 A flowchart of a robot remote operation and control method based on a VR headset;

[0070] Figure 7 Flowchart for determining whether the average distance between all fingertips is less than a preset threshold;

[0071] Figure 8 Flowchart for calculating the hand pose data into the target pose of the dual robotic arm end effectors;

[0072] Figure 9 Flowchart for gripper compliance control;

[0073] Figure 10 Flowchart for controlling the wheelchair to move in accordance with the user's head movements;

[0074] Figure 11 Flowchart for data annotation;

[0075] Figure 12 A flowchart for data evaluation;

[0076] Figure 13 A schematic diagram of computer equipment;

[0077] Figure 14 This is a schematic diagram of a computer-readable storage medium. Detailed Implementation

[0078] The present invention will now be further described with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0079] Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this invention.

[0080] The drawing numbers in this application are only used to distinguish the steps in the scheme and are not used to limit the execution order of the steps. The specific execution order is as described in the specification.

[0081] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0082] Example 1

[0083] A robot remote operation control system based on a VR headset, such as Figure 1 As shown, it includes:

[0084] The wheelchair body includes a chassis 102, a seat 110, and a support member 103 for connecting the chassis and the seat;

[0085] The dual-arm actuator includes two lightweight multi-axis robotic arms 107, which are symmetrically mounted on the front sides of the seat 110. Each robotic arm has a visual sensor integrated on its end effector for sensing environmental information.

[0086] VR headsets are worn on the user's head and have built-in sensors for real-time capture of the user's hand, fingertips and head spatial posture.

[0087] The main control system is communicatively connected to the VR headset, the dual robotic arm actuators, and the drive motors of the wheelchair body. The main control system is configured to execute a robot remote operation control method based on the VR headset.

[0088] In this embodiment, the chassis 102 of the wheelchair body adopts a layout combining differential drive wheels 101 and omnidirectional wheels 104. The differential drive wheels 101 are front-mounted, meaning the drive wheels are installed at the front of the chassis. Compared to the traditional scheme where the rotation center of the rear wheels is located, this front-mounted drive method places the rotation center of the entire system in the middle of the chassis, significantly reducing the radius of motion of the body when turning on the spot and increasing steering flexibility. The rear wheels are omnidirectional wheels 104, whose axles are fixedly connected to the chassis 102, enabling passive following in any direction, and working in conjunction with the front differential drive wheels to achieve flexible steering.

[0089] The support member 103 is used to connect the chassis 102 and the seat 110. Its interior adopts a hollow reinforcing rib design. While ensuring the support strength, the hollow part is used to house electrical components such as the dual-arm six-axis robot driver, chassis motor drive and power supply, so as to achieve a compact structure and high space utilization.

[0090] In this embodiment, each robotic arm's end effector integrates a two-finger parallel gripper 108 for grasping and manipulating objects; a camera 109 is fixed above the gripper 108 to perceive changes in the external environment around the gripper in real time and provide visual feedback to the operator.

[0091] The dual robotic arm actuator is fixed to the support member via an adjustable mounting adapter. The mounting adapter is provided with multiple height-adjustable mounting holes to accommodate the operating height requirements of different users.

[0092] like Figure 1 As shown, the end of the robotic arm is fixedly connected to the end of the adapter 106 by bolts, and the fixed part of the robotic arm base is aligned with the front drive wheel on the same axis in the vertical plane. The adapter 106 is fixedly connected to the support member 103 via part 105, which has multiple height adjustment mounting holes. By selecting different mounting holes, the height of the six-axis robotic arm relative to the seat can be adjusted to suit the operating height requirements of different users, allowing the robotic arm's working range to cover the operating area in front of the seat.

[0093] The VR headset 112 uses Apple Vision Pro and is worn on the head of user 111. This headset integrates multiple cameras, an IMU (Inertial Measurement Unit), and other sensors to capture the spatial pose of the user's hands and head in real time. Through built-in algorithms, the VR headset can calculate six-degree-of-freedom pose data for key points relative to a fixed reference coordinate system, including the spatial position and orientation of the wrist, fingertips, and head.

[0094] The main control system uses Jetson Orin NX as the core processor and communicates with the drive motors of the VR headset, the dual robotic arm actuators, and the wheelchair itself.

[0095] like Figure 2 As shown, the main control system and the VR headset are located on the same local area network and establish wireless communication via the TCP / IP protocol. Specifically, a TCP server is established in the VR headset, and a TCP client runs on the main control system. A TCP-based wireless connection is established through a unique IP address to achieve high-frequency, low-latency data transmission.

[0096] The six-axis robotic arm utilizes a commercially available lightweight six-axis design, such as those from Songling Robotics or Ark Wireless, and establishes a communication connection with the main control system via a USB-to-CAN module. A camera at the end of the robotic arm is directly connected to the main control system via a USB interface for transmitting real-time image data.

[0097] The wheelchair's hub motors utilize high-precision encoders and servo motors, employing FOC (Field Orientation Control) via UVW three-phase lines. The encoders transmit position information to the motor controllers through a high-speed SPI interface. Both motor controllers communicate with the main control system via the same CAN bus, enabling precise control of the wheelchair's movement.

[0098] The main control system is configured to execute the method of the present invention, specifically including the following functional modules:

[0099] The pose receiving module is used to receive real-time pose data, including wrist, fingertips and head, sent by the VR headset;

[0100] The enable judgment module is used to calculate the relative distance between the fingertips based on the fingertip pose data of the five fingers, and to determine whether the average distance between all fingertips is less than a preset threshold. If so, the VR control mode is activated.

[0101] The robotic arm control module is used to calculate the hand pose data into the target pose of the dual robotic arm end effectors during VR control mode activation, and generate control commands to drive the dual robotic arms to follow the user's hand movements in real time.

[0102] The wheelchair control module is used to calculate the head posture data into the speed and direction of the wheelchair body, and generate control commands to drive the wheelchair body to move in accordance with the user's head movements.

[0103] The data recording module is used to synchronously collect and timestamp-align human instructional intent data with machine execution status data during VR control mode activation, forming a training dataset for storage or uploading. Preferably, the collected data is serialized and packaged using the ROS Bag format, and the naming rules for the data packets include user identifier, action name, and date and time information.

[0104] With the above-described structure and functional configuration, the system in this embodiment can achieve intuitive control and high-quality data acquisition of a dual-arm wheelchair robot.

[0105] For a detailed description of the robot remote operation and control method based on VR headset, please refer to the corresponding description in the following method embodiments, which will not be repeated here.

[0106] Example 2

[0107] A method for remote operation and control of a robot based on a VR headset is disclosed, applied to a remote operation and control system for a robot based on a VR headset. For a detailed description of the system, please refer to the corresponding description in the above system embodiments, which will not be repeated here. Figure 6 As shown, the method includes the following steps:

[0108] S1. Real-time collection of the user's hand pose data, fingertip pose data, and head pose data through a VR headset worn on the user's head.

[0109] like Figure 3 As shown, this embodiment employs a multi-layer coordinate system to achieve a unified representation and transformation of pose. 301 is a fixed reference coordinate system, maintaining its origin throughout the process; all other coordinate systems use this coordinate system as their reference. 302 is the head coordinate system, recording the position and orientation of the head relative to the fixed reference coordinate system 301. 303 is the wrist coordinate system, recording the position and orientation of the wrist relative to the fixed reference coordinate system 301. Furthermore, the position and orientation of each finger joint relative to the fixed reference coordinate system 301 are also recorded. Figure 3 In Figure 304, points 1 to 24 represent finger joints whose spatial poses can be captured.

[0110] Specifically, the VR headset (in this embodiment, Apple Vision Pro) uses multiple cameras and IMU sensors integrated into the device to capture the user's hand and head movements in real time. The VR headset obtains the coordinates relative to a fixed reference system (i.e., ...) through the API interface provided by visionOS. Figure 3The six-degree-of-freedom pose data of each key point in the fixed coordinate system 301 includes the pose of the head relative to the fixed reference coordinate system, including position and orientation; the pose of the left and right wrists relative to the fixed reference coordinate system; and the spatial coordinates of the fingertips of a total of 10 fingers of both hands relative to the fixed reference coordinate system.

[0111] VR headsets use the API provided by visionOS to acquire real-time position and posture data of the head, wrist, and fingertips relative to a fixed reference coordinate system. For example... Figure 4 As shown, the VR headset program first performs initialization operations, establishing a TCP server, setting the port number, and then repeatedly listening for client connection requests. When a client connects, it obtains the necessary spatial data for the wrist, head, and fingertips through the API interface and sends it to the client, repeating the above operations until the program exits.

[0112] To avoid accidental operation, this embodiment uses enable judgment logic based on five fingers together. For example... Figure 5 As shown, after the main control system program on the wheelchair starts, it initializes (establishes TCP client, initializes wheel hub motor, initializes six-axis robotic arm), and then establishes a connection with the TCP server on the VR headset. If the connection is successful, it enters the main program.

[0113] S2. Calculate the relative distance between the fingertips based on the fingertip pose data, and determine whether the average distance between all fingertips is less than a preset threshold. If so, activate the VR control mode.

[0114] In some embodiments, such as Figure 7 As shown, the step of determining whether the average distance between all fingertips is less than a preset threshold includes:

[0115] S21. Obtain the spatial coordinates of the five fingertips relative to a fixed reference coordinate system collected by the VR headset, including the fingertips of the thumb, index finger, middle finger, ring finger, and little finger.

[0116] Specifically, according to Figure 3 The spatial coordinates of the numbers 4 (thumb tip), 9 (index finger tip), 14 (middle finger tip), 19 (ring finger tip), and 24 (little finger tip) shown in 304 are calculated. The five distance values ​​are averaged.

[0117] S22. Calculate the spatial distance between adjacent fingertips. A total of five distance values ​​are calculated, including adjacent fingertips, such as the fingertips of the thumb and index finger, the fingertips of the index finger and middle finger, the fingertips of the middle finger and ring finger, the fingertips of the ring finger and little finger, and the fingertips of the little finger and thumb.

[0118] Taking the tip of the thumb (fingertip 4) and the tip of the index finger (fingertip 9) as an example, the distance calculation formula is:

[0119]

[0120] in, These represent the spatial position of fingertip 4 relative to the reference coordinate system. The spatial position of fingertip 9 relative to the reference coordinate system.

[0121] S23. Calculate the arithmetic mean of the five distance values;

[0122] Specifically, the distances between the five fingertips are recorded as follows: The mean distance is:

[0123]

[0124] S24. When the arithmetic mean is less than the preset distance threshold, it is determined that the user is in a clenched fist or fist-like state, and the VR control mode is activated; S25. Otherwise, it is determined to be in a non-control state, and the motion mapping is stopped.

[0125] In this embodiment, the preset threshold D is set to 0.02 meters. When When the value is less than 0.02, the system determines that the user is in a clenched fist or fist-like position. At this time, the VR control mode is activated, mapping hand movements to robotic arm movements and head movements to chassis movements. When the value is ≥0.02, it is determined to be in an uncontrolled state, and the robotic arm and chassis stop responding to motion mapping, thereby effectively avoiding misoperation caused by the user's unconscious actions.

[0126] S3. During the activation of the VR control mode, the hand pose data is calculated into the target pose of the dual robotic arm end effectors, and the dual robotic arms are controlled to follow the user's hand movements in real time; at the same time, the head pose data is calculated into the movement speed and direction of the wheelchair body, and the wheelchair body is controlled to follow the user's head movements.

[0127] After the VR control mode is activated, the main control system converts the received pose data into control commands for the robotic arm through coordinate transformation. In some embodiments, such as Figure 8 As shown, the steps for converting the hand pose data into the target pose of the dual-arm end effector include:

[0128] S31. First, using the odometer information provided by the chassis, obtain the homogeneous transformation matrix of the odometer relative to the fixed reference coordinate system. ;

[0129] S32. Convert the homogeneous transformation matrix of the wrist and fingers relative to the fixed reference coordinate system obtained through the VR headset into a local homogeneous transformation matrix relative to the wheelchair body odometer coordinate system.

[0130] Specifically, the head ,wrist Fingertips Use uniformly This is represented as a homogeneous transformation matrix relative to the wheelchair body odometer coordinate system:

[0131]

[0132] S33. Calculate the pose increment between the current time and the previous time.

[0133] Specifically, by Converting the representation to spatial position and rotation around three axes, we can obtain the pose increments at the current time t and the previous time t-1.

[0134]

[0135] S34. Obtain the actual pose of the current end effector of the robotic arm, and superimpose the pose increment with the actual pose of the current end effector to obtain the target pose of the robotic arm at the next moment.

[0136] Specifically, the current pose of the end effector gripper is obtained through the robotic arm interface. Add this to the desired pose increment to obtain the target pose of the robotic arm's end effector at the next moment:

[0137]

[0138] S35. Solve the joint control target through inverse kinematics to drive the joint movement of the robotic arm.

[0139] Specifically, the control targets for each joint are obtained through the inverse kinematics solution interface of the robotic arm, and commands are issued to drive the robotic arm joints to move to the target positions.

[0140] For the control of the end effector gripper, the spatial distance between the tips of the thumb and index finger, captured by the VR headset, is calculated. This target opening and closing distance is used as the gripper's target opening and closing distance. Control can be completed by sending the target opening and closing distance to the gripper through the gripper control API interface.

[0141] In some embodiments, such as Figure 9 As shown, when controlling the dual robotic arms to follow the user's hand movements in real time, the process also includes S36, the gripper compliance control step:

[0142] S361. The target opening and closing angle of the end gripper is determined by calculating the spatial distance between the tip of the thumb and the tip of the index finger captured by the VR headset.

[0143] S362. Set a maximum torque threshold on the gripper motor driver. When the gripper cannot reach the target position due to contact with the object during its movement, the motor maintains a constant torque mode instead of forcibly executing the position control mode.

[0144] S4. During the activation of the VR control mode, the first type of data and the second type of data are collected and time-stamped simultaneously. The first type of data is the original pose data of the human hand and head collected by the VR headset, and the second type of data is the actual execution status data fed back by the dual robotic arm actuators during the following motion.

[0145] To ensure high consistency and time correlation of the collected data, this embodiment employs a unified timestamp mechanism. Upon receiving each data packet from the VR headset (typically at 90Hz or higher), the main control system immediately records a high-precision timestamp from its local system. In each sampling period The data recording module synchronously collects the following two types of data:

[0146] The first type of data is human instructional intent data. This type of data comes from VR headsets and reflects the operator's subjective operational intent. Specifically, it includes head pose data, that is, the homogeneous transformation matrix of the head relative to a fixed reference coordinate system. , including location and posture And fine motor data of the hands, namely the transformation matrices of the left and right wrists relative to a fixed reference coordinate system. And the spatial coordinates of the fingertips of all 10 fingers on both hands. To enrich the data dimensions, optional data collection can be performed on the proximal and distal interphalangeal joints.

[0147] The second type of data is machine execution status data. This type of data comes from real-time feedback from the robotic arm and chassis, reflecting the actual execution results of the machine. Specifically, it includes the joint status of the robotic arm, namely the actual angle values ​​of the six joints of each arm. and Joint speed, joint current, end effector status, i.e., the actual opening and closing distance of the left and right grippers. The system includes clamping force feedback values, end-effector visual information (i.e., real-time image frames captured by the end-effector camera of the robotic arm, which strictly correspond to the pose data at the current moment, saved in JPEG or raw image data format, and accompanied by intrinsic and extrinsic parameter information, as well as chassis motion status, i.e., the odometer information of the wheelchair itself. Actual linear velocity and angular velocity .

[0148] During VR control mode activation, the main control system simultaneously calculates head pose data into motion control commands for the wheelchair body. In some embodiments, such as Figure 10 As shown, the steps for controlling the wheelchair body to move in accordance with the user's head movements include:

[0149] S37. Using the coordinate transformation method described above, obtain the homogeneous transformation matrix of the head relative to the wheelchair's odometer coordinate system. ;

[0150] S38. Extract the pitch angle p and yaw angle y;

[0151] S39. Calculate the pitch angle increment and yaw angle increment between the current time and the previous time.

[0152] S310. Multiply the pitch angle increment by the first control coefficient to obtain the target linear velocity of the wheelchair body, and multiply the yaw angle increment by the second control coefficient to obtain the target angular velocity of the wheelchair body.

[0153] Specifically, this is achieved by calculating the pitch angle increment and yaw angle increment between the current moment and the previous moment. and Multiply the pitch angle increment by the first control coefficient The target linear velocity of the wheelchair is obtained, and the yaw angle increment is multiplied by the second control coefficient. The target angular velocity of the wheelchair body is obtained:

[0154]

[0155] This allows users to intuitively control the wheelchair by tilting their head forward and backward, and by turning their head to control its steering.

[0156] To prevent rigid collisions when the robotic arm comes into contact with a human body or object, this embodiment employs a quasi-direct drive joint motor and a compliant control strategy. The robotic arm joints utilize a quasi-direct drive design, combining a direct drive motor with a small reduction ratio reducer. While this joint motor module has slightly lower precision, it ensures compliance when in contact with the human body. In some embodiments, when controlling the dual robotic arms to follow the user's hand movements in real time, a compliant control step is also included:

[0157] When the robotic arm joint motor is set to impedance control mode, the joint torque output satisfies the formula:

[0158]

[0159] in, This is the final torque exerted by the motor. and The current position and speed of the motor. and For the target position and velocity, To compensate for the bias caused by gravity and friction. and These are the stiffness coefficient and the damping coefficient, respectively.

[0160] The key to current compliance control lies in and Setting two parameters. This embodiment provides a parameter adjustment method: first, set a small value. Value, and set a suitable one. The value is used to provide damping; then a fixed target position is sent, and the robotic arm joint is manually pushed to feel its stiffness; the parameters are gradually adjusted based on the feeling, and if the joint feels too stiff, it is reduced. If you feel any shaking, increase the value. Value; iterate continuously until a satisfactory level of smoothness is found.

[0161] For compliant control of the gripper, this embodiment sets a maximum torque threshold on the motor driver. When the gripper cannot reach the target position due to contact with the object during its movement, the motor will not generate a large torque to force the position control mode. Instead, it will maintain the torque at the set limit value, maintaining a constant torque mode. This ensures the stability of the gripping while preventing damage to the object or the gripper itself due to forced closure.

[0162] To achieve the acquisition of high-quality training data, this embodiment adds two functional modules—semantic annotation and data evaluation—to the data acquisition process. In some embodiments, such as… Figure 11 As shown, after the synchronous acquisition and timestamp alignment of the first and second types of data, the process further includes S41, a data annotation step:

[0163] S411. Through the virtual environment interface of the VR headset, receive the semantic tag selection of the current operation action by the user in the form of eye contact or gesture, and insert the selected semantic tag into the data stream of the corresponding time period;

[0164] And / or, S412, receive the user's verbal commands in real time through the voice recognition interface of the VR headset and convert them into text semantic tags, inserting them into the data stream of the corresponding time period.

[0165] This embodiment provides two annotation methods: visual annotation and voice annotation. The visual annotation method involves designing a floating menu of action labels within the virtual environment of the VR headset. Before starting an action, such as opening a bottle, the operator can click the "open bottle" label using their gaze or gesture. The data stream subsequently collected by the system is automatically tagged with the semantic label "open bottle."

[0166] The voice annotation method involves capturing the operator's verbal commands in real time using the microphone and voice recognition API built into the VR headset. When the operator says commands such as "pick up a water glass," the voice recognition is automatically converted into text and inserted into the corresponding data stream.

[0167] The two annotation methods can be used individually or simultaneously. When used simultaneously, the system fuses the semantic labels selected by gaze or gesture with the text semantic labels from speech recognition to form multimodal annotation data, further improving the accuracy and richness of the annotations.

[0168] To automatically determine the success or failure of an operation, in some embodiments, such as Figure 12 As shown, after the synchronous acquisition and timestamp alignment of the first and second types of data, the data evaluation step is further included in step S42:

[0169] S421. In real time, compare the final hand pose in the first type of data with the final robotic arm pose in the second type of data, and calculate the error between the two.

[0170] S422. When the error exceeds a preset threshold, or when the robotic arm's end-effector detects that the target object slips or is lost after being grasped, the data for the corresponding time period is marked as a failure case and removed from the standard training set and stored in a failure case library. Failure cases are removed from the standard training set and stored in a dedicated failure case library for subsequent failure analysis or as negative samples for model training.

[0171] S5. Combine and package the first type of data with the second type of data to form a training dataset that contains a one-to-one correspondence between human teaching intentions and machine execution states, and store or upload it.

[0172] During each sampling period, the main control system serializes and packages all the above data according to a predetermined protocol. This embodiment preferably uses the ROS Bag format for storage because ROS Bag can efficiently store multi-sensor data (images, point clouds, poses) and time-series data, and has a complete open-source toolchain for easy subsequent data parsing and playback. The naming convention for the data packets is subject_id_action_name_datetime.bag, for easy later classification and management.

[0173] During non-operational periods, such as when the wheelchair is charging, the main control system automatically or manually uploads the locally stored Bag file to the cloud data center via Wi-Fi or 5G modules. Cloud storage enables centralized data management and cross-device sharing, providing data support for subsequent model training and algorithm iteration.

[0174] like Figure 5 As shown, after the wheelchair-side main control system program starts, it initializes by establishing a TCP client, initializing the wheel hub motors, and initializing the six-axis robotic arm. Then, it establishes a connection with the VR headset's TCP server. If the connection is successful, it enters the main program. The main program first determines whether VR control is enabled, i.e., it executes step S2. If enabled successfully, it executes steps S3 and S4. Simultaneously, the data recording module is automatically triggered, executing step S5. The entire process is executed cyclically, achieving continuous remote operation control and data acquisition.

[0175] This invention enhances operational capabilities and applicability: by employing a dual-arm collaborative design, it can mimic human hands to complete complex daily life tasks such as opening bottles, wringing towels, and carrying large objects, greatly expanding the practicality of wheelchair robotic arm systems.

[0176] This invention can improve control precision and efficiency: by using a high-precision VR headset to directly acquire hand and head postures, there is no need for complex image recognition algorithms to calculate intentions. The control latency is low and the precision is high, realizing intuitive WYSIWYG control and significantly improving task execution efficiency.

[0177] This invention enhances safety and reliability: by using enable judgment logic based on the distance between the five fingertips, it effectively distinguishes between the user's intended operation and unintentional actions, avoiding maloperation of the robotic arm caused by sensor misidentification or unconscious user actions, thus ensuring safe use.

[0178] This invention addresses data bottlenecks and drives intelligentization: the system is not only an operating terminal but also a natural high-quality data acquisition platform. By synchronously collecting high-precision teaching and execution data, it can construct large-scale, high-quality human-machine collaborative operation datasets at low cost and high efficiency. This provides the necessary data foundation for subsequently introducing general-purpose large models and realizing the autonomous intelligence of wheelchair robotic arms, breaking the deadlock of traditional systems' inability to improve their intelligence level.

[0179] Example 3

[0180] A computer device 600, such as Figure 13As shown, the system includes a memory 610, a processor 620, and a computer program 630 stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of a method for remotely controlling a robot based on a VR headset. For a detailed description of the method, please refer to the corresponding description in the above method embodiments; it will not be repeated here.

[0181] Example 4

[0182] A computer-readable storage medium, such as Figure 14 As shown, a computer program is stored thereon. When executed by a processor, the computer program implements the steps of a method for remote operation and control of a robot based on a VR headset. For a detailed description of the method, please refer to the corresponding description in the above method embodiments, which will not be repeated here.

[0183] The number of devices and processing scale described herein are for the purpose of simplifying the description of the invention. Applications, modifications, and variations of the invention will be readily apparent to those skilled in the art.

[0184] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

[0185] The apparatus, computer device, and non-volatile computer storage medium and method provided in the embodiments of this specification are corresponding. Therefore, the apparatus, computer device, and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, computer device, and non-volatile computer storage medium will not be repeated here.

[0186] Those skilled in the art will also know that, besides implementing the controller in the form of purely computer-readable program code, the same functions can be achieved by logically programming the method steps, making the controller take the form of logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers (PLCs), and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the devices included within it for implementing various functions can also be considered structures within that hardware component. Alternatively, the devices for implementing various functions can be considered as both software units implementing the method and structures within a hardware component.

[0187] The systems, apparatuses, or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. For ease of description, the above apparatuses are described separately as various units based on their functions. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0188] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0189] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0190] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0191] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0192] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0193] This specification may be described in the general context of computer-executable instructions, such as program units, that are executed by a computer. Generally, program units include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification may also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program units may reside in local and remote computer storage media, including storage devices.

[0194] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0195] The above description is merely an embodiment of this specification and is not intended to limit the scope of one or more embodiments of this specification. Various modifications and variations can be made to one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of one or more embodiments of this specification should be included within the scope of the claims of one or more embodiments of this specification.

Claims

1. A method for remote operation and control of a robot based on a VR headset, characterized in that, Includes the following steps: The VR headset worn on the user's head collects the user's hand pose data, fingertip pose data, and head pose data in real time. The relative distance between the five fingertips is calculated based on the fingertip pose data, and it is determined whether the average distance between all fingertips is less than a preset threshold. If so, the VR control mode is activated. During the activation of the VR control mode, the hand pose data is calculated into the target pose of the dual robotic arm end effectors, and the dual robotic arms are controlled to follow the user's hand movements in real time; at the same time, the head pose data is calculated into the speed and direction of the wheelchair body, and the wheelchair body is controlled to follow the user's head movements. During the activation of the VR control mode, the first type of data and the second type of data are collected and time-stamped and aligned. The first type of data is the original pose data of the human hand and head collected by the VR headset, and the second type of data is the actual execution status data fed back by the dual robotic arm actuators during the follow-up movement. The first type of data and the second type of data are combined and packaged to form a training dataset that contains a one-to-one correspondence between human teaching intentions and machine execution states, and then stored or uploaded. The step of determining whether the average distance between all fingertips is less than a preset threshold includes: Obtain the spatial coordinates of the five fingertips relative to a fixed reference coordinate system, as captured by the VR headset; Calculate the spatial distance between adjacent fingertips. A total of five distance values ​​are calculated, including the fingertips of the thumb and index finger, the fingertips of the index finger and middle finger, the fingertips of the middle finger and ring finger, the fingertips of the ring finger and little finger, and the fingertips of the little finger and thumb. Calculate the arithmetic mean of the five distance values; When the arithmetic mean is less than a preset distance threshold, it is determined that the user is in a clenched fist or fist-like state, and the VR control mode is activated; otherwise, it is determined to be in a non-control state, and motion mapping is stopped. The steps for controlling the wheelchair body to move in accordance with the user's head movements include: Obtain the homogeneous transformation matrix of the head relative to the wheelchair body odometer coordinate system; Extract the pitch and yaw angles; Calculate the pitch angle increment and yaw angle increment between the current time and the previous time. The target linear velocity of the wheelchair body is obtained by multiplying the pitch angle increment by the first control coefficient, and the target angular velocity of the wheelchair body is obtained by multiplying the yaw angle increment by the second control coefficient.

2. The robot remote operation and control method based on a VR headset as described in claim 1, characterized in that, The steps for converting the hand pose data into the target pose of the dual robotic arm end effector include: Obtain the homogeneous transformation matrix of the odometer relative to the fixed reference coordinate system; The homogeneous transformation matrix of the wrist and fingers relative to a fixed reference coordinate system, obtained through the VR headset, is converted into a local homogeneous transformation matrix relative to the wheelchair body odometer coordinate system. Calculate the pose increment between the current time step and the previous time step; The actual pose of the current end effector of the robotic arm is obtained, and the pose increment is superimposed with the actual pose of the current end effector to obtain the target pose of the robotic arm at the next moment. The joint control target is solved by inverse kinematics, which drives the joint movement of the robotic arm.

3. The robot remote operation and control method based on a VR headset as described in claim 2, characterized in that, When controlling the dual robotic arms to follow the user's hand movements in real time, a compliant control step is also included: When the robotic arm joint motor is set to impedance control mode, the joint torque output satisfies the formula: in, This is the final torque exerted by the motor. and The current position and speed of the motor. and For the target position and velocity, To compensate for the bias caused by gravity and friction. and These are the stiffness coefficient and the damping coefficient, respectively. The stiffness coefficient and damping coefficient It is configured to make the robotic arm compliant when in contact with objects, avoiding rigid collisions.

4. The robot remote operation and control method based on a VR headset as described in claim 3, characterized in that, When controlling the dual robotic arms to follow the user's hand movements in real time, the gripper compliance control step is also included: The target opening and closing angle of the end gripper is determined by calculating the spatial distance between the tip of the thumb and the tip of the index finger captured by the VR headset. A maximum torque threshold is set on the gripper motor driver. When the gripper cannot reach the target position due to contact with the object during its movement, the motor maintains a constant torque mode instead of forcibly executing the position control mode.

5. The robot remote operation and control method based on a VR headset as described in claim 1, characterized in that, The second type of data includes one or more of the following: the actual angle values ​​of each joint of the dual robotic arms, joint speed, joint current, actual opening and closing distance of the end gripper, gripping force feedback value, real-time image frames collected by the end vision sensor of the robotic arm, odometer information of the wheelchair body, and actual linear and angular velocities of the wheelchair body.

6. The robot remote operation and control method based on a VR headset as described in claim 1, characterized in that, After the synchronous acquisition and timestamp alignment of the first and second types of data, a data annotation step is also included: The VR headset's virtual environment interface receives semantic tag selections from the user via eye contact or gestures for the current operation, and inserts the selected semantic tags into the data stream for the corresponding time period. And / or, through the voice recognition interface of the VR headset, the user's verbal commands are received in real time and converted into text semantic tags, which are then inserted into the data stream for the corresponding time period.

7. The robot remote operation and control method based on a VR headset as described in claim 1, characterized in that, Following the synchronous acquisition and timestamp alignment of the first and second types of data, a data evaluation step is also included: The final hand pose in the first type of data is compared with the final robotic arm pose in the second type of data in real time, and the error between the two is calculated. When the error exceeds a preset threshold, or when the vision sensor at the end of the robotic arm detects that the target object slips or is lost after being gripped, the data for the corresponding time period is marked as a failure case and removed from the standard training set and stored in the failure case library.

8. A robot remote operation control system based on a VR headset, characterized in that, include: The wheelchair body includes a chassis, a seat, and a support member for connecting the chassis and the seat; The dual-arm actuator includes two lightweight multi-axis robotic arms, which are symmetrically mounted on the front sides of the seat. The end effector of each robotic arm integrates a visual sensor for sensing environmental information. VR headsets are worn on the user's head and have built-in sensors for real-time capture of the user's hand, fingertips and head spatial posture. The main control system is communicatively connected to the drive motors of the VR headset, the dual robotic arm actuators, and the wheelchair body, and is configured to perform the method as described in any one of claims 1 to 7.

9. A robot remote operation control system based on a VR headset as described in claim 8, characterized in that, The chassis of the wheelchair body adopts a layout that combines differential drive wheels and omnidirectional wheels. The differential drive wheels are front-mounted, so that the rotation center of the system is located in the middle of the chassis.

10. A robot remote operation control system based on a VR headset as described in claim 8, characterized in that, The dual robotic arm actuator is fixed to the support member via an adjustable mounting adapter. The mounting adapter is provided with multiple height-adjustable mounting holes to accommodate the operating height requirements of different users.

11. The robot remote operation control system based on a VR headset as described in claim 8, characterized in that, The main control system includes a data recording module, which is automatically triggered during VR control mode. The data recording module serializes and packages the collected data using the ROS Bag format. The naming rules for the data packets include user identifier, action name, and date and time information.

12. A computer-readable storage medium, characterized in that, It stores program instructions that, when executed, implement the method as described in any one of claims 1 to 7.

Images