Mechanical arm motion control method and system, equipment and medium

Abstract

The invention discloses a mechanical arm motion control method and system, equipment and a medium, and relates to the technical field of mechanical arm control. The method comprises the steps that after a host starts a virtual reality interaction device and a mechanical arm, modeling is conducted according to real joint angle information and structure information of the mechanical arm to obtain a machine virtual model of the mechanical arm and the current pose of the machine virtual model in a virtual space coordinate system; the host continuously reads the current state information of the handle device; under the condition that the key state information indicates that a control key of the handle device is triggered, the host carries out first frame pose alignment processing and pose determination processing according to the handle pose information and the key state information, and the target pose of the mechanical arm in the virtual space coordinate system is determined; and inverse kinematics solving processing is carried out according to the current pose and the target pose, an optimized joint angle is obtained, and a motion control instruction is generated to control the mechanical arm to move. And the universality and the real-time performance can be considered in the remote operation process of the mechanical arm.

Description

Technical Field

[0001] This application relates to the field of robotic arm control technology, and in particular to a robotic arm motion control method, system, device and medium. Background Technology

[0002] Virtual Reality (VR) teleoperation technology refers to a technical system in which an operator, wearing a VR headset and controllers, controls the movement of a remote physical robot in real time within a virtual environment. This technology integrates human-computer interaction, robot kinematics, real-time communication, and computer vision, aiming to solve problems such as insufficient immersion, low operational accuracy, and high learning costs associated with traditional teach pendants or 2D screen-controlled robots. In terms of technical implementation, VR teleoperation systems typically follow a closed-loop process: "data acquisition → motion mapping → inverse kinematics solution → robot control → visual feedback." The operator's head and hand poses are captured by the VR device and mapped to the robot's workspace through coordinate system transformation; the inverse kinematics algorithm converts the end-effector poses into joint angle commands; after the robot executes the commands, it feeds back the status to the VR device via cameras or sensor networks, forming a closed loop. The core challenges in this field include physiological adaptation of motion redirection (heterogeneous mapping between human hands and robotic arms), visual immersion (synchronization of first-person perspective with the operator's line of sight), and system versatility (adaptation to different robot configurations).

[0003] Traditional robotic arm teleoperation typically employs a combination of button panels, joysticks, keyboards and mice, or monitoring videos. This approach is difficult to use and struggles to complete complex trajectories or delicate maneuvers. Furthermore, it's challenging to balance versatility and real-time performance during teleoperation; and it requires multiple systems or platforms to successfully implement robotic arm teleoperation, resulting in system complexity and high learning costs. Summary of the Invention

[0004] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a robotic arm motion control method, system, device, and medium, which can synchronously achieve low-latency and high-stability motion control of the robotic arm based on a virtual reality interactive device, taking into account both versatility and real-time performance during the teleoperation of the robotic arm.

[0005] In a first aspect, embodiments of this application provide a robotic arm motion control method, applied to a robotic arm motion control system; the robotic arm motion control system includes: a host, a robotic arm, a virtual reality interaction device, and a router; the host, the robotic arm, and the virtual reality interaction device are all communicatively connected to the router; the virtual reality interaction device includes: a binocular virtual reality head-mounted display and a controller, the controller being provided with control buttons; the virtual reality interaction device is configured with a VR remote collaboration component program; After the host computer starts the virtual reality interaction device and the robotic arm, it performs modeling processing based on the actual joint angle information and structural information of the robotic arm to obtain the machine virtual model of the robotic arm and the current pose of the machine virtual model in the virtual space coordinate system. The host continuously reads the current status information of the gamepad device; the current status information includes: button status information and gamepad pose information in the virtual space coordinate system; When the button status information indicates that the control button of the handle device is triggered, the host performs first frame pose alignment processing and pose determination processing based on the handle pose information and the button status information to determine the target pose of the robotic arm in the virtual space coordinate system. The host performs inverse kinematics calculation based on the current pose and the target pose to obtain the optimized joint angles. The host computer generates motion control commands based on the optimized joint angles to control the movement of the robotic arm.

[0006] Secondly, embodiments of this application provide a robotic arm motion control system, comprising: a host computer, a robotic arm, a virtual reality interaction device, and a router; the host computer, the robotic arm, and the virtual reality interaction device are all communicatively connected to the router; the virtual reality interaction device includes: a binocular virtual reality head-mounted display and a controller, the controller being provided with control buttons; in the robotic arm motion control system, the host computer, the robotic arm, the virtual reality interaction device, and the router cooperate with each other to implement the robotic arm motion control method as described in any one of the embodiments of the first aspect.

[0007] Thirdly, embodiments of this application provide an electronic device, characterized in that it includes at least one processor and a memory for communicatively connecting to the at least one processor; the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the robotic arm motion control method as described in any of the embodiments of the first aspect.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium, characterized in that the computer-readable storage medium stores computer-executable instructions, which are used to cause a computer to perform the robotic arm motion control method as described in any one of the first aspect embodiments.

[0009] The embodiments of this application include: The robotic arm motion control system includes: a host computer, a robotic arm, a virtual reality (VR) interaction device, and a router; the host computer, robotic arm, and VR interaction device are all communicatively connected to the router; the VR interaction device includes: a binocular VR headset and a controller, with control buttons on the controller; the VR interaction device is configured with a VR remote collaboration component program; this robotic arm motion control system directly utilizes inter-process communication, avoiding the need for ROS operating system development; using the VR remote collaboration component program for data acquisition and sharing of the VR interaction device makes the overall communication of the robotic arm motion control system more flexible, more compatible, and has lower latency; during the process of using the robotic arm motion control system to control the movement of the robotic arm, firstly, after the host computer starts the VR interaction device and the robotic arm, it then... The system performs modeling processing on the joint angle and structural information to obtain the robot arm's virtual model and its current pose in the virtual space coordinate system. Next, the host continuously reads the current state information of the handle device, including button state information and handle pose information in the virtual space coordinate system. Then, when the button state information indicates that the handle device's control button has been triggered, the host performs first-frame pose alignment and pose determination processing based on the handle pose information and button state information to determine the robot arm's target pose in the virtual space coordinate system. Then, the host performs inverse kinematics solving based on the current pose and target pose to obtain optimized joint angles. Finally, the host generates motion control commands based on the optimized joint angles to control the robot arm's movement, achieving low-latency, high-stability motion control. In other words, this embodiment can synchronously achieve low-latency, high-stability motion control of a robot arm based on a virtual reality interactive device, balancing versatility and real-time performance during remote operation of the robot arm.

[0010] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the description and the accompanying drawings. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the structure of a robotic arm motion control system provided in one embodiment of this application; Figure 2 This is a schematic diagram of the collaborative process steps of various devices in a robotic arm motion control system provided in one embodiment of this application; Figure 3 This is a schematic diagram of coordinate system alignment provided in one embodiment of this application; Figure 4 This is a flowchart illustrating a robotic arm motion control method provided in one embodiment of this application; Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of this application. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments.

[0013] It should be noted that although a logical order is shown in the flowcharts in this application, in some cases, the steps shown or described may be performed in a different order than that shown in the flowcharts. In the description of this application, "several" means one or more, and "more" means two or more. The terms "first" and "second" are used only to distinguish technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of technical features indicated, or implicitly indicating the order in which the technical features are indicated.

[0014] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0015] First, let me explain some of the terms used in this application: VR stands for Virtual Reality. VR is a comprehensive technology that uses computers to generate and simulate three-dimensional environments.

[0016] WebSocket is a network communication protocol that supports full-duplex, bidirectional communication; it allows a client (such as a browser) to establish a connection with a server and then conduct continuous, low-latency data exchange.

[0017] URDF stands for Unified Robot Description Format.

[0018] Pinocchio: A library for robot dynamics.

[0019] CasADi: An open-source software library for optimization, automatic differentiation, and numerical computation.

[0020] Unity is a tool platform used by creators to make games and 3D interactive content.

[0021] ROS stands for Robot Operating System.

[0022] Traditional teleoperation of robotic arms typically employs a combination of button panels, joysticks, keyboards and mice, or monitoring videos. This approach has significant drawbacks: first, it lacks intuitive operation, requiring extensive professional training for users; second, it limits the field of view, making it difficult to accurately judge the robotic arm's status, especially in environments with complex obstructions or confined spaces; and third, it is difficult to control, hindering the completion of complex trajectories or delicate maneuvers. Furthermore, it struggles to balance the versatility and real-time performance of teleoperation; and it requires multiple systems or platforms to successfully implement robotic arm teleoperation, resulting in system complexity and high learning costs.

[0023] Based on this, this application provides a robotic arm motion control method, a robotic arm motion control system, an electronic device, and a computer-readable storage medium, relating to the field of robotic arm control technology. It enables low-latency, high-stability motion control of the robotic arm synchronously using a virtual reality interactive device, balancing versatility and real-time performance during remote operation of the robotic arm.

[0024] The embodiments of this application will be further described below with reference to the accompanying drawings.

[0025] like Figure 1 As shown, the robotic arm motion control system 1000 includes: a host computer 100, a robotic arm 200, a virtual reality interaction device 300, and a router 400; the host computer 100, robotic arm 200, and virtual reality interaction device 300 are all communicatively connected to the router 400; the virtual reality interaction device 300 includes: a binocular virtual reality head-mounted display device 310 and a controller device 320, with control buttons 321 provided on the controller device 320. The virtual reality interaction device 300 is configured with a VR remote collaboration component software plugin.

[0026] The robotic arm motion control system 1000 also includes a camera device 500, which is communicatively connected to the host 100; the camera device 500 is used to collect real-time image data toward the robotic arm 200.

[0027] It is understood that the virtual reality interaction device 300 specifically includes: a set of binocular virtual reality head-mounted display devices 310; a pair of hand controllers 320 with posture tracking capabilities, the hand controllers 320 including a left hand controller and a right hand controller; a high-speed shared memory data channel; and an independently running posture acquisition thread.

[0028] It is understandable that the communication logic between the various devices in the robotic arm motion control system 1000 is as follows: Figure 1 As shown, the host 100, robotic arm 200, virtual reality interaction device 300, router 400, and camera communicate in the same network environment.

[0029] Specifically, in one embodiment, the robotic arm motion control system 1000 employs a Pico4ultra binocular VR headset and a handle device 320 (including a left handle and a handle with handles) as a virtual reality interaction device 300. The dual-arm robot includes two 6-DOF industrial-grade robotic arms 200 and a host computer 100 running the Ubuntu system. Optionally, the robotic arm motion control system 1000 can display the position, pose, etc. of the robotic arms 200 and their target end effectors through a 3D browser's visual interface.

[0030] Specifically, the controller device 320 is equipped with a VR posture acquisition module, which updates the controller posture data at a frequency of 60 Hz.

[0031] Specifically, the robotic arm motion control system 1000 also includes a locally configured inverse kinematics solver, which is used to perform optimization solutions locally. The typical solution time is less than 5 ms, and it can respond to the dual-arm control requirements in real time.

[0032] In actual operation, the operator wears a binocular virtual reality headset 310 and uses the left and right handles respectively to control the end effector movements of the left and right robotic arms 200. When the grip button on either handle is pressed, that handle enters the control mode of the corresponding robotic arm 200; releasing the button stops control. As the operator moves the handles in VR space, the end effector of the robotic arm 200 follows the translational movement of the handles in a non-linear scaling manner, maintaining the stability and high responsiveness of the real-time image data feedback.

[0033] It is understood that this application enables remote operation of a lightweight dual-arm robot based on virtual reality interaction through the robotic arm motion control system 1000. The virtual reality interaction device 300 collects the operator's handle pose information and converts it into motion control commands for the end effector of the dual-arm robotic arm 200 in real time. Through the interaction of the VR device and the end effector of the robotic arm 200 in virtual / physical space with 6 degrees of freedom, the operation intention is accurately mapped.

[0034] The collaborative process steps of the various devices in the motion control system of the robotic arm in this application are as follows: Figure 2 As shown; Step S1: The host computer starts up and initializes the virtual reality interaction device and the robotic arm.

[0035] Step S2: The handle device continuously sends current status information.

[0036] Step S3: After the operator triggers the control button through the handle device, the robot arm is modeled using the Pinocchio+CasaDi hybrid optimization framework to obtain a virtual model of the machine, and the joint angle information is obtained by real-time inverse kinematics optimization.

[0037] Step S4: The joint angle information obtained from the solution is input into the robotic arm interface to drive the real robotic arm movement in real time and update the virtual model of the machine synchronously.

[0038] It is understandable that the host, robotic arm, virtual reality interaction device and router in the robotic arm motion control system cooperate with each other to realize the robotic arm motion control method provided in this embodiment. It can realize low-latency and high-stability motion control of the robotic arm synchronously based on the virtual reality interaction device, and take into account both versatility and real-time performance in the process of remote operation of the robotic arm.

[0039] Those skilled in the art will understand that the system structure shown in the figures does not constitute a limitation on the embodiments of this application, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0040] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0041] It will be understood by those skilled in the art that the system architecture and application scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. It is known by those skilled in the art that with the evolution of system architecture and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0042] Based on the above system structure, various embodiments of the robotic arm motion control method of this application are presented below.

[0043] like Figure 3 As shown, this robotic arm motion control method can be applied to, for example... Figure 1 The illustrated robotic arm motion control system includes: a host computer, a robotic arm, a virtual reality (VR) interaction device, and a router; the host computer, robotic arm, and VR interaction device are all communicatively connected to the router; the VR interaction device includes: a binocular VR headset and a controller, with control buttons on the controller; the VR interaction device is configured with a VR remote collaboration component software plugin. This robotic arm motion control method may include, but is not limited to, steps S100 to S500.

[0044] Step S100: After the host starts the virtual reality interaction device and the robotic arm, it performs modeling processing based on the actual joint angle information and structural information of the robotic arm to obtain the machine virtual model of the robotic arm and the current pose of the machine virtual model in the virtual space coordinate system.

[0045] It is understandable that the host is equipped with a dual-arm robotic arm modeling module, and step S100 is executed through the dual-arm robotic arm modeling module.

[0046] According to some embodiments of this application, step S100 is further described. Step S100: Modeling is performed based on the actual joint angle information and structural information of the robotic arm to obtain the virtual machine model of the robotic arm and the current pose of the virtual machine model in the virtual space coordinate system, including but not limited to steps S110 to S140.

[0047] Step S110: When the robotic arm starts, obtain the actual joint angle information of the robotic arm and obtain the structural information of the robotic arm from the URDF description file.

[0048] Step S120: Construct a virtual machine model based on the structural information.

[0049] Step S130: Determine the initial pose of the machine virtual model based on the actual joint angle information.

[0050] Step S140: In the initial pose, calculate the current pose of the machine virtual model using a forward kinematics algorithm; the current pose is used to indicate the pose of each joint of the robotic arm and the end effector in space.

[0051] Specifically, the virtual machine model includes information such as linkage structure, joint type, and end effector.

[0052] Specifically, forward kinematics is defined as the process of calculating the position and orientation of an end effector (or other link) in space using known joint parameters (such as rotation angles and translation distances). The core of forward kinematics algorithms is establishing kinematic equations through coordinate transformations, deriving the pose of the robotic arm's end effector relative to a fixed coordinate system, and forming a complete computational system with inverse kinematics.

[0053] This application, through steps S110 to S140, during the startup phase of the robotic arm motion control system, inputs the actual joint angle information of the physical robotic arm. When the robotic arm starts, it retrieves the structural information of the robotic arm from the URDF description file. Based on the structural information, a unified dynamics and kinematics description is established for each joint, generating a virtual machine model, i.e., a robot model. Simultaneously, based on the actual joint angle information of the robotic arm, the joint states of the robotic arm are initialized to the initial posture set by the virtual machine model. Subsequently, a forward kinematics algorithm is used to calculate the pose of each joint and the end effector in space for subsequent alignment with the virtual reality coordinate system.

[0054] It should be noted that this application provides an alignment mechanism between the VR device coordinate system and the robot arm coordinate system. Specifically, this embodiment achieves consistency between the motion direction of the virtual controller and the motion direction of the robot end effector by defining a unified transformation matrix between the VR device coordinate system and the robot arm coordinate system. The VR device coordinate system typically refers to the coordinate system defined by the virtual reality device (including headsets and controllers) in the tracking system (such as SteamVR base stations, Oculus sensors, or Inside-Out tracking). The robot arm coordinate system refers to the real physical coordinate system.

[0055] This embodiment uses the following 3×3 rotation matrix as the coordinate alignment relationship: ; In this 3×3 rotation matrix, each column represents the orientation of the X, Y, and Z axes of the VR device's coordinate system within the robot's coordinate system. Specifically, as shown... Figure 4 As shown, the 3×3 rotation matrix achieves the following mapping relationship: the X-axis of the VR device coordinate system corresponds to the -X-axis of the robot arm coordinate system; the Y-axis of the VR device coordinate system corresponds to the Z-axis of the robot arm coordinate system; and the Z-axis of the VR device coordinate system corresponds to the Y-axis of the robot arm coordinate system. The 3×3 rotation matrix completes the mapping of the X, Y, and Z axes of the VR / OpenXR coordinate system to the robot coordinate system, enabling the spatial motion of the VR controller to accurately drive the corresponding motion direction of the robot's end effector in the real mechanical coordinate system, thereby ensuring the naturalness and consistency of the teleoperation experience.

[0056] Step S200: The host continuously reads the current status information of the gamepad device; the current status information includes: button status information and gamepad pose information in the virtual space coordinate system.

[0057] Specifically, button status information includes: squeeze, trigger, joystick operation, etc.

[0058] Understandably, when the operator operates the left and right handles of the controller device, the controller device with posture tracking capabilities tracks and obtains the handle pose information and captures the button status information. Furthermore, the VR remote collaboration component program in the controller device collects button status information and handle pose information in real time at a fixed sampling period and stores it in a posture buffer in shared memory. This allows the main control program in the host machine to synchronously read the current state information from the posture buffer, mapping the operator's gestures and control commands to the joint space of the physical dual-arm robotic arms, thus completing the real-time motion control of the robotic arms. In this way, high-precision mapping and low-latency response are achieved between the operator's hand movements and the movements of the robotic arm's end effector.

[0059] Step S300: When the button status information indicates that the control button of the handle device is triggered, the host performs first frame pose alignment processing and pose determination processing based on the handle pose information and button status information to determine the target pose of the robotic arm in the virtual space coordinate system.

[0060] According to some embodiments of this application, step S300 is further described. Step S300: First frame pose alignment processing and pose determination processing are performed based on handle pose information and button status information to determine the target pose of the robotic arm in the virtual space coordinate system, including but not limited to steps S310 to S340.

[0061] Step S310: When the button status information indicates that the control button is triggered for the first time, the host obtains and records the first initial matrix T_vr0 corresponding to the initial posture of the handle device in the virtual space coordinate system from the handle pose information; and simultaneously records the second initial matrix T_r0 corresponding to the current pose of the corresponding robotic arm in the real space mechanical coordinate system. Step S320: Each time the control button is triggered, align the handle device with the starting point of the robotic arm and obtain the current attitude matrix T_vr(t) of the handle device at time t. Step S330: Multiply the current attitude matrix T_vr(t) by the inverse of the first initial matrix T_vr0 to obtain the motion increment of the handle device; Step S340: Multiply the motion increment and the second initial matrix T_r0 to obtain the target pose of the robotic arm. .

[0062] Steps S310 to S340 achieve pose alignment and determine the target pose, laying the foundation for subsequent calculation of optimized joint angles to control the movement of the robotic arm. Furthermore, the alignment calibration between the handle device and the end effector of the robotic arm is triggered by a button press, preventing sudden movements or jumps in the robotic arm.

[0063] Specifically, the pose is aligned in the first frame after each button press to avoid drastic changes in the robot due to the setting of absolute position. This design performs button alignment each time the button is pressed (i.e., the starting point of VR and robot is consistent at the moment the user presses the button). When the operator presses the trigger button on the controller for the first time, the system records the pose of the VR controller in the virtual space in the current frame as the first initial matrix T_vr0 corresponding to the initial pose; simultaneously records the second initial matrix T_r0 corresponding to the current pose of the corresponding robotic arm end effector; and converts all subsequent VR controller movements into a change matrix relative to this initial pose, mapping it to the robotic arm end effector space. Specifically, the calculation formula is: The motion increment of the virtual reality device is ΔT=inv(T_vr0)*T_vr(t); where inv is the inverse matrix obtained by performing the inverse operation on the matrix; the target pose of the robotic arm is T_target=T_r0*ΔT.

[0064] Step S400: The host performs inverse kinematics calculation based on the current pose and the target pose to obtain the optimized joint angles.

[0065] According to some embodiments of this application, step S400 is further described, and step S400 includes, but is not limited to, steps S410 to S430.

[0066] Step S410: Calculate the pose error between the current pose and the target pose.

[0067] According to some embodiments of this application, step S410 includes, but is not limited to, the following steps: obtaining the first pose matrix corresponding to the current pose and the second pose matrix corresponding to the target pose; multiplying the second pose matrix by the inverse of the first pose matrix to obtain the pose error.

[0068] Specifically, the error calculation formula involved in step S410 is as follows: ; Where T(q) is the first pose matrix corresponding to the current pose, Ttarget is the second pose matrix corresponding to the target pose, and the pose error is... .

[0069] Step S420: Substitute the pose error into the preset joint increment optimization formula for calculation to obtain the optimized second joint increment.

[0070] According to some embodiments of this application, the joint increment optimization formula is as follows: ; in, It is the optimized second joint increment; It is the term that minimizes the joint increment; This is the end position error term; These are pre-configured weighting coefficients; It is the initial first joint increment to be optimized; It is the terminal total Jacobian matrix; This is the pose error. In the end-effector total Jacobian matrix J, the first three rows represent position, and the last three rows represent pose.

[0071] Understandably, this step uses the j(q) Jacobi matrix constructed by pinocchio to optimize the joint increment using CasADi.

[0072] It should be noted that the weighting coefficients can be adjusted.

[0073] It should be noted that the robotic arm motion control system of this application is also equipped with an optimization-based inverse kinematics solver, which implements steps S410 to S430. The inverse kinematics solver uses a nonlinear optimization method to calculate joint increments, thereby achieving accurate tracking of the target pose by the end effector. The solution method includes: using the joint increment vectors of the dual-arm robotic arm as the variables to be optimized; calculating the position Jacobian matrix of the end effector through a model; and constructing an objective function containing two terms: a joint increment minimization term and an end-effector position error term.

[0074] Step S430: Perform integral calculation based on the second joint increment to obtain the optimized joint angle.

[0075] The formula for the integral operation involved in step S430 is: ; This usually indicates a view on q. Integrate the points.

[0076] Nonlinear optimization solutions are achieved through steps S410 to S430. Compared with the traditional Jacobian inverse matrix method, this method has the following advantages: it is applicable to dual-arm robotic arms with arbitrary configurations; it can automatically find the optimal solution of joints in redundant degrees of freedom; it has higher numerical stability; and it can adjust the weight coefficients to achieve control precision adjustment.

[0077] This application, through steps S100 to S400, independently sets the URDF structure and uses pinocchio+CasADi to implement a general inverse kinematics method, which can automatically find the optimal joint solution in redundant degrees of freedom; it has higher numerical stability; and it can adjust the control precision according to the task weight.

[0078] Step S500: The host computer generates motion control commands based on the optimized joint angles to control the movement of the robotic arm.

[0079] According to some embodiments of this application, the robotic arm motion control system further includes: a camera device, which is communicatively connected to the host computer; the robotic arm motion control method further includes: the host computer acquiring real-time image data towards the robotic arm through the camera device; and the VR remote collaboration component program loading the real-time image data and displaying the real-time image data on a visual interface.

[0080] Furthermore, the virtual reality data interaction mechanism provided in the embodiments of this application will be further explained.

[0081] It is understood that the virtual reality interaction device of this application includes: a set of binocular virtual reality head-mounted display devices; a pair of controllers with posture tracking capabilities; a high-speed shared memory data channel; and an independently running posture acquisition thread.

[0082] In practice, operators wear binocular virtual reality headsets and use VR remote collaboration components to acquire and load real-time image data from the robotic arm's perspective, allowing them to observe the robotic arm's real-time status in the workspace from a first-person perspective. Real-time image data is transmitted using shared memory, ensuring low latency and high frame rate updates, enabling operators to intuitively perceive the robotic arm's real-time position, posture, and working environment.

[0083] During operation, the handgrip pose information and button status information generated when the operator operates the left and right handgrips are collected in real time by the VR remote collaboration component program at a fixed sampling period and stored in the pose buffer in shared memory. The host's main control program can synchronously read data from this buffer and map the operator's gestures and control commands to the joint space of the physical dual-arm robotic arms, completing the real-time motion control of the robotic arms. In this way, high-precision mapping and low-latency response between the operator's hand movements and the movement of the robotic arm's end effector are achieved.

[0084] Specifically, the data output by the VR remote collaboration component program includes: the 3D spatial position and pose matrix of the left and right controllers; the grasping action status of the left and right hands or controllers (such as pinch, squeeze, etc.); and the trigger buttons, squeeze buttons, joystick direction, and button status on the controllers. Simultaneously, the VR remote collaboration component program internally updates the device status in real time through an asynchronous event callback mechanism and combines this with a shared memory mechanism to achieve multi-process or multi-threaded data synchronization. This ensures that the pose and operational status of the virtual reality interactive device can be efficiently, continuously, and with low latency provided to the robot control system, enabling the operator to precisely control the physical robotic arm in the virtual environment.

[0085] In this embodiment, the robotic arm motion control system includes: a host computer, a robotic arm, a virtual reality interaction device, and a router; the host computer, robotic arm, and virtual reality interaction device are all communicatively connected to the router; the virtual reality interaction device includes: a binocular virtual reality head-mounted display and a controller, with control buttons on the controller; the virtual reality interaction device is configured with a VR remote collaboration component program; in this robotic arm motion control system, inter-process communication is directly utilized, avoiding the development using the ROS operating system; the VR remote collaboration component program is used for data acquisition and sharing of the virtual reality interaction device, making the overall communication of the robotic arm motion control system more flexible, more compatible, and with lower latency; through steps S100 to S500, during the process of using the robotic arm motion control system to control the motion of the robotic arm, firstly, the host computer starts the virtual reality interaction device and the robotic arm. Next, modeling is performed based on the actual joint angle and structural information of the robotic arm to obtain the virtual machine model of the robotic arm and its current pose in the virtual space coordinate system. Then, the host continuously reads the current state information of the handle device, including button state information and handle pose information in the virtual space coordinate system. Following this, when the button state information indicates that the control button of the handle device has been triggered, the host performs first-frame pose alignment and pose determination processing based on the handle pose information and button state information to determine the target pose of the robotic arm in the virtual space coordinate system. Then, the host performs inverse kinematics solving based on the current pose and the target pose to obtain optimized joint angles. Finally, the host generates motion control commands based on the optimized joint angles to control the movement of the robotic arm, achieving low-latency and high-stability motion control. In other words, this embodiment of the application can synchronously achieve low-latency and high-stability motion control of a robotic arm based on a virtual reality interactive device, balancing versatility and real-time performance during the teleoperation of the robotic arm.

[0086] It is important to emphasize that existing patented technologies use ROS for module communication, Unity for virtual simulation, and Movelt for control. In contrast, this application uses a web-based interface, WebSocket, and shared memory to build a lightweight teleoperation system, reducing system complexity and lowering the learning curve. The inverse kinematics calculations in this application are implemented using a self-developed system with Pinocchio and CasADi, rather than relying on MoveIt / BioIK, making it cross-platform, cross-structure, and Unity-free, resulting in extremely high adaptability for the robotic arm. Compared to teleoperation methods that incorporate exoskeleton-like devices, which require a one-to-one correspondence between arm joints and robotic arm joints, resulting in poor versatility, this application does not require exoskeleton technology. Therefore, the teleoperation method for the robotic arm is lower in cost and more versatile. This embodiment directly utilizes inter-process communication to avoid the development of a ROS operating system, achieving VR-based synchronous control of the robotic arm's two ends, characterized by low latency, high stability, and a good sense of immersion. Furthermore, the use of VR remote collaboration component programs to achieve data acquisition and sharing for VR devices offers greater flexibility, stronger compatibility, and lower latency.

[0087] like Figure 5 As shown, the present invention also provides an electronic device, comprising: The processor 501 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 502 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 502 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 502 and is called and executed by the processor 501 using the robotic arm motion control method of the embodiments of this application. The input / output interface 503 is used to implement information input and output; The communication interface 504 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 505 transmits information between various components of the device (e.g., processor 501, memory 502, input / output interface 503, and communication interface 504); The processor 501, memory 502, input / output interface 503, and communication interface 504 are connected to each other within the device via bus 505.

[0088] This application embodiment also provides a storage medium, which is a computer-readable storage medium, storing a computer program that, when executed by a processor, implements the above-described robotic arm motion control method.

[0089] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof. The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate, and may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0090] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0091] The above provides a detailed description of the preferred embodiments of this application. However, this application is not limited to the above-described embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application. All such equivalent modifications or substitutions are included within the scope defined by this application.

Claims

1. A method for controlling the motion of a robotic arm, characterized in that, This invention relates to a robotic arm motion control system, comprising: a host computer, a robotic arm, a virtual reality interaction device, and a router; the host computer, the robotic arm, and the virtual reality interaction device are all communicatively connected to the router; the virtual reality interaction device includes: a binocular virtual reality head-mounted display and a controller, the controller being equipped with control buttons; and a VR remote collaboration component program is configured in the virtual reality interaction device. After the host computer starts the virtual reality interaction device and the robotic arm, it performs modeling processing based on the actual joint angle information and structural information of the robotic arm to obtain the machine virtual model of the robotic arm and the current pose of the machine virtual model in the virtual space coordinate system. The host continuously reads the current status information of the gamepad device; the current status information includes: button status information and gamepad pose information in the virtual space coordinate system; When the button status information indicates that the control button of the handle device is triggered, the host performs first frame pose alignment processing and pose determination processing based on the handle pose information and the button status information to determine the target pose of the robotic arm in the virtual space coordinate system. The host performs inverse kinematics calculation based on the current pose and the target pose to obtain the optimized joint angles. The host computer generates motion control commands based on the optimized joint angles to control the movement of the robotic arm.

2. The robotic arm motion control method according to claim 1, characterized in that, The step of modeling based on the actual joint angle information and structural information of the robotic arm to obtain the virtual machine model of the robotic arm and the current pose of the virtual machine model in the virtual space coordinate system includes: When the robotic arm is started, the actual joint angle information of the robotic arm is obtained, and the structural information of the robotic arm is obtained from the URDF description file; The machine virtual model is constructed based on the structural information; The initial posture of the machine virtual model is determined based on the actual joint angle information; In the initial posture, the current pose of the virtual machine model is calculated using a forward kinematics algorithm; the current pose is used to indicate the pose of each joint of the robotic arm and the end effector in space.

3. The robotic arm motion control method according to claim 1, characterized in that, The step of performing first-frame pose alignment and pose determination processing based on the handle pose information and button status information to determine the target pose of the robotic arm in the virtual space coordinate system includes: When the button status information indicates that the control button is triggered for the first time, the host obtains and records the first initial matrix T_vr0 corresponding to the initial posture of the handle device in the virtual space coordinate system from the handle pose information; and simultaneously records the second initial matrix T_r0 corresponding to the current pose of the robotic arm in the real space mechanical coordinate system. Each time the control button is triggered, the handle device is aligned with the starting point of the robotic arm, and the current posture matrix T_vr(t) of the handle device at time t is obtained. Multiply the current attitude matrix T_vr(t) on the left by the inverse of the first initial matrix T_vr0 to obtain the motion increment of the handle device; The target pose of the robotic arm is obtained by multiplying the motion increment by the second initial matrix T_r0. .

4. The robotic arm motion control method according to claim 1, characterized in that, The host computer performs inverse kinematics calculations based on the current pose and the target pose to obtain optimized joint angles, including: Calculate the pose error between the current pose and the target pose; The pose error is substituted into a preset joint increment optimization formula for calculation to obtain the optimized second joint increment; The optimized joint angle is obtained by performing an integral operation based on the second joint increment.

5. The robotic arm motion control method according to claim 4, characterized in that, The calculation of the pose error between the current pose and the target pose includes: Obtain the first pose matrix corresponding to the current pose and the second pose matrix corresponding to the target pose; The pose error is obtained by left-multiplying the second pose matrix by the inverse of the first pose matrix.

6. The robotic arm motion control method according to claim 4, characterized in that, The joint increment optimization formula is as follows: ； in, It is the optimized second joint increment; It is the term that minimizes the joint increment; This is the end position error term; These are pre-configured weighting coefficients; It is the initial first joint increment to be optimized; It is the terminal total Jacobian matrix; It is the pose error.

7. The robotic arm motion control method according to claim 1, characterized in that, The robotic arm motion control system further includes: a camera device, which is communicatively connected to the host computer; the method further includes: The host computer acquires real-time image data towards the robotic arm via the camera device; The VR remote collaboration component program loads the real-time image data and displays the real-time image data on a visual interface.

8. A motion control system for a robotic arm, characterized in that, include: The robotic arm motion control system includes: a host computer, a robotic arm, a virtual reality interaction device, and a router; the host computer, the robotic arm, and the virtual reality interaction device are all communicatively connected to the router; the virtual reality interaction device includes: a binocular virtual reality head-mounted display and a controller device, the controller device being provided with control buttons; in the robotic arm motion control system, the host computer, the robotic arm, the virtual reality interaction device, and the router cooperate with each other to realize the robotic arm motion control method as described in any one of claims 1 to 7.

9. An electronic device, characterized in that, It includes at least one processor and a memory for communicatively connecting to the at least one processor; the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform the robotic arm motion control method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the robotic arm motion control method as described in any one of claims 1 to 7.

Images