Mechanical arm assembling method and system based on hand-eye collaborative virtual reality technology

Through hand-eye collaborative virtual reality technology, virtual reality equipment and helmets are combined with robotic arms, solving the complexity and inefficiency of traditional robotic arm control systems, achieving more accurate and efficient robotic arm assembly operations.

CN118906056APending Publication Date: 2024-11-08SHANGHAI MARITIME UNIVERSITY
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411022780.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Traditional robotic arm control systems are complex and inflexible, limiting their application in the assembly field, resulting in complex operation and inefficient efficiency.

Method used

The robotic arm assembly method based on hand-eye collaborative virtual reality technology is adopted to obtain operation control information through virtual reality equipment, the computer issues robotic arm instructions, and the virtual reality helmet supervises the robotic arm movement to achieve more precise and natural control.

Benefits of technology

It improves the control accuracy and operating efficiency of the robotic arm, reduces the risk of human errors and safety accidents, simplifies the operation process, and adapts to different working conditions and equipment scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118906056A_ABST
    Figure CN118906056A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a mechanical arm assembling method and system based on a hand-eye collaborative virtual reality technology. The method comprises the steps that a virtual reality environment is established; acquiring motion information through a handle; the motion information is processed, and a corresponding motion instruction is issued to the mechanical arm through a computer; mapping the motion information of the mechanical arm into the helmet; and the mechanical arm is controlled to grab the materials and assemble the materials. The virtual reality technology and the mechanical arm intelligent control technology are combined, and therefore the problems that under the current virtual reality environment, the mechanical arm control accuracy is low, and supervision of the mechanical arm motion working condition is lacked are solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of robot arm assembly, and in particular to a robot arm assembly method and system based on hand-eye collaborative virtual reality technology. Background Art

[0002] In modern manufacturing, the application of robotic arms has penetrated into various fields, and they play an important role in improving production efficiency, ensuring product quality, and reducing labor costs. However, traditional robotic arm control systems usually rely on complex programming or operation through computers and consoles, which not only limits the flexibility and adaptability of robotic arms, but also increases the difficulty of operation, affecting their application in a wider range of fields.

[0003] With the rapid development of virtual reality technology, people have begun to explore new ways to combine virtual reality technology with robotic arm control. Virtual reality technology enables users to interact with the robotic arm in a more intuitive and natural way by providing an immersive interactive environment. In particular, by using a virtual reality helmet and handle, users can observe the operating environment of the robotic arm in real time and control the movements of the robotic arm with gestures, thereby achieving more flexible and precise control.

[0004] In the specific field of assembly, traditional assembly methods may face problems such as complex operation and low efficiency. In order to solve these problems, the present invention proposes a robot arm assembly method and system based on hand-eye collaborative virtual reality technology. Summary of the invention

[0005] The purpose of the present invention is to solve the problems raised in the above background technology. The present invention provides a method and system for assembling a robot arm based on hand-eye collaborative virtual reality technology. The operation control information is obtained through the virtual reality device. After receiving the control information of the handle, the computer issues instructions to the robot arm to control the robot arm to complete the corresponding operation. At the same time, the movement of the robot arm is supervised through the helmet, and finally the robot arm is controlled by the virtual reality device for assembly. The present invention combines virtual reality technology with robot arm intelligent control technology, thereby improving the problems of low robot arm control accuracy and lack of supervision of robot arm movement conditions in the current virtual reality environment.

[0006] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0007] A method for assembling a robotic arm based on hand-eye collaborative virtual reality technology, comprising:

[0008] Step 1: Create a virtual reality environment;

[0009] Step 2: Get motion information through the handle;

[0010] Step 3: Process the motion information and issue corresponding motion instructions to the robotic arm through the computer;

[0011] Step 4: Map the motion information of the robotic arm to the helmet;

[0012] Step 5: Control the robotic arm to grab the material and assemble it.

[0013] An embodiment of the present invention further discloses a robot arm assembly system based on hand-eye collaborative virtual reality technology, including a virtual reality device, a middleware, an industrial camera, a scene-related device and a robot arm, wherein the virtual reality device is connected to the middleware, the industrial camera and the scene-related device, the middleware is connected to the industrial camera, the scene-related device and the robot arm, and the robot arm is connected to the industrial camera and the scene-related device, and the middleware includes a virtual reality environment module and a robot arm motion environment module, wherein:

[0014] The virtual reality environment module includes a computer and a virtual scene model, the computer is connected to left and right handle controllers and a head mounted display, the left and right handle controllers are connected to a client, the virtual scene model is connected to a model parsing module, and the model parsing module is connected to the head mounted display and the client;

[0015] The robot arm motion environment module includes a robot arm motion controller, and the robot arm motion controller is connected to the robot arm.

[0016] Beneficial effects:

[0017] 1. The immersive environment and intuitive handle operation provided by virtual reality enable the operator to control the robot arm more accurately. By processing the trajectory information, the operation difficulty is reduced and the accuracy is improved. It can not only improve working conditions, reduce labor intensity, and reduce production costs, but also greatly improve production efficiency and quality, and reduce safety accidents caused by human negligence.

[0018] 2. The virtual reality helmet can reflect the real-time motion information and scenes of the robotic arm, thereby assisting in the supervision of the robotic arm assembly work and improving the accuracy of the robotic arm's remote teleoperation assembly work.

[0019] 3. Simplified operation process and real-time feedback mechanism significantly reduce operation time and improve production efficiency. Through the virtual reality helmet, the operator can get an immersive experience, improve the naturalness and comfort of operation, and enhance the user experience. The operator can control the robot arm in a safe environment, reducing the risk of human error and accidents, enhancing safety, and can adapt to different working conditions and working scenarios under different equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 This is a schematic diagram of a robotic arm assembly method based on hand-eye collaborative virtual reality technology according to an embodiment of the present invention;

[0022] Figure 2 This is a schematic diagram of a robotic arm assembly system based on hand-eye collaborative virtual reality technology according to an embodiment of the present invention;

[0023] Figure 3 This is a schematic diagram of an assembly process of an embodiment of the present invention;

[0024] Figure 4 This is a diagram of trajectory information filtering results according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0026] The embodiment of the present invention discloses a method for assembling a mechanical arm based on hand-eye collaborative virtual reality technology, comprising:

[0027] S1: Establishing virtual reality environment;

[0028] S2: Get motion information through the handle;

[0029] S3: Process motion information and issue corresponding motion instructions to the robot arm through the computer;

[0030] S4: Mapping the motion information of the robotic arm to the helmet;

[0031] S5: Control the robotic arm to grab materials and assemble them.

[0032] The specific steps of establishing the virtual reality environment in step 1 include:

[0033] Step 1.1: Basic equipment connection: connect the virtual reality device to the computer, and the computer compiles the basic virtual reality scene to create the basic picture of the virtual reality environment helmet;

[0034] Step 1.2: The scene model is established. The three-dimensional model of the robotic arm and the surrounding environment model are arranged. The operator sees the real-time rendering image of the robotic arm through the helmet and operates it through the handle;

[0035] Step 1.3: User interaction design: develop an intuitive user interface, integrate it into the VR helmet, display the robot status, control options and feedback information, integrate a control panel into the VR interface to adjust the robot parameters and execute special commands, and establish a basic VR environment.

[0036] Step 2 is based on the virtual reality environment and obtains the user's control information through the virtual reality handle to perform real-time mapping of motion control information. Obtaining motion information includes the following steps:

[0037] Step 2.1: Action mapping, mapping the physical movement and rotation of the handle to the corresponding action of the robotic arm. For example, pushing the handle forward moves the robotic arm forward, and rotating the handle rotates the robotic arm around a specific axis.

[0038] Step 2.2: Button Functions. The buttons on the compile handle are used to perform specific functions, such as opening and closing the gripper, emergency stop, or mode switching.

[0039] Step 2.3: Synchronous acquisition of motion information. The operator sends the motion control information intention of the robot arm by moving the handle, and the system obtains the trajectory information of the handle and processes it.

[0040] Step 3 proposes to process the acquired handle motion information and issue corresponding motion instructions to the robotic arm through a computer. The specific steps include:

[0041] Step 3.1: Spatial posture synchronization: convert the spatial coordinate system of the robot arm in different environments so that the motion information of the robot and the operator are synchronized in the spatial coordinate system;

[0042] Step 3.2: Based on the kinematics solution in the virtual reality environment, the robot's motion control is performed by controlling the position of the robot's end effector. The inverse kinematics is used to reverse the angles of each joint of the robot through the position and posture of the robot's end effector, thereby solving the angles of each joint of the robot;

[0043] Step 3.3: Trajectory information filtering. Kalman filtering is used to reduce the impact of jitter on the handle position information, so as to obtain a smooth motion trajectory, reduce the processing time of the robot, and improve the smoothness of the robot's synchronous handle controller operation.

[0044] Step 3.4: Solve the control information of the robot arm, communicate with the computer and the robot arm to realize the control of the robot by the virtual reality device, connect the virtual reality environment and the robot arm environment, and then realize the control of the robot in the robot arm control system.

[0045] Step 4 is used to map the motion information of the robotic arm to the helmet. The specific steps include:

[0046] Step 4.1: Scene switching: Switch different working scenes of the robot in the virtual reality environment, and realize the motion monitoring in different scenes through the interaction between the handle and the helmet;

[0047] Step 4.2: Map the virtual assembly process and call the images in the robot motion simulation software to display them in the virtual reality environment to achieve more intuitive visualization. By capturing the output of the simulation information as a video stream and transmitting it in the virtual reality environment, an immersive visualization display of the robot motion status at the software level can be achieved;

[0048] Step 4.3: Map the virtual-real assembly process, combine the on-site monitoring in the real scene with the robot motion state interface in the simulation software scene, so as to realize the visualization of the assembly process in the virtual-real environment of robot motion, and be able to simultaneously detect the system's working environment and the robot's motion state.

[0049] Step 5 is used to control the robot arm to grab the target such as a blade and assemble it. The specific steps include:

[0050] Step 5.1: The robot moves. After receiving the control information from the handle, the target posture of the robot, i.e., the point movement of the robot, is determined by the joint rotation angle solved by the inverse kinematics of the robot. The robot reaches the target point through the movement of the end effector, and the robot starts to execute the control movement of the handle to grab the material.

[0051] Step 5.2: The gripping of the gripper, the opening and closing of the gripper is realized by compiling the handle button, and the handle button is operated when the blade and the turntable are reached to clamp and release the blade;

[0052] Step 5.3: Real-time display of the robot arm's motion information. The virtual display helmet displays the robot arm's motion information and the actual working scene position in real time.

[0053] When filtering the trajectory information, the extended Kalman filter method is used to process the trajectory information. The processing steps of the extended Kalman filter are as follows:

[0054] (1) Initialization: Define the state vector x and the state covariance matrix P. The state vector contains the internal state of the system. Define the process noise covariance matrix Q and the observation noise covariance matrix R, which represent the noise levels in the process model and the observation model, respectively.

[0055] (2) Linearization: At each time step, the EKF linearizes the nonlinear system by calculating the Jacobian matrix of the state transfer function and the observation function. The Jacobian matrix is ​​the derivative matrix of the first-order Taylor expansion of the function around the current estimation point.

[0056] (3) Prediction step: Use the linearized state transition model to predict the state of the next time step:

[0057] x_pred=f(x,u)

[0058] Where f is the linearized state transfer function, x is the current state, and u is the control input;

[0059] The updated state covariance matrix is:

[0060] P_pred=F*P*F^T+Q

[0061] Where F is the Jacobian matrix of the state transition model;

[0062] (4) Update steps:

[0063] Use the linearized observation model to predict observations:

[0064] z_pred=h(x)

[0065] Where h is the Jacobian matrix of the observation function;

[0066] Calculate the Kalman gain:

[0067] K=P_pred*H^T*(H*P_pred*H^T+R)^(-1)

[0068] where H is the Jacobian matrix of the observation model,

[0069] Update the state estimate:

[0070] x=x_pred+K*(z-z_pred)

[0071] Where z is the actual observation value, update the state covariance matrix:

[0072] P=(IK*H)*P_pred

[0073] Where I is the identity matrix;

[0074] (5) Iteration: Repeat the prediction and update steps to process each new observation.

[0075] According to the process, the state transition model captures the dynamic changes of the handle position over time, and the observation model associates the real-time position readings of the handle. The state vector x in the state transition model contains the position and velocity information of the handle in three-dimensional space, that is:

[0076] x=[p x p y p z v x v y v z ] T .

[0077] When the computer receives the operator's control command, it processes it according to the following logic:

[0078] First, the button command is identified. If a unique command can be identified, the command is directly executed;

[0079] If the controller button data corresponds to multiple commands, the system will instead identify the helmet command and execute the command corresponding to the helmet interaction;

[0080] If the helmet interaction also corresponds to multiple commands, the system will output the command that matches both the handle and the helmet and execute it;

[0081] If there is a conflict or uncertainty in instructions, the system will prompt the operator to confirm and execute the corresponding instructions based on the operator's choice.

[0082] An embodiment of the present invention further discloses a robot arm assembly system based on hand-eye collaborative virtual reality technology, including a virtual reality device 1, a middleware 2, an industrial camera 3, a scene-related device 4 and a robot arm 5, wherein the virtual reality device 1 is connected to the middleware 2, the industrial camera 3 and the scene-related device 4, the middleware 2 is connected to the industrial camera 3, the scene-related device 4 and the robot arm 5, the robot arm 5 is connected to the industrial camera 3 and the scene-related device 4, the middleware 2 includes a virtual reality environment module and a robot arm motion environment module, wherein:

[0083] The virtual reality environment module includes a computer and a virtual scene model, the computer is connected to left and right handle controllers and a head mounted display, the left and right handle controllers are connected to a client, the virtual scene model is connected to a model parsing module, and the model parsing module is connected to the head mounted display and the client;

[0084] The robot arm motion environment module includes a robot arm motion controller, and the robot arm motion controller is connected to the robot arm.

[0085] The present invention uses the position information of the handle of the virtual reality device to control the robot arm. Considering that controlling the angles of each joint of the robot arm will increase the complexity and response time of the system, other joints are driven to perform motion transformation after inverse kinematics calculation. More precise control accuracy can be achieved through point-to-point control of the handle and the end effector, while also improving the usability of the system. The handle device can provide the operator with position information and command information, and other operations can also be achieved in the system through button interaction of the handle, such as opening and closing the gripper.

[0086] The virtual reality scene can capture the images and information of the simulation software in the robotic arm operating platform and display them in the developed virtual scene. At the same time, the system can connect to other infrastructure for visualization optimization. For example, by connecting to an industrial camera in an actual scene, the camera's monitoring screen can be directly displayed in the scene in the virtual environment, thereby realizing the simultaneous observation of the robot's actual movement and simulated movement in the virtual scene, as well as various data information during movement.

[0087] The present invention sets the logic of controlling the handle position information to start recording the handle motion information and converting it into a motion control signal when it detects that the button for controlling the position application is pressed. In addition, based on the virtual scene arrangement of the game engine Unity, the motion logic of each component in the virtual scene of the system can be closer to the actual scene. The forward and inverse kinematics settings and binding of the robot can make the robot in the virtual scene have the same motion logic as the actual robot. The operator can do things in the live scene in the virtual scene. At the same time, in some dangerous areas of the actual scene, the operator will also mark them out in the virtual scene, but the operator will not be prohibited from accessing this area, because considering the simulation effect in the offline scene, so when some movements are wrong and problems occur, the operator can intuitively check the problems of the equipment.

[0088] The present invention is different from the single control and display of ordinary data on a single screen in ordinary human-machine collaboration. The system can provide more diversified operation interactions and intuitive data displays. The interactivity of the system can provide more convenient and effective control and combination of some common operations or displays. In addition to the functions of the system itself, it is also relatively convenient to continue to develop other functions on this basis, including motion control and data visualization, monitoring and other functions commonly used in blade assembly.

[0089] This system visualizes the operation page, and uses menus and prompt boxes to simplify the operation interface. In addition to restoring the physical scene and some professional debugging and simulation functions as much as possible, other operations and functions are simplified to reduce the difficulty and threshold of operation, thereby greatly reducing the scope of application of the system. The operator only needs to be familiar with a few buttons and different functions corresponding to the interface to quickly access the virtual scene, perform offline scene simulation debugging, and directly control the actual robot.

[0090] This system uses a stable and effective communication protocol to ensure that the communication interface and data transmission process used in the system are linear. The operator can transmit models and different control instruction information in different scenarios through the default framework of the system, and still maintain accurate and effective communication levels. Even for other types of robots or other application scenarios and work tasks, the system can provide good compatibility for operators to develop and set up, which also greatly reduces the operator's learning time on the robot system platform. At the same time, the computer-based interface can provide good compatibility for the system to be combined with other software and hardware. In addition to virtual reality devices, computers and robots as the core hardware of the system, it can also connect and communicate with multi-platform devices and hardware, thereby improving the scalability and compatibility of the system, and can be combined with industrial cameras or other expansion components of virtual reality devices.

[0091] The present invention first creates and starts a virtual reality environment. The operator enters the preset virtual environment by wearing a virtual reality helmet. In the virtual reality interface, specific motion control information is mapped to the action of the robotic arm through a handle and a helmet device. The operator uses the virtual reality handle to perform the movement. The system recognizes the information and converts it into a control instruction of the robotic arm. The operator can clearly see the position of the target by using the display in the virtual reality helmet and accurately locate it by using the handle. The operator controls the robotic arm and the gripper to grab the target by using the handle and places it at a designated assembly position in the virtual environment.

[0092] The handle controller compiles the system functions in advance. When the operator executes these button commands in the virtual environment, the system will recognize and control the robotic arm and gripper to perform corresponding actions.

[0093] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A robotic arm assembly method based on hand-eye collaborative virtual reality technology, It is characterized in that include: Step 1: Create a virtual reality environment; Step 2: Get motion information through the handle; Step 3: Process the motion information and issue corresponding motion instructions to the robotic arm through the computer; Step 4: Map the motion information of the robotic arm to the helmet; Step 5: Control the robotic arm to grab the material and assemble it.

2. According to claim 1, a robotic arm assembly method based on hand-eye collaborative virtual reality technology, It is characterized in that The specific steps of establishing the virtual reality environment in step 1 include: Step 1.1: Basic equipment connection: connect the virtual reality device to the computer, and the computer compiles the basic virtual reality scene to create the basic picture of the virtual reality environment helmet; Step 1.2: The scene model is established. The three-dimensional model of the robotic arm and the surrounding environment model are arranged. The operator sees the real-time rendering image of the robotic arm through the helmet and operates it through the handle; Step 1.3: User interaction design: develop an intuitive user interface, integrate it into the VR helmet, display the robot status, control options and feedback information, integrate a control panel into the VR interface to adjust the robot parameters and execute special commands, and establish a basic VR environment.

3. According to claim 1, a method for assembling a robotic arm based on hand-eye collaborative virtual reality technology, It is characterized in that Step 2 is based on the virtual reality environment and obtains the user's control information through the virtual reality handle to perform real-time mapping of motion control information. Obtaining motion information includes the following steps: Step 2.1: Action mapping, mapping the physical movement and rotation of the handle to the corresponding action of the robotic arm; Step 2.2: Button functions, the buttons on the compilation handle are used to perform specific functions; Step 2.3: Synchronous acquisition of motion information. The operator sends the motion control information intention of the robot arm by moving the handle, and the system obtains the trajectory information of the handle and processes it.

4. According to claim 1, a method for assembling a robotic arm based on hand-eye collaborative virtual reality technology, It is characterized in that The specific steps of step 3 include: Step 3.1: Spatial posture synchronization: convert the spatial coordinate system of the robot arm in different environments so that the motion information of the robot and the operator are synchronized in the spatial coordinate system; Step 3.2: Based on the kinematics solution in the virtual reality environment, the robot's motion control is performed by controlling the position of the robot's end effector. The inverse kinematics is used to reverse the angles of each joint of the robot through the position and posture of the robot's end effector, thereby solving the angles of each joint of the robot; Step 3.3: Trajectory information filtering: Kalman filtering is used to reduce the impact of jitter on the handle position information to obtain a smooth motion trajectory; Step 3.4: Solve the control information of the robotic arm, communicate with the computer and the robotic arm to realize the control of the robot by the virtual reality device, connect the virtual reality environment and the robotic arm environment, and then realize the control of the robot in the robotic arm control system.

5. According to claim 1, a method for assembling a robotic arm based on hand-eye collaborative virtual reality technology, It is characterized in that The specific steps of step 4 include: Step 4.1: Scene switching: Switch different working scenes of the robot in the virtual reality environment, and realize the motion monitoring in different scenes through the interaction between the handle and the helmet; Step 4.2: Map the virtual assembly process, call the images in the robot motion simulation software to display in the virtual reality environment, and realize the immersive visualization display of the robot motion status at the software level by capturing the output of the simulation information as a video stream and transmitting it in the virtual reality environment; Step 4.3: Map the virtual-real assembly process, combine the on-site monitoring in the real scene with the robot motion state interface in the simulation software scene, realize the visualization of the assembly process in the virtual-real environment of robot motion, and detect the working environment of the system and the motion state of the robot at the same time.

6. The method for assembling a robotic arm based on hand-eye collaborative virtual reality technology according to claim 1, It is characterized in that The specific steps of step 5 include: Step 5.1: The robot moves. After receiving the control information from the handle, the target posture of the robot, i.e., the point movement of the robot, is determined by the joint rotation angle solved by the inverse kinematics of the robot. The robot reaches the target point through the movement of the end effector, and the robot starts to execute the control movement of the handle to grab the material. Step 5.2: The gripper is opened and closed by compiling the handle button, and the gripper is clamped and placed by operating the handle button when the material is at the material position; Step 5.3: Real-time display of the robot arm's motion information. The virtual display helmet displays the robot arm's motion information and the actual working scene position in real time.

7. The method for assembling a robotic arm based on hand-eye collaborative virtual reality technology according to claim 4, It is characterized in that When filtering the trajectory information, the extended Kalman filter method is used to process the trajectory information. The processing steps of the extended Kalman filter are as follows: (1) Initialization: Define the state vector x and the state covariance matrix P. The state vector contains the internal state of the system. Define the process noise covariance matrix Q and the observation noise covariance matrix R, which represent the noise levels in the process model and the observation model, respectively. (2) Linearization: At each time step, the EKF linearizes the nonlinear system by calculating the Jacobian matrix of the state transfer function and the observation function. The Jacobian matrix is ​​the derivative matrix of the first-order Taylor expansion of the function around the current estimation point. (3) Prediction step: Use the linearized state transition model to predict the state of the next time step: x_pred=f(x,u) Where f is the linearized state transfer function, x is the current state, and u is the control input; The updated state covariance matrix is: P_pred=F*P*F^T+Q Where F is the Jacobian matrix of the state transition model; (4) Update steps: Use the linearized observation model to predict observations: z_pred=h(x) Where h is the Jacobian matrix of the observation function; Calculate the Kalman gain: K=P_pred*H^T*(H*P_pred*H^T+R)^(-1) where H is the Jacobian matrix of the observation model, Update the state estimate: x=x_pred+K*(z-z_pred) Where z is the actual observation value, update the state covariance matrix: P=(IK*H)*P_pred Where I is the identity matrix; (5) Iteration: Repeat the prediction and update steps to process each new observation.

8. The method for assembling a robotic arm based on hand-eye collaborative virtual reality technology according to claim 7, It is characterized in that According to the process, the state transition model captures the dynamic changes of the handle position over time, and the observation model associates the real-time position readings of the handle. The state vector x in the state transition model contains the position and velocity information of the handle in three-dimensional space, that is: x=[p x p y p z v x v y v z ] T 。 9. The method for assembling a robotic arm based on hand-eye collaborative virtual reality technology according to claim 1, It is characterized in that When the computer receives the operator's control command, it processes it according to the following logic: First, the button command is identified. If a unique command can be identified, the command is directly executed; If the controller button data corresponds to multiple commands, the system will instead identify the helmet command and execute the command corresponding to the helmet interaction; If the helmet interaction also corresponds to multiple commands, the system will output the command that matches both the handle and the helmet and execute it; If there is a conflict or uncertainty in instructions, the system will prompt the operator to confirm and execute the corresponding instructions based on the operator's choice.

10. A robotic arm assembly system based on hand-eye collaborative virtual reality technology, It is characterized in that It includes virtual reality equipment, middleware, industrial camera, scene-related equipment and robotic arm. The virtual reality equipment is connected with the middleware, industrial camera and scene-related equipment. The middleware is connected with the industrial camera, scene-related equipment and robotic arm. The robotic arm is connected with the industrial camera and scene-related equipment. The middleware includes a virtual reality environment module and a robotic arm motion environment module, wherein: The virtual reality environment module includes a computer and a virtual scene model, the computer is connected to left and right handle controllers and a head mounted display, the left and right handle controllers are connected to a client, the virtual scene model is connected to a model parsing module, and the model parsing module is connected to the head mounted display and the client; The robot arm motion environment module includes a robot arm motion controller, and the robot arm motion controller is connected to the robot arm.

Citation Information

Patent Citations

  • Method and system for controlling virtual reality device based on intelligent wearable device

    CN106371602A

  • Multi-sensor handle controller hybrid tracking method and device

    CN111949123A

  • Optimization control method for motion trail of bionic mechanical arm

    CN118003324A

  • An enhanced reality underwater maintenance syestem by using a virtual reality manipulator (VRM)

    WO2020094205A1