A force feedback based robotic arm teleoperation training method

Abstract

The application discloses a kind of mechanical arm remote operation training methods based on force feedback, which is built by scene simulation platform Unity including hot room virtual environment of mechanical arm model and operating object, and joint position is configured according to real mechanical arm DH parameter;Dynamics simulation platform CoppeliaSim builds corresponding dynamics model, and collision detection function is configured inverse kinematics solution;Integrated force feedback device, construct workspace mapping model conversion hand movement as mechanical arm end desired pose;Two platforms synchronize data through dynamic link library, object control and force sensation calculation module algorithm collision force, bolt installation and disassembly torque and are transmitted to force feedback device.The application does not need to rely on real mechanical arm, avoids safety risk and high cost, improves training repeatability and pertinence, enhances immersion and operation precision through force sensation and visual synchronous feedback, shortens training cycle, and is suitable for fine operation training of mechanical arm in dangerous environment.

Description

Technical Field

[0001] This invention relates to the fields of force-tactile simulation, robot teleoperation, and virtual training systems, specifically to a force feedback-based robotic arm teleoperation training method. Background Technology

[0002] Training for operating robotic arms in hazardous environments such as high-temperature radiation and confined spaces often improves safety and effectiveness by combining virtual simulation with force feedback. These technologies have been applied in aerospace, industrial robotic arm training, and other scenarios.

[0003] Currently, existing solutions that are similar to the technical solution of this invention include disclosed patented technologies and traditional training methods. Among them, the traditional hot chamber robotic arm training method mainly relies on the downtime in the production process to allow operators to directly control the real robotic arm for training. Although this method allows operators to come into contact with real equipment, the downtime is short, and if the operator's skills are not proficient and the robotic arm collides, it may not only cause equipment damage, but also increase maintenance and training costs. At the same time, it is difficult to realistically reproduce diverse operating environments, and the training has poor relevance and repeatability.

[0004] The published patent CN106504605A proposes a simulation control system for force feedback teleoperation training of a space station robotic arm. This system combines a force feedback handle with a virtual simulation platform to realize inverse kinematics, path planning, collision detection, and force feedback output of the space station robotic arm. Its advantage is that it can conduct virtual operation training before missions, improving astronauts' proficiency with the space station robotic arm. However, the system is mainly designed for aerospace application scenarios, the construction of the virtual environment is relatively crude, the visual immersion is limited, and it fails to simulate redundant robotic arms such as ten joints. The force feedback form is also relatively simple, only reflecting basic collision resistance, and it fails to achieve realistic force interaction for complex operation tasks such as bolt tightening.

[0005] Another published patent, CN110039561A, proposes a point cloud-based training system for remote operators of live-line working robots. This system includes a depth camera, an industrial computer, and a main robotic arm telescopic joystick with force feedback. It can quickly reconstruct the actual working scene in three dimensions, giving trainees a sense of force presence, improving training effectiveness, and avoiding direct operation that could damage the actual robotic arm. However, the force feedback of this system is still relatively simple, and it does not involve the dual-platform coupling of high-fidelity scene rendering and dynamics solution platform. It cannot achieve a more realistic simulation effect through dual-platform collaboration, making it difficult to meet the needs of training in dangerous environments for high immersion and realistic force feedback.

[0006] In summary, existing technologies related to remote operation training of robotic arms generally suffer from problems such as high cost due to reliance on physical equipment and safety risks, or insufficient immersion in the virtual environment of virtual simulation systems, limited force feedback, lack of simulation for redundant robotic arms, and lack of dual-platform coupling to improve simulation effects. These issues make it difficult to meet the requirements of safety, high fidelity, and repeatability for robotic arm operation training in dangerous environments. Summary of the Invention

[0007] The purpose of this invention is to provide a force feedback-based remote operation training method for robotic arms, which solves the problems of traditional training relying on physical equipment, posing safety risks, and making it difficult to realistically reproduce the operating environment.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] In a first aspect, the present invention provides a method for training teleoperation of a robotic arm based on force feedback, comprising the following steps:

[0010] Modeling and visual rendering are performed based on real hot chamber scene data, and virtual environment data of the hot chamber containing robotic arms and manipulated objects are generated on the scene simulation platform.

[0011] Based on the data of the robotic arm and the object being manipulated in the virtual environment data of the hot chamber, dynamic modeling is performed, and kinematic solving and collision detection functions are configured on the dynamic simulation platform to generate kinematic and dynamic simulation scene data of the robotic arm.

[0012] The force feedback device data and the virtual environment data of the hot chamber are integrated and processed to construct a workspace mapping model between the force feedback device and the virtual robotic arm, and the force feedback device posture data is converted to generate the workspace mapping model and control algorithm data.

[0013] The interaction interface between the scene simulation platform and the dynamics simulation platform is encapsulated to generate a dynamic link library. Operation data and simulation data are transmitted through the dynamic link library to obtain synchronized data between the two platforms.

[0014] Force model calculations are performed on the interaction information between the robotic arm and the virtual environment in the dual-platform synchronized data to generate force feedback data. The force feedback data is then transmitted to the force feedback device to obtain the robotic arm teleoperation training interaction results.

[0015] In one possible implementation, the modeling and visual rendering based on real hot chamber scene data includes: performing 3D modeling processing on the structural data, tooling data, and object data of the real hot chamber to obtain basic scene model data; configuring the robot arm joint parameters in the basic scene model data so that the rotation center and movement axis of the robot arm joints conform to the real robot arm parameters to obtain precise robot arm model data; and performing high-fidelity visual rendering processing on the basic scene model data and the precise robot arm model data to generate the virtual environment data of the hot chamber.

[0016] In one possible implementation, the dynamic modeling based on the robotic arm and manipulated object data in the hot chamber virtual environment data includes: performing corresponding modeling processing on the robotic arm and manipulated object data in the hot chamber virtual environment data to obtain basic dynamic model data; configuring a robotic arm inverse kinematics solving module on the basic dynamic model data to obtain robotic arm motion control data; configuring collision detection attributes on the robotic arm end effector and manipulated object in the basic dynamic model data to generate the robotic arm kinematics and dynamics simulation scene data.

[0017] In one possible implementation, the integration processing of the force feedback device data and the virtual environment data of the hot chamber includes: parsing the operator's hand position and posture data collected by the force feedback device to obtain raw hand movement data; performing mapping operations on the raw hand movement data and the virtual robotic arm workspace data, using an offset-based variable-scale mapping method to obtain the expected position data of the robotic arm end effector; performing attitude transformation processing on the handle rotation angle data of the force feedback device end effector to convert it into Euler angles to obtain the expected attitude data of the robotic arm end effector; and integrating the expected position data and the expected attitude data to generate the workspace mapping model and control algorithm data.

[0018] In one possible implementation, the offset-based variable-scale mapping method includes: performing velocity calculation processing on the position coordinate data in the original hand motion data to obtain force feedback position velocity data; comparing the force feedback position velocity data with a preset movement threshold; if it is greater than the threshold, calculating the offset to obtain the robotic arm end-effector position offset data; superimposing the current frame end-effector position data of the robotic arm with the position offset data to obtain the next frame end-effector position coordinate data of the robotic arm; if it is less than or equal to the threshold, keeping the current frame end-effector position data of the robotic arm unchanged, and generating the expected end-effector position data of the robotic arm.

[0019] In one possible implementation, the encapsulation of the interaction interface between the scene simulation platform and the dynamics simulation platform includes: extracting and processing the interface functions of the two platforms to obtain interface function set data; encapsulating the interface function set data to generate the dynamic link library; transmitting the operation data of the force feedback device to the dynamics simulation platform through the dynamic link library, and obtaining the joint angle data of the robotic arm through inverse kinematics solution; and transmitting the joint angle data, the pose data of the manipulated object, and the collision data back to the scene simulation platform through the dynamic link library to generate the dual-platform synchronization data.

[0020] In one possible implementation, the force perception model calculation of the robotic arm-virtual environment interaction information in the dual-platform synchronous data includes: extracting and processing collision depth data and collision object elastic coefficient data from the dual-platform synchronous data to obtain collision parameter data; performing a product operation on the collision parameter data to obtain collision force perception data; extracting and processing bolt screwing angle data, initial torque data, and torque coefficient data from the dual-platform synchronous data to obtain bolt operation parameter data; performing calculations on the bolt operation parameter data, subtracting the product of the torque coefficient and the screwing angle from the initial torque to obtain bolt installation and removal torque data; and integrating the collision force perception data and the bolt installation and removal torque data to generate the force perception feedback data.

[0021] In one possible implementation, the extraction and processing of collision depth data and collision object elastic coefficient data in the dual-platform synchronization data includes: detecting and processing the contact point data between the robotic arm and the virtual object in the dual-platform synchronization data to obtain contact position data; performing depth calculation processing on the contact position data to obtain the collision depth data; performing parameter configuration processing on the material property data of the collision object to obtain the collision object elastic coefficient data; and performing a multiplication operation on the collision depth data and the elastic coefficient data to generate the collision force data.

[0022] In one possible implementation, the extraction and processing of bolt advance angle data, initial torque data, and torque coefficient data from the dual-platform synchronous data includes: real-time acquisition and processing of bolt rotation information in the dual-platform synchronous data to obtain the bolt advance angle data; preset processing of torque parameters for bolt assembly / disassembly tasks to obtain the initial torque data and torque coefficient data; and subtracting the product of the torque coefficient data and the advance angle data from the initial torque data to obtain the bolt assembly / disassembly torque data, wherein the bolt assembly / disassembly torque data is used to simulate the frictional changes during bolt assembly / disassembly.

[0023] In one possible implementation, transmitting the force feedback data to the force feedback device includes: performing signal conversion processing on the force feedback data to obtain a force feedback device drive signal; transmitting the drive signal to the force feedback device for drive processing to cause the force feedback device to generate corresponding tactile feedback; performing real-time rendering output processing on the virtual environment data of the hot chamber to obtain a visual interactive screen; and integrating the tactile feedback and the visual interactive screen to generate the robotic arm teleoperation training interactive result for the operator to perform skills training.

[0024] Secondly, the present invention provides a force feedback-based robotic arm teleoperation training device, comprising:

[0025] A scene simulation platform is used to generate a hot chamber virtual environment that includes a robotic arm model and an operation object model, and the scene simulation platform outputs hot chamber virtual environment data.

[0026] A dynamic simulation platform is used to construct a dynamic model of the robotic arm and a dynamic model of the manipulated object corresponding to the virtual environment of the hot chamber. The dynamic simulation platform receives pose data transmitted from the outside and outputs simulation data.

[0027] A force feedback device is used to collect the operator's hand movement information and outputs the hand movement information to an external control module.

[0028] The force feedback control module is connected to the force feedback device and the scene simulation platform respectively. The force feedback control module receives the hand movement information and the hot chamber virtual environment data, constructs a workspace mapping model and outputs the expected pose data of the robotic arm end effector. At the same time, it receives external force data and outputs it to the force feedback device.

[0029] The CoppeliaSim control module is connected to the force feedback control module, the dynamics simulation platform, and the scene simulation platform. The CoppeliaSim control module receives the desired pose data of the robotic arm end effector and outputs it to the dynamics simulation platform. It also receives the simulation data output by the dynamics simulation platform and outputs it to the scene simulation platform.

[0030] The dynamic link library is connected to the scene simulation platform, the CoppeliaSim control module, and the dynamic simulation platform respectively. The dynamic link library is used to transmit interactive data between the scene simulation platform and the dynamic simulation platform.

[0031] The object control and force calculation module is connected to the scene simulation platform, the dynamics simulation platform, and the force feedback control module. The object control and force calculation module receives the simulation data, calculates the force data and outputs it to the force feedback control module, and simultaneously outputs object control data to the scene simulation platform.

[0032] Compared with the prior art, the advantages of this invention are as follows:

[0033] Current common methods for training robotic arms in hot chambers primarily rely on downtime during production processes. Operators must directly control the actual robotic arm for training during these downtime periods. However, the duration of these downtime periods is relatively short, making it difficult to meet sufficient training needs. Furthermore, if operators are not skilled enough, collisions with the robotic arm during operation can damage the actual equipment and further increase equipment maintenance and overall training costs. The method and system provided by this invention fully reproduce the hot chamber operation scenario in a virtual environment and combine force feedback and visual feedback. This allows operators to conduct highly realistic training under safe and repeatable conditions without relying on a real robotic arm or downtime, thereby reducing training costs and effectively improving training efficiency.

[0034] For most industrial operations, especially in hazardous environments like high-radiation hot chambers, visual feedback alone is often insufficient to accurately reflect the interaction between the robotic arm and the object being manipulated. Take, for example, a task requiring precise force control, such as tightening bolts in a virtual environment. Without force feedback, operators struggle to determine if the tightening torque is sufficient, potentially leading to operation failure due to improper torque control. In severe cases, this could even damage equipment in the virtual environment, impacting training effectiveness. This invention introduces force feedback into the system, enabling operators to realistically perceive the contact state between the robotic arm and walls / targets in a virtual environment. They can also clearly sense the torque changes during bolt tightening. Experienced operators can adaptively adjust the robotic arm's trajectory and applied torque based on this real-time force feedback information, achieving a training process highly consistent with real-world operations. This ensures operator safety while enhancing operational accuracy and immersion during training. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1This is a flowchart illustrating the force feedback-based robotic arm teleoperation training method in an embodiment of the present invention.

[0037] Figure 2 This is a field debugging diagram of the force feedback-based robotic arm teleoperation training system in an embodiment of the present invention.

[0038] Figure 3 This refers to the virtual simulation scene in the scene simulation platform of this invention.

[0039] Figure 4 This refers to the virtual simulation scene in the dynamics simulation platform within the scene simulation platform of this invention embodiment;

[0040] Figure 5 This is a schematic diagram of the data interaction method of the simulation software in an embodiment of the present invention. Detailed Implementation

[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0042] Example:

[0043] It should be noted that the terms "comprising" and "having" and any variations thereof in the embodiments of the present invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products, or devices.

[0044] Figure 1 This is a flowchart illustrating the force feedback-based virtual sensory integration training method in an embodiment of the present invention; see also Figure 1 This invention provides a force feedback-based method for training remote operation of a robotic arm, comprising the following steps:

[0045] Step 1: Model and visually render based on real hot chamber scene data, and generate virtual hot chamber environment data containing robotic arms and manipulated objects on the scene simulation platform.

[0046] In this step, the modeling and visual rendering based on real hot chamber scene data includes: performing 3D modeling processing on the structural data, tooling data, and object data of the real hot chamber to obtain basic scene model data; configuring the robot arm joint parameters in the basic scene model data so that the rotation center and movement axis of the robot arm joints conform to the real robot arm parameters to obtain precise robot arm model data; and performing high-fidelity visual rendering processing on the basic scene model data and the precise robot arm model data to generate the virtual environment data of the hot chamber.

[0047] Step 2: Perform dynamic modeling based on the robotic arm and manipulated object data in the virtual environment data of the hot chamber, configure kinematics solving and collision detection functions on the dynamic simulation platform, and generate kinematic and dynamic simulation scene data of the robotic arm.

[0048] In this step, the dynamic modeling based on the robotic arm and manipulated object data in the hot chamber virtual environment data includes: performing corresponding modeling processing on the robotic arm and manipulated object data in the hot chamber virtual environment data to obtain basic dynamic model data; configuring a robotic arm inverse kinematics solution module on the basic dynamic model data to obtain robotic arm motion control data; configuring collision detection attributes on the robotic arm end effector and manipulated object in the basic dynamic model data to generate the robotic arm kinematics and dynamics simulation scene data.

[0049] Step 3: Integrate the force feedback device data and the virtual environment data of the hot chamber to construct a workspace mapping model between the force feedback device and the virtual robotic arm, convert the force feedback device posture data, and generate the workspace mapping model and control algorithm data.

[0050] In this step, the integration processing of the force feedback device data and the virtual environment data of the hot chamber includes: parsing the operator's hand position and posture data collected by the force feedback device to obtain raw hand movement data; performing mapping operations on the raw hand movement data and the virtual robotic arm workspace data, using an offset-based variable-scale mapping method to obtain the expected position data of the robotic arm end effector; performing posture transformation processing on the handle rotation angle data of the force feedback device end effector to convert it into Euler angles to obtain the expected posture data of the robotic arm end effector; and integrating the expected position data and the expected posture data to generate the workspace mapping model and control algorithm data.

[0051] Furthermore, the offset-based variable-scale mapping method includes: performing velocity calculation processing on the position coordinate data in the original hand motion data to obtain force feedback position velocity data; comparing the force feedback position velocity data with a preset movement threshold; if it is greater than the threshold, calculating the offset to obtain the robotic arm end-effector position offset data; superimposing the current frame end-effector position data of the robotic arm with the position offset data to obtain the next frame end-effector position coordinate data of the robotic arm; if it is less than or equal to the threshold, keeping the current frame end-effector position data of the robotic arm unchanged, and generating the expected end-effector position data of the robotic arm.

[0052] Step 4: Encapsulate the interaction interface between the scene simulation platform and the dynamics simulation platform to generate a dynamic link library. Transmit operation data and simulation data through the dynamic link library to obtain synchronized data between the two platforms.

[0053] In this step, the encapsulation of the interaction interface between the scene simulation platform and the dynamics simulation platform includes: extracting and processing the interface functions of the two platforms to obtain interface function set data; encapsulating the interface function set data to generate the dynamic link library; transmitting the operation data of the force feedback device to the dynamics simulation platform through the dynamic link library, and obtaining the joint angle data of the robotic arm through inverse kinematics solution; and transmitting the joint angle data, the pose data of the manipulated object, and the collision data back to the scene simulation platform through the dynamic link library to generate the dual-platform synchronization data.

[0054] Step 5: Calculate the force model of the interaction information between the robotic arm and the virtual environment in the dual-platform synchronized data, generate force feedback data, transmit the force feedback data to the force feedback device, and obtain the robotic arm teleoperation training interaction results.

[0055] In this step, the force perception model calculation of the robotic arm and virtual environment interaction information in the dual-platform synchronous data includes: extracting and processing the collision depth data and the elastic coefficient data of the colliding object from the dual-platform synchronous data to obtain collision parameter data; performing a product operation on the collision parameter data to obtain collision force perception data; extracting and processing the bolt screwing angle data, initial torque data, and torque coefficient data from the dual-platform synchronous data to obtain bolt operation parameter data; performing calculations on the bolt operation parameter data, subtracting the product of the torque coefficient and the screwing angle from the initial torque to obtain bolt installation and removal torque data; and integrating the collision force perception data and the bolt installation and removal torque data to generate the force perception feedback data.

[0056] Furthermore, the extraction and processing of collision depth data and collision object elastic coefficient data in the dual-platform synchronization data includes: detecting and processing the contact point data between the robotic arm and the virtual object in the dual-platform synchronization data to obtain contact position data; performing depth calculation processing on the contact position data to obtain the collision depth data; performing parameter configuration processing on the material property data of the collision object to obtain the collision object elastic coefficient data; and performing a multiplication operation on the collision depth data and the elastic coefficient data to generate the collision force data.

[0057] Furthermore, the extraction and processing of bolt advance angle data, initial torque data, and torque coefficient data from the dual-platform synchronous data includes: real-time acquisition and processing of bolt rotation information in the dual-platform synchronous data to obtain the bolt advance angle data; preset processing of torque parameters for bolt assembly and disassembly tasks to obtain the initial torque data and torque coefficient data; and subtracting the product of the torque coefficient data and the advance angle data from the initial torque data to obtain the bolt assembly and disassembly torque data, which is used to simulate the frictional force changes during bolt assembly and disassembly.

[0058] Furthermore, transmitting the force feedback data to the force feedback device includes: performing signal conversion processing on the force feedback data to obtain a force feedback device drive signal; transmitting the drive signal to the force feedback device for drive processing to enable the force feedback device to generate corresponding tactile feedback; performing real-time rendering output processing on the virtual environment data of the hot chamber to obtain a visual interactive screen; and integrating the tactile feedback and the visual interactive screen to generate the robotic arm teleoperation training interactive result for the operator to conduct skills training.

[0059] Based on the same inventive concept, embodiments of the present invention also provide a force feedback-based robotic arm teleoperation training device, comprising:

[0060] A scene simulation platform is used to generate a hot chamber virtual environment that includes a robotic arm model and an operation object model, and the scene simulation platform outputs hot chamber virtual environment data.

[0061] A dynamic simulation platform is used to construct a dynamic model of the robotic arm and a dynamic model of the manipulated object corresponding to the virtual environment of the hot chamber. The dynamic simulation platform receives pose data transmitted from the outside and outputs simulation data.

[0062] A force feedback device is used to collect the operator's hand movement information and outputs the hand movement information to an external control module.

[0063] The force feedback control module is connected to the force feedback device and the scene simulation platform respectively. The force feedback control module receives the hand movement information and the hot chamber virtual environment data, constructs a workspace mapping model and outputs the expected pose data of the robotic arm end effector. At the same time, it receives external force data and outputs it to the force feedback device.

[0064] The CoppeliaSim control module is connected to the force feedback control module, the dynamics simulation platform, and the scene simulation platform. The CoppeliaSim control module receives the desired pose data of the robotic arm end effector and outputs it to the dynamics simulation platform. It also receives the simulation data output by the dynamics simulation platform and outputs it to the scene simulation platform.

[0065] The dynamic link library is connected to the scene simulation platform, the CoppeliaSim control module, and the dynamic simulation platform respectively. The dynamic link library is used to transmit interactive data between the scene simulation platform and the dynamic simulation platform.

[0066] The object control and force calculation module is connected to the scene simulation platform, the dynamics simulation platform, and the force feedback control module. The object control and force calculation module receives the simulation data, calculates the force data and outputs it to the force feedback control module, and simultaneously outputs object control data to the scene simulation platform.

[0067] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0068] Figure 2 This is a field debugging diagram of the force feedback-based robotic arm teleoperation training system in an embodiment of the present invention. Figure 3 This refers to the virtual simulation scene in the scene simulation platform of this invention. Figure 4 This refers to the virtual simulation scene in the dynamics simulation platform within the scene simulation platform of this invention embodiment; Figure 5 This is a schematic diagram illustrating the data interaction method of the simulation software in an embodiment of the present invention. See also... Figures 2 to 5As a preferred example, this embodiment provides a teleoperation training method for a robotic arm based on force feedback. This embodiment takes the virtual training task of a hot chamber robotic arm performing tightening and loosening of bolts on the top cover of a target in a high-temperature radiation and confined space environment as the application scenario.

[0069] The method in this embodiment integrates VR devices and force feedback devices. Operators can immerse themselves in the virtual environment by using the realistic visual rendering effects provided by the scene simulation platform and the real force feedback rendering information transmitted by the force feedback device. This allows them to carry out robotic arm operation skills training under safe and controllable conditions, effectively solving the problems of traditional training such as reliance on physical equipment, high training costs, safety risks, and difficulty in realistically reproducing the operating environment. At the same time, it improves the repeatability and relevance of training, which is of great significance for improving workers' operating skills, shortening the training cycle, and ensuring work safety.

[0070] In this embodiment, a force feedback device independently developed in the laboratory is used to remotely control the virtual industrial robotic arm Willis Miller A1000s, focusing on training in tightening and loosening bolts. The specific implementation process is as follows:

[0071] Based on the structural characteristics and operational requirements of the actual hot chamber, a virtual hot chamber environment model was built in the Unity simulation platform. This model includes elements such as a ten-joint robotic arm, a target body, a target cover, bolts, nuts, and operational fixtures. When building the ten-joint robotic arm model, the position of each joint must be set according to the DH parameters of the real robotic arm to ensure that the rotation center and movement axis of each joint are consistent with the real equipment, thus avoiding the impact of model parameter deviations on the training effect.

[0072] In the CoppeliaSim dynamics simulation platform, a ten-joint robotic arm model corresponding to the virtual hot chamber environment model in the Unity scene, along with models of manipulated objects such as bolts and targets, were established. Simultaneously, an inverse kinematics solution module for the robotic arm was configured for the platform to ensure that the robotic arm can adjust joint angles in real time according to the desired end-effector pose. Furthermore, collision detection attributes need to be defined for the bolt and the robotic arm's end effector. When the end-effector tool comes into contact with the bolt, the CoppeliaSim platform can calculate the collision state, relative velocity, and contact depth at the contact point in real time, providing basic data for subsequent force feedback calculations.

[0073] This training system has three main modules in the Unity scene simulation platform. These three modules use the observer pattern to transfer data, which ensures that information can be transmitted between different modules in real time without significant delay.

[0074] The first module is the force feedback control module. This module acquires the position and attitude information transmitted by the force feedback device in real time through a preset interface. Then, by constructing a workspace mapping model between the force feedback device and the virtual robot, and using a control algorithm, it transforms the operator's hand movement information collected by the force feedback device, including hand position, hand posture, and hand speed, into the desired pose of the robotic arm's end effector. The workspace mapping model adopts an offset-based variable-scale mapping method, which is implemented as follows: when the ratio of the magnitude of the force feedback position coordinate to its own magnitude is greater than the movement threshold k0, the next frame end effector position coordinate P of the robotic arm... t+1 Equal to the position coordinate P of the end of the current frame t Add the overall velocity mapping coefficient k and the force feedback position coordinate v t Subtract the movement threshold k0 multiplied by v t The product of the ratio of the force feedback position coordinate to its own modulus; when the ratio of the force feedback position coordinate to its own modulus is less than or equal to the movement threshold k0, the next frame end position coordinate P of the robotic arm. t+1 Maintain the current frame end position coordinates P t constant.

[0075] Right now:

[0076]

[0077] In the formula, P t+1 P represents the coordinates of the robotic arm's end effector position in the next frame. t Let v be the coordinates of the end effector position of the robotic arm in this frame, k be the overall velocity mapping coefficient, and v be the position coordinates of the end effector position. t Here are the force feedback position coordinates, and k0 is the movement threshold.

[0078] In this method, the movement threshold k0 is set to 20mm. Its purpose is to ensure that the hand's speed is 0 when it is within a 20mm distance from the force feedback center point, thus preventing slight hand tremors from interfering with the robotic arm control. When the hand moves more than 20mm, the vector truncated to 20mm is used as the mapping direction to ensure control accuracy. The overall speed mapping coefficient k can be adjusted according to actual needs. For large-space movements, k can be set to a larger value; for fine movements, k can be set to 1 to satisfy the one-to-one mapping relationship of positions. For the posture control of the robotic arm's end effector, precise control of the end effector's posture can be achieved by collecting the rotation angle information of the handle at the end of the force feedback device and converting this information into Euler angles. Simultaneously, the force feedback control module continuously monitors the data transmitted by the force calculation module and outputs the calculated force information to the force feedback device in real time, allowing the user to experience realistic force sensations when interacting with the virtual environment during operation.

[0079] The second module is the CoppeliaSim control module. This module enables bidirectional data communication with the Unity scene simulation platform by calling pre-packaged dynamic link libraries. Specifically, the CoppeliaSim control module first monitors the end-effector position and attitude information transmitted by the force feedback control module in real time, and then transmits this information to the CoppeliaSim dynamics simulation platform via the dynamic link library. The CoppeliaSim platform uses its built-in inverse kinematics algorithm to iteratively converge on the received end-effector pose information, ultimately obtaining ten joint angle values ​​that meet the pose accuracy requirements. After the calculation is completed, the CoppeliaSim control module acquires the real-time ten joint angle values, as well as the pose information of other manipulated objects (including bolt position information), and sends this data back to the Unity scene simulation platform via the dynamic link library, ensuring that the model motion states of the two platforms remain synchronized.

[0080] The third module is the object control and force calculation module, which includes an object control submodule and a force calculation submodule. The object control submodule continuously monitors data transmitted from the CoppeliaSim mechanics simulation platform and assigns this data to the corresponding objects in the Unity scene, thus visually presenting the movement of the robotic arm on the display screen, allowing the operator to clearly observe the robotic arm's trajectory. The force calculation submodule calculates force parameters using collision force models and friction force models, and outputs the calculation results to the force feedback device, achieving synchronous output of visual and force feedback, thereby training the operator's hand-eye coordination. The force models in this training system are mainly divided into collision force models and friction force models. The collision force model calculates the force generated when the robotic arm comes into contact with other scene models, while the friction force model calculates the force generated when the robotic arm installs or removes bolts. The reaction force F generated by the collision... c Through formula F c =k w d is calculated to obtain F in the formula c k represents the force sensation generated by the collision. w d represents the elastic coefficient of the colliding objects, and d represents the collision depth.

[0081] That is: F c =k w d;

[0082] In the formula, F c For the force sensation generated by the collision, k w d is the elastic coefficient of the colliding object, and d is the collision depth.

[0083] The force sensing model for bolt installation and removal mainly plays a role when the robotic arm tightens the bolts on the target body, and it is expressed by formula T. c=T0-mθ is achieved, where T c T0 represents the output torque, m represents the initial torque, and θ represents the precession angle.

[0084] That is: T c =T0-mθ;

[0085] In the formula, T c The output torque is T0, the initial torque is m, the torque coefficient is θ, and the precession angle is θ.

[0086] This torque calculation formula can simulate the change in friction between the robotic arm and the bolt, and the magnitude of the force is related to the angle of the bolt being screwed in. It can simulate a realistic force feeling effect of screwing out and tightening, making the force feedback during the bolt installation and removal process closer to real operation.

[0087] Based on the collision force feedback and highly realistic visual feedback provided by the system, the operator can clearly perceive the interaction between the robotic arm and the virtual environment. In this way, the operator can stably and accurately remotely operate the robotic arm to train for bolt assembly and disassembly in dangerous environments such as high temperature radiation and confined spaces, and gradually improve the robotic arm operation capabilities in dangerous scenarios.

[0088] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0089] The above embodiments are merely illustrative of the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made based on the essence of the content of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for training remote operation of a robotic arm based on force feedback, characterized in that, Includes the following steps: Modeling and visual rendering are performed based on real hot chamber scene data, and virtual environment data of the hot chamber containing robotic arms and manipulated objects are generated on the scene simulation platform. Based on the data of the robotic arm and the object being manipulated in the virtual environment data of the hot chamber, dynamic modeling is performed, and kinematic solving and collision detection functions are configured on the dynamic simulation platform to generate kinematic and dynamic simulation scene data of the robotic arm. The force feedback device data and the virtual environment data of the hot chamber are integrated and processed to construct a workspace mapping model between the force feedback device and the virtual robotic arm, and the force feedback device posture data is converted to generate the workspace mapping model and control algorithm data. The interaction interface between the scene simulation platform and the dynamics simulation platform is encapsulated to generate a dynamic link library. Operation data and simulation data are transmitted through the dynamic link library to obtain synchronized data between the two platforms. Force model calculation is performed on the interaction information between the robotic arm and the virtual environment in the dual-platform synchronous data to generate force feedback data. The force feedback data is then transmitted to the force feedback device to obtain the robotic arm teleoperation training interaction results. The calculation of the force model for the interaction information between the robotic arm and the virtual environment in the dual-platform synchronized data includes: The collision depth data and the elastic coefficient data of the colliding object in the dual-platform synchronization data are extracted and processed to obtain collision parameter data; The collision parameter data is multiplied to obtain the collision force perception data; The bolt screwing angle data, initial torque data, and torque coefficient data in the dual-platform synchronous data are extracted and processed to obtain bolt operation parameter data; The bolt operation parameter data is processed by subtracting the product of the torque coefficient and the screw-in angle from the initial torque to obtain the bolt installation and removal torque data. The collision force data and bolt installation / removal torque data are integrated to generate the force feedback data; The extraction and processing of collision depth data and collision object elastic coefficient data from the dual-platform synchronization data includes: The contact point data between the robotic arm and the virtual object in the dual-platform synchronization data are detected and processed to obtain the contact position data; The contact position data is processed by depth calculation to obtain the collision depth data; the material property data of the colliding object is processed by parameter configuration to obtain the elastic coefficient data of the colliding object. The collision depth data and the elastic coefficient data are multiplied to generate the collision force data. The extraction and processing of bolt insertion angle data, initial torque data, and torque coefficient data from the dual-platform synchronous data includes: The bolt rotation information in the dual-platform synchronous data is collected and processed in real time to obtain the bolt screw-in angle data. The torque parameters for bolt assembly and disassembly are preset to obtain the initial torque data and torque coefficient data; The bolt assembly / disassembly torque data is obtained by subtracting the product of the torque coefficient data and the screw-in angle data from the initial torque data. The bolt assembly / disassembly torque data is used to simulate the change in friction force during the bolt assembly / disassembly process.

2. The method for training remote operation of a robotic arm based on force feedback according to claim 1, characterized in that, The modeling and visual rendering based on real hothouse scene data includes: The structural data, tooling data, and object data of the actual hot chamber are processed into three-dimensional models to obtain basic scene model data. The robot arm joint parameters in the basic scene model data are configured to make the rotation center and movement axis of the robot arm joint conform to the real robot arm parameters, thus obtaining accurate robot arm model data. The basic scene model data and the precision robotic arm model data are subjected to high-fidelity visual rendering to generate the virtual environment data of the hot chamber.

3. The method for training remote operation of a robotic arm based on force feedback according to claim 1, characterized in that, The dynamic modeling based on the robotic arm and manipulated object data in the hot chamber virtual environment data includes: The data of the robotic arm and the object being operated in the virtual environment data of the hot chamber are modeled accordingly to obtain the basic dynamic model data. Configure the inverse kinematics solution module for the robotic arm based on the dynamic basic model data to obtain the motion control data of the robotic arm; The end effector of the robotic arm and the object being manipulated in the basic dynamic model data are configured with collision detection attributes to generate the kinematic and dynamic simulation scene data of the robotic arm.

4. The method for training remote operation of a robotic arm based on force feedback according to claim 1, characterized in that, The integration and processing of the force feedback device data and the hot chamber virtual environment data includes: The operator's hand position and posture data collected by the force feedback device are analyzed and processed to obtain the raw hand movement data; The original hand motion data and the virtual robotic arm workspace data are mapped and processed using an offset-based variable-scale mapping method to obtain the expected data of the robotic arm end position. The rotation angle data of the end handle of the force feedback device is processed by attitude conversion to convert it into Euler angles to obtain the expected attitude data of the end of the robotic arm. The position expectation data and attitude expectation data are integrated to generate the workspace mapping model and control algorithm data.

5. The method for training remote operation of a robotic arm based on force feedback according to claim 4, characterized in that, The offset-based variable-scale mapping method includes: The position coordinate data in the original hand movement data is processed by velocity calculation to obtain force feedback position velocity data; The force feedback position velocity data is compared with a preset movement threshold. If the value is greater than the threshold, the offset is calculated to obtain the position offset data of the robotic arm end effector. The current frame end position data of the robotic arm is superimposed with the position offset data to obtain the end position coordinate data of the robotic arm in the next frame. If the value is less than or equal to the threshold, the current frame end position data of the robotic arm remains unchanged, and the expected end position data of the robotic arm is generated.

6. The method for training remote operation of a robotic arm based on force feedback according to claim 1, characterized in that, The encapsulation of the interaction interface between the scene simulation platform and the dynamics simulation platform includes: The interface functions of the two platforms are extracted and processed to obtain a set of interface function data; The interface function set data is encapsulated to generate the dynamic link library; the operation data of the force feedback device is transmitted to the dynamic simulation platform through the dynamic link library, and the joint angle data of the robot arm is obtained by inverse kinematics solution. The joint angle data, the pose data of the manipulated object, and the collision data are transmitted back to the scene simulation platform through the dynamic link library to generate the dual-platform synchronized data.

7. The method for training remote operation of a robotic arm based on force feedback according to claim 1, characterized in that, The step of transmitting the force feedback data to the force feedback device includes: The force feedback data is processed by signal conversion to obtain the force feedback device drive signal; The driving signal is transmitted to the force feedback device for driving processing, so that the force feedback device generates corresponding tactile feedback. The virtual environment data of the hot chamber is rendered and output in real time to obtain a visual interactive screen; By integrating the tactile feedback and visual interaction, the remote operation training interaction results of the robotic arm are generated for operators to conduct skills training.

Images