Wireless teleoperation method and system for augmented reality robots

Abstract

The application discloses a wireless remote operation method and system of an augmented reality robot, relates to the technical field of augmented reality and robot control, and aims to improve the problems of "hand-eye incoordination" of an operator, low remote operation accuracy and efficiency caused by visual and tactile perception time delay and wireless transmission time delay. The method comprises the following steps: acquiring motion information of a multi-degree-of-freedom force feedback controller remote operation, and generating physical robot control instructions; controlling the physical robot to perform remote operation according to the physical robot control instructions, receiving environmental contact force information and displacement increment of the physical robot; calculating virtual additional force information, superimposing the environmental contact force information and the virtual additional force information, and generating tactile feedback information; calculating a displacement correction amount of a virtual robot, superimposing the displacement correction amount and the displacement increment of the virtual robot, obtaining virtual robot control instructions, and determining position information of each joint according to the virtual robot control instructions.

Description

Technical Field

[0001] This application relates to the fields of augmented reality technology and robot control technology, and in particular to a wireless teleoperation method and system for an augmented reality robot. Background Technology

[0002] Teleoperated robots have been widely used in extreme and hazardous environments (such as deep space exploration, deep-sea exploration, nuclear power plant maintenance, and explosive ordnance disposal) to replace humans in on-site operations, thereby reducing labor costs and mission risks. With the rapid development of Augmented Reality (AR) technology, a completely new experience has been brought to robot teleoperation, providing operators with a direct and immersive visual experience. Furthermore, AR technology can also provide interactive virtual haptic feedback. By using haptic gloves, force feedback devices, or haptic sensors, operators can receive haptic feedback during teleoperation, allowing them to perceive the feel and force of real-world operations.

[0003] However, during robot teleoperation, when the master controller and the slave robot are far apart, transmission delays are unavoidable due to many unknown external interference factors during the transmission of interactive information. This means that by the time the operator receives the robot's posture information through AR glasses and performs virtual modeling to obtain visual feedback, the actual robot state may have already changed. Furthermore, due to the delay in physiological feedback, the operator's hand movements may not be completely synchronized with the actual perceived tactile feedback, leading to "hand-eye incoordination." Consequently, the operator finds it difficult to accurately perceive and control the robot's position, direction, and force, reducing the accuracy and efficiency of teleoperation. Summary of the Invention

[0004] In view of this, this application provides a wireless teleoperation method and system for augmented reality robots. The main purpose is to improve the problems of "hand-eye incoordination" and low teleoperation accuracy and efficiency caused by the operator's difficulty in accurately perceiving and controlling the robot's position, direction and force due to visual and tactile perception delays and wireless transmission delays.

[0005] According to one aspect of this application, a wireless teleoperation method for an augmented reality robot is provided, comprising:

[0006] The system acquires motion information from the teleoperation of the multi-degree-of-freedom force feedback controller, generates control commands for the physical robot based on the motion information, and transmits the control commands to the physical robot wirelessly.

[0007] The system controls the physical robot to perform the teleoperation according to the physical robot control command, and receives the environmental contact force information and the displacement increment of the physical robot collected by the physical robot wirelessly.

[0008] The virtual applied force information is calculated based on the displacement increment of the physical robot. The environmental contact force information and the virtual applied force information are superimposed to generate the tactile feedback information of the multi-degree-of-freedom force feedback controller for tactile correction.

[0009] The displacement correction amount of the virtual robot is calculated based on the displacement increment of the multi-degree-of-freedom force feedback controller and the displacement increment of the physical robot. The displacement correction amount is then superimposed with the displacement increment of the virtual robot to obtain the virtual robot control command. The virtual robot control command is transmitted to the virtual robot wirelessly, so that the virtual robot can determine the position information of each joint according to the virtual robot control command for visual correction. The position information is obtained by limiting the range of joint movement angles based on the virtual three-dimensional model of the physical robot built in the augmented reality environment.

[0010] Preferably, the step of acquiring the motion information of the multi-degree-of-freedom force feedback controller for teleoperation, generating control commands for the physical robot based on the motion information, and transmitting the control commands to the physical robot wirelessly specifically includes:

[0011] Acquire motion information of the multi-degree-of-freedom force feedback controller for teleoperation;

[0012] The end effector acceleration of the multi-degree-of-freedom force feedback controller is calculated based on the operating force information and damping information contained in the motion information.

[0013] The displacement increment of the multi-degree-of-freedom force feedback controller is calculated based on the end-effector acceleration and velocity information.

[0014] Based on the displacement increment of the multi-degree-of-freedom force feedback controller, the teleoperation is divided into operation stages according to a pre-generated operation stage division standard to obtain at least one operation stage.

[0015] The standard displacement and corresponding gain coefficient of each operation stage are calculated based on the number of operation stages. The displacement increment of the physical robot in each operation stage is calculated based on the gain coefficient and the displacement increment of the multi-degree-of-freedom force feedback controller, so as to generate physical robot control instructions based on the displacement increment of each physical robot.

[0016] The control commands for the physical robot are encapsulated and transmitted to the physical robot wirelessly.

[0017] Preferably, before dividing the teleoperation into operation stages based on the displacement increment of the multi-degree-of-freedom force feedback controller and a pre-generated operation stage division standard to obtain at least one operation stage, the method further includes generating an operation stage division standard, specifically including:

[0018] The number of preset operation stages is determined according to the control requirements, and the operation stages are divided according to the preset number of operation stages;

[0019] The standard displacement and corresponding gain coefficient of each operation stage are calculated according to the preset number of operation stages. An operation stage division standard is generated based on the mapping relationship between the standard displacement and corresponding gain coefficient of each operation stage, and the operation stage to which it belongs is determined. The standard displacement is used to characterize the expected motion range of the operation stage, and the gain coefficient is used to characterize the proportional relationship between the control signal and the output.

[0020] Preferably, the step of calculating virtual applied force information based on the displacement increment of the physical robot, superimposing the environmental contact force information and the virtual applied force information to generate tactile feedback information of the multi-degree-of-freedom force feedback controller for tactile correction specifically includes:

[0021] The virtual applied force information is calculated based on the gain coefficient and the displacement increment of the physical robot.

[0022] The difference between the environmental contact force information and the operational force information is calculated and then superimposed with the virtual additional force information to generate the tactile feedback information of the multi-degree-of-freedom force feedback controller.

[0023] Preferably, determining the position information of each joint according to the virtual robot control instructions for visual correction specifically includes:

[0024] The inverse solution formula is constructed based on the virtual robot control instructions using the analytical method.

[0025] Matrix transformation and inverse transformation operations are performed on the inverse solution formula to obtain multiple sets of inverse solutions for each joint of the virtual robot;

[0026] Based on the joint connection relationships of the virtual robot, the range of motion angles of each joint is restricted to filter out the unique inverse solution of each joint from the multiple sets of inverse solutions, thereby obtaining the position information of each joint. The joint connection relationships are obtained based on the virtual three-dimensional model of the physical robot built in the augmented reality environment.

[0027] Preferably, before limiting the range of motion angles of each joint based on the joint connection relationships of the virtual robot to filter out the unique inverse solution for each joint from the multiple sets of inverse solutions and obtain the position information of each joint, the method further includes establishing a virtual three-dimensional model of the physical robot in an augmented reality environment, specifically including:

[0028] Based on the physical structure of the physical robot, establish the joint coordinate system of each joint of the virtual robot.

[0029] The transformation matrix of adjacent joints is determined based on the transformation relationship between the joint coordinate systems of two adjacent joints, so as to construct the kinematic relationship of the virtual robot;

[0030] Based on the kinematic relationship, the position information of each joint in the virtual robot is calculated;

[0031] A virtual 3D model of the physical robot is built in an augmented reality environment based on the kinematic relationships and positional information of each joint.

[0032] Preferably, before calculating the position information of each joint of the virtual robot based on the kinematic relationship, the method further includes determining the position information of the end effector of the virtual robot, specifically including:

[0033] The coordinate system corresponding to each end effector is determined based on the physical structure of the physical robot;

[0034] The coordinate system is replaced with the joint coordinate system of the last joint of the virtual robot to obtain the position information of the end effector of the virtual robot.

[0035] According to another aspect of this application, a wireless teleoperation system for an augmented reality robot is provided, comprising:

[0036] Multi-degree-of-freedom force feedback controller, physical robot, high-speed wireless communication module, AR glasses, virtual robot, multi-dimensional force sensor;

[0037] The multi-degree-of-freedom force feedback controller is used to collect motion information from remote operation, generate control commands for the physical robot based on the motion information, and provide tactile feedback information to the operator based on environmental contact force information and virtual additional force information.

[0038] The physical robot is used to perform the teleoperation according to the physical robot control command, and to collect the displacement increment of the physical robot and the environmental contact force information sensed by the multi-dimensional force sensor;

[0039] The AR glasses are used to construct the kinematic relationship of the virtual robot based on the physical structure of the physical robot, and to build a virtual three-dimensional model of the physical robot in an augmented reality environment to construct the virtual robot.

[0040] The virtual robot is used to determine the position information of each joint according to virtual robot control commands for visual correction. The virtual robot control commands are calculated based on the displacement increment of the multi-degree-of-freedom force feedback controller and the displacement increment of the physical robot to obtain the displacement correction amount of the virtual robot, and then the displacement correction amount is superimposed with the displacement increment of the virtual robot.

[0041] The high-speed wireless communication module is used to transmit the physical robot control commands to the physical robot, receive the environmental contact force information collected by the physical robot and the displacement increment of the physical robot, and transmit the virtual robot control commands to the virtual robot.

[0042] By employing the above technical solutions, the technical solutions provided in the embodiments of this application have at least the following advantages:

[0043] This application provides a wireless teleoperation method and system for augmented reality robots. First, it acquires motion information from a multi-degree-of-freedom force feedback controller, generates physical robot control commands based on the motion information, and wirelessly transmits these commands to the physical robot. Second, it controls the physical robot to perform the teleoperation according to the physical robot control commands, and wirelessly receives environmental contact force information and displacement increments collected by the physical robot. Finally, it calculates virtual additional force information based on the displacement increments of the physical robot, and superimposes the environmental contact force information and the virtual additional force information to generate the multi-degree-of-freedom force feedback controller. The tactile feedback information of the multi-degree-of-freedom force feedback controller is used for tactile correction. The displacement correction amount of the virtual robot is calculated based on the displacement increment of the multi-degree-of-freedom force feedback controller and the displacement increment of the physical robot. The displacement correction amount is superimposed with the displacement increment of the virtual robot to obtain the virtual robot control command. The virtual robot control command is transmitted to the virtual robot wirelessly so that the virtual robot can determine the position information of each joint according to the virtual robot control command for visual correction. The position information is obtained by limiting the range of joint movement angles based on the virtual three-dimensional model of the physical robot built in the augmented reality environment. Compared with existing technologies, the embodiments of this application divide the operation stages according to the end-effector acceleration of the master multi-degree-of-freedom force feedback controller and dynamically adjust the gain coefficient to achieve variable proportional incremental control of the slave physical robot, solving the cross-scale motion problem and achieving matching of hand movements and robot movements in each operation stage. Furthermore, the tactile feedback information is corrected based on virtual additional force information, and the position information of the slave virtual robot is corrected based on the displacement increment of the master multi-degree-of-freedom force feedback controller and the displacement increment of the slave physical robot, thereby improving the "hand-eye asynchrony" situation, optimizing the synchronization effect of visual-tactile feedback and robot teleoperation, and improving the accuracy and efficiency of teleoperation.

[0044] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0045] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0046] Figure 1A flowchart illustrating a wireless teleoperation method for an augmented reality robot according to an embodiment of this application is shown.

[0047] Figure 2 A flowchart illustrating the generation of control instructions provided in an embodiment of this application is shown;

[0048] Figure 3 This document shows a flowchart illustrating the solution process for the unique inverse solution provided in an embodiment of this application.

[0049] Figure 4 A block diagram of a wireless teleoperation system for an augmented reality robot provided in an embodiment of this application is shown. Detailed Implementation

[0050] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0051] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0052] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the scope of this application and its application or use.

[0053] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0054] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0055] The embodiments of this application can be applied to computer systems / servers that can operate with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with computer systems / servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems, etc.

[0056] Computer systems / servers can be described in the general context of computer system executable instructions (such as program modules) executed by the computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are performed by remote processing devices linked through a communication network. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0057] This application provides a wireless teleoperation method for augmented reality robots, such as... Figure 1 As shown, the method includes:

[0058] 101. Obtain motion information from the multi-degree-of-freedom force feedback controller for remote operation, generate control commands for the physical robot based on the motion information, and transmit the control commands to the physical robot wirelessly.

[0059] In this embodiment, the current execution end can be the control module of the wireless teleoperation system of the augmented reality robot. By acquiring the action information of the operator performing remote operation based on the multi-degree-of-freedom force feedback controller, including the action completed by the operator holding the multi-degree-of-freedom force feedback controller, the force applied to the multi-degree-of-freedom force feedback controller, displacement increment, operation force information, damping information, etc., control instructions for the physical robot are generated.

[0060] The control commands for the physical robot are transmitted to the physical robot via a high-speed wireless communication module. When transmitting the control commands to the physical robot, the control commands can be transmitted to both the main body of the physical robot and the end effector (gripper) to enable free control of the end effector and improve the control accuracy of the physical robot.

[0061] 102. Control the physical robot to perform remote operation according to the physical robot control instructions, and receive the environmental contact force information and displacement increment of the physical robot collected by the physical robot wirelessly.

[0062] In this embodiment, the robot is used to control a physical robot to perform remote operations. At the same time, the physical robot also collects environmental contact force information and obtains the displacement increment of the physical robot through a multi-dimensional force sensor, and transmits it to the current execution end through a high-speed wireless communication module.

[0063] 103. Calculate virtual additional force information based on the displacement increment of the physical robot, superimpose the environmental contact force information and the virtual additional force information to generate tactile feedback information for the multi-degree-of-freedom force feedback controller for tactile correction.

[0064] The location information is obtained by limiting the range of joint movement angles based on a virtual 3D model of the physical robot built in an augmented reality environment.

[0065] It should be noted that due to the time delay of physiological feedback, the operator's hand movements may not be completely synchronized with the tactile feedback sensed by the multi-degree-of-freedom force feedback controller, thus affecting the operator's perception of the physical robot's state and force. To overcome the above problem, in this embodiment, virtual additional force information can be calculated based on the physical robot's displacement increment, and then tactile error compensation can be performed on the received environmental contact force information collected by the physical robot based on the virtual additional force information. That is, the environmental contact force information and the virtual additional force information are superimposed to generate the tactile feedback information of the multi-degree-of-freedom force feedback controller. Specifically, the virtual additional force information f v According to the formula The tactile feedback information F of the multi-degree-of-freedom force feedback controller is calculated. m It can be calculated using the following formula.

[0066]

[0067] Among them, f s Indicates environmental contact force information, f m It represents the force information so that the tactile feedback information fed back to the operator by the multi-degree-of-freedom force feedback controller is synchronized with the operator's hand movements, thereby accurately sensing the force of the physical robot and improving the accuracy and efficiency of teleoperation.

[0068] 104. Calculate the displacement correction amount of the virtual robot based on the displacement increment of the multi-degree-of-freedom force feedback controller and the displacement increment of the physical robot. Superimpose the displacement correction amount with the displacement increment of the virtual robot to obtain the virtual robot control command. Transmit the virtual robot control command to the virtual robot wirelessly so that the virtual robot can determine the position information of each joint according to the virtual robot control command for visual correction.

[0069] Furthermore, due to the latency in information transmission and processing, by the time the operator receives the posture information of the physical robot through AR glasses and performs virtual modeling to obtain visual feedback from the virtual robot, the state of the physical robot may have already changed. To overcome this problem, in this embodiment, the displacement correction amount of the virtual robot can be calculated based on the displacement increment of the multi-degree-of-freedom force feedback controller and the displacement increment of the physical robot, according to a pre-built error correction model. The displacement increment of the virtual robot is then corrected based on this displacement correction amount, i.e., the displacement correction amount is superimposed on the displacement increment of the virtual robot (which is the same as the displacement increment of the physical robot) to obtain the virtual robot control command. Specifically, the error correction model is Δx. vx (t)=βKΔx m (t)+(βKΔx m (t-2τ)-Δx s (t-τ)), where Δx m This represents the displacement increment Δx based on the multi-degree-of-freedom force feedback controller. s Δx represents the displacement increment of the physical robot. vx This represents the displacement correction amount of the virtual robot. Finally, the aforementioned virtual robot control commands are transmitted wirelessly to the virtual robot, enabling it to determine the position information of each joint based on these commands. This ensures that the visual feedback received by the operator using AR glasses regarding the virtual robot remains synchronized with the state of the physical robot, thereby accurately perceiving the position and orientation of the physical robot and improving the accuracy and efficiency of teleoperation.

[0070] Compared with existing technologies, the embodiments of this application divide the operation stages according to the end-effector acceleration of the master multi-degree-of-freedom force feedback controller and dynamically adjust the gain coefficient to achieve variable proportional incremental control of the slave physical robot, solving the cross-scale motion problem and achieving matching of hand movements and robot movements in each operation stage. Furthermore, the tactile feedback information is corrected based on virtual additional force information, and the position information of the slave virtual robot is corrected based on the displacement increment of the master multi-degree-of-freedom force feedback controller and the displacement increment of the slave physical robot, thereby improving the "hand-eye asynchrony" situation, optimizing the synchronization effect of visual-tactile feedback and robot teleoperation, and improving the accuracy and efficiency of teleoperation.

[0071] In one embodiment of this application, for further definition and explanation, such as Figure 2 As shown, step 101 of the embodiment obtains the motion information of the multi-degree-of-freedom force feedback controller for teleoperation, generates control commands for the physical robot based on the motion information, and transmits the control commands to the physical robot wirelessly. Specifically, this includes:

[0072] 201. Obtain the action information of the multi-degree-of-freedom force feedback controller for teleoperation.

[0073] 202. Calculate the end effector acceleration of the multi-degree-of-freedom force feedback controller based on the operating force information and damping information contained in the motion information.

[0074] 203. Calculate the displacement increment of the multi-degree-of-freedom force feedback controller based on the end-effector acceleration and velocity information.

[0075] 204. Based on the displacement increment of the multi-degree-of-freedom force feedback controller, the teleoperation is divided into operation stages according to the pre-generated operation stage division criteria to obtain at least one operation stage.

[0076] 205. Calculate the standard displacement and corresponding gain coefficient of each operation stage according to the number of operation stages, and calculate the physical robot displacement increment of each operation stage according to each gain coefficient and the displacement increment of the multi-degree-of-freedom force feedback controller, so as to generate physical robot control commands according to each physical robot displacement increment.

[0077] 206. Encapsulate the control commands for the physical robot and transmit them to the physical robot wirelessly.

[0078] In this embodiment of the application, the teleoperation action information of the multi-degree-of-freedom force feedback controller includes the operating force information f. m and damping information f z End-effector acceleration of a multi-degree-of-freedom force feedback controller Where m represents the end mass of the multi-degree-of-freedom force feedback controller, determined by the operating damping and the inherent end mass. Furthermore, the displacement increment of the master-end multi-degree-of-freedom force feedback controller... Among them, v m Velocity information can be directly obtained through a force feedback controller; t represents the motion time. Furthermore, based on the displacement increment Δx of the multi-degree-of-freedom force feedback controller... m Based on a pre-generated operation stage division standard, the current teleoperation is divided into operation stages, resulting in at least one operation stage. The operation stage division standard records the mapping relationship between the standard displacement and the corresponding gain coefficient for each operation stage, used to determine the operation stage. The standard displacement characterizes the expected range of motion for the operation stage, and the gain coefficient characterizes the proportional relationship between the control signal and the output. For example, assuming a preset number of operation stages M = 3, three standard displacements J0, J1, and J2 can be calculated. Therefore, the operation stage division standard includes three operation stages: 0-J0, J0-J1, and J1-J2, each corresponding to a gain coefficient K0, K1, and K2. When the displacement increment Δx of the multi-degree-of-freedom force feedback controller... mWhen the operation is within the range of 0-J0 in the first operation stage, the first gain coefficient K0 corresponding to the first operation stage is selected, and the formula is used. Calculate the displacement increment Δx of the physical robot during the first operation phase. s Where β represents the base scaling factor, t represents the current time, and τ represents the one-way transmission delay; one-way transmission delay Among them, t s t represents the command transmission time of the multi-degree-of-freedom force feedback controller. r This indicates the time when feedback information is received. Similarly, when the displacement increment Δx of the multi-degree-of-freedom force feedback controller... m When the robot is within the range of J0-J1 in the second operation stage, the first gain coefficient K1 corresponding to the first operation stage is selected, and the displacement increment Δx of the physical robot in the second operation stage is calculated. s Finally, control commands for the physical robot are generated based on the displacement increments of the physical robot at each operation stage to achieve variable proportional incremental control, thereby solving the cross-scale motion problem, achieving matching between hand movements and robot movements at each operation stage, and encapsulating the above-mentioned physical robot control commands and transmitting them to the physical robot wirelessly.

[0079] In one embodiment of this application, for further definition and explanation, before step 204 of the embodiment divides the teleoperation into operation stages based on the displacement increment of the multi-degree-of-freedom force feedback controller and a pre-generated operation stage division standard to obtain at least one operation stage, the embodiment method further includes generating an operation stage division standard, specifically including: determining a preset number of operation stages according to control requirements, and dividing the operation stages according to the preset number of operation stages; calculating the standard displacement and corresponding gain coefficient of each operation stage according to the preset number of operation stages, and generating an operation stage division standard based on the mapping relationship between the standard displacement and corresponding gain coefficient of each operation stage, for determining the operation stage to which it belongs.

[0080] Wherein, the standard displacement is used to characterize the desired range of motion in the operation stage; the gain coefficient is used to characterize the proportional relationship between the control signal and the output; the control requirements are preset according to the production scenario. For example, in a grinding scenario, due to the small amplitude of the movement, higher control precision is required. In this case, the value of the preset number of operation stages M can be set relatively large to achieve higher control precision and finer control. In the embodiments of this application, the standard displacement J k (k = 0, 1, ..., M-1) can be calculated according to the formula The calculation yields M, where M represents the number of preset operation stages; the gain coefficient M can be calculated using the formula... The calculations were performed. Finally, a criterion for dividing the operation stages was generated based on the mapping relationship between the standard displacements of each operation stage and the corresponding gain coefficients.

[0081] In one embodiment of this application, for further definition and explanation, step 103 of the embodiment calculates virtual additional force information based on the displacement increment of the physical robot, and superimposes the environmental contact force information and the virtual additional force information to generate tactile feedback information of the multi-degree-of-freedom force feedback controller. Specifically, this includes: calculating virtual additional force information based on the gain coefficient and the displacement increment of the physical robot; calculating the difference between the environmental contact force information and the operating force information, and then superimposing it with the virtual additional force information to generate tactile feedback information of the multi-degree-of-freedom force feedback controller for tactile correction.

[0082] Specifically, virtual applied force information f v According to the formula The tactile feedback information F of the multi-degree-of-freedom force feedback controller is calculated. m It can be calculated using the following formula.

[0083]

[0084] Among them, f s Indicates environmental contact force information, f m This indicates the operating force information.

[0085] In one embodiment of this application, for further definition and explanation, such as Figure 3 As shown, in step 103 of the embodiment, the position information of each joint is determined according to the virtual robot control instructions for visual correction, specifically including:

[0086] 301. Create a virtual 3D model of a physical robot in an augmented reality environment.

[0087] In this embodiment, firstly, based on the physical structure of the physical robot, multiple links and joints are sequentially connected to form a virtual robot. The joint coordinate system can be represented as a quadruple (l, α, d, θ), where l represents the link length, α represents the rotation angle between two adjacent links around the Z-axis, d represents the offset of two adjacent links on the Z-axis, and θ represents the joint angle. Further, the transformation matrix between adjacent joints is determined based on the transformation relationship between their coordinate systems to construct the kinematic relationship of the virtual robot. Specifically, the transformation matrix can be a 4×4 homogeneous transformation matrix used to describe the relative position and orientation from joint i to joint i-1, and can be represented as...

[0088]

[0089] Furthermore, based on the physical structure of the physical robot, the coordinate system corresponding to each end effector is determined, and this coordinate system is replaced with the coordinate system of the last joint to obtain the pose information of the virtual robot's end effector, that is, the position and orientation of the last link's coordinate system relative to the Cartesian space coordinate system, which can be represented as:

[0090]

[0091] Finally, based on the kinematic relationships and pose information of each joint, the pre-made component models of the physical robot are combined to create a virtual 3D model of the physical robot in an augmented reality environment.

[0092] Accordingly, step 301 of the embodiment specifically includes: establishing the joint coordinate system of each joint of the virtual robot based on the physical structure of the physical robot; determining the transformation matrix of adjacent joints based on the transformation relationship between the joint coordinate systems of two adjacent joints to construct the kinematic relationship of the virtual robot; determining the coordinate system corresponding to each end effector according to the physical structure of the physical robot; replacing the coordinate system with the joint coordinate system of the last joint of the virtual robot to obtain the position information of the end effector of the virtual robot; calculating the position information of each joint of the virtual robot based on the kinematic relationship; and establishing a virtual three-dimensional model of the physical robot in an augmented reality environment based on the kinematic relationship and position information of each joint.

[0093] 302. Based on the analytical method, construct the inverse solution formula according to the control instructions of the virtual robot to solve the problem.

[0094] 303. Perform matrix transformation and inverse transformation operations on the inverse solution formula to obtain multiple sets of inverse solutions for each joint of the virtual robot.

[0095] 304. Based on the connection relationship of each joint of the virtual robot, the range of motion angle of each joint is restricted in order to select the unique inverse solution of each joint from multiple sets of inverse solutions and obtain the position information of each joint.

[0096] The connection relationships between the joints are obtained based on the virtual 3D model of the physical robot built in an augmented reality environment.

[0097] It should be noted that virtual robots need to be controlled by solving the inverse equation for each joint. Specifically, since virtual robots satisfy the Pieper criterion, the inverse equation formula can be constructed analytically based on the control commands. By making the rows and columns of both matrices equal, multiple inverse solutions for each joint of the virtual robot are obtained using matrix transformations and inverse transformations. Furthermore, based on the connection relationships between the virtual robot's joints, the range of motion angles for each joint is restricted, and a unique inverse solution for each joint is selected to obtain the position information of each joint. This ensures that the visual feedback received by the operator through AR glasses regarding the virtual robot remains synchronized with the state of the physical robot, thereby accurately perceiving the position and orientation of the physical robot and improving the accuracy and efficiency of teleoperation.

[0098] This application provides a wireless teleoperation method for an augmented reality robot. First, it acquires motion information from a multi-degree-of-freedom force feedback controller (MDF controller), generates control commands for the physical robot based on the motion information, and transmits these commands wirelessly to the physical robot. Second, it controls the physical robot to perform the teleoperation according to the control commands, and wirelessly receives environmental contact force information and displacement increments collected by the physical robot. Finally, it calculates virtual additional force information based on the displacement increments of the physical robot, superimposes the environmental contact force information and the virtual additional force information to generate tactile feedback information from the MDF controller for tactile correction, calculates a displacement correction amount for the virtual robot based on the displacement increments of the MDF controller and the physical robot, and superimposes the displacement correction amount with the displacement increments of the virtual robot to obtain virtual robot control commands. These virtual robot control commands are then wirelessly transmitted to the virtual robot, enabling the virtual robot to determine the position information of each joint for visual correction. The position information is obtained by limiting the range of joint movement angles based on a virtual 3D model of the physical robot built in the augmented reality environment. Compared with existing technologies, the embodiments of this application divide the operation stages according to the end-effector acceleration of the master multi-degree-of-freedom force feedback controller and dynamically adjust the gain coefficient to achieve variable proportional incremental control of the slave physical robot, solving the cross-scale motion problem and achieving matching of hand movements and robot movements in each operation stage. Furthermore, the tactile feedback information is corrected based on virtual additional force information, and the position information of the slave virtual robot is corrected based on the displacement increment of the master multi-degree-of-freedom force feedback controller and the displacement increment of the slave physical robot, thereby improving the "hand-eye asynchrony" situation, optimizing the synchronization effect of visual-tactile feedback and robot teleoperation, and improving the accuracy and efficiency of teleoperation.

[0099] Furthermore, as a response to the above Figure 1 The implementation of the method shown in this application provides a wireless teleoperation system for augmented reality robots, such as... Figure 4 As shown, the system includes:

[0100] 41. Multi-degree-of-freedom force feedback controller; 42. Physical robot; 43. High-speed wireless communication module; 44. Virtual robot; 45. Multi-dimensional force sensor; 46. AR glasses;

[0101] The multi-degree-of-freedom force feedback controller is used to collect motion information from remote operation, generate control commands for the physical robot based on the motion information, and provide tactile feedback information to the operator based on environmental contact force information and virtual additional force information.

[0102] The physical robot is used to perform the teleoperation according to the physical robot control command, and to collect the displacement increment of the physical robot and the environmental contact force information sensed by the multi-dimensional force sensor;

[0103] The AR glasses are used to construct the kinematic relationship of the virtual robot based on the physical structure of the physical robot, and to build a virtual three-dimensional model of the physical robot in an augmented reality environment to construct the virtual robot.

[0104] The virtual robot is used to determine the position information of each joint according to the virtual robot control command for visual correction. The virtual robot control command is calculated based on the displacement increment of the multi-degree-of-freedom force feedback controller and the displacement increment of the physical robot to obtain the displacement correction amount of the virtual robot, and then the displacement correction amount is superimposed with the displacement increment of the virtual robot.

[0105] The high-speed wireless communication module is used to transmit the physical robot control commands to the physical robot, receive the environmental contact force information collected by the physical robot and the displacement increment of the physical robot, and transmit the virtual robot control commands to the virtual robot.

[0106] Optionally, the system may also include an end effector for performing the gripping action during the teleoperation.

[0107] In specific application scenarios, the wireless teleoperation system of an augmented reality robot can consist of multiple parts. The master end, the operator's end, can include a multi-DOF force feedback controller (such as a seven-DOF force feedback controller) to sense the operator's movements and the forces applied to them, generating control commands to control the robot's movement and receiving tactile feedback from the robot regarding the forces acting on it. AR glasses are used to virtually model the physical robot in the operator's field of vision, overlaying virtual information and scenes to create an immersive experience. The slave end, the robot's end, can include a physical robot (such as a six-DOF robot) to perform the same teleoperations as the master end's multi-DOF force feedback controller according to control commands; a virtual robot (i.e., a three-dimensional model of the physical robot in the AR glasses) to simulate the physical robot and move synchronously with it; and a multi-dimensional force sensor (such as a six-dimensional force sensor) to collect environmental contact force information between the robot's end effector and the environment. The master and slave ends may also include a high-speed wireless communication module (such as a 5G module) to provide low-latency and highly reliable data transmission between the master and slave ends; and the slave end may also include an end device (such as a gripper) to perform grasping tasks.

[0108] This application provides a wireless teleoperation system for an augmented reality robot, including a multi-degree-of-freedom force feedback controller, a physical robot, a high-speed wireless communication module, AR glasses, a virtual robot, and a multi-dimensional force sensor. The multi-degree-of-freedom force feedback controller is used to collect motion information from the teleoperation, generate control commands for the physical robot based on the motion information, and provide tactile feedback information to the operator based on environmental contact force information and virtual additional force information. The physical robot is used to execute the teleoperation according to the physical robot control commands, and collect the displacement increment of the physical robot and the environmental contact force information sensed by the multi-dimensional force sensor. The AR glasses are used to construct the kinematics of the virtual robot based on the physical structure of the physical robot. The system establishes a virtual 3D model of the physical robot in an augmented reality environment to construct a virtual robot. The virtual robot is used to determine the position information of each joint according to virtual robot control commands for visual correction. The virtual robot control commands are calculated based on the displacement increment of the multi-degree-of-freedom force feedback controller and the displacement increment of the physical robot, and then the displacement correction is superimposed on the displacement increment of the virtual robot. The high-speed wireless communication module is used to transmit the physical robot control commands to the physical robot, receive environmental contact force information collected by the physical robot and the displacement increment of the physical robot, and transmit the virtual robot control commands to the virtual robot. Compared with existing technologies, the embodiments of this application divide the operation stages according to the end-effector acceleration of the master multi-degree-of-freedom force feedback controller and dynamically adjust the gain coefficient to achieve variable proportional incremental control of the slave physical robot, solving the cross-scale motion problem and achieving matching of hand movements and robot movements in each operation stage. Furthermore, the tactile feedback information is corrected based on virtual additional force information, and the position information of the slave virtual robot is corrected based on the displacement increment of the master multi-degree-of-freedom force feedback controller and the displacement increment of the slave physical robot, thereby improving the "hand-eye asynchrony" situation, optimizing the synchronization effect of visual-tactile feedback and robot teleoperation, and improving the accuracy and efficiency of teleoperation.

[0109] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0110] The methods and systems of this application may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this application are not limited to the order specifically described above, unless otherwise specifically stated. Furthermore, in some embodiments, this application may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this application. Thus, this application also covers recording media storing programs for performing the methods according to this application.

[0111] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0112] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A wireless teleoperation method for an augmented reality robot, characterized in that, include: The system acquires motion information from the teleoperation of the multi-degree-of-freedom force feedback controller, generates control commands for the physical robot based on the motion information, and transmits the control commands to the physical robot wirelessly. The system controls the physical robot to perform the teleoperation according to the physical robot control instructions, and receives the environmental contact force information and the displacement increment of the physical robot collected by the physical robot wirelessly. The virtual applied force information is calculated based on the displacement increment of the physical robot. The environmental contact force information and the virtual applied force information are superimposed to generate the tactile feedback information of the multi-degree-of-freedom force feedback controller for tactile correction. The displacement correction amount of the virtual robot is calculated based on the displacement increment of the multi-degree-of-freedom force feedback controller and the displacement increment of the physical robot. The displacement correction amount is then superimposed with the displacement increment of the virtual robot to obtain the virtual robot control command. The virtual robot control command is transmitted to the virtual robot wirelessly so that the virtual robot can determine the position information of each joint according to the virtual robot control command for visual correction. The position information is obtained by limiting the range of joint movement angles based on the virtual three-dimensional model of the physical robot built in the augmented reality environment. The process of acquiring motion information from the multi-degree-of-freedom force feedback controller, generating control commands for the physical robot based on the motion information, and transmitting the control commands to the physical robot wirelessly includes: Acquire motion information of the multi-degree-of-freedom force feedback controller for teleoperation; The end effector acceleration of the multi-degree-of-freedom force feedback controller is calculated based on the operating force information and damping information contained in the motion information. The displacement increment of the multi-degree-of-freedom force feedback controller is calculated based on the end-effector acceleration and velocity information. Based on the displacement increment of the multi-degree-of-freedom force feedback controller, the teleoperation is divided into operation stages according to a pre-generated operation stage division standard to obtain at least one operation stage. The standard displacement and corresponding gain coefficient of each operation stage are calculated based on the number of operation stages. The displacement increment of the physical robot in each operation stage is calculated based on the gain coefficient and the displacement increment of the multi-degree-of-freedom force feedback controller, so as to generate physical robot control instructions based on the displacement increment of each physical robot. The control commands for the physical robot are encapsulated and transmitted wirelessly to the physical robot. Before dividing the teleoperation into operation stages based on the displacement increment of the multi-degree-of-freedom force feedback controller and a pre-generated operation stage division standard to obtain at least one operation stage, the method further includes generating an operation stage division standard, specifically including: The number of preset operation stages is determined according to the control requirements, and the operation stages are divided according to the preset number of operation stages; The standard displacement and corresponding gain coefficient of each operation stage are calculated according to the preset number of operation stages. The operation stage division criteria are generated based on the mapping relationship between the standard displacement and corresponding gain coefficient of each operation stage, and the operation stage to which it belongs is determined. The standard displacement is used to characterize the expected range of motion of the operation stage, and the gain coefficient is used to characterize the proportional relationship between the control signal and the output. The step of calculating virtual applied force information based on the displacement increment of the physical robot, superimposing the environmental contact force information and the virtual applied force information to generate tactile feedback information for the multi-degree-of-freedom force feedback controller for tactile correction specifically includes: The virtual applied force information is calculated based on the gain coefficient and the displacement increment of the physical robot. The difference between the environmental contact force information and the operational force information is calculated and then superimposed with the virtual additional force information to generate the tactile feedback information of the multi-degree-of-freedom force feedback controller. The step of determining the position information of each joint according to the virtual robot control instructions for visual correction specifically includes: The inverse solution formula is constructed based on the virtual robot control instructions using the analytical method. Matrix transformation and inverse transformation operations are performed on the inverse solution formula to obtain multiple sets of inverse solutions for each joint of the virtual robot; Based on the connection relationship of each joint of the virtual robot, the range of motion angle of each joint is restricted, so as to filter out the unique inverse solution of each joint from the multiple sets of inverse solutions and obtain the position information of each joint. The connection relationship of each joint is obtained based on the virtual three-dimensional model of the physical robot established in the augmented reality environment. Before restricting the range of motion angles of each joint based on the joint connection relationships of the virtual robot to filter out the unique inverse solution for each joint from the multiple sets of inverse solutions and obtain the position information of each joint, the method further includes establishing a virtual three-dimensional model of the physical robot in an augmented reality environment, specifically including: Based on the physical structure of the physical robot, establish the joint coordinate system of each joint of the virtual robot; The transformation matrix of adjacent joints is determined based on the transformation relationship between the joint coordinate systems of two adjacent joints, so as to construct the kinematic relationship of the virtual robot; Based on the kinematic relationship, the position information of each joint in the virtual robot is calculated; A virtual 3D model of the physical robot is built in an augmented reality environment based on the kinematic relationships and positional information of each joint. Before calculating the position information of each joint of the virtual robot based on the kinematic relationship, the method further includes determining the position information of the end effector of the virtual robot, specifically including: The coordinate system corresponding to each end effector is determined based on the physical structure of the physical robot; The coordinate system is replaced with the joint coordinate system of the last joint of the virtual robot to obtain the position information of the end effector of the virtual robot.

2. A wireless teleoperation system for an augmented reality robot, used to execute the wireless teleoperation method for an augmented reality robot as described in claim 1, characterized in that, include: Multi-degree-of-freedom force feedback controller, physical robot, high-speed wireless communication module, AR glasses, virtual robot, multi-dimensional force sensor; The multi-degree-of-freedom force feedback controller is used to collect motion information from remote operation, generate control commands for the physical robot based on the motion information, and provide tactile feedback to the operator based on environmental contact force information and virtual additional force information. The physical robot is used to perform the teleoperation according to the physical robot control command, and to collect the displacement increment of the physical robot and the environmental contact force information sensed by the multi-dimensional force sensor; The AR glasses are used to construct the kinematic relationship of the virtual robot based on the physical structure of the physical robot, and to build a virtual three-dimensional model of the physical robot in an augmented reality environment to construct the virtual robot. The virtual robot is used to determine the position information of each joint according to the virtual robot control command for visual correction. The virtual robot control command is calculated based on the displacement increment of the multi-degree-of-freedom force feedback controller and the displacement increment of the physical robot to obtain the displacement correction amount of the virtual robot, and then the displacement correction amount is superimposed with the displacement increment of the virtual robot. The high-speed wireless communication module is used to transmit the physical robot control commands to the physical robot, receive the environmental contact force information collected by the physical robot and the displacement increment of the physical robot, and transmit the virtual robot control commands to the virtual robot.

Images