Force tactile feedback teleoperation system and method for remotely controlling manipulator

Abstract

The embodiment of the invention discloses a force tactile feedback teleoperation system and a method for remotely controlling a manipulator. One specific embodiment of the system comprises a haptic feedback glove, a remote control manipulator and a control device, wherein the haptic feedback glove is configured to execute the following capturing steps: capturing human hand action information; the control device is configured to execute the following mapping steps: receiving human hand action information; generating mapping motion information of the manipulator; the remote control manipulator is controlled to do motion corresponding to the manipulator mapping motion information; the mapping motion information of the manipulator is sent to the remote control manipulator; the remote control manipulator is configured to execute the following feedback information acquisition steps: acquiring motion feedback information; acquiring tactile information when the robot is in contact with an object in the movement process; the motion feedback information and the touch information are sent to control equipment; the control device is further configured to perform a haptic rendering operation on the human hand. According to the embodiment, the user experience is improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a force tactile feedback teleoperation system and a remote control method for a manipulator. Background Art

[0002] With the continuous advancement of artificial intelligence technology, manipulators have been widely used in all aspects of life. A force tactile feedback teleoperation system is a system used to control the movement of a manipulator (for example, to control the manipulator to make a specified action). At present, when controlling the movement of a manipulator, the commonly used method is to control the movement trajectory and action of the manipulator through a pre-written program.

[0003] However, when the above method is used to control the movement of the robot, the following technical problems often occur:

[0004] First, the motion trajectory and movements of the robot are controlled by a pre-written program. The movements of the robot are usually fixed, and the flexibility of the robot's movements is poor. At the same time, the motion trajectory and movements of the robot are controlled by a pre-written program. The user cannot intuitively feel the movement of the robot and the interaction with the spatial environment during operation, and the user experience is poor.

[0005] Continuing, in the process of adopting technical solutions to solve the above-mentioned technical problem 1, the following technical problems often occur: in the process of making the manipulator contact various objects in the space environment, it is usually necessary to feed back the tactile sensation and feedback force when the manipulator contacts various objects in the space environment to the human hand that controls the movement of the manipulator. The conventional solution to feed back the tactile sensation and feedback force when the manipulator contacts various objects in the space environment to the human hand that controls the movement of the manipulator is generally to install ropes that bind the fingers and palms in the gloves that control the movement of the manipulator, and to make the human hand feel the feedback force by tightening the ropes. However, the following problems still exist:

[0006] Second, by installing ropes to bind fingers and palms in the gloves that control the movement of the robot, the human hand can feel the feedback force by tightening the ropes. There may be a nonlinear relationship between the tension of the ropes and the magnitude of the feedback force received by the robot during the movement of the robot, resulting in poor authenticity of the feedback force received by the human hand during the movement of the robot. Installing ropes to bind fingers and palms in gloves cannot feedback the tactile sensation of contact with spatial objects during the movement of the robot to the human hand. At the same time, using ropes to bind fingers and palms may give users a sense of restraint and discomfort, resulting in a poor user experience.

[0007] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0008] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0009] Some embodiments of the present disclosure propose a force tactile feedback teleoperation system and a remote control manipulator method to solve one or more of the technical intentions mentioned in the above background technology section.

[0010] In a first aspect, some embodiments of the present disclosure provide a force tactile feedback teleoperation system, the system comprising: a force tactile feedback glove, a remote-controlled manipulator, and a control device, wherein: the force tactile feedback glove is configured to perform the following capture steps: capture human hand motion information; transmit the human hand motion information to the control device; the control device is configured to perform the following mapping steps: receive the human hand motion information transmitted by the force tactile feedback glove; generate manipulator mapping motion information based on the human hand motion information; control the remote-controlled manipulator to make a motion corresponding to the manipulator mapping motion information, wherein the manipulator mapping motion information is mapping motion information of each mechanical joint or spatial coordinate information of each manipulator fingertip; send the manipulator mapping motion information to the remote-controlled manipulator; the remote-controlled manipulator is configured to perform the following feedback steps: Information collection step: in response to determining that the above-mentioned manipulator motion information is mapped motion information of each mechanical joint, collecting each current manipulator joint motion information of the manipulator joint where the preset manipulator angle sensor is located as motion feedback information through at least one preset manipulator angle sensor; in response to determining that the above-mentioned manipulator motion information is mapped as each manipulator fingertip spatial coordinate information, collecting each force information of the manipulator fingertip where the force sensor is located as motion feedback information through at least one force sensor on the above-mentioned at least one manipulator fingertip; collecting tactile information when contacting with an object during movement through at least one tactile feedback information collector; sending the above-mentioned motion feedback information and tactile information to the above-mentioned control device; the above-mentioned control device is further configured to perform force tactile reproduction operation on the human hand through the above-mentioned force tactile feedback glove based on the above-mentioned motion feedback information and tactile information.

[0011] In a second aspect, some embodiments of the present disclosure provide a method for remotely controlling a robot, the method comprising: receiving human hand motion information transmitted by the above-mentioned force tactile feedback glove; generating robot mapping motion information based on the above-mentioned human hand motion information; and controlling the above-mentioned remotely controlled robot to make movements corresponding to the above-mentioned robot mapping motion information.

[0012] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the force tactile feedback teleoperation system of some embodiments of the present disclosure, the flexibility of the manipulator movement and the user experience are improved. Specifically, the reasons for the poor flexibility and user experience of the manipulator movement are: the movement trajectory and movement of the manipulator are controlled by a pre-written program, the movement of the manipulator is usually relatively fixed, and the flexibility of the manipulator movement is poor. At the same time, the movement trajectory and movement of the manipulator are controlled by a pre-written program. The user cannot intuitively feel the movement of the manipulator and the interaction with the space environment during the operation process, and the user experience is poor. Based on this, the force tactile feedback teleoperation system of some embodiments of the present disclosure includes: a force tactile feedback glove, a remote control manipulator, and a control device, wherein: first, the force tactile feedback glove is configured to perform the following capture steps: capture human hand movement information; transmit the human hand movement information to the control device. Thus, the human hand movement information used to control the movement of the remote control manipulator can be captured by the force tactile feedback glove. Then, the control device is configured to perform the following mapping steps: receiving the human hand motion information transmitted by the force tactile feedback glove; generating the manipulator mapping motion information based on the human hand motion information; controlling the remote-controlled manipulator to make a motion corresponding to the manipulator mapping motion information, wherein the manipulator mapping motion information is the mapping motion information of each mechanical joint or the spatial coordinate information of each manipulator fingertip; sending the manipulator mapping motion information to the remote-controlled manipulator. Thus, the manipulator mapping motion information can be generated by the control device, and the remote-controlled manipulator movement can be controlled based on the manipulator mapping motion information. The remote-controlled manipulator is controlled by the force tactile feedback glove and the control device to make a motion consistent with the human hand motion information, thereby improving the flexibility of the manipulator movement. Then, the remote control manipulator is configured to perform the following feedback information collection steps: in response to determining that the motion information mapped by the manipulator is the motion information mapped by each mechanical joint, the motion information of each current manipulator joint where the preset manipulator angle sensor is located is collected as motion feedback information through at least one preset manipulator angle sensor; in response to determining that the motion information mapped by the manipulator is the spatial coordinate information of each manipulator fingertip, the force information of each manipulator fingertip where the force sensor is located is collected as motion feedback information through at least one force sensor on the fingertip of at least one manipulator; the tactile information when contacting with the object during the motion is collected through at least one tactile feedback information collector; the motion feedback information and tactile information are sent to the control device. Thus, the motion feedback information and tactile information of the remote control manipulator interacting with the spatial environment during the motion can be collected through at least one preset manipulator angle sensor or at least one force sensor. Afterwards, the control device is further configured to perform force tactile reproduction operation on the human hand through the force tactile feedback glove based on the motion feedback information and tactile information.Therefore, the motion feedback information and tactile information of the remote-controlled manipulator's interaction with the spatial environment during movement can be applied to the human hand through the force tactile feedback gloves through the control device, so that the user can intuitively feel the movement of the manipulator and the interaction with the spatial environment during operation, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0014] Figure 1 is an architectural diagram of an exemplary system of a force haptic feedback teleoperation system according to the present disclosure;

[0015] Figure 2 Flow charts of some embodiments of the remote control manipulator method according to the present disclosure;

[0016] Figure 3 is an internal test diagram of some embodiments of the force tactile feedback teleoperation system according to the present disclosure;

[0017] Figure 4 is an internal product diagram of some embodiments of the force tactile feedback gloves according to the present disclosure. DETAILED DESCRIPTION

[0018] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0019] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0020] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0021] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0022] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0023] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0024] Figure 1 An exemplary system architecture 100 is shown to which some embodiments of the present disclosure may be applied for a force haptic feedback teleoperation system.

[0025] like Figure 1 As shown, the system architecture 100 may include: a force tactile feedback glove 102, a remote control manipulator 101, and a control device 103. The force tactile feedback glove 102 and the control device 103 may be connected via a medium of a communication link. The remote control manipulator 101 and the control device 103 may be connected via a medium of a communication link. The medium of the communication link may be a physical medium or method for transmitting data, such as a wired or wireless communication link or an optical fiber cable, etc. The control device may be a computing device.

[0026] Figure 4 is an internal product diagram of some embodiments of the force tactile feedback glove 102 according to the present disclosure.

[0027] In some embodiments, the force tactile feedback glove 102 may be configured to perform the following capture steps:

[0028] The first step is to capture the hand movement information. The force tactile feedback glove 102 includes at least one angle sensor (for example, the thumb of the force tactile feedback glove 102 is equipped with four angle sensors, and each of the remaining fingers is equipped with three angle sensors). A fingertip space position collector can also be installed at the preset position of the fingertip of each finger of the force tactile feedback glove. A wrist space position collector can also be installed in the wrist area of ​​the force tactile feedback glove. The fingertip space position collector and the wrist space position collector can be, but are not limited to, one of the following: an ultrasonic tracking sensor, a GPS positioning sensor.

[0029] The second step is to transmit the above-mentioned human hand movement information to the above-mentioned control device 103.

[0030] In some optional implementations of some embodiments, the force tactile feedback glove 102 may obtain the human hand motion information through the following steps, including:

[0031] In the first step, each of the above-mentioned angle sensors collects the joint motion information of the hand joint where the angle sensor is located. The joint motion information corresponds to the angle sensor identifier of the angle sensor. The joint motion information can be the angle collected by the angle sensor that represents the hand joint at a moment. For example, the joint motion information can be "the angle of the hand joint corresponding to hand joint 1 at the time point 5 seconds is 85 degrees". The hand joint 1 can be a hand joint identifier.

[0032] The second step is to determine the collected joint motion information as hand motion information, wherein each preset manipulator angle sensor in at least one preset manipulator angle sensor corresponds to one of the above angle sensors, and each current manipulator joint motion information in the above current manipulator joint motion information corresponds to one of the above joint motion information.

[0033] In some optional implementations of some embodiments, the force tactile feedback glove 102 may capture human hand motion information through the following steps, including:

[0034] The first step is to collect the spatial point position information of the human wrist through the wrist spatial position collector. The spatial point position information of the human wrist includes the wrist spatial point coordinates. The spatial point position information of the human wrist can represent the spatial coordinates of a preset point in the human wrist. The wrist spatial point coordinates can be the spatial coordinates of the preset point.

[0035] In the second step, each of the fingertip spatial position collectors collects the fingertip spatial position information of the fingertip where it is located, and obtains each fingertip spatial position information. Among them, each fingertip spatial position information in the above-mentioned fingertip spatial position information includes fingertip spatial point coordinates. The above-mentioned fingertip spatial position information can represent the spatial position of a preset point in the fingertip of the force tactile feedback glove. The above-mentioned fingertip spatial point coordinates can represent the spatial coordinates of the preset point in the above-mentioned fingertip.

[0036] In the third step, the spatial position information of each fingertip and the spatial position information of the wrist of the human hand are determined as the human hand motion information.

[0037] In some embodiments, the control device 103 may be configured to perform the following mapping steps:

[0038] The first step is to receive the human hand movement information transmitted by the force tactile feedback gloves.

[0039] The second step is to generate robot arm mapping motion information based on the above-mentioned human hand motion information.

[0040] The third step is to control the remote-controlled manipulator to make a movement corresponding to the mapped motion information of the manipulator. The mapped motion information of the manipulator is the mapped motion information of each mechanical joint or the spatial coordinate information of each manipulator fingertip. Each mechanical joint mapped motion information in the mapped motion information of each mechanical joint can represent the angle of the mechanical joint in space. For example, the mechanical joint mapped motion information can be "mechanical joint 1, angle 90 degrees". In practice, in response to determining that the mapped motion information of the manipulator is the mapped motion information of each mechanical joint, each mechanical joint included in the remote-controlled manipulator is controlled to make an angle represented by the corresponding mechanical joint mapped motion information. In response to determining that the mapped motion information of the manipulator is the spatial coordinate information of each manipulator fingertip, the remote-controlled manipulator is controlled to make a movement corresponding to the mapped motion information of the manipulator according to the mapped motion information of the manipulator through the inverse kinematics technology.

[0041] The fourth step is to send the above-mentioned robot mapping motion information to the above-mentioned remote-controlled robot.

[0042] In some optional implementations of some embodiments, the control device 103 may generate the robot arm mapping motion information based on the human hand motion information through the following steps, including:

[0043] The first step is to obtain the preset human hand motion limit range information corresponding to the above-mentioned force tactile feedback glove and the preset motion range information of the above-mentioned remote-controlled manipulator. Among them, the above-mentioned preset human hand motion limit range information includes the motion range information of each joint of each joint, and the above-mentioned preset motion range information includes the motion range information of each mechanical joint of each remote-controlled manipulator joint. Each joint motion range information in the above-mentioned joint motion range information corresponds to each mechanical joint motion range information in the above-mentioned mechanical joint motion range information. Each joint motion range information in the above-mentioned joint motion range information corresponds to each joint motion information in the above-mentioned joint motion range information. In practice, the above-mentioned control device 103 can obtain the preset human hand motion limit range information of the force tactile feedback glove and the preset motion range information of the above-mentioned remote-controlled manipulator from a preset file. Each joint motion range information in the above-mentioned joint motion range information of each joint can represent the bending degree range of the human hand joint. Each mechanical joint motion range information in the above-mentioned mechanical joint motion range information can represent the bending degree range of the mechanical joint. For example, the above-mentioned joint motion range information can be "0 degrees to 60 degrees". The above-mentioned mechanical joint motion range information may be “0 degrees to 60 degrees”.

[0044] In the second step, for each joint motion information included in the above-mentioned human hand motion information, the following steps are performed:

[0045] In the first sub-step, the joint motion range information corresponding to the above-mentioned joint motion range information is determined as the target joint motion range information.

[0046] The second sub-step is to generate joint activity ratio information based on the joint motion information and the target joint motion range information. In practice, the control device 103 may determine the absolute value of the difference between the upper limit value and the lower limit value corresponding to the target joint motion range information as the first value. Then, the control device 103 may refer to the joint angle corresponding to the joint motion information as the second value. Afterwards, the control device 103 may determine the ratio of the second value to the first value as the joint activity ratio information. As an example, the target joint motion range information may be "0 degrees to 60 degrees". The first value may be 60 degrees. The joint motion information may be "the angle of the hand joint corresponding to the time point 5 seconds is 30 degrees". The first value may be 30 degrees. The joint activity ratio information may be one-half (i.e., 30 degrees / 60 degrees).

[0047] In a third sub-step, the mechanical joint motion range information corresponding to the target joint motion range information among the above-mentioned mechanical joint motion range information is determined as the target mechanical joint motion range information.

[0048] The fourth sub-step is to generate mechanical joint mapping motion information based on the above-mentioned joint activity ratio information and the above-mentioned target mechanical joint motion range information. In practice, the above-mentioned control device 103 can determine the absolute value of the difference between the upper limit value and the lower limit value of the above-mentioned target mechanical joint motion range information as the motion range span value. Afterwards, the above-mentioned control device 103 can determine the product of the numerical value corresponding to the joint activity ratio information and the above-mentioned motion range span value as the target numerical value. Afterwards, the above-mentioned control device 103 can determine the sum of the lower limit value of the target mechanical joint motion range information and the above-mentioned target numerical value as the mechanical joint mapping motion information.

[0049] The third step is to determine the generated mechanical joint mapping motion information as the manipulator mapping motion information, wherein each current manipulator joint motion information in the above-mentioned current manipulator joint motion information corresponds to one mechanical joint mapping motion information in the above-mentioned mechanical joint mapping motion information.

[0050] In some optional implementations of some embodiments, the control device 103 may generate the robot arm mapping motion information based on the human hand motion information through the following steps, including:

[0051] The first step is to determine the wrist space point coordinates included in the above-mentioned human hand movement information as the reference wrist space point coordinates.

[0052] The second step is to collect the spatial coordinates of the preset point in the wrist of the manipulator as the spatial coordinates of the wrist of the manipulator. In practice, the control device 103 can collect the spatial coordinates of the wrist of the manipulator through a wrist spatial position collector installed at the preset point in the wrist of the manipulator.

[0053] In the third step, for each piece of fingertip spatial position information included in the above-mentioned human hand motion information, the following steps are performed:

[0054] The first sub-step is to generate relative spatial coordinate information between the reference wrist spatial point coordinates and the fingertip spatial position information based on the reference wrist spatial point coordinates and the fingertip spatial position information; in practice, the control device 103 can determine the difference between the horizontal coordinate value of the reference wrist spatial point coordinates and the horizontal coordinate value contained in the fingertip spatial position information as the relative horizontal coordinate difference. Afterwards, the vertical coordinate value of the reference wrist spatial point coordinates and the vertical coordinate value contained in the fingertip spatial position information can be determined as the relative vertical coordinate difference. Next, the vertical coordinate value of the reference wrist spatial point coordinates and the vertical coordinate value contained in the fingertip spatial position information can be determined as the relative vertical coordinate difference. Finally, the control device 103 can determine the relative horizontal coordinate difference, the relative vertical coordinate difference, and the relative vertical coordinate difference as the relative spatial coordinate information between the reference wrist spatial point coordinates and the fingertip spatial position information.

[0055] The second sub-step is to generate the manipulator fingertip spatial coordinate information based on the manipulator wrist spatial point coordinates and the relative spatial coordinate information. In practice, the control device 103 can determine the sum of the horizontal coordinate value included in the manipulator wrist spatial point coordinates and the relative horizontal coordinate difference included in the relative spatial coordinate information as the manipulator fingertip spatial horizontal coordinate. Afterwards, the sum of the vertical coordinate value included in the manipulator wrist spatial point coordinates and the relative vertical coordinate difference included in the relative spatial coordinate information is determined as the manipulator fingertip spatial vertical coordinate. Next, the sum of the vertical coordinate value included in the manipulator wrist spatial point coordinates and the relative vertical coordinate difference included in the relative spatial coordinate information can be determined as the manipulator fingertip spatial vertical coordinate. Finally, the control device 103 can determine the manipulator fingertip spatial horizontal coordinate, the manipulator fingertip spatial vertical coordinate and the manipulator fingertip spatial vertical coordinate as the manipulator fingertip spatial coordinate information.

[0056] The fourth step is to determine the generated spatial coordinate information of each robot fingertip as the robot mapping motion information.

[0057] In some embodiments, the remote control manipulator 101 may be configured to perform the following steps:

[0058] The first step, in response to determining that the above-mentioned manipulator mapping motion information is the mapping motion information of each mechanical joint, each current manipulator joint motion information of the manipulator joint where the preset manipulator angle sensor is located is collected as motion feedback information through at least one preset manipulator angle sensor. Among them, each current manipulator joint motion information in the above-mentioned current manipulator joint motion information can be the angle of bending or tilt that characterizes the manipulator joint collected by the above-mentioned preset manipulator angle sensor. For example, the above-mentioned current manipulator joint motion information can be "the angle of the mechanical joint corresponding to the manipulator joint 1 at the time point 6 seconds is 85 degrees." The above-mentioned at least one preset manipulator angle sensor can be at least one angle sensor installed in the remote-controlled manipulator (for example, the manipulator thumb of the remote-controlled manipulator 101 is installed with four angle sensors, and the remaining manipulator fingers are each installed with three angle sensors). The above-mentioned manipulator joint 1 can be an identifier of the manipulator joint.

[0059] In the second step, in response to determining that the motion information mapped by the manipulator is the spatial coordinate information of each manipulator fingertip, at least one force sensor on the at least one manipulator fingertip collects the force information of the manipulator fingertip where the force sensor is located as motion feedback information. The at least one manipulator fingertip corresponds one-to-one to at least one human fingertip corresponding to at least one finger of the force tactile feedback glove.

[0060] The third step is to collect tactile information when the remote control manipulator 101 contacts an object during the movement by at least one tactile feedback information collector, wherein the tactile feedback information collector may be a tactile sensor for collecting tactile information when the remote control manipulator 101 contacts an object in space.

[0061] The fourth step is to send the motion feedback information and tactile information to the control device.

[0062] In some embodiments, the control device 103 may be further configured to perform force tactile reproduction operation on the human hand through the force tactile feedback glove based on the motion feedback information and tactile information. Wherein, a tactile feedback device is installed in a preset area of ​​the fingertip of each finger in the force tactile feedback glove, a force feedback actuator is installed in a preset area of ​​each finger in the force tactile feedback glove, each force feedback actuator in the at least one force feedback actuator included in the force tactile feedback glove corresponds to each force sensor in the at least one force sensor, the force feedback actuator is composed of a steering wheel, a winch, and a tendon rope, the tactile information includes at least one tactile sub-information collected by the at least one tactile feedback information collector, and each tactile feedback information collector in the at least one tactile feedback information collector corresponds one-to-one to each tactile feedback device in the at least one tactile feedback glove. The tactile feedback device may be a device that transmits tactile sensation by expressing vibration, electrical stimulation, ultrasound, array, etc. For example, the tactile feedback device may be, but is not limited to, one of the following: an ERM or LRA vibration motor. The force feedback actuator may be a device that applies feedback force to a human hand. Each tactile sub-information in the at least one tactile sub-information may be information collected by one of the at least one tactile feedback devices.

[0063] In some optional implementations of some embodiments, the control device 103 may be further configured to perform a force tactile reproduction operation on a human hand through the force tactile feedback glove based on the motion feedback information and the tactile information, including:

[0064] In the first step, in response to determining that the motion feedback information maps motion information to each mechanical joint, based on the motion feedback information, the following first force feedback reproduction operation is performed:

[0065] The first sub-step is to perform the following steps for each current manipulator joint motion information in each current manipulator joint motion information included in the motion feedback information:

[0066] Sub-step 1: determining the mechanical joint mapping motion information corresponding to the current manipulator joint motion information among the above-mentioned various mechanical joint mapping motion information as the target mechanical joint mapping motion information.

[0067] Sub-step 2, based on the current manipulator joint motion information and the target mechanical joint mapping motion information, generate torque force information. Wherein, the torque force information corresponds to the angle sensor, and the angle sensor corresponds to the force feedback actuator. In practice, the control device 103 can determine the angle of the mechanical joint corresponding to the current manipulator joint motion information as the first target angle. Then, the control device 103 can determine the angle represented by the target mechanical joint mapping motion information as the second target angle. After that, the control device 103 can determine the absolute value of the difference between the second angle and the first angle as the target angle difference. After that, the control device 103 can query the preset angle difference and torque relationship table to obtain the torque corresponding to the target angle difference. After that, the control device 103 can determine the length of the mechanical joint where the preset manipulator angle sensor that collects the current manipulator joint motion information is located. Finally, the product of the length of the mechanical joint and the torque is determined as the torque force information. Wherein, the preset angle difference and torque relationship table can be a preset table representing the one-to-one correspondence between angle difference and torque. As an example, the target angle difference can be 5 degrees. The torque corresponding to the 5 degrees may be 3Nm. The length corresponding to the length of the mechanical joint may be 0.5m. The torque force information may be "3Nm×0.5m=1.5N".

[0068] Sub-step three, controlling the force feedback actuator corresponding to the torque force information in the force tactile feedback glove to apply the feedback force corresponding to the torque force information to the human hand. In practice, the control device 103 can determine the movement direction of the hand joint corresponding to the mechanical joint through the gesture tracking sensor (for example, optical fiber sensor) in the force tactile feedback glove. Afterwards, the feedback force corresponding to the movement direction and torque force information can be applied to the hand joint of the human hand corresponding to the mechanical joint through the force feedback actuator. Among them, the mechanical hand joints correspond one-to-one to the hand joints of the human hand corresponding to the force tactile feedback glove.

[0069] In the second step, in response to determining that the motion feedback information is each piece of force information, based on the motion feedback information, the following second force feedback reproduction operation is performed:

[0070] The first sub-step is to perform the following steps for each piece of force information included in the force feedback information:

[0071] Sub-step 1: determining a force sensor for collecting the above force information.

[0072] Sub-step two: determining a force feedback actuator corresponding to the force sensor among the at least one force feedback actuator as a target force feedback actuator.

[0073] Sub-step three, control the target force feedback actuator to apply the feedback force corresponding to the force information to the human hand. In practice, the control device 103 can determine the direction of movement of the fingertips of the human hand through a gesture tracking sensor (e.g., an optical fiber sensor) in a force tactile feedback glove. Among them, the fingertips of the human hand can be the fingertips of the human hand corresponding to the fingertips of the mechanical finger where the force sensor for collecting the force information is located. Afterwards, the control device 103 can apply the feedback force corresponding to the direction of movement and the force information to the fingertips of the human hand through the force feedback actuator. Among them, each of the at least one force feedback actuator corresponds one-to-one to the at least one force sensor collecting force sensor.

[0074] In the third step, for each tactile sub-information in the at least one tactile sub-information included in the tactile information, the following steps are performed:

[0075] The first sub-step is to determine a tactile feedback information collector that collects the tactile sub-information;

[0076] In a second sub-step, a tactile feedback device corresponding to the tactile feedback information collector among at least one tactile feedback device in the force tactile feedback glove is determined as a target tactile feedback device;

[0077] The third sub-step is to control the target tactile feedback device to apply the tactile effect corresponding to the tactile sub-information to a preset area of ​​the fingertip where the tactile feedback device is located.

[0078] The above technical solution and its related contents, as an inventive point of an embodiment of the present disclosure, solve the second technical problem mentioned in the background technology: "By installing a rope for binding fingers and palms in a glove for controlling the movement of a manipulator, the human hand is made to feel the feedback force by tightening the rope. There may be a nonlinear relationship between the tension of the rope and the magnitude of the feedback force received by the manipulator during the movement of the manipulator, resulting in poor authenticity of the feedback force received by the human hand during the movement of the manipulator. The rope for binding fingers and palms installed in the glove cannot feedback the tactile sensation of contact with space objects during the movement of the manipulator to the human hand. At the same time, using ropes to bind fingers and palms may bring a sense of restraint and discomfort to the user, resulting in a poor user experience." The factors that lead to a poor user experience are often as follows: By installing a rope for binding fingers and palms in a glove for controlling the movement of a manipulator, the human hand is made to feel the feedback force by tightening the rope. There may be a nonlinear relationship between the tension of the rope and the magnitude of the feedback force received by the manipulator during the movement of the manipulator, resulting in poor authenticity of the feedback force received by the human hand during the movement of the manipulator. The ropes installed in the gloves to bind the fingers and palms cannot feedback the tactile sensation of the manipulator contacting the space object during the movement to the human hand. At the same time, the use of ropes to bind the fingers and palms may bring a sense of restraint and discomfort to the user, resulting in a poor user experience. If the above factors are solved, the effect of improving the user experience can be achieved. In order to achieve this effect, the first step is to respond to the determination that the motion feedback information is the motion information mapped for each mechanical joint, and based on the motion feedback information, the following first force feedback reproduction operation is performed: the first sub-step, for each current manipulator joint motion information included in the above motion feedback information, the following steps are performed: sub-step one, the mechanical joint mapping motion information corresponding to the above current manipulator joint motion information in the above mechanical joint mapping motion information is determined as the target mechanical joint mapping motion information. Sub-step two, based on the above current manipulator joint motion information and the above target mechanical joint mapping motion information, torque force information is generated, wherein the above torque force information corresponds to the angle sensor, and the angle sensor corresponds to the force feedback actuator. Sub-step three, control the force feedback actuator corresponding to the torque force information in the force tactile feedback glove to apply the feedback force corresponding to the torque force information to the human hand. Thus, the magnitude of the feedback force actually applied to the current manipulator joint during the movement, i.e., the torque force information, can be determined based on the difference between each current manipulator joint motion information (current actual motion position) included in the motion feedback information and the corresponding mechanical joint mapping motion information (target motion position) through the above steps. The actual feedback force applied to the human hand by the force feedback actuator included in the force tactile feedback glove improves the authenticity of the feedback force, reduces the sense of restraint and discomfort brought to the user, and improves the user experience.The second step, in response to determining that the motion feedback information is each force information, based on the motion feedback information, performs the following second force feedback reproduction operation: the first sub-step, for each force information included in the above force feedback information, performs the following steps: sub-step one, determine the force sensor that collects the above force information. Sub-step two, determine the force feedback actuator corresponding to the above force sensor in the at least one force feedback actuator as the target force feedback actuator. Sub-step three, control the above target force feedback actuator to apply the feedback force corresponding to the above force information to the human hand. In this way, when the motion feedback information is each force information, the actual feedback force can be applied to the human hand through the force feedback actuator included in the force tactile feedback glove, thereby improving the authenticity of the feedback force, reducing the sense of restraint and discomfort brought to the user, and improving the user experience. The third step is to perform the following steps for each tactile sub-information in at least one tactile sub-information included in the tactile information: the first sub-step is to determine the tactile feedback information collector that collects the tactile sub-information; the second sub-step is to determine the tactile feedback device corresponding to the tactile feedback information collector in at least one tactile feedback device in the force tactile feedback glove as the target tactile feedback device; the third sub-step is to control the target tactile feedback device to apply the tactile action corresponding to the tactile sub-information to the preset fingertip area where the tactile feedback device is located. In this way, the tactile action corresponding to the tactile information collected by the remote-controlled manipulator when it contacts an object during movement can be applied to the human hand through the tactile feedback device included in the force tactile feedback glove, thereby improving the user experience.

[0079] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the force tactile feedback teleoperation system of some embodiments of the present disclosure, the flexibility of the manipulator movement and the user experience are improved. Specifically, the reasons for the poor flexibility and user experience of the manipulator movement are: the movement trajectory and movement of the manipulator are controlled by a pre-written program, the movement of the manipulator is usually relatively fixed, and the flexibility of the manipulator movement is poor. At the same time, the movement trajectory and movement of the manipulator are controlled by a pre-written program. The user cannot intuitively feel the movement of the manipulator and the interaction with the space environment during the operation process, and the user experience is poor. Based on this, the force tactile feedback teleoperation system of some embodiments of the present disclosure includes: a force tactile feedback glove, a remote control manipulator, and a control device, wherein: first, the force tactile feedback glove is configured to perform the following capture steps: capture human hand movement information; transmit the human hand movement information to the control device. Thus, the human hand movement information used to control the movement of the remote control manipulator can be captured by the force tactile feedback glove. Then, the control device is configured to perform the following mapping steps: receiving the human hand motion information transmitted by the force tactile feedback glove; generating the manipulator mapping motion information based on the human hand motion information; controlling the remote-controlled manipulator to make a motion corresponding to the manipulator mapping motion information, wherein the manipulator mapping motion information is the mapping motion information of each mechanical joint or the spatial coordinate information of each manipulator fingertip; sending the manipulator mapping motion information to the remote-controlled manipulator. Thus, the manipulator mapping motion information can be generated by the control device, and the remote-controlled manipulator movement can be controlled based on the manipulator mapping motion information. The remote-controlled manipulator is controlled by the force tactile feedback glove and the control device to make a motion consistent with the human hand motion information, thereby improving the flexibility of the manipulator movement. Then, the remote control manipulator is configured to perform the following feedback information collection steps: in response to determining that the motion information mapped by the manipulator is the motion information mapped by each mechanical joint, the motion information of each current manipulator joint where the preset manipulator angle sensor is located is collected as motion feedback information through at least one preset manipulator angle sensor; in response to determining that the motion information mapped by the manipulator is the spatial coordinate information of each manipulator fingertip, the force information of each manipulator fingertip where the force sensor is located is collected as motion feedback information through at least one force sensor on the fingertip of at least one manipulator; the tactile information when contacting with the object during the motion is collected through at least one tactile feedback information collector; the motion feedback information and tactile information are sent to the control device. Thus, the motion feedback information and tactile information of the remote control manipulator interacting with the spatial environment during the motion can be collected through at least one preset manipulator angle sensor or at least one force sensor. Afterwards, the control device is further configured to perform force tactile reproduction operation on the human hand through the force tactile feedback glove based on the motion feedback information and tactile information.Therefore, the motion feedback information and tactile information of the remote-controlled manipulator's interaction with the spatial environment during movement can be applied to the human hand through the force tactile feedback gloves through the control device, so that the user can intuitively feel the movement of the manipulator and the interaction with the spatial environment during operation, thereby improving the user experience.

[0080] Figure 2 The process 200 of some embodiments of the remote control manipulator method using the control device included in the force tactile feedback teleoperation system according to the present disclosure is shown. The remote control manipulator method includes the following steps:

[0081] Step 201: receiving human hand motion information transmitted by a force tactile feedback glove.

[0082] In some embodiments, the execution entity of the remote-controlled manipulator (eg, the control device included in the force tactile feedback teleoperation system) can receive the human hand motion information transmitted by the force tactile feedback glove.

[0083] Step 202: Generate robot arm mapping motion information based on human hand motion information.

[0084] In some embodiments, the execution entity may generate robot arm mapping motion information based on the human hand motion information.

[0085] Step 203, controlling the remote-controlled manipulator to make a movement corresponding to the manipulator mapped movement information.

[0086] In some embodiments, the execution subject may control the remote-controlled robot to make movements corresponding to the motion information mapped by the robot.

[0087] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the remote control manipulator method applied to the control device included in the above-mentioned force tactile feedback teleoperation system in some embodiments of the present disclosure, the flexibility of the manipulator movement is improved. Specifically, the reason for the poor flexibility of the manipulator movement is that the movement trajectory and movement of the manipulator are controlled by a pre-written program, the movement of the manipulator is usually relatively fixed, and the flexibility of the manipulator movement is poor. Based on this, the remote control manipulator method applied to the control device included in the above-mentioned force tactile feedback teleoperation system in some embodiments of the present disclosure includes: first, receiving the human hand movement information transmitted by the above-mentioned force tactile feedback glove. Thus, the human hand movement information representing the real movement of the human hand can be collected through the force tactile feedback glove. Then, based on the above-mentioned human hand movement information, the manipulator mapping movement information is generated. Thus, the manipulator mapping movement information for controlling the movement of the remote control manipulator can be generated according to the human hand movement information. Afterwards, the above-mentioned remote control manipulator is controlled to make a movement corresponding to the above-mentioned manipulator mapping movement information. Thus, the remote control manipulator can be controlled to make a movement corresponding to the above-mentioned manipulator mapping movement information. Also, by collecting the human hand motion information of the real movement of the human hand, based on the human hand motion information, the robot mapping motion information for controlling the movement of the robot is generated. The robot can move accordingly following the movement of the human hand, thereby improving the flexibility of the robot movement.

[0088] Figure 3 is an internal test diagram of some embodiments of the force tactile feedback teleoperation system according to the present disclosure, Figure 3 A test teleoperated manipulator 301 and a test force haptic feedback glove 302 are shown under test.

[0089] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of technical features, but should also cover other technical solutions formed by any combination of technical features or their equivalent features without departing from the inventive concept. For example, a technical solution formed by replacing the features with (but not limited to) the technical features with similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A force tactile feedback teleoperation system, characterized in that: The force tactile feedback teleoperation system comprises: a force tactile feedback glove, a remote control manipulator, and a control device, wherein: The force tactile feedback glove is configured to perform the following capture steps: Capture human hand motion information; transmitting the human hand motion information to the control device; The control device is configured to perform the following mapping steps: Receiving human hand motion information transmitted by the force tactile feedback glove; Based on the human hand motion information, generating robot arm mapping motion information; Controlling the remote-controlled manipulator to make a movement corresponding to the manipulator mapped motion information, wherein the manipulator mapped motion information is the mapped motion information of each mechanical joint or the spatial coordinate information of each manipulator fingertip; Sending the manipulator mapping motion information to the remote-controlled manipulator; The remote control manipulator is configured to perform the following feedback information collection steps: In response to determining that the manipulator mapping motion information is the mapping motion information of each mechanical joint, collecting, through at least one preset manipulator angle sensor, each current manipulator joint motion information of the manipulator joint where the preset manipulator angle sensor is located as motion feedback information; In response to determining that the motion information mapped by the manipulator is spatial coordinate information of each manipulator fingertip, collecting, through at least one force sensor on at least one manipulator fingertip, each force information of the manipulator fingertip where the force sensor is located as motion feedback information; Collecting tactile information when contacting an object during movement through at least one tactile feedback information collector; Sending the motion feedback information and tactile information to the control device; The control device is further configured to perform a force tactile reproduction operation on a human hand through the force tactile feedback glove based on the motion feedback information and the tactile information.

2. The force tactile feedback teleoperation system according to claim 1, characterized in that: The force tactile feedback glove includes various angle sensors, and the force tactile feedback glove is further configured to capture human hand motion information through the following steps, including: Collecting joint motion information of the hand joint where the angle sensor is located through each of the angle sensors, wherein the joint motion information corresponds to the angle sensor identifier of the angle sensor; The collected joint motion information is determined as hand motion information, wherein each preset manipulator angle sensor of at least one preset manipulator angle sensor corresponds to one of the angle sensors, and each current manipulator joint motion information of the current manipulator joint motion information corresponds to one joint motion information of the joint motion information.

3. The force tactile feedback teleoperation system according to claim 2, characterized in that: The control device is further configured to generate robot arm mapping motion information based on the human hand motion information through the following steps, including: Obtaining preset human hand motion limit range information corresponding to the force tactile feedback glove and preset motion range information of the remote-controlled manipulator, wherein the preset human hand motion limit range information includes each joint motion range information of each joint, the preset motion range information includes each mechanical joint motion range information of each remote-controlled manipulator joint, each joint motion range information in the each joint motion range information corresponds to each mechanical joint motion range information in the each mechanical joint motion range information, and each joint motion range information in the each joint motion range information corresponds to each joint motion information in the each joint motion information; For each joint motion information in each joint motion information included in the human hand motion information, the following steps are performed: determining the joint motion range information corresponding to the joint motion information among the respective joint motion range information as the target joint motion range information; generating joint range of motion ratio information based on the joint motion information and the target joint motion range information; Determine the mechanical joint motion range information corresponding to the target joint motion range information among the respective mechanical joint motion range information as the target mechanical joint motion range information; Generate mechanical joint mapping motion information based on the joint activity ratio information and the target mechanical joint motion range information; The generated respective mechanical joint mapping motion information is determined as manipulator mapping motion information, wherein each current manipulator joint motion information in the respective current manipulator joint motion information corresponds to one mechanical joint mapping motion information in the respective mechanical joint mapping motion information.

4. The force tactile feedback teleoperation system according to claim 1, characterized in that: The force tactile feedback glove comprises a fingertip spatial position collector and a wrist spatial position collector, and the force tactile feedback glove is further configured to capture human hand motion information through the following steps: Collecting the spatial point position information of a human wrist through the wrist spatial position collector, wherein the spatial point position information of a human wrist includes the coordinates of the wrist spatial point; By collecting the fingertip spatial position information of the fingertip where the fingertip is located by each of the fingertip spatial position collectors, each fingertip spatial position information is obtained, wherein each fingertip spatial position information of the fingertip spatial position information includes the coordinates of the fingertip spatial point; The spatial position information of each fingertip and the spatial position information of the human wrist are determined as human hand motion information.

5. The force tactile feedback teleoperation system according to claim 4, characterized in that: The control device is further configured to generate robot arm mapping motion information based on the human hand motion information through the following steps, including: Determine the wrist space point coordinates included in the human hand motion information as reference wrist space point coordinates; Collecting the spatial coordinates of a preset point in the manipulator wrist as the spatial point coordinates of the manipulator wrist; For each piece of fingertip spatial position information included in the human hand motion information, the following steps are performed: Based on the reference wrist space point coordinates and the fingertip space position information, generating relative space coordinate information between the reference wrist space point coordinates and the fingertip space position information; Generate manipulator fingertip spatial coordinate information based on the manipulator wrist spatial point coordinates and the relative spatial coordinate information; The generated spatial coordinate information of each robot fingertip is determined as robot mapping motion information.

6. A method for remotely controlling a manipulator, applied to a control device included in a force tactile feedback teleoperation system according to any one of claims 1 to 5, the method comprising: Receiving human hand motion information transmitted by the force tactile feedback glove; Based on the human hand motion information, generating robot arm mapping motion information; The remote-controlled manipulator is controlled to make a movement corresponding to the motion information mapped by the manipulator.

Images