Electromagnetic force-based haptic interaction method, system, and electromagnetic force generation device

Abstract

The present disclosure provides a method, system and electromagnetic force generating device for haptic interaction. The method comprises: obtaining an operation posture of an operation subject for controlling a remote robot operation; determining, based on an environmental force borne by the remote robot operation and the operation posture, a desired position of a magnetic field generating element group and a set of electrical parameter values to be applied to the magnetic field generating element group, the magnetic field generating element group corresponding to the set of electrical parameter values one-to-one; driving the magnetic field generating element group to move based on the desired position, and applying electrical parameters with corresponding electrical parameter values to each magnetic field generating element, wherein the magnetic field generating elements in the magnetic field generating element group each generate a magnetic field distribution in different directions based on the applied electrical parameters, and the magnetic field distributions in different directions interact with a magnetic element arranged at the operation subject to synthesize electromagnetic forces in different directions at the operation subject.

Description

Technical Field

[0001] This disclosure relates to the field of robotics, and more specifically, to a tactile interaction method based on electromagnetic force, a tactile interaction system based on electromagnetic force, and an electromagnetic force generating device. Background Technology

[0002] With the rapid development of artificial intelligence and robotics, robots are already able to replace humans in performing some simple, repetitive tasks, leveraging their advantages in speed and precision. However, when performing complex decisions, judgments, and tasks requiring high maneuverability, most robots still require human guidance to complete the task.

[0003] In human-guided robot interaction, the application of feedback force is crucial for flexible operation within limited spaces. It allows operators to understand more operational states, improves task performance, and achieves a more realistic and natural sense of presence and immersion in human-computer interaction. For example, feedback force can be applied in underwater drone control, remote operation of space machines, remote medical surgery, and robot-assisted tumor localization. Feedback force in human-computer interaction requires the use of feedback force generation devices to reproduce it.

[0004] In recent years, research on feedback force generation devices has seen applications in various fields, including joystick-based and wearable-based devices. Wearable feedback force generation devices are mechanisms fixed to the operator's hands, limbs, or body, simulating the force exerted on the human body by providing driving forces to multiple joints of the linkage. However, these feedback force generation devices suffer from drawbacks such as frictional errors, accumulated connection errors, limited operating space, and restrictions on hand operation, failing to meet the evolving demands of natural human-computer interaction technology for robot interaction.

[0005] Therefore, a feedback force generation technology is needed to solve key problems such as slow response, limited space, large error, and poor user-friendliness in the feedback force generation process. Summary of the Invention

[0006] According to one aspect of this disclosure, a tactile interaction method based on electromagnetic force is provided, comprising: acquiring the operating posture of an operating subject for controlling the operation of a remote robot; determining, based on the environmental forces experienced by the remote robot during operation and the operating posture, a desired position of a magnetic field generating element group and a set of electrical parameter values ​​to be applied to the magnetic field generating element group, wherein the magnetic field generating element group corresponds one-to-one with the set of electrical parameter values; driving the magnetic field generating element group to move based on the desired position, and applying electrical parameters with corresponding electrical parameter values ​​to each magnetic field generating element, wherein each magnetic field generating element in the magnetic field generating element group generates a magnetic field distribution in different directions based on the applied electrical parameters, and the magnetic field distribution in different directions interacts with a magnetic element disposed at the operating subject to synthesize electromagnetic forces in different directions at the operating subject.

[0007] According to another aspect of this disclosure, an electromagnetic force-based tactile interaction system is provided, comprising: a controller configured to execute the electromagnetic force-based tactile interaction method as described above; a pose sensor for acquiring the operating posture of an operating subject; a remote robot for operating according to the operating posture of the operating subject under the control of the controller, wherein environmental forces experienced by the remote robot during operation are used to generate electromagnetic forces to be fed back to the operating subject; an operating-side end effector for tracking the operating subject under the control of the controller; and a magnetic field generating element group disposed on the operating-side end effector, wherein each magnetic field generating element in the magnetic field generating element group generates a magnetic field distribution in different directions based on applied electrical parameters, and the magnetic field distributions in different directions interact with magnetic elements disposed at the operating subject to synthesize electromagnetic forces in different directions at the operating subject.

[0008] According to another aspect of this disclosure, an electromagnetic force generating device is provided, the device comprising: an insulating housing including a central block and a protruding block connected to the central block, the protruding block forming a concave region with the central block; a magnetic field generating element group disposed in the concave region, one magnetic field generating element of the magnetic field generating element group being disposed in the central block as a central magnetic field generating element, and the remaining magnetic field generating elements of the magnetic field generating element group being disposed at equal intervals in the protruding block as peripheral magnetic field generating elements, wherein each magnetic field generating element in the magnetic field generating element group generates a magnetic field based on an applied electrical parameter, the magnetic field having a magnetic field direction associated with the arrangement direction of the magnetic field generating element and a magnetic field distribution associated with the value of the applied electrical parameter.

[0009] The electromagnetic force-based tactile interaction method disclosed herein uses electromagnetic force for feedback and only requires magnetic elements on the operating subject. This reduces friction error, reduces delay, and allows the operator's movements to be unrestricted by space or physical limitations. Furthermore, by employing a magnetic field generating element group capable of generating magnetic fields in different directions, electromagnetic feedback forces in different directions (multi-dimensional electromagnetic forces, also referred to as electromagnetic forces in this paper) can be generated, thereby further enhancing the operator's sense of immersion. Attached Figure Description

[0010] Figure 1 A schematic diagram of a tactile interaction system based on electromagnetic force according to an embodiment of the present disclosure is shown.

[0011] Figure 2 A flowchart illustrating a tactile interaction method based on electromagnetic force according to an embodiment of the present disclosure is shown.

[0012] Figure 3 A schematic diagram of the magnetic field distribution of the coil after it is energized is shown.

[0013] Figure 4 A flowchart illustrating the process of preprocessing the operating posture of the operating subject according to an embodiment of the present disclosure is shown.

[0014] Figures 5A-5B A schematic diagram illustrating the process of determining the desired location and electrical parameters of a magnetic field generating element group according to an embodiment of the present disclosure is shown.

[0015] Figure 5C This is a schematic diagram of the structure of a neural network model.

[0016] Figures 6A-6B A schematic diagram of another process for determining the desired location and electrical parameters of a magnetic field generating element group according to an embodiment of the present disclosure is shown.

[0017] Figures 7A-7B A schematic diagram is shown illustrating yet another process for determining the desired location and electrical parameters of a magnetic field generating element assembly according to an embodiment of the present disclosure.

[0018] Figure 8 It shows that according to Figures 2-7B A schematic process flow of a specific example of the electromagnetic force-based tactile interaction method.

[0019] Figure 9 An example structure of a magnetic field generating element group is shown.

[0020] Figure 10 A structural block diagram of a computing device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0021] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and is not intended to limit the scope and application of the present invention.

[0022] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure. It should be understood that this disclosure is not limited to the exemplary embodiments described herein.

[0023] In this specification and accompanying drawings, steps and elements that are substantially the same or similar are indicated by the same or similar reference numerals, and repeated descriptions of these steps and elements are omitted. The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the invention's embodiments; "multiple" means two or more unless otherwise explicitly specified.

[0024] To address the aforementioned problems, non-contact feedback force generation devices can be employed. For example, because electromagnetic force can generate strong forces over short distances, create smooth, continuous force fields in space, and effectively control and regulate the output electromagnetic force through current or voltage, electromagnetic tactile interaction devices can be designed based on electromagnetic control principles to achieve non-contact feedback force generation. This feedback force offers advantages such as low latency, small error, and unrestricted movement by space or physical limitations. However, the required feedback force may vary in both direction and magnitude, a point that current non-contact feedback force generation devices do not consider, making it difficult to generate multi-dimensional feedback forces.

[0025] Therefore, this disclosure takes into account the different directions and magnitudes of feedback forces in the interactive operation space and proposes a scheme that can generate multi-dimensional (directional) electromagnetic feedback forces.

[0026] Figure 1 A schematic diagram of a tactile interaction system based on electromagnetic force according to an embodiment of the present disclosure is shown.

[0027] like Figure 1 As shown, the electromagnetic force-based tactile interaction system 100 includes a pose sensor 1, a remote manipulated object 2, a magnetic field generating element group 3, an end effector 4, and a controller (not shown).

[0028] The pose sensor 1 is used to acquire the operating posture of the subject.

[0029] Optionally, the operating subject can be a human hand, limbs, etc. For illustrative purposes, this disclosure will use the term "human hand" extensively in the following description; however, those skilled in the art will understand that the operating subject can be any body part that can be moved to control the robot's operation and can sense or measure electromagnetic feedback forces. Even in some scenarios, the operating subject can be an object that has the ability to sense electromagnetic feedback forces and notify the operator of these forces.

[0030] The operating posture can include the position and orientation of the operating subject. The posture sensor 1 can measure the position, velocity, acceleration, orientation, angular velocity, etc. of the operating subject.

[0031] The pose sensor 1 can be a LeapMotion sensor, fixed to the end effector 4. There can be one or two pose sensors. With one sensor, it can be fixed directly above the center of the end effector 4, and this sensor has a high-precision measurement area. With two sensors, they are placed at a 45° angle towards the end effector 4, with overlapping measurement spaces. When the user is within this overlapping space, the data from the two sensors are fused to improve measurement accuracy. This overlapping space is also called the high-precision measurement area, meaning that when the user moves within this area, the position and orientation of the user can be sensed effectively. For simplicity, the following description assumes a single pose sensor, where all movements of the user can be well sensed by the pose sensor. This is predictable; as described later, when the user moves, the end effector on the platform (operation side) (on which the pose sensor is fixed) will follow the user's movement, thus ensuring that the user remains within the high-precision measurement area.

[0032] The remote manipulated object 2 is used to operate according to the posture of the operating subject under the control of the controller. The remote manipulated object 2 can be the end effector of a remote robot (for convenience, it is also referred to as a remote robot). When the remote manipulated object 2 is operating, it may be subjected to environmental forces, which can be the interaction force between the manipulated object and the environment. These environmental forces need to be fed back to the operating subject so that the operator can understand the operating status.

[0033] The magnetic field generating element group 3 can also be disposed on the aforementioned end effector 4. The magnetic field generating elements therein generate a magnetic field based on applied electrical parameters. This generated magnetic field interacts with a magnetic element disposed on the operating body, thereby providing electromagnetic feedback force to the operator's hand. The electrical parameters applied to the magnetic field generating element group are related to the environmental forces experienced by the distal operated object 2. The magnetic element can be a magnet, such as a permanent magnet.

[0034] For example, one of the magnetic field generating elements in the magnetic field generating element group serves as the central magnetic field generating element; the remaining magnetic field generating elements in the magnetic field generating element group are arranged at equal intervals around the central magnetic field generating element as peripheral magnetic field generating elements, wherein the lower surface of the peripheral magnetic field generating elements and the lower surface of the central magnetic field generating element form a concave region, and the magnetic field generating element group is located in this concave region.

[0035] For example, the magnetic field generating element can be a coil, such as a coil with an iron core (which can enhance the magnetic field strength and concentrate the magnetic field lines). When energized, the coil generates a magnetic field in a direction perpendicular to its cross-section. More details about the magnetic field generating element assembly will be described later.

[0036] The controller is used to control the operation of the remotely operated object 2 according to the operation posture and to control the electromagnetic feedback force generation process.

[0037] For example, when the remotely manipulated object 2 (such as the end effector of a remote robot) comes into contact with the surrounding environment, a force sensor, such as a three-degree-of-freedom force sensor on the remotely manipulated object 2, first measures the force exerted by the environment on the remotely manipulated object 2 and provides this information to the controller. Then, based on the measured force and the current position or position and orientation of the operator, the controller can determine the displacement and a set of electrical parameter values ​​(this process can be implemented by a neural network model). Finally, the magnetic field generating element group is moved to the corresponding position and provides the corresponding current, so that the operator can feel an electromagnetic feedback force that is essentially the same as the force exerted on the remotely manipulated object 2. This enhances the operator's immersion, not only improving the operator's accuracy in controlling the remotely manipulated object 2, but also preventing damage to the remotely manipulated object 2 or the workpiece due to collisions between the environment and the remotely manipulated object 2.

[0038] After briefly introducing the electromagnetic force-based tactile interaction system above, the following section will combine... Figure 2 An explanation of the associated electromagnetic force-based tactile interaction method.

[0039] Figure 2 A flowchart illustrating a tactile interaction method based on electromagnetic force according to an embodiment of the present disclosure is shown. As an example, the method can be executed by a controller.

[0040] like Figure 2 As shown, in step S210, the operating posture of the operating subject used to control the operation of the remote robot is obtained.

[0041] Optionally, the operating posture can be determined by, for example: Figure 1The pose sensor, as shown, detects the position and then the controller obtains the detected operation posture information (e.g., the position and orientation information of the operating subject) from the pose sensor at a preset sampling frequency.

[0042] Specifically, obtaining the operating posture may include: performing coordinate calibration on the obtained position and orientation of the operating subject; and filtering the position and orientation of the coordinate-calibrated operating subject to obtain the estimated position and estimated orientation of the operating subject.

[0043] like Figure 1 As shown, the entire electromagnetic force-based haptic interaction system includes multiple coordinate systems, such as the operator's coordinate system (H), the platform's end effector coordinate system (E), the world coordinate system (W), and the pose sensor coordinate system (L). Therefore, to obtain effective operator tracking performance, each coordinate system needs to be unified to a reference coordinate system for control purposes. Furthermore, although the pose sensor can effectively sense the operator's pose, it is still susceptible to external noise interference, which affects its sensing accuracy. Therefore, it is necessary to filter the operator's pose sensed by the pose sensor (after coordinate registration). The specific processes of coordinate registration and filtering will be discussed later. Figure 4 A detailed introduction will be provided.

[0044] In step S220, based on the environmental forces experienced by the remote robot during operation and the operating posture, the desired position of the magnetic field generating element group and a set of electrical parameter values ​​to be applied to the magnetic field generating element group are determined, and the magnetic field generating element group corresponds one-to-one with the set of electrical parameter values.

[0045] Optionally, each of the magnetic field generating elements in the magnetic field generating element group generates a magnetic field distribution in different directions based on the applied electrical parameters. The magnetic field distributions in different directions interact with the magnetic elements disposed at the operating body to synthesize electromagnetic forces in different directions at the operating body.

[0046] The resultant force of the various electromagnetic forces in different directions at the operator's location is correlated with (e.g., equal to) the environmental forces, enabling more realistic force feedback to be provided to the operator. The values ​​of the electrical parameters applied to each magnetic field generating element are determined by the controller based on the operator's posture and the environmental forces. The magnetic field distribution generated by each magnetic field generating element includes the magnetic field intensity distribution.

[0047] As the distance between the magnetic field generating element group and the operating body increases due to the movement of the operating body, the electromagnetic feedback force at the operating body will decrease while the magnetic field strength generated by each magnetic field generating element remains constant. Furthermore, the magnetic field strength generated by the magnetic field generating elements is positively correlated with the value of the applied electrical parameter (e.g., current or voltage), and the magnetic field direction is related to the arrangement of the magnetic field generating elements. For example, as... Figure 1 As shown, a coil located at the center is arranged perpendicular to the horizontal plane of the end effector. When current is applied, it generates a magnetic field in the vertical direction. The magnetic field distribution (which can also be understood as the distribution of magnetic field lines and magnetic field strength) can be as follows: Figure 3 As shown in Figure 1; and for any of the other coils shown in Figure 1, which are arranged at a certain angle to the horizontal plane of the end effector, a magnetic field is generated in the direction at the angle to the horizontal plane when a current is applied, and the magnetic field distribution is as follows: Figure 3 The results are similar, except that the magnetic field strength may vary depending on the magnitude of the current.

[0048] As can be seen from the above, the electromagnetic force synthesized at the operating body for feedback can be generated based on the distance between the magnetic field generating element group and the operating body (e.g., the vertical distance between the palm of a human hand and the geometric center of the magnetic field generating element at the center of the magnetic field generating element group) and the magnetic field strength generated by each magnetic field generating element (corresponding to the value of the applied electrical parameter). In other words, after knowing the environmental forces experienced by the remote robot, the desired position of the magnetic field generating element group should be determined to generate an electromagnetic feedback force consistent with that environmental force, and the value of the electrical parameter that should be applied to each magnetic field generating element when it is in that desired position.

[0049] Further details of step S230 will be provided later. Figures 5A-7B Describe it.

[0050] In step S230, the magnetic field generating element group is driven to move based on the desired position, and an electrical parameter with a corresponding electrical parameter value is applied to each magnetic field generating element.

[0051] As described above, to generate the desired feedback force, the magnetic field generating element assembly should be positioned at the desired location with the required relative distance to the human hand, and a corresponding electrical parameter value should be applied to each magnetic field generating element, thereby synthesizing electromagnetic forces in different directions (multidimensional electromagnetic forces) at the human hand. For example, the controller can issue commands to drive the three-degree-of-freedom robotic arm of the end effector of the control platform to move, thereby moving the magnetic field generating element assembly.

[0052] Through reference Figure 2The electromagnetic force-based tactile interaction method uses electromagnetic force for feedback and only requires magnetic elements on the operating subject. Therefore, it can reduce friction error, reduce delay, and allow the operator's movements to be unrestricted by space or physical limitations. Furthermore, because it uses a magnetic field generating element group that can generate magnetic fields in different directions, it can generate electromagnetic feedback forces in different directions (multi-dimensional electromagnetic forces), thereby further increasing the operator's sense of immersion.

[0053] Figure 4 Further details regarding the acquisition of the operational posture involved in step S210 are shown.

[0054] like Figure 4 As shown, the coordinate registration steps may include the following sub-steps S210-1 to S210-3. The following example uses a human hand as the operator.

[0055] In sub-step S210-1, the human hand coordinate system (H), the platform end effector coordinate system (E), the world coordinate system (W), and the pose sensor coordinate system (L) are defined as a set of coordinate systems.

[0056] by Figure 1 Taking an electromagnetic force-based tactile interaction system as an example, the magnetic field generating element group and the pose sensor are mounted on the end effector of the platform. For instance, the magnetic field generating element group is placed on the end effector of the platform, while the two pose sensors are placed at an angle of 45° towards the end effector. When there is only one pose sensor, it can be placed directly above the center of the end effector. The end effector on the platform can be fixed to the platform via a robotic arm with multiple degrees of freedom (e.g., three degrees of freedom), so that the end effector can translate, rotate, etc. in space.

[0057] Optionally, the hand coordinate system is defined as X H Y H Z H The coordinate system of the platform's end effector is defined as X. E Y E Z E The world coordinate system is defined as X W Y W Z W The pose sensor coordinate system is defined as X L Y L Z L .

[0058] The specific meanings of each coordinate system are as follows:

[0059] The human hand coordinate system has its origin at the center of the palm. The three axes of the human hand coordinate system are defined as follows: Y... H With the positive direction perpendicular to the back of the hand and upward, the direction of the middle finger is X.H The positive direction, Z H Perpendicular to X H -Y H Plane. When a person moves their hand, the coordinates relative to the three axes of the hand's coordinate system will change.

[0060] The coordinate system of the platform's end effector has its origin at the center of the portion where the lower surface of the magnetic field generating element assembly coincides with the horizontal plane of the platform; the three coordinate axes of the platform's end effector are defined as follows: X E The positive direction is along the longer side of the end effector and opposite to the direction of the unengaged boundary of the end effector, Y E The positive direction is perpendicular to the upper surface of the end effector and outwards, Z E Perpendicular to X E -Y E flat.

[0061] The origin of the pose sensor's coordinate system is its geometric center; the three coordinate axes of the pose sensor are defined as follows: X L The positive direction is along the longer side of the pose sensor, Y L The positive direction is perpendicular to the front of the pose sensor and faces outwards, Z L Perpendicular to X L -Y L Plane. The posture information of a human hand measured by a pose sensor is based on its own coordinate system, but it can also be transformed to the world coordinate system as needed.

[0062] The world coordinate system uses the upper right corner of the platform as its origin, Z... W The positive direction is vertically upward, opposite to the direction of gravity; X W The positive direction is from the upper right corner of the platform to the lower right corner, Y W Perpendicular to Z W -X W flat.

[0063] In sub-step S210-2, one of the coordinate systems in the set of coordinate systems is determined as the reference coordinate system for the set of coordinate systems.

[0064] Optionally, the reference coordinate system is the world coordinate system.

[0065] Of course, any coordinate system can be used as the reference coordinate system. You just need to transform the coordinates of different coordinate systems in the set of coordinate systems to the reference coordinate system before performing the calculation.

[0066] In sub-step S210-3, the position and orientation of the hand are registered to the reference coordinate system based on the transformation matrix between the hand coordinate system and the reference coordinate system.

[0067] Because the pose sensor is fixed to the end effector of the platform, X L Y L Z L and X E Y E Z E Transformation matrix between Pre-registration is possible. X E Y E Z E and X W Y W Z W Transformation matrix between The system updates in real time based on the actual displacement of the platform's end effector. H Y H Z H and X L Y L Z L Transformation matrix between It can be measured by a pose sensor. Therefore, X H Y H Z H and X W Y W Z W Transformation matrix between It can be by and The transformation yields, i.e. Similarly, the transformation matrix between any two coordinate systems can be obtained in a similar way. With the world coordinate system as the reference coordinate system, the position and orientation of the human hand sensed by the pose sensor can be transformed into the world coordinate system, and the position and orientation of the human hand in the X coordinate system can be determined. W Y W Z W The movement of the hand controls the movement of the platform's end effector, allowing the end effector to follow the movement of the hand.

[0068] On the other hand, regarding filtering the data sensed by the pose sensor, such as... Figure 4 As shown, it may include sub-steps S210-4 and S210-5.

[0069] In sub-step S210-4, the position of the hand after coordinate registration is filtered based on the interval Kalman filter (IKF) technique to obtain the estimated position.

[0070] In other words, the position of the human hand sensed by the pose sensor and registered with coordinates is an observed value, which may have a certain error with the actual value. Therefore, the IKF algorithm can be used to estimate the actual value to improve the accuracy of the calculation of the human hand position.

[0071] The following description uses the IKF algorithm for filtering as an example. However, it should be understood that other methods can also be used for filtering, such as conventional Kalman filters, particle filters, etc.

[0072] In this disclosure, when using the IKF algorithm, the following procedures may be included.

[0073] Let s k Let be the state of manpower at time k, and s k It can be represented as:

[0074] s k =[p x,k ,v x,k ,a x,k ,p y,k ,v y,k ,a y,k ,p z,k ,v z,k ,z z,k (1)

[0075] Where p x,k ,v x,k and a x,k Let p represent the position, velocity, and acceleration of the hand on the x-axis at time k (world coordinate system). Similarly, p y,k ,v y,k ,a y,k and p z,k ,v z,k ,z z,k These represent the position, velocity, and acceleration of the hand on the y and z axes at time k, respectively.

[0076] Therefore, during the movement of the hand, the velocity components of the hand on each axis can be expressed as:

[0077]

[0078] The relationship between acceleration and velocity on each axis can be expressed as:

[0079]

[0080] According to equations (2) and (3), the state transition matrix Φ of the hand from the state at time k-1 to the state at time k is known. k for:

[0081]

[0082] Where t is the sampling time interval, and since there is no system input during the hand movement, the input matrix of IKF can be written as:

[0083] Γk ·u k-1 =[0,0,0,0,0,0,0,0,0] T (5)

[0084] Based on the input matrix and the state transition matrix between two adjacent states, the state s at time k can be obtained. k s k =Φ k ·s k-1 +Γ k ·u k-1 +b k-1 b k-1 It is random noise.

[0085] Using a registered and initialized pose sensor, acceleration and position components can be calculated. Therefore, the observation matrix for position estimation, representing the mapping of the true state space to the observation space at time k, can be expressed as:

[0086]

[0087] via z k =H k ·s k +m k m k For random noise, z k For state s k The observed values ​​are used to obtain the best estimate of the position of the tracked hand at time k.

[0088] P k ′=(p x,k ′,p y,k ′,p z,k ′) (7)

[0089] In sub-step S210-5, the direction of the hand after coordinate registration is filtered based on the improved particle filter algorithm (IPF) to obtain the estimated direction.

[0090] Because the posture of a human hand is dynamically variable and its trajectory is non-linear, this method uses an Inverter-Focused Field (IPF) with angular velocity information to accurately estimate the hand's orientation (angular velocity information can be obtained from a pose sensor). By using Euler's theorem for finite rotations, Euler angles can be converted into quaternions, as follows:

[0091]

[0092] The relationship between the quaternions Q = [q0, q1, q2, q3] is as follows:

[0093]

[0094] in, θ and This indicates the roll angle, yaw angle, and pitch angle of the human hand.

[0095] To improve the accuracy of orientation estimation, in the step of acquiring the operating posture from the pose sensor, a finite number of samples (i.e., multiple particles) and normalized weights can be used to approximate the posterior probability using the IPF method. The input parameters of the IPF method include quaternions and angular velocity. Therefore, the quaternion Q representing the orientation of the human hand by the i-th particle at time k can be obtained.

[0096]

[0097] Since the quaternion Q is used to represent the direction of the human hand, the quaternion Q must satisfy:

[0098]

[0099] Therefore, the quaternion of the i-th particle at time k+1 can be calculated using the following formula:

[0100]

[0101] Where t represents the sampling interval, ω x,k ,ω y,k ,ω z,k These represent the angular velocity components along the x-axis, y-axis, and z-axis, respectively. Therefore, the quaternion calculated for the i-th particle can be obtained through formula (12), and the corresponding estimated directions (roll angle, yaw angle, and pitch angle) can be obtained.

[0102] We need to determine the weight of each particle in each batch, and then obtain the weight and estimated direction for each particle in that batch to arrive at the final estimated direction. The specific process is as follows.

[0103] First, the acceleration of the target (which can be for each particle) is obtained using formula (13):

[0104]

[0105] Among them, f V Represents velocity within the global coordinate system (e.g., the world coordinate system). Represents acceleration within the global coordinate system. This is the transformation matrix between the current coordinate system and the global coordinate system. b A represents the acceleration measurement value in the current coordinate system (e.g., obtained through a pose sensor), f g It is a global gravity variable.

[0106] The acceleration calculation error increases with the error of the transformation matrix (used for coordinate system transformation). The cumulative position difference between the calculated position of the i-th particle and the estimated position of the IKF can be used to assign weights to each particle.

[0107]

[0108] in M is the cumulative position error of the i-th particle during the s-th iteration of the direction of the human hand. s =ΔT s / t, ΔT s The preset time period is defined as t, where t is the sampling time interval, and the preset window time is defined as ΔT. s The cumulative position error of the i-th particle is calculated at multiple time points within a given time period. This is the estimated position of the i-th particle on the x-axis at time k, estimated using IKF. Similarly... y and z are the estimated positions of the i-th particle estimated using IKF, respectively. The position of the i-th particle at time k can be obtained from the acceleration obtained in formula (13) and time k, and is used as the calculated position of the i-th particle. Similarly, y and z are the calculated positions of the i-th particle estimated using IKF, respectively.

[0109] Thus, the smaller the accumulated position error, the higher the weight is assigned to the particle. The weight can be assigned using the following posterior probability formula:

[0110]

[0111] Therefore, by setting the window time (ΔT) s The estimated directions of each particle within the range are weighted to obtain the estimated direction at time k.

[0112] By reference Figure 4 The described method involves coordinate registration and filtering of the hand posture information obtained from a pose sensor to obtain hand posture information that is easy to calculate and has high accuracy.

[0113] The following provides a detailed description of the process in step S230 where the desired position of the magnetic field generating element group and a set of related electrical parameter values ​​are obtained based on the position and orientation of the operating subject (hereinafter also using a human hand as an example).

[0114] First, as mentioned earlier, since the human hand is equipped with magnetic components that interact with the magnetic field generating components, and these magnetic components are subject to gravity, it is necessary to compensate for this gravity in order to more accurately reflect the expected feedback force and avoid errors caused by gravity.

[0115] Therefore, one approach is to obtain the total electromagnetic feedback force based on the expected feedback force and the gravity of the magnetic element, and then use the total electromagnetic feedback force to determine the above-mentioned desired position and electrical parameter values. That is, step S230 can be further defined as determining the desired position of the magnetic field generating element group and a set of electrical parameter values ​​to be applied to the magnetic field generating element group based on the total electromagnetic feedback force and the operating posture.

[0116] For example, with F E F represents the expected feedback force (e.g., the environmental force experienced by the remote manipulated object). T For the total electromagnetic feedback force, F G If the magnitude of gravity G is the same but the direction is opposite (these forces can all be converted into components on the coordinate axes of the reference coordinate system), then the following relationship exists:

[0117] F T =F E +F G And F G =-G.

[0118] It is evident that even when the expected feedback force is absent, electrical parameters can still be applied to the magnetic field generating element group to generate a force that cancels out the gravity of the magnetic element, so that the human hand will not feel the gravity of the magnetic element.

[0119] Another approach is to fix a magnet satisfying a preset magnetic condition on the upper surface of the central magnetic field generating element in the magnetic field generating element group. This magnet interacts with a magnetic element on the user's hand to counteract the gravity of that magnetic element. In this case, the total electromagnetic feedback force is the same as the expected feedback force.

[0120] Optionally, the magnet that meets the preset magnetic conditions is a magnet with sufficiently strong magnetism and a relatively uniform magnetic field strength, such as a strong rubidium magnet.

[0121] When using this method, if the line connecting the center of gravity of the magnetic element in the hand and the center of the central magnetic field generating element is not perpendicular to the world coordinate system, while offsetting or compensating for the gravity of the magnetic element in the hand, there will still be a force component in the horizontal direction of the world coordinate system, even without the expected feedback force. This will lead to an error in the electromagnetic feedback force felt by the hand. Therefore, this method is generally only applied when the hand is moving and the magnetic field generating element does not rotate (i.e., the end effector of the platform does not rotate), as will be discussed below. Figures 5A-5B And as described in 6A-6B.

[0122] The following is in conjunction with the appendix Figures 5A-5B The determination process of the desired location and electrical parameter values ​​of the magnetic field generating element group is described in 6A-6B and 7A-7B.

[0123] As mentioned earlier, during the movement of the human hand, not only should the remote object being manipulated follow the movement of the human hand, but the end effector on the platform should also follow the movement of the human hand. This is partly to ensure that the human hand is always within the high-precision measurement area of ​​the posture sensor, and partly to enable the magnetic field generating element group fixed on the end effector on the platform to provide electromagnetic feedback force to the human hand.

[0124] In addition, in some implementation scenarios, the movement of a human hand may only be a translation of the human hand on the x, y, and z axes. In other implementation scenarios, the human hand can rotate, but the platform's end effector can move with the human hand but will not rotate with the human hand's rotation. In yet another implementation scenario, the platform's end effector can move with the human hand and rotate with the human hand.

[0125] The following will describe these three scenarios separately.

[0126] First, in the scenario where the palm of the hand does not rotate and is parallel to the end effector of the platform during the movement of the hand, step S230 may include the following sub-steps.

[0127] like Figure 5A As shown, in sub-step S230-1, based on the estimated position and the components of the total electromagnetic feedback force on the x, y, and z axes, the desired vertical distance between the human hand and the magnetic field generating element group, as well as a set of electrical parameter values ​​to be applied to the magnetic field generating element group, are obtained. Here, the x, y, and z axes are coordinate axes in the reference coordinate system (e.g., the world coordinate system) of the entire electromagnetic force feedback system.

[0128] In other words, for example, if a hand moves from position (px, py, pz) to an estimated position (px', py', pz'), then it has moved Δx on the x-axis, Δy on the y-axis, and Δz on the z-axis. Since the magnetic field generating element group follows the movement of the hand, it can be determined that the magnetic field generating element group also needs to move Δx on the x-axis and Δy on the y-axis. That is, the x- and y-axis components of the desired position of the magnetic field generating element group are also px' and py'. The vertical distance movement of the magnetic field generating element on the z-axis needs to be considered in conjunction with the electromagnetic feedback force that needs to be fed back.

[0129] Optionally, this step, as will be referenced later, is also included. Figures 6A-6B The steps S230-1' described and S230-1” described in references 7A-7B can be implemented using a trained neural network model.

[0130] For example, such as Figure 5CAs shown, the trained neural network model can sequentially include an input layer, a hidden layer, and an output layer, with fully connected connections between the hidden layer and the input layer, as well as between the hidden layer and the output layer. The input layer can include a number of nodes corresponding to the number of components of the total feedback force and the operating posture on the x, y, and z axes. The output of the input layer is provided to the hidden layer after passing through an activation function. The hidden layer contains at least two linear layers, and the output of each linear layer is provided to the next layer after passing through an activation function. The output layer includes N+1 neuron nodes and is fully connected to the hidden layer. It receives the output from the hidden layer and outputs the desired vertical distance and a set of electrical parameter values, where N is the number of magnetic field generating elements included in the set of magnetic field generating elements to which the set of electrical parameter values ​​are applied.

[0131] Specifically, in this scenario, since the end effector of the magnetic field generating element group moves with the hand without rotating the hand, the estimated position (px', py', pz') and the total electromagnetic feedback force (Fx, Fy, Fz) can be used as inputs to the neural network model. Therefore, the number of nodes in the input layer is six. The neural network model outputs the expected vertical distance and the values ​​of the electrical parameters (e.g., voltage and current) applied to each magnetic field generating element in the magnetic field generating element group. Of course, to ensure the uniformity of the training sample parameters for model training, the angular component (φ) can also be used. x , φ y , φ z ) is also used as input, but in this case it should be 0.

[0132] For example, by way of example and not limitation, the neural network model can be a backpropagation neural network, which can include an input layer, hidden layers, and an output layer. The input layer is a linear layer with six nodes, used to process input data (px', py', pz', Fx, Fy, Fz). The activation function between the input layer and the hidden layer is the ReLU function, and the connection between the input layer and the hidden layer is a fully connected structure. The hidden layer contains at least two linear processing layers to improve performance, each fully connected to the previous layer, with the activation functions of each linear processing layer set to Sigmoid, Tanh, and Tanh, respectively. The output layer is a Softmax layer containing N+1 neurons, used to output the desired vertical distance D between the hand and the magnetic field generating element group, and the power supply strategy for the N magnetic field generating elements included in the magnetic field generating element group. An output of 1 indicates that the magnetic field generating element is provided with the maximum electrical parameter, an output of 0 indicates that the magnetic field generating element is not provided with the electrical parameter, and an intermediate value between 0 and 1 indicates that the magnetic field generating element is provided with an electrical parameter having a value between 0 and the maximum value. The connection between the output layer and the hidden layer is fully connected.

[0133] In sub-step S230-2, the desired position is determined based on the components of the estimated position on the x-axis, y-axis, and z-axis, as well as the desired vertical distance.

[0134] For example, the x-axis and y-axis components of the desired position to which the magnetic field generating element group needs to be moved are px' and py', and the z-axis distance to which the magnetic field generating element group needs to be moved can be determined based on the desired vertical distance and the vertical distance of the magnetic field generating element group when it is not moved.

[0135] Specifically, the magnetic field generating element group needs to move Δx on the x-axis and Δy on the y-axis. The magnetic field generating element group is generally located directly below the human hand, and at a position D away from the human hand in the vertical direction.

[0136] In other words, the coordinates (px', py', pz') of the estimated position of the hand can be used to determine the coordinates (px', py', pz'-D) of the desired position of the magnetic field generating element group. Therefore, the controller can drive the magnetic field generating element group to move to this desired position.

[0137] Figure 5B It shows Figure 5A A schematic diagram illustrating the situation for which the described method is applied.

[0138] like Figure 5B As shown, the hand moves from position (px, py, pz) to the estimated position (px', py', pz'), which translates to a movement of Δx on the x-axis, Δy on the y-axis, and Δz on the z-axis. The hand is parallel to the end effector of the platform, and the movement is purely translational. The magnetic field generating element group (only three elements are shown for simplicity, but more can be included to achieve multidimensional electromagnetic force) follows the hand's movement, also moving Δx on the x-axis and Δy on the y-axis. The required vertical distance D between the magnetic field generating element group and the hand on the z-axis is determined based on the electromagnetic feedback force. The oblique forces shown in the figure represent the electromagnetic forces of the coils other than the central coil. Furthermore, electrical parameters are applied to each magnetic field generating element group based on a calculated set of electrical parameter values ​​(the dashed lines in the figure represent the electromagnetic forces provided by each magnetic field generating element, illustrating the synthesis of a multidimensional electromagnetic feedback force).

[0139] It should be noted that the vertical distance between the human hand and the magnetic field generating element group mentioned in the context can refer to the vertical distance between the palm of the human hand and the center of the lower surface of the magnetic field generating element group, or the vertical distance between the palm of the human hand and the geometric center or the center of the upper surface of the central magnetic field generating element group, etc. This disclosure does not limit this, as long as it can be used to define the relative position of the human hand and the magnetic field generating element group.

[0140] In addition, in the scenario where the palm of the hand rotates during the movement of the hand, but the end effector of the platform moves along with it without rotating, step S230 may include the following sub-steps.

[0141] like Figure 6A As shown, in sub-step S230-1', based on the estimated position, the estimated direction, and the components of the total electromagnetic feedback force on the x, y, and z axes, the desired vertical distance between the human hand and the magnetic field generating element group on the z-axis, and a set of electrical parameter values ​​to be applied to the magnetic field generating element group are obtained. Here, the x, y, and z axes are coordinate axes in the reference coordinate system (e.g., the world coordinate system) of the entire electromagnetic force feedback system.

[0142] Compared to sub-step S230-1, sub-step S230-1' adds the consideration of the estimated direction of the hand when determining the desired vertical distance. Based on the estimated direction of the hand, the changes in the hand's roll angle, yaw angle, and pitch angle can be determined (which can also be converted into angular components about the x, y, z axes of the world coordinate system).

[0143] Alternatively, this step can also be implemented using a trained neural network model. Compared to the previous case, the estimated direction needs to be included as input to the neural network model, i.e., the estimated position (px', py', pz') and the estimated direction (φ). x , φ y , φ z The total electromagnetic feedback force (Fx, Fy, Fz) is used as input to the neural network model. Similarly, the neural network model outputs the desired vertical distance and the values ​​of the electrical parameters (e.g., voltage and current) applied to each magnetic field generating element in the magnetic field generating element group. Here, the desired vertical distance is the distance along the z-axis in the world coordinate system.

[0144] For example, the neural network model can also be a backpropagation neural network, and it can be compared with the previous reference. Figure 5A The backpropagation neural network described is basically the same in structure, the difference being that the input layer is a linear layer with nine nodes, used to process the input data (px', py', pz', φ). x , φ y , φ z ,Fx,Fy,Fz).

[0145] In sub-step S230-2', the desired position is determined based on the components of the estimated position on the x-axis, y-axis, and z-axis and the desired vertical distance.

[0146] For example, the x-axis and y-axis components of the desired position to which the magnetic field generating element group needs to be moved are px' and py', and the z-axis distance to which the magnetic field generating element group needs to be moved can be determined based on the desired vertical distance.

[0147] Specifically, the magnetic field generating element assembly also needs to be moved by Δx on the x-axis and Δy on the y-axis, and also needs to be rotated (φ). x , φ y , φ z Furthermore, the magnetic field generating element group is located directly below the human hand. Therefore, it can be determined that the desired position of the magnetic field generating element group is directly below the palm of the human hand, and at a position D away from the palm of the human hand in the vertical direction.

[0148] In other words, the coordinates (px', py', pz') of the estimated position of the human hand can be used to determine the coordinates (px', py', pz'-D) of the desired position of the magnetic field generating element group. Therefore, the controller can drive the magnetic field generating element group to move to this desired position and, by adjusting the values ​​of the electrical parameters of each magnetic field generating element, generate an electromagnetic feedback force at the palm of the human hand that is consistent with the environmental force.

[0149] Figure 6B It shows Figure 6A A diagram illustrating the situation for which the described method is applied.

[0150] like Figure 6B As shown, the hand moves from position (px,py,pz) to the estimated position (px',py',pz') and rotates (30°,0,0). This results in a movement of Δx on the x-axis, Δy on the y-axis, and Δz on the z-axis. The magnetic field generating element group (only three elements are shown for simplicity, but more could be included to achieve multidimensional electromagnetic force) follows the hand's movement, also moving Δx on the x-axis and Δy on the y-axis, but does not rotate with the hand. The required electromagnetic feedback force determines the vertical distance D between the magnetic field generating element group and the palm of the hand on the z-axis. At this point, the electrical parameter values ​​applied to each magnetic field generating element differ from those applied before the hand's movement, thus generating different electromagnetic feedback forces (the dashed lines in the figure represent the electromagnetic forces provided by each magnetic field generating element, illustrating the synthesis of a multidimensional electromagnetic feedback force).

[0151] In addition, for scenarios where the platform rotates while the user's hand is moving, the following combination can be used. Figures 7A-7B The described method determines the desired location and electrical parameter values ​​of the magnetic field generating element assembly.

[0152] like Figure 7A The step S230 may include the following sub-steps.

[0153] In sub-step S230-1", based on the estimated position, the estimated direction, and the components of the total electromagnetic feedback force on the x, y, and z axes, the desired vertical distance between the magnetic field generating element group and the hand (e.g., the desired vertical distance between the center of the lower surface, the center of the upper surface of the central magnetic field generating element, or the geometric center of the palm, etc., which is not limited in this disclosure) and a set of electrical parameter values ​​to be applied to the magnetic field generating element group are obtained. Here, the x, y, and z axes are coordinate axes in the reference coordinate system (e.g., the world coordinate system) of the entire electromagnetic force feedback system.

[0154] Similarly, this process can also be achieved using a neural network model.

[0155] In sub-step S230-2", the desired position is determined based on the components of the estimated position and the estimated direction on the x-axis, y-axis, and z-axis, and the desired vertical distance.

[0156] In this case, the positional components of the desired location to which the magnetic field generating element group is to be moved, along the x and y axes, may no longer be the same as the positional components of the human hand along the x and y axes after the movement, because the desired vertical distance must be satisfied as in sub-step S230-2. Once the desired vertical distance, the estimated position of the human hand, and the components of the estimated direction along the x and y axes are known, the positional components of the desired location to which the magnetic field generating element group is to be moved, along the x, y, and z axes, can be calculated.

[0157] Figure 7B It shows Figure 7A A schematic diagram illustrating the situation for which the described method is applied.

[0158] like Figure 7B As shown, the hand moves from position (px, py, pz) to the estimated position (px', py', pz') and rotates (30°, 0, 0). The magnetic field generating element group (only one element is shown for clarity, but multiple elements should be included to achieve multidimensional electromagnetic force) follows the hand's movement and also rotates (30°, 0, 0). Assuming the total electromagnetic feedback force required for feedback determines the direction perpendicular to the plane of the hand, and the center of the lower surface of the magnetic field generating element group needs to be a desired vertical distance D from the palm of the hand, for the x-axis as shown, the desired coordinate of the center of the lower surface of the magnetic field generating element group on the x-axis is px' - D / 2. The y-axis remains unchanged, therefore it is the same as the coordinate of the estimated position of the hand, i.e., py', and its desired coordinate on the x-axis is... That is, the coordinates of the desired position of the magnetic field generating element group are (px'-D / 2, py', ).

[0159] Furthermore, training a neural network model requires a large number of samples. Therefore, the samples needed for training can be generated in the following ways.

[0160] Samples can be generated through closed-loop control, utilizing a traditional PID controller integrated into an electromagnet on the operating unit. The input to the PID controller is the total electromagnetic feedback force F. T The hand position P and direction O (the angle value of the direction is 0 when the hand movement does not involve rotation) are determined. The current I (multiple) and distance D are dynamically adjusted to generate an electromagnetic force from the coil, which the operator wearing a magnet (magnetic element) can feel. The electromagnet is fixed to the operating body (palm) and does not obstruct operation. The adjustment of the current allows for the measurement of the electromagnetic feedback force F. h Approximately equal to the total electromagnetic feedback force F T The deviation between two forces can be expressed as |F|. T -F h |≤e. In summary, given random sampled data (P, O, and F) T ), D and I are regulated by a PID controller. When F h and F T When the deviation between them is less than the threshold e, the adjustment is appropriate, and the sample (P, O, F) is collected. T (D, I). Optionally, e is set to 0.01N.

[0161] Alternatively, the neural network model can be derived from a reference. Figure 1 The controller described herein can be trained and used locally, or the controller can retrieve a pre-trained neural network model from external storage for use, or the controller can retrieve the computation results of the trained neural network model from other computing devices; this disclosure does not limit this.

[0162] The above combination Figure 5A-7B An exemplary method for obtaining the desired location and corresponding electrical parameters of a magnetic field generating element group is presented. In this method, a trained neural network model is used to obtain the required output, which can be conveniently predicted. For the desired location of the magnetic field generating element group, only one distance (such as the vertical distance described in 5A-7B or the distance of the vertical line connecting the center point) is determined by the neural network model. The remaining required distances are obtained through simple mathematical derivation, which can reduce the computational load of the neural network model and improve the prediction speed.

[0163] Figure 8 It shows according to Figures 2-7B A schematic process flow of a specific example of the electromagnetic force-based tactile interaction method.

[0164] like Figure 8As shown, in process S810, the pose sensor (LeapMotion) captures the hand pose; in process S820, the controller acquires the hand pose (e.g., the position and orientation of the hand at this time based on a preset sampling frequency); in process S830, the controller performs coordinate registration on the acquired hand pose; in processes S840-1 and S840-2, the position and orientation of the coordinate-registered hand are estimated using IKF and IPF respectively, yielding estimated position and estimated orientation; in process S850, the controller acquires the environmental force from the force sensor of the remotely operated object as the expected feedback force; in process S860, the controller determines the desired vertical distance between the hand and the magnetic field generating element group (coil group) based on the estimated position, estimated orientation, and expected feedback force (including the gravity of the magnetic element on the hand). Figures 5A-6B ) or the expected vertical distance of the center perpendicular line ( Figures 7A-7B The controller determines the desired position of the coil group by moving the coil group to the desired position (or additionally, in the same direction as the palm) to track the hand. In process 880, the controller drives the power supply unit to supply power to each coil according to the current value to be applied to each coil in the coil group, which was also determined in process 860. This causes each coil to generate a magnetic field in a different direction (the strength of which is determined by the magnitude of the current value). Ultimately, these magnetic fields in different directions interact with the magnetic elements on the hand as a composite magnetic field, generating a multi-dimensional electromagnetic feedback force at the hand.

[0165] In other respects, as described above, the magnetic field generating element group is a crucial component in generating electromagnetic feedback force and can be included in an electromagnetic force generating device. Each magnetic field generating element in the magnetic field generating element group generates a magnetic field based on applied electrical parameters. This magnetic field has a magnetic field direction associated with the arrangement orientation of the magnetic field generating elements and a magnetic field distribution associated with the value of the applied electrical parameters. Therefore, the following... Figure 9 Describe an example structure of a magnetic field generating element group.

[0166] like Figure 9 As shown, the electromagnetic force generating device 900 includes an insulating housing 90 and a magnetic field generating element group.

[0167] The insulating housing 90 includes a central block and a protruding block connected to the central block, the protruding block forming a concave region with the central block. The protruding block can be a single unit or can be divided into multiple parts.

[0168] like Figure 9As shown, the protruding block is divided into four parts for placing four magnetic field generating elements. When the protruding block is a single unit, multiple magnetic field generating elements can be arranged at equal intervals on the protruding block. All groups of magnetic field generating elements are located in concave regions.

[0169] like Figure 9 As shown, the magnetic field generating element group includes five magnetic field generating elements. One magnetic field generating element 901 is disposed in the central block of the insulating shell 90, serving as the central magnetic field generating element. The remaining magnetic field generating elements (902-905) are spaced apart and disposed in the protruding blocks surrounding the central magnetic field generating element, serving as the peripheral magnetic field generating elements. That is, the peripheral magnetic field generating elements (902-905) surround the central magnetic field generating element (901) and are disposed inward.

[0170] With such settings, such as Figure 9 As shown by the middle arrow, when a current is applied to each magnetic field generating element, a magnetic field distribution in different directions will be formed. When there is a magnetic element above, electromagnetic forces in multiple directions will be generated, that is, multidimensional electromagnetic forces.

[0171] Optionally, since the magnetic field generating element group needs to interact with the magnetic element on the operating body, and the magnetic element is subject to gravity, it is necessary to compensate for the gravity. In this case, the upper surface of the central magnetic field generating element in the magnetic field generating element group is fixed with a magnet that meets the preset magnetic conditions. The magnet that meets the preset magnetic conditions interacts with the magnetic element on the operating body to counteract the gravity of the magnetic element.

[0172] For example, a magnet that meets the preset magnetic conditions is a magnet with sufficiently strong magnetism and a relatively uniform magnetic field strength, such as a strong rubidium magnet.

[0173] When using a central magnetic field generating element with this structure, if the line connecting the magnetic element on the operating body and the center of the central magnetic field generating element is not perpendicular to the world coordinate system, while offsetting or compensating for the weight of the magnetic element on the user's hand, there will still be a force component in the horizontal direction of the world coordinate system, even without the expected feedback force. This will lead to an error in the electromagnetic feedback force felt by the operating body. Therefore, this method is generally only applicable when the operating body is moving and the magnetic field generating element group does not rotate, as will be discussed earlier. Figures 5A-5B As described in 6A-6B.

[0174] Furthermore, this disclosure primarily describes the process of generating an electromagnetic feedback force that is fed back to the operator's hand. Visual feedback can also be added to allow the operator to observe the details of the operation on the remotely operated object.

[0175] According to another aspect of this disclosure, a computing device is also provided. This computing device may include functions for implementing or performing various operations of the controller as described above.

[0176] Figure 10 A structural block diagram of a computing device 1000 according to an embodiment of the present disclosure is shown.

[0177] The computing device includes a processor and a memory storing instructions that, when executed by the processor, cause the processor to perform various operations involved in the process of the controller generating electromagnetic feedback force as described above.

[0178] The computing device can be a computer terminal, a mobile terminal, or other devices.

[0179] The processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the steps and logic block diagrams of the operations performed by the identification unit and optionally the signal processing unit in the embodiments of this disclosure. The general-purpose processor can be a microprocessor or any conventional processor, and can be based on an x84 architecture or an ARM architecture.

[0180] The memory can be non-volatile memory, such as read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. It should be noted that the memory used in the methods described in this disclosure is intended to include, but is not limited to, these and any other suitable types of memory.

[0181] The display screen of a computing device can be an LCD screen or an e-ink screen. For example, the screen can display operation details of a remotely operated object to provide visual feedback to the operator. The input device of the computing device can be a touch layer covering the display screen, buttons, trackballs, or touchpads on the terminal casing, or external keyboards, touchpads, or mice.

[0182] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains at least one executable instruction for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0183] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0184] The exemplary embodiments of this disclosure described in detail above are merely illustrative and not restrictive. Those skilled in the art will understand that various modifications and combinations can be made to these embodiments or their features without departing from the principles and spirit of this disclosure, and such modifications should fall within the scope of this disclosure.

Claims

1. A tactile interaction method based on electromagnetic force, comprising: The operating posture of the operating subject used to control the operation of a remote robot is obtained, wherein the operating posture includes the position and orientation of the operating subject; Based on the environmental forces experienced by the remote robot during operation and the operating posture, the desired position of the magnetic field generating element group should be determined when an electromagnetic feedback force consistent with the environmental forces is to be generated at the operating body. A set of electrical parameter values ​​should also be applied to the magnetic field generating element group when it is in the desired position. The magnetic field generating element group includes multiple magnetic field generating elements with different magnetic field directions, and each of the multiple magnetic field generating elements corresponds one-to-one with the set of electrical parameter values. Determining the desired position of the magnetic field generating element group includes: determining the desired vertical distance between the magnetic field generating element group and the operating body in a direction perpendicular to the plane where the operating body is located, based on the position, direction, and environmental forces of the operating body; and determining the desired position based on the position, direction, and desired vertical distance of the operating body. Based on the desired position, the magnetic field generating element group is moved, and an electrical parameter with a corresponding electrical parameter value is applied to each magnetic field generating element. In this configuration, each magnetic field generating element in the magnetic field generating element group generates a magnetic field distribution in different directions based on the applied electrical parameters. The magnetic field distributions in different directions interact with the magnetic elements disposed at the operating body to generate electromagnetic forces in different directions at the operating body. The electromagnetic forces in different directions are combined to realize the electromagnetic feedback force.

2. The tactile interaction method based on electromagnetic force according to claim 1, wherein, The magnetic field generating element group is mounted on the end effector on the operating side. One of the magnetic field generating elements in the magnetic field generating element group is disposed on the end effector as the central magnetic field generating element; The remaining magnetic field generating elements in the magnetic field generating element group are disposed on the end effector and are equally spaced around the central magnetic field generating element, serving as surrounding magnetic field generating elements. The lower surface of the surrounding magnetic field generating element and the lower surface of the central magnetic field generating element form a concave region, and the magnetic field generating element group is located in the concave region.

3. The tactile interaction method based on electromagnetic force according to claim 2, wherein, A magnet satisfying a preset magnetic condition is fixed to the upper surface of the central magnetic field generating element. The magnet satisfying the preset magnetic condition interacts with the magnetic element on the operating body to counteract the gravity of the magnetic element. Specifically, based on the environmental forces experienced by the remote robot during operation and the operating posture, the desired position of the magnetic field generating element group is determined when an electromagnetic feedback force consistent with the environmental forces is to be generated at the operating body, and a set of electrical parameter values ​​to be applied to the magnetic field generating element group when the magnetic field generating element group is in the desired position, including: The environmental force is taken as the total feedback force; and Based on the total feedback force and the operating posture, the desired position of the magnetic field generating element group and a set of electrical parameter values ​​to be applied to the magnetic field generating element group are determined.

4. The tactile interaction method based on electromagnetic force according to claim 2, wherein, Based on the environmental forces experienced by the remote robot during operation and the operating posture, the desired position of the magnetic field generating element group should be determined when an electromagnetic feedback force consistent with the environmental forces is to be generated at the operating body, and a set of electrical parameter values ​​to be applied to the magnetic field generating element group when the magnetic field generating element group is in the desired position, including: The total feedback force is obtained based on the environmental force and the gravity of the magnetic element installed on the operating body; Based on the total feedback force and the operating posture, the desired position of the magnetic field generating element group and a set of electrical parameter values ​​to be applied to the magnetic field generating element group are determined.

5. The tactile interaction method based on electromagnetic force according to claim 3 or 4, wherein, Acquire the operating posture of the operator used to control the operation of the remote robot, including: The obtained position and orientation of the operating subject are registered in coordinates; and The position and orientation of the operating subject after coordinate calibration are filtered to obtain the estimated position and orientation of the operating subject.

6. The tactile interaction method based on electromagnetic force according to claim 5, wherein, The position and orientation of the operating subject after coordinate calibration are filtered to obtain the estimated position and orientation of the operating subject, including: The estimated position is obtained by filtering the position of the operating subject after coordinate calibration using interval Kalman filtering technology. The estimated direction is obtained by filtering the orientation of the operating subject after coordinate calibration using an improved particle filter (IPF) algorithm.

7. The tactile interaction method based on electromagnetic force according to claim 5, wherein, Based on the total feedback force and the operating posture, the desired position of the magnetic field generating element group and a set of electrical parameter values ​​to be applied to the magnetic field generating element group are determined, including: Based on the estimated position and the components of the total feedback force on the x, y, and z axes, the desired vertical distance between the operating subject and the magnetic field generating element group on the z axis, and a set of electrical parameter values ​​to be applied to the magnetic field generating element group are obtained, where the x, y, and z axes are coordinate axes in the reference coordinate system; The desired position is determined based on the components of the estimated position on the x-axis, y-axis, and z-axis, as well as the desired vertical distance.

8. The tactile interaction method based on electromagnetic force according to claim 5, wherein, Based on the total feedback force and the operating posture, the desired position of the magnetic field generating element group and a set of electrical parameter values ​​to be applied to the magnetic field generating element group are determined, including: Based on the estimated position, the estimated direction, and the components of the total feedback force on the x, y, and z axes, the desired vertical distance between the operating subject and the magnetic field generating element group on the z axis, and a set of electrical parameter values ​​to be applied to the magnetic field generating element group are obtained, where the x, y, and z axes are coordinate axes in the reference coordinate system; The desired position is determined based on the components of the estimated position on the x-axis, y-axis, and z-axis and the desired vertical distance.

9. The tactile interaction method based on electromagnetic force according to claim 5, wherein, Based on the total feedback force and the operating posture, the desired position of the magnetic field generating element group and a set of electrical parameter values ​​to be applied to the magnetic field generating element group are determined, including: Based on the estimated position, the estimated direction, and the components of the total feedback force on the x, y, and z axes, the desired vertical distance between the magnetic field generating element group and the operating body, and a set of electrical parameter values ​​to be applied to the magnetic field generating element group are obtained in the direction perpendicular to the plane where the operating body is located, where the x, y, and z axes are coordinate axes in the reference coordinate system. The desired position is determined based on the components of the estimated position and the estimated direction on the x-axis, y-axis, and z-axis, and the desired vertical distance.

10. The tactile interaction method based on electromagnetic force according to any one of claims 7-9, wherein, The desired position and the set of electrical parameter values ​​are determined using a neural network model, based on the total feedback force and the operating posture. The neural network model comprises an input layer, a hidden layer, and an output layer, wherein the connections between the hidden layer and the input layer, as well as between the hidden layer and the output layer, are fully connected. The input layer includes a number of nodes corresponding to the number of components of the total feedback force and the operating posture on the x-axis, y-axis, and z-axis, and the output of the input layer is provided to the hidden layer through an activation function; The hidden layer contains at least two linear layers, and the output of each linear layer is provided to the next layer after being activated by an activation function; The output layer includes N+1 neuron nodes and is fully connected to the hidden layer. It is used to receive the output from the hidden layer and output the desired vertical distance and the set of electrical parameter values, where N is the number of magnetic field generating elements included in the magnetic field generating element group to which the set of electrical parameter values ​​are applied.

11. The tactile interaction method based on electromagnetic force according to claim 1, wherein, The electrical parameters are current or voltage, and the magnetic field generating element group includes multiple coils. Each coil is capable of generating a magnetic field when a current or voltage is applied, the magnetic field having a magnetic field direction associated with the arrangement direction of the coil and a magnetic field distribution associated with the applied current or voltage value.

12. A tactile interaction system based on electromagnetic force, comprising: A controller for performing the electromagnetic force-based tactile interaction method as described in any one of claims 1-11; A pose sensor is used to acquire the operating posture of the subject. A remote robot is used to operate according to the operating posture of the operating subject under the control of the controller, wherein the environmental forces experienced by the remote robot during operation are used to generate electromagnetic forces to be fed back to the operating subject; An end effector on the operating side is used to track the operating entity under the control of the controller; A magnetic field generating element group is disposed on the end effector on the operating side. Each magnetic field generating element in the magnetic field generating element group generates a magnetic field distribution in different directions based on the applied electrical parameters. The magnetic field distribution in different directions interacts with the magnetic element disposed at the operating body to synthesize electromagnetic forces in different directions at the operating body.

13. The tactile interaction system based on electromagnetic force according to claim 12, wherein, One of the magnetic field generating elements in the magnetic field generating element group is disposed on the end effector as the central magnetic field generating element; The remaining magnetic field generating elements in the magnetic field generating element group are disposed on the end effector and are equally spaced around the central magnetic field generating element, serving as surrounding magnetic field generating elements. The lower surface of the surrounding magnetic field generating element and the lower surface of the central magnetic field generating element form a concave region, and the magnetic field generating element group is located in this concave region.

14. The tactile interaction system based on electromagnetic force according to claim 13, wherein, A magnet satisfying a preset magnetic condition is fixed on the upper surface of the central magnetic field generating element. The magnet satisfying the preset magnetic condition interacts with the magnetic element on the operating body to counteract the gravity of the magnetic element.

15. An electromagnetic force generating device, comprising: An insulating housing includes a central block and a protruding block connected to the central block, the protruding block forming a concave region with the central block; A magnetic field generating element group is disposed in the concave region, comprising multiple magnetic field generating elements with different magnetic field directions. One magnetic field generating element in the group is disposed in the central block as the central magnetic field generating element, and the remaining magnetic field generating elements in the group are equally spaced in the protruding blocks as surrounding magnetic field generating elements. Each magnetic field generating element in the magnetic field generating element group generates a magnetic field based on the applied electrical parameters, and the magnetic field has a magnetic field direction associated with the arrangement direction of the magnetic field generating element and a magnetic field distribution associated with the value of the applied electrical parameters. The electrical parameters applied to the magnetic field generating elements in the magnetic field generating element group are used to generate an electromagnetic feedback force at the operating body used to control the operation of the remote robot, which is consistent with the environmental force experienced by the remote robot during operation. This force is determined based on the environmental force and the operating posture of the operating body, together with the desired position where the magnetic field generating element group should be located. The operating posture includes the position and orientation of the operating body. The desired position where the magnetic field generating element group should be located is determined based on: the desired vertical distance between the magnetic field generating element group and the operating body in a direction perpendicular to the plane where the operating body is located, based on the position, orientation, and environmental force of the operating body; and the desired position, based on the position, orientation, and desired vertical distance of the operating body. In this configuration, each magnetic field generating element in the magnetic field generating element group generates a magnetic field distribution in different directions based on the applied electrical parameters. The magnetic field distributions in different directions interact with the magnetic elements disposed at the operating body to generate electromagnetic forces in different directions at the operating body. The electromagnetic forces in different directions are combined to realize the electromagnetic feedback force.

16. The electromagnetic force generating device according to claim 15, wherein, The upper surface of the central magnetic field generating element in the magnetic field generating element group is fixed with a magnet that meets a preset magnetic condition. The magnet that meets the preset magnetic condition interacts with the magnetic element on the operating body to counteract the gravity of the magnetic element.

Images