Prosthetic wrist joint coordinated control method, device and readable medium based on tactile feedback

Abstract

The present invention discloses a method, device, and readable medium for collaborative control of a prosthetic wrist joint based on tactile feedback. The method comprises the following steps: obtaining a tactile image collected when the back of a prosthetic hand contacts an object to be grasped, preprocessing the tactile image to obtain a processed image; establishing a back-of-hand coordinate system, and calculating the first displacement required to move from the origin of the back-of-hand coordinate system to the contact position with the object to be grasped based on the processed image; determining a second displacement when the prosthetic palm rotates without touching the object, and setting a third displacement based on the distance between the grasping center position of the prosthetic palm and the origin of the back-of-hand coordinate system; establishing a prosthetic kinematic model, and controlling the prosthetic palm to grasp the object to be grasped based on the prosthetic kinematic model in combination with the first, second, and third displacements. Thus, tactile perception control and re-grasp planning are used to assist the user in operating the prosthetic hand to grasp an object, providing the user with a simpler and more flexible prosthetic hand operation method.

Description

Technical Field

[0001] The present invention relates to the field of prosthetic limb control, and in particular to a method, device and readable medium for coordinated control of a prosthetic wrist joint based on tactile feedback. Background Art

[0002] Compared to traditional mechanical prostheses, intelligent prostheses are unique in that they are equipped with various sensors for human-computer interaction, such as myoelectric sensors and tactile sensors, and each joint is controlled by a corresponding motor or mechanical structure. Using myoelectric sensors, a computer can read the user's operating intentions and control the prosthetic joints. Using tactile sensors to obtain tactile information not only provides users with real-time tactile perception, but also can be combined with a computer to control the arm, wrist, and finger joints for coordinated control, achieving grasping behaviors that are closer to human instinct.

[0003] Current intelligent prosthetic hands mostly rely on using electromyographic or electroencephalographic sensors to sense the user's intention to manipulate the prosthetic hand, or tactile sensors to sense the hand's grasping state, allowing for different control methods to manipulate the rotation of the finger joints and achieve grasping. However, this requires the user to accurately align the prosthetic hand with the object before the next grasping action can be performed. Such control methods lack coordinated control of the entire prosthetic joint and require the user to have a certain level of operational experience. They simply optimize the grasping behavior itself, without considering the intelligent and simplified operation of the prosthetic hand before grasping the object. Summary of the Invention

[0004] In response to the above-mentioned technical problems, the embodiment of the present application aims to propose a method, device and readable medium for coordinated control of a prosthetic wrist joint based on tactile feedback, so as to solve the technical problems mentioned in the above background technology section.

[0005] In a first aspect, the present invention provides a method for coordinated control of a prosthetic wrist joint based on tactile feedback, comprising the following steps:

[0006] S1, acquiring a tactile image collected when the back of the prosthetic hand contacts the object to be grasped, and preprocessing the tactile image to obtain a processed image;

[0007] S2, establishing a hand back coordinate system, and calculating the first displacement required to move from the origin of the hand back coordinate system to the contact position with the object to be grasped based on the processed image;

[0008] S3, determining a second displacement when the prosthetic palm rotates without touching the object, and setting a third displacement according to the distance between the grasping center position of the prosthetic palm and the origin of the hand back coordinate system;

[0009] S4, establishing a prosthetic kinematic model, and controlling the prosthetic hand to grasp the object according to the prosthetic kinematic model in combination with the first displacement, the second displacement, and the third displacement.

[0010] Preferably, the tactile image is collected by a flexible tactile array sensor installed on the back of the palm and the back of the fingers of the prosthesis, and the flexible tactile array sensor includes a plurality of sensing units distributed in an array.

[0011] Preferably, preprocessing the tactile image in step S1 to obtain a processed image specifically includes:

[0012] Normalize the value of each pixel in the tactile image and represent it as a grayscale image;

[0013] A threshold is set to separate the contact area and non-contact area of ​​the grayscale image. The area where the pixel value in the grayscale image is higher than or equal to the threshold is determined as the contact area, and the area where the pixel value in the grayscale image is lower than the threshold is determined as the non-contact area. The pixel values ​​in the non-contact area are reset to zero to obtain the processed image.

[0014] Preferably, step S2 specifically includes:

[0015] S21, with the center of the flexible tactile array sensor coverage area as the origin, the direction of the four fingers of the prosthetic hand as the X-axis, the direction along the back of the prosthetic hand and perpendicular to the X-axis as the Y-axis, and the direction of the prosthetic palm as the Z-axis, establish a back-of-hand coordinate system on the small curvature surface where the back of the prosthetic palm and the back of the fingers are located;

[0016] S22, calculate the center position P(X, Y, 0) of the contact area based on the processed image, where M 00 =∑ x ∑ y f(x,y),M 10 =∑ x ∑ y xf(x,y),M 01 =∑ x ∑ y yf(x,y), f(x,y) is the value of the pixel in the processed image;

[0017] S23, based on the center position P(X, Y, 0) of the contact area, calculate the first displacement that the prosthetic hand needs to move when the origin of the hand back coordinate system moves to the contact position with the object to be grasped

[0018] Preferably, step S3 specifically includes:

[0019] S31. Define the hand width dimension of the prosthetic hand as a, and calculate the vertical distance from the center position of the contact area to the edge of the index finger side of the back of the hand and the vertical distance to the edge of the little finger side of the back of the hand If l1 ≥ l2, define the second displacement when the prosthetic hand rotates without touching the object If l1 < l2, define the second displacement when the prosthetic hand rotates without touching the object

[0020] S32. Determine the range of the maximum diameter object that the prosthetic hand can grasp according to the spatial range of the circumference that the prosthetic hand can achieve to enclose the object to be grasped Set the center of the range of the maximum diameter object that the prosthetic hand can grasp as the grasping center position of the prosthetic hand. The distance between the grasping center position of the prosthetic hand and the origin of the coordinate system of the back of the hand is a fixed value b, and set the third displacement

[0021]

[0022] Preferably, in step S4, establish a prosthetic kinematic model, specifically including:

[0023] Define the number of connecting rods and rotary joints of the prosthetic kinematic model according to the structure of the human upper arm;

[0024] Set the initial kinematic parameters of the prosthetic according to the motion dimensions of the actual prosthetic connecting rods, including the rod lengths, connecting rod offsets, and connecting rod torsion angles of each connecting rod;

[0025] Set the joint coordinate system of the first connecting rod as the base coordinate, and set the end center of the last connecting rod as the origin of the end coordinate system;

[0026] Use the joint angles of the three joints when the user operates the prosthetic and the prosthetic back of the hand is in stable contact with the object to be grasped read by the angle sensor as the initial joint angles of the three joints;

[0027] Read the rotation angle of the current wrist joint as the torsion angle between the third connecting rod and the end coordinate system, so that the three-axis orientations of the end coordinate system and the coordinate system of the back of the prosthetic hand correspond; [[ID=三十五]]

[0028] Establish a prosthetic kinematic model with the coordinate system of the first joint as the base coordinate system {b}; set the origin position of the current end coordinate system {e} in the base coordinate system {b} as the starting point of motion b A0(X0, Y0, Z0);

[0029] Convert the coordinates of the starting and ending points of each displacement of the prosthetic hand from the end coordinate system {e} to the coordinates in the base coordinate system {b}, and construct the attitude matrix of the end coordinate system {e} with respect to the base coordinate system {b} according to the joint angles read by the angle sensor and the prosthetic kinematic parameters and position vector in, is the direction cosine matrix of the three unit orthogonal principal vectors of the end coordinate system {e} and the base coordinate system {b}, is the origin O of the end coordinate system {e} e The position vector in the base coordinate system {b}, then the coordinates of the starting point and end point of each displacement in the end coordinate system {e} in the base coordinate system {b} are i=0,1,2,3, will b A i The input is sent to the controller to drive the prosthetic joint to the target position.

[0030] Preferably, in step S4, controlling the prosthetic hand to grasp the object to be grasped according to the prosthetic kinematic model in combination with the first displacement, the second displacement, and the third displacement specifically includes:

[0031] S41, driving the prosthetic hand to break away from the contact with the object to be grasped, and moving along the center of the current prosthetic hand toward the Z axis in the end coordinate system {e} Arrived at the end of the first displacement e A1(X1, Y1, Z1) = (0, 0, l1) or (0, 0, l2). After reaching the position, the wrist joint of the prosthetic limb is driven to rotate 180° so that the palm of the prosthetic limb is roughly facing the object to be grasped.

[0032] S42, driving the prosthetic hand to move along the plane where the current prosthetic palm is located in the end coordinate system {e} Arrived at the end of the second displacement e A2(X2, Y2, Z2) = (X, -Y, 0), so that the center of the prosthetic palm is directly at the actual contact point on the surface of the object to be grasped;

[0033] S43: Drive the prosthetic hand to approach the object to be grasped and move along the center of the prosthetic palm toward the Z axis in the end coordinate system {e}. Or (0,0,l2-b), so that the prosthetic hand reaches the end point of the third displacement e A3(X3, Y3, Z3) = (0, 0, l1-b) or (0, 0, l2-b), which means it reaches the valid grasping position;

[0034] S44, after reaching the effective grasping position, the prosthetic hand is operated to grasp the object through the myoelectric control system.

[0035] In a second aspect, the present invention provides a prosthetic wrist joint coordinated control device based on tactile feedback, comprising:

[0036] an image acquisition module configured to acquire a tactile image collected when the back of the prosthetic hand contacts the object to be grasped, and pre-process the tactile image to obtain a processed image;

[0037] a first displacement calculation module configured to establish a hand back coordinate system and calculate, based on the processed image, a first displacement required for moving from the origin of the hand back coordinate system to a contact position with the object to be grasped;

[0038] a second displacement calculation module configured to determine a second displacement of the prosthetic hand when the palm rotates without touching the object, and to set a third displacement according to a distance between a grasping center position of the prosthetic hand and an origin of a hand back coordinate system;

[0039] The grasping module is configured to establish a prosthetic limb kinematic model, and control the prosthetic limb palm to grasp the object to be grasped according to the prosthetic limb kinematic model combined with the first displacement, the second displacement and the third displacement.

[0040] In a third aspect, the present invention provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0041] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] (1) The present invention analyzes the relative posture of the current prosthetic limb and the object to be grasped based on the tactile image collected by the flexible tactile array sensor, and performs collaborative control of the intelligent prosthetic hand-wrist joint, which can provide users with a simpler and more flexible prosthetic hand operation method.

[0044] (2) The present invention can plan the movement of the prosthetic hand-wrist joint according to the current posture of the prosthetic limb, and calculate the activity space required for the rotation of the prosthetic wrist joint; then the posture of the prosthetic limb is controlled, and the system operates the prosthetic limb to perform multiple displacements so that the prosthetic limb reaches the appropriate grasping position, thereby realizing the use of tactile perception control technology to assist the user in operating the prosthetic limb to grasp objects.

[0045] (3) The present invention takes into account the operational performance of the prosthesis and the influence of the user's own operational experience on the use and grasping of the prosthesis, and uses tactile perception control and re-grasping planning to assist the user in operating the prosthesis to grasp objects, which significantly reduces the operational experience threshold required for the user to use the prosthesis to grasp objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 is a diagram of an exemplary device architecture to which an embodiment of the present application may be applied;

[0048] Figure 2 Schematic diagram of a flow chart of a method for coordinated control of a prosthetic wrist joint based on tactile feedback according to an embodiment of the present application;

[0049] Figure 3 A schematic diagram of the hand back coordinate system and some dimensions of the prosthetic wrist joint collaborative control method based on tactile feedback according to an embodiment of the present application;

[0050] Figure 4 Schematic diagram of the control flow of the prosthetic wrist joint coordinated control method based on tactile feedback according to an embodiment of the present application;

[0051] Figure 5 Schematic diagram of a prosthetic wrist joint collaborative control device based on tactile feedback according to an embodiment of the present application;

[0052] Figure 6 It is a structural diagram of a computer device suitable for implementing the electronic device of the embodiment of the present application. DETAILED DESCRIPTION

[0053] To make the objectives, technical solutions, and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings. It is apparent that the embodiments described are only some, not all, of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.

[0054] Figure 1 An exemplary device architecture 100 is shown to which a method for collaborative control of a prosthetic wrist joint based on tactile feedback or a device for collaborative control of a prosthetic wrist joint based on tactile feedback according to an embodiment of the present application can be applied.

[0055] like Figure 1 As shown, the device architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0056] Users can use terminal devices 101, 102, 103 to interact with server 105 via network 104 to receive or send messages, etc. Various applications, such as data processing applications and file processing applications, can be installed on terminal devices 101, 102, 103.

[0057] Terminal devices 101, 102, and 103 can be hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices, including but not limited to smartphones, tablet computers, laptop computers, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or they can be implemented as a single software or software module. No specific limitations are given here.

[0058] The server 105 may be a server that provides various services, such as a background data processing server that processes files or data uploaded by the terminal devices 101, 102, and 103. The background data processing server may process the acquired files or data and generate processing results.

[0059] It should be noted that the prosthetic wrist joint collaborative control method based on tactile feedback provided in the embodiment of the present application can be executed by the server 105, or by the terminal devices 101, 102, and 103. Accordingly, the prosthetic wrist joint collaborative control device based on tactile feedback can be set in the server 105, or in the terminal devices 101, 102, and 103.

[0060] It should be understood that Figure 1 The number of terminal devices, networks, and servers in the above description is merely illustrative. Any number of terminal devices, networks, and servers may be provided as needed. If the processed data does not need to be acquired remotely, the above-described apparatus architecture may not include a network, but only require servers or terminal devices.

[0061] Figure 2 A method for collaborative control of a prosthetic wrist joint based on tactile feedback is provided in an embodiment of the present application, comprising the following steps:

[0062] S1, obtaining a tactile image collected when the back of the prosthetic hand contacts the object to be grasped, and preprocessing the tactile image to obtain a processed image.

[0063] In a specific embodiment, the tactile image is collected by a flexible tactile array sensor installed on the back of the palm and the back of the fingers of the prosthesis, and the flexible tactile array sensor includes a plurality of sensing units distributed in an array.

[0064] In a specific embodiment, preprocessing the tactile image in step S1 to obtain a processed image specifically includes:

[0065] Normalize the value of each pixel in the tactile image and represent it as a grayscale image;

[0066] A threshold is set to separate the contact area and non-contact area of ​​the grayscale image. The area where the pixel value in the grayscale image is higher than or equal to the threshold is determined as the contact area, and the area where the pixel value in the grayscale image is lower than the threshold is determined as the non-contact area. The pixel values ​​in the non-contact area are reset to zero to obtain the processed image.

[0067] Specifically, refer to Figure 3 The prosthesis consists of a prosthetic hand mounted on a prosthetic arm, and flexible tactile array sensors are installed on the back of the palm and the back of the fingers of the prosthetic hand. The wrist joint of the prosthetic hand is the terminal joint connecting the prosthetic arm and the prosthetic hand. The prosthesis can be operated by the user using electromyographic signals to rotate each joint, and each joint is equipped with an angle sensor, which is driven by a corresponding motor. The flexible tactile array sensor has array-distributed sensing units and can output tactile images. Flexible pads are placed under the flexible tactile array sensors installed on the back of the palm and the back of the fingers of the prosthetic hand, so that the spatial distribution of the flexible tactile array sensors has a certain curvature, which prevents the contact points between the user's operation of the prosthetic hand and the object to be grasped from being concentrated on the edge of the sensor, thereby achieving more accurate contact area measurement.

[0068] refer to Figure 4 ,This method mainly includes the back of hand contact recognition stage, ,prosthetic limb posture estimation stage, re-grasping planning stage and prosthetic limb posture control stage.

[0069] During the back-of-hand contact recognition phase, the user operates the prosthetic hand through a myoelectric control system and makes contact with the object to be grasped on the back of the prosthetic hand. The point of contact between the object to be grasped and the prosthetic hand is located in the area covered by flexible tactile array sensors on the back of the palm and fingers of the prosthetic hand. After the prosthetic hand makes contact with the object to be grasped, the system extracts the tactile image output by the flexible tactile array sensors after the contact stabilizes. The tactile images of the flexible tactile array sensors installed on the back of the palm and fingers of the prosthetic hand are collected, and the value of each pixel in the tactile image is normalized and represented as a grayscale image. A threshold is set to separate the contact area from the non-contact area. Pixels with values ​​below the threshold are judged to be non-contact areas, and the pixel values ​​in the non-contact area are reset to zero.

[0070] S2, establishing a hand back coordinate system, and calculating a first displacement required for moving from the origin of the hand back coordinate system to the contact position with the object to be grasped based on the processed image.

[0071] In a specific embodiment, step S2 specifically includes:

[0072] S21: Taking the center of the coverage range of the flexible tactile array sensor as the origin, the direction of the four fingers of the prosthetic finger as the X-axis, the direction along the back of the prosthetic hand and perpendicular to the X-axis as the Y-axis, and the direction of the palm of the prosthetic hand as the Z-axis, establish a coordinate system on the small-curvature surface where the back of the prosthetic hand and the back of the fingers are located;

[0073] S22: Calculate the center position P(X, Y, 0) of the contact area according to the processed image, where M 00 = ∑ x ∑ y f(x, y), M 10 = ∑ x ∑ y xf(x, y), M 01 = ∑ x ∑ y yf(x, y), and f(x, y) is the value of the pixel point in the processed image;

[0074] S23: Calculate the first displacement that the prosthetic hand needs to move when the origin of the coordinate system on the back of the hand moves to the contact position with the object to be grasped according to the center position P(X, Y, 0) of the contact area

[0075] Specifically, the above steps are executed in the prosthetic hand pose estimation stage to analyze the relative pose between the current prosthetic hand and the object to be grasped.

[0076] S3: Determine the second displacement when the prosthetic hand palm rotates without touching the object, and set the third displacement according to the distance between the grasping center position of the prosthetic hand palm and the origin of the coordinate system on the back of the hand.

[0077] In a specific embodiment, step S3 specifically includes:

[0078] S31: Define the hand width size of the prosthetic hand as a, and calculate the vertical distance from the center position of the contact area to the side edge of the index finger on the back of the hand and the vertical distance to the side edge of the little finger on the back of the hand If l1 ≥ l2, then define the second displacement when the prosthetic hand palm rotates without touching the object If l1 < l2, then define the second displacement when the prosthetic hand palm rotates without touching the object

[0079] S32: According to the prosthetic hand's ability to enclose the object to be grasped The spatial range of the above circumference determines the range of the maximum diameter object that the prosthetic hand can grasp. The center of the range of the maximum diameter object that the prosthetic hand can grasp is set as the grasping center position of the prosthetic palm. The distance between the grasping center position of the prosthetic palm and the origin of the hand back coordinate system is a fixed value b. The third displacement is set

[0080]

[0081] Specifically, the re-grasp planning phase primarily involves executing the above steps. Grasp planning determines the second displacement of the prosthetic hand, which prevents the hand from contacting the object during rotation, and the third displacement, which moves from the origin of the back-of-hand coordinate system to the grasping center of the prosthetic hand. Subsequent steps require maintaining the position of the object in contact with the back of the prosthetic hand while moving the prosthetic hand away from the object and flipping it. After flipping, the prosthetic hand is then moved back to the grasping center.

[0082] S4, establishing a prosthetic kinematic model, and controlling the prosthetic hand to grasp the object according to the prosthetic kinematic model in combination with the first displacement, the second displacement, and the third displacement.

[0083] In a specific embodiment, establishing a prosthetic limb kinematic model in step S4 specifically includes:

[0084] The number of links and rotational joints in the limb kinematic model is set according to the structure of the human upper arm;

[0085] The initial kinematic parameters of the prosthesis are set according to the actual motion dimensions of the prosthetic links, including the length of each link, link offset, and link torsion angle;

[0086] Set the joint coordinate system of the first link as the base coordinate system, and set the end center of the last link as the origin of the end coordinate system;

[0087] The angle sensor reads the joint angles of the three joints when the user operates the prosthetic limb and uses the back of the prosthetic hand to stably contact the object to be grasped as the initial joint angles of the three joints;

[0088] The current wrist joint rotation angle is read as the torsion angle between the third link and the end coordinate system, so that the three-axis orientation of the end coordinate system and the back hand coordinate system of the prosthesis correspond to each other;

[0089] The coordinate system of the first joint is used as the base coordinate system {b} to establish the prosthetic limb kinematic model; the origin position of the current end coordinate system {e} is set as the starting point of the movement in the base coordinate system {b} b A0(X0,Y0,Z0);

[0090] The coordinates of the starting and ending points of each displacement of the prosthetic hand are converted from the end coordinate system {e} to the coordinates of the base coordinate system {b}, and the posture matrix of the end coordinate system {e} to the base coordinate system {b} is constructed according to the joint angles read by the angle sensor and the kinematic parameters of the prosthetic hand. and position vector in, is the direction cosine matrix of the three unit orthogonal principal vectors of the end coordinate system {e} and the base coordinate system {b}, is the origin O of the end coordinate system {e} e The position vector in the base coordinate system {b}, then the coordinates of the starting point and end point of each displacement in the end coordinate system {e} in the base coordinate system {b} are i=0,1,2,3, will b A i The input is sent to the controller to drive the prosthetic joint to the target position.

[0091] Specifically, a kinematic model of the limb is designed based on the structure of the human upper arm. The human upper arm structure is set with three links and three rotational joints. The shoulder joint consists of two rotational joints and two links with two degrees of freedom, and the elbow joint consists of one rotational joint and one link with one degree of freedom. The link torsion angle of the first and second links is set to 90°, and the link torsion angle of the second and third links is set to 0°. In the subsequent process, the starting point and end point of each displacement in the end coordinate system {e} are first determined. e A i , and then according to and The starting and ending points of each displacement segment in the end coordinate system {e} e A i Convert to coordinates in the base coordinate system {b}, b A i The input is sent to the controller to drive the prosthetic joint to the target position.

[0092] In a specific embodiment, step S4 controls the prosthetic hand to grasp the object to be grasped based on the prosthetic kinematic model in combination with the first displacement, the second displacement, and the third displacement, and specifically includes:

[0093] S41, driving the prosthetic hand to break away from the contact with the object to be grasped, and moving along the center of the current prosthetic hand toward the Z axis in the end coordinate system {e} Arrived at the end of the first displacement e A1(X1, Y1, Z1) = (0, 0, l1) or (0, 0, l2). After reaching the position, the wrist joint of the prosthetic limb is driven to rotate 180° so that the palm of the prosthetic limb is roughly facing the object to be grasped.

[0094] S42, driving the prosthetic hand to move along the plane where the current prosthetic palm is located in the end coordinate system {e} Arrived at the end of the second displacement e A2(X2, Y2, Z2) = (X, -Y, 0), so that the center of the prosthetic palm is directly at the actual contact point on the surface of the object to be grasped;

[0095] S43: Drive the prosthetic hand to approach the object to be grasped and move along the center of the prosthetic palm toward the Z axis in the end coordinate system {e}. Or (0,0,l2-b), so that the prosthetic hand reaches the end point of the third displacement e A3(X3, Y3, Z3) = (0, 0, l1-b) or (0, 0, l2-b), which means it reaches the valid grasping position;

[0096] S44, after reaching the effective grasping position, the prosthetic hand is operated to grasp the object through the myoelectric control system.

[0097] Specifically, the above steps are performed in the prosthetic posture control stage, and the effective grasping position reached by the prosthetic hand is located at a position where the grasping center of the prosthetic palm, the origin of the hand back coordinate system, and the actual contact point of the hand back on the surface of the object to be grasped are in a straight line. Therefore, the movement of the prosthetic wrist joint can be planned according to the current prosthetic posture, and the activity space required for the rotation of the prosthetic wrist joint can be calculated; then the prosthetic posture is controlled, and the system operates the prosthetic to perform multiple displacements so that the prosthetic reaches a suitable grasping position, thereby realizing the use of tactile perception control technology to assist the user in operating the prosthetic to grasp objects. The embodiment of the present application optimizes the process of the user operating the prosthetic limb, so that the user does not need to accurately align the prosthetic limb with the object to be grasped. The user only needs to contact the back of the prosthetic hand with the object to be grasped, and the prosthetic palm can be aligned with the object to be grasped according to the contact position, so that the user's operation of the prosthetic hand is more in line with the grasping behavior of the human hand relying on intuition and skin perception.

[0098] Further references Figure 5 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a prosthetic wrist joint collaborative control device based on tactile feedback, which is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0099] The present application provides a tactile feedback-based coordinated control device for a prosthetic wrist joint, comprising:

[0100] Image acquisition module 1 is configured to acquire a tactile image collected when the back of the prosthetic hand contacts the object to be grasped, and pre-process the tactile image to obtain a processed image;

[0101] A first displacement calculation module 2 is configured to establish a hand back coordinate system and calculate a first displacement required for moving from the origin of the hand back coordinate system to a contact position with the object to be grasped based on the processed image;

[0102] A second displacement calculation module 3 is configured to determine a second displacement when the prosthetic palm rotates without touching the object, and set a third displacement according to the distance between the grasping center position of the prosthetic palm and the origin of the hand back coordinate system;

[0103] The grasping module 4 is configured to establish a prosthetic limb kinematic model, and control the prosthetic limb palm to grasp the object to be grasped according to the prosthetic limb kinematic model in combination with the first displacement, the second displacement and the third displacement.

[0104] Reference below Figure 6 , which shows an electronic device (eg Figure 1 A structural diagram of a computer device 600 (a server or terminal device as shown). Figure 6 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0105] like Figure 6 As shown, the computer device 600 includes a central processing unit (CPU) 601 and a graphics processing unit (GPU) 602, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 603 or the program loaded from the storage part 609 to the random access memory (RAM) 604. Various programs and data required for the operation of the device 600 are also stored in the RAM 604. The CPU 601, GPU 602, ROM 603 and RAM 604 are connected to each other through a bus 605. An input / output (I / O) interface 606 is also connected to the bus 605.

[0106] The following components are connected to the I / O interface 606: an input section 607 including a keyboard, a mouse, and the like; an output section 608 including a display such as a liquid crystal display (LCD), a speaker, and the like; a storage section 609 including a hard disk and the like; and a communication section 610 including a network interface card such as a LAN card or a modem. The communication section 610 performs communication processing via a network such as the Internet. A drive 611 may also be connected to the I / O interface 606 as needed. A removable medium 612, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like, is installed in the drive 611 as needed, so that a computer program read therefrom can be installed into the storage section 609 as needed.

[0107] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 610, and / or installed from a removable medium 612. When the computer program is executed by the central processing unit (CPU) 601 and the graphics processing unit (GPU) 602, the above-mentioned functions defined in the method of the present application are executed.

[0108] It should be noted that the computer-readable medium described in this application may be a computer-readable signal medium or a computer-readable medium, or any combination thereof. Computer-readable media may be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor devices, apparatuses, or components, or any combination thereof. More specific examples of computer-readable media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution device, apparatus, or component. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution apparatus, device, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical cable, RF, or any suitable combination thereof.

[0109] Computer program code for performing the operations of the present application can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0110] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions and operations of the devices, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of code, and the module, program segment or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based device that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0111] The modules involved in the embodiments described in this application may be implemented in software or hardware, and may also be set in a processor.

[0112] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or it may exist independently and not be assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by the electronic device, the electronic device: obtains a tactile image collected when the back of the prosthetic hand contacts the object to be grasped, pre-processes the tactile image, and obtains a processed image; establishes a back-of-the-hand coordinate system, and calculates the first displacement required to move from the origin of the back-of-the-hand coordinate system to the contact position with the object to be grasped based on the processed image; determines the second displacement of the prosthetic palm when it rotates without touching the object, and sets the third displacement based on the distance between the grasping center position of the prosthetic palm and the origin of the back-of-the-hand coordinate system; establishes a prosthetic kinematic model, and controls the prosthetic palm to grasp the object to be grasped based on the prosthetic kinematic model in combination with the first displacement, the second displacement, and the third displacement.

[0113] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned inventive concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A method for coordinated control of a prosthetic wrist joint based on tactile feedback, characterized in that: The following steps are involved: S1, acquiring a tactile image collected when the back of the prosthetic hand contacts the object to be grasped, and preprocessing the tactile image to obtain a processed image; S2, establishing a hand back coordinate system, and calculating a first displacement required for moving from the origin of the hand back coordinate system to a contact position with the object to be grasped based on the processed image; S3, determining a second displacement when the prosthetic palm rotates without touching the object, and setting a third displacement according to the distance between the grasping center position of the prosthetic palm and the origin of the hand back coordinate system; S4, establishing a prosthetic limb kinematic model, and controlling the prosthetic limb palm to grasp the object to be grasped according to the prosthetic limb kinematic model in combination with the first displacement, the second displacement and the third displacement.

2. The method for coordinated control of a prosthetic wrist joint based on tactile feedback according to claim 1, characterized in that: The tactile image is collected by a flexible tactile array sensor installed on the back of the palm and the back of the fingers of the prosthesis, and the flexible tactile array sensor includes a plurality of sensing units distributed in an array.

3. The method for coordinated control of a prosthetic wrist joint based on tactile feedback according to claim 1, characterized in that: The step S1 pre-processes the tactile image to obtain a processed image, specifically comprising: Normalizing the value of each pixel in the tactile image and expressing it as a grayscale image; A threshold is set to separate the contact area and non-contact area of ​​the grayscale image, and the area in the grayscale image where the pixel value is higher than or equal to the threshold is determined as the contact area, and the area in the grayscale image where the pixel value is lower than the threshold is determined as the non-contact area, and the pixel value of the non-contact area is reset to zero to obtain the processed image.

4. The method for coordinated control of a prosthetic wrist joint based on tactile feedback according to claim 2, characterized in that: The step S2 specifically includes: S21, establishing a back-of-hand coordinate system on the small curvature surface where the back of the prosthetic palm and the back of the fingers are located, with the center of the coverage area of ​​the flexible tactile array sensor as the origin, the direction of the four fingers of the prosthetic hand as the X-axis, the direction along the back of the prosthetic hand and perpendicular to the X-axis as the Y-axis, and the direction of the prosthetic palm as the Z-axis; S22, calculating the center position P(X, Y, 0) of the contact area based on the processed image, where: M 00 =∑ x ∑ y f(x,y),M 10 =∑ x ∑ y xf(x,y),M 01 =∑ x x y yf(x, y), f(x, y) is the value of the pixel in the processed image; S23, calculating the first displacement required for the prosthetic hand to move when the origin of the hand back coordinate system moves to the contact position with the object to be grasped based on the center position P(X, Y, 0) of the contact area 5. The method for coordinated control of a prosthetic wrist joint based on tactile feedback according to claim 4, characterized in that: The step S3 specifically includes: S31, define the hand width as a, and calculate the vertical distance from the center of the contact area to the edge of the index finger on the back of the hand The vertical distance from the edge of the little finger to the back of the hand If l1 ≥ l2, define the second displacement of the prosthetic hand when it rotates without touching the object If l1 < l2, define the second displacement of the prosthetic hand when it rotates without touching the object S32, according to the ability of the prosthetic hand to envelop the object to be grasped The spatial range of the above circumference determines the range of the maximum diameter object that the prosthetic hand can grasp. The center of the range of the maximum diameter object that the prosthetic hand can grasp is set as the grasping center position of the prosthetic palm. The distance between the grasping center position of the prosthetic palm and the origin of the hand back coordinate system is a fixed value b. The third displacement is set 6. The method for coordinated control of a prosthetic wrist joint based on tactile feedback according to claim 1, characterized in that: The step S4 of establishing the prosthetic limb kinematic model specifically includes: The number of links and rotational joints in the limb kinematic model is set according to the structure of the human upper arm; The initial kinematic parameters of the prosthesis are set according to the actual motion dimensions of the prosthetic links, including the length of each link, link offset, and link torsion angle; Set the joint coordinate system of the first link as the base coordinate system, and set the end center of the last link as the origin of the end coordinate system; The angle sensor reads the joint angles of the three joints when the user operates the prosthetic limb and uses the back of the prosthetic hand to stably contact the object to be grasped as the initial joint angles of the three joints; The current wrist joint rotation angle is read as the torsion angle between the third link and the end coordinate system, so that the three-axis orientation of the end coordinate system and the back hand coordinate system of the prosthesis correspond to each other; The coordinate system of the first joint is used as the base coordinate system {b} to establish the prosthetic limb kinematic model; the origin position of the current end coordinate system {e} is set as the starting point of the movement in the base coordinate system {b} b A0(X0, Y0, Z0); The coordinates of the starting and ending points of each displacement of the prosthetic hand are converted from the end coordinate system {e} to the coordinates of the base coordinate system {b}, and the posture matrix of the end coordinate system {e} to the base coordinate system {b} is constructed according to the joint angles read by the angle sensor and the kinematic parameters of the prosthetic hand. and position vector in, is the direction cosine matrix of the three unit orthogonal principal vectors of the end coordinate system {e} and the base coordinate system {b}, is the origin O of the end coordinate system {e} e The position vector in the base coordinate system {b}, then the coordinates of the starting point and end point of each displacement in the end coordinate system {e} in the base coordinate system {b} are i=0,1,2,3, will b A i The input is sent to the controller to drive the prosthetic joint to the target position.

7. The method for coordinated control of a prosthetic wrist joint based on tactile feedback according to claim 6, characterized in that: In step S4, controlling the prosthetic palm to grasp the object to be grasped according to the prosthetic kinematic model in combination with the first displacement, the second displacement, and the third displacement specifically includes: S41, driving the prosthetic hand to break away from the contact with the object to be grasped, and moving along the center of the current prosthetic hand toward the Z axis in the end coordinate system {e} Arrived at the end of the first displacement e A1(X1, Y1, Z1) = (0, 0, l1) or (0, 0, l2). After reaching the position, the wrist joint of the prosthetic limb is driven to rotate 180° so that the palm of the prosthetic limb is roughly facing the object to be grasped. S42, driving the prosthetic hand to move along the plane where the current prosthetic palm is located in the end coordinate system {e} Arrived at the end of the second displacement e A2(X2, Y2, Z2) = (X, -Y, 0), so that the center of the prosthetic palm is exactly at the actual contact point on the surface of the object to be grasped; S43, driving the prosthetic hand to approach the object to be grasped, and moving along the center of the current prosthetic palm toward the Z axis in the end coordinate system {e} Or (0, 0, l2-b), so that the prosthetic hand reaches the end point of the third displacement e A3(X3, Y3, Z3) = (0, 0, l1-b) or (0, 0, l2-b), i.e. reaching the valid grasping position; S44, after reaching the effective grasping position, the prosthetic hand is operated to grasp the object to be grasped through the myoelectric control system.

8. A prosthetic wrist joint coordinated control device based on tactile feedback, characterized in that: include: an image acquisition module configured to acquire a tactile image collected when the back of the prosthetic hand contacts the object to be grasped, and pre-process the tactile image to obtain a processed image; a first displacement calculation module configured to establish a hand back coordinate system and calculate a first displacement required for moving from the origin of the hand back coordinate system to a contact position with the object to be grasped based on the processed image; a second displacement calculation module configured to determine a second displacement of the prosthetic hand when the palm rotates without touching the object, and to set a third displacement according to a distance between a grasping center position of the prosthetic hand and an origin of a hand back coordinate system; The grasping module is configured to establish a prosthetic limb kinematic model, and control the prosthetic limb palm to grasp the object to be grasped according to the prosthetic limb kinematic model in combination with the first displacement, the second displacement and the third displacement.

9. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images