Data glove and dexterous hand pose mapping method and system

Abstract

This invention discloses a method and system for mapping pose between a data glove and a dexterous hand. The method includes: detecting the start time and movement beat interval of each channel; arranging all start times of the same finger into a start sequence and all movement beat intervals into a beat interval sequence; combining elements at corresponding positions of the two sequences into tuples to obtain the finger's motion pattern encoding sequence; calculating the edit distance between any two finger motion pattern encoding sequences to obtain an inter-finger edit distance matrix; extracting the principal component variance contribution rate of the edit distance matrix as the hand motion pattern dispersion; when the dispersion is less than a threshold, matching the finger motion pattern encoding sequence with a preset dexterous hand motion pattern library, and selecting the joint angle trajectory corresponding to the matching template as the control command output; when the dispersion is not less than the threshold, each finger is mapped independently. By using the start sequence and beat interval encoding motion patterns, adaptive mapping from a data glove to a dexterous hand is achieved.

Description

Technical Field

[0001] This invention belongs to the field of human-computer interaction and robot control technology, and particularly relates to a method and system for mapping the pose of a data glove and a dexterous hand. Background Technology

[0002] A data glove is a human-computer interaction device that can collect the bending angles of various joints in the human hand in real time. It is widely used in fields such as teleoperated robots, virtual reality, rehabilitation training, and prosthetic control. As the control object of the data glove, the dexterous hand needs to convert the bending angle signals of various joints of the human hand into angle commands for each joint of the dexterous hand.

[0003] Existing methods for mapping pose between data gloves and dexterous hands mainly suffer from the following technical shortcomings: First, direct mapping based on angle amplitude loses motion pattern information. Existing methods directly convert the bending angle amplitude of each joint into the corresponding angle command of the dexterous hand through linear transformation or nonlinear mapping, utilizing only the single dimension of angle amplitude information. During human hand movement, dynamic pattern information such as the activation sequence of each joint and the movement rhythm is completely ignored, resulting in a lack of naturalness in the dexterous hand's movements.

[0004] Second, each finger is mapped independently, failing to consider hand movement coordination. Existing methods treat the five fingers as independent control units, mapping each finger to a corresponding finger in the dexterous hand. When the hand is in cooperative grasping mode, the movements of each finger are interconnected; when the hand is in fine manipulation mode, the movements of each finger are independent. Existing methods cannot automatically identify the current hand movement coordination state and employ a fixed mapping strategy, resulting in a lack of coordination in the mapped movements of each finger in cooperative grasping mode.

[0005] Third, the mapping parameters rely on manual calibration. Existing methods require calibration for each operator by collecting sensor readings and their correspondence with joint angles under specific postures to determine the mapping parameters. The calibration process is cumbersome and has poor individual adaptability.

[0006] Therefore, there is a need for a method that can automatically identify the coordination state of hand movements and adaptively select a mapping strategy by utilizing the initiation sequence and beat interval information of human hand movements. Summary of the Invention

[0007] This invention provides a method and system for mapping the pose of a data glove and a dexterous hand, to solve at least one of the above-mentioned technical problems.

[0008] In a first aspect, the present invention provides a method for mapping the pose of a data glove and a dexterous hand, comprising: Acquire the bending angle signals of each sensor channel of the data glove; The moment when the angle value in the bending angle signal of each channel first exceeds the preset start threshold is taken as the start time of each channel. The time interval between two consecutive adjacent local extreme points in the bending angle signal of the same channel is taken as the motion beat interval of each channel. Taking all the start times corresponding to the same finger as the processing object, the start times are arranged in chronological order to obtain the start sequence sequence corresponding to the same finger. Taking all the movement beat intervals corresponding to the same finger as the processing object, the movement beat intervals are arranged in the corresponding joint order to obtain the beat interval sequence corresponding to the same finger. Each element in the start sequence is combined with the corresponding element in the beat interval sequence to form a tuple. All tuples are arranged in order to obtain the motion pattern encoding sequence corresponding to the same finger. Using the motion pattern coding sequences of all fingers as the processing object, the edit distance between any two motion pattern coding sequences of fingers is calculated to obtain the inter-finger edit distance matrix; Extract the principal components of the edit distance matrix, use the variance contribution rate of the principal components as the dispersion of the hand movement pattern, and determine whether the dispersion of the hand movement pattern is less than a preset dispersion threshold. When the dispersion of the hand movement pattern is less than the preset dispersion threshold, the movement pattern encoding sequence of each finger is used as input, and the preset dexterous hand action pattern library is used as the matching target. The movement pattern encoding sequence of each finger is matched with each action pattern encoding template in the action pattern library. The joint angle trajectory corresponding to the action pattern encoding template with the highest matching degree is selected as the control command of the dexterous hand and output to the dexterous hand joint controller for execution.

[0009] Secondly, the present invention provides a data glove and dexterous hand pose mapping system, comprising: The acquisition module is configured to acquire the bending angle signals of each sensor channel of the data glove; The detection module is configured to detect the moment when the angle value in the bending angle signal of each channel first exceeds the preset start threshold, and use it as the start time of each channel; and to detect the time interval between two consecutive adjacent local extreme points in the bending angle signal of the same channel, and use it as the motion beat interval of each channel. The processing module is configured to take all the start times corresponding to the same finger as the processing object, arrange all the start times in chronological order to obtain the start sequence sequence corresponding to the same finger, and take all the movement beat intervals corresponding to the same finger as the processing object, arrange all the movement beat intervals in the corresponding joint order to obtain the beat interval sequence corresponding to the same finger. The sorting module is configured to combine each element in the start sequence with the corresponding element in the beat interval sequence into a tuple, and arrange all the tuples in order to obtain the motion pattern encoding sequence corresponding to the same finger. The calculation module is configured to take the motion pattern encoding sequences of all fingers as the processing object, calculate the edit distance between any two motion pattern encoding sequences of fingers, and obtain the inter-finger edit distance matrix. The judgment module is configured to extract the principal components of the edit distance matrix, use the variance contribution rate of the principal components as the dispersion of the hand movement pattern, and determine whether the dispersion of the hand movement pattern is less than a preset dispersion threshold. The output module is configured to, when the dispersion of the hand movement pattern is less than a preset dispersion threshold, take the movement pattern encoding sequence of each finger as input, take the preset dexterous hand action pattern library as the matching target, match the movement pattern encoding sequence of each finger with each action pattern encoding template in the action pattern library, select the joint angle trajectory corresponding to the action pattern encoding template with the highest matching degree as the control command of the dexterous hand, and output it to the dexterous hand joint controller for execution.

[0010] Thirdly, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the data glove and dexterous hand pose mapping method of any embodiment of the present invention.

[0011] Fourthly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the program instructions are executed by a processor, the processor performs the steps of the data glove and dexterous hand pose mapping method of any embodiment of the present invention.

[0012] The data glove and dexterity hand pose mapping method and system of this application have the following beneficial effects: Instead of using the bending angle amplitude as the mapping basis, this method extracts the initiation time and beat interval of each joint, and uses a combination of initiation sequence and beat interval to encode and describe the finger's movement pattern. The initiation sequence reflects the sequential relationship of each joint's initiation and is a temporal topological feature of motion control; the beat interval reflects the rhythm of each joint's movement and is a rate distribution feature of motion control. The combination of the two fully describes the finger's control strategy during the initiation phase.

[0013] The edit distance matrix between the encoded sequences of each finger's motion pattern is calculated, and the variance contribution rate of the first principal component of the edit distance matrix is ​​extracted as the dispersion of the hand motion pattern through principal component analysis. When the dispersion is low, the encoded sequences of each finger's motion pattern tend to be consistent, and the hand is in a state of coordinated movement; when the dispersion is high, the encoded sequences of each finger's motion pattern show significant differences, and the hand is in a state of independent movement. This determination process does not rely on any pre-set gesture template.

[0014] When the hand is in a coordinated movement state, the present invention matches the current hand movement mode encoding sequence with the template in the preset action mode library, and uses the predefined joint angle trajectory corresponding to the successfully matched template as the control command of the dexterous hand. This mapping method converts the signal of the data glove into the overall movement mode of the dexterous hand, rather than the independent angle values ​​of each joint, making the movement of the dexterous hand more natural and coordinated.

[0015] When the hand is in an independent movement state, the bending angle signal of each finger is mapped to the joint angle command of the corresponding finger of the dexterous hand, realizing precise independent finger control. The two strategies of cooperative mapping and independent mapping are automatically switched according to the dispersion of hand movement mode, realizing adaptive selection of mapping strategy. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a method for mapping the pose of a data glove and a dexterous hand according to an embodiment of the present invention; Figure 2 This is a structural block diagram of a data glove and dexterous hand pose mapping system provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Please see Figure 1 The diagram shows a flowchart of a data glove and dexterous hand pose mapping method according to this application.

[0020] like Figure 1 As shown, the method for mapping the pose of a data glove to a dexterous hand specifically includes the following steps: Step S101: Obtain the bending angle signal of each sensor channel of the data glove.

[0021] In this step, bending angle signals of various joints of the human hand are collected by bending sensors in the data glove. The data glove contains multiple bending sensors installed at the metacarpophalangeal joints and interphalangeal joints of the thumb, index finger, middle finger, ring finger and little finger. Each sensor synchronously collects bending angle values ​​at a preset sampling frequency, and the bending angle signals of each sensor channel are aligned on the time axis.

[0022] Step S102: Detect the moment when the angle value in the bending angle signal of each channel first exceeds the preset start threshold, and take it as the start time of each channel. Detect the time interval between two consecutive adjacent local extreme points in the bending angle signal of the same channel, and take it as the motion beat interval of each channel.

[0023] In this step, for the bending angle signal of each channel, the sampling point is scanned sequentially starting from the signal start position. The sampling moment when the angle value first exceeds the preset start threshold is detected and taken as the start time of the corresponding channel. The preset start threshold is determined based on the noise level of the channel in the resting state, and is usually taken as 3 times the standard deviation of the angle value in the resting state. At the same time, all local extreme points are detected in the bending angle signal of the channel. The local extreme point is the position where the first derivative of the bending angle signal is zero. For two consecutive adjacent local extreme points, the time difference between the two local extreme points is calculated as the motion beat interval corresponding to the channel. All adjacent local extreme point pairs of the same channel are traversed to obtain all motion beat intervals corresponding to the channel.

[0024] Step S103: Taking all the start times corresponding to the same finger as the processing object, arrange all the start times in chronological order to obtain the start sequence sequence corresponding to the same finger. Taking all the movement beat intervals corresponding to the same finger as the processing object, arrange all the movement beat intervals in the corresponding joint order to obtain the beat interval sequence corresponding to the same finger.

[0025] In this step, for the same finger, the activation times of all the joints contained in the finger are arranged in chronological order to obtain the activation sequence of the finger. Each element in the activation sequence is a joint identifier, and the position in the sequence represents the order in which the joints are activated. The joint at the beginning of the sequence is the first joint activated by the finger, and the joint at the end of the sequence is the last joint activated by the finger. At the same time, the movement beat intervals corresponding to all the joints contained in the finger are arranged in the physiological order of the joints (from the metacarpophalangeal joints to the distal interphalangeal joints) to obtain the beat interval sequence of the finger. Each element in the beat interval sequence is the corresponding beat interval value.

[0026] Step S104: Combine each element in the start sequence with the corresponding element in the beat interval sequence into a tuple, and arrange all the tuples in order to obtain the motion pattern encoding sequence corresponding to the same finger.

[0027] In this step, the joints corresponding to the same finger are used as indices to extract the start sequence value corresponding to each joint from the start sequence sequence, and the beat interval value corresponding to each joint is extracted from the beat interval sequence. For each joint, the start sequence value corresponding to the joint is used as the first component of the tuple, and the beat interval value corresponding to the joint is quantized and used as the second component of the tuple to obtain the tuple corresponding to each joint. The quantization method is to compare the beat interval value with multiple preset beat level thresholds to determine the beat level to which the beat interval belongs, and use the level number of the beat level as the second component of the tuple. Arrange the binary pairs corresponding to each joint according to the physiological order of each joint in the same finger to obtain the motion pattern encoding sequence corresponding to the same finger.

[0028] In one specific embodiment, the starting sequence sequence and beat interval sequence corresponding to the same finger obtained in step S103 are first obtained. Each element in the starting sequence sequence is a joint identifier, and the position in the sequence represents the order in which each joint is started. That is, the joint in the first position is the joint that is started first by the finger, and the joint in the second position is the joint that is started second by the finger. Each element in the beat interval sequence is the movement beat interval value corresponding to each joint. The beat interval value is the time difference between two consecutive adjacent local extreme points in the bending angle signal of the same channel.

[0029] Using the joints corresponding to the same finger as indices, the numerical values ​​corresponding to each joint are extracted from the start sequence sequence and the beat interval sequence, where the joints are metacarpophalangeal joints, proximal interphalangeal joints, and distal interphalangeal joints.

[0030] For each joint, obtain the position number of the joint in the startup sequence. For example, if a joint is the first in the startup sequence, the startup sequence value corresponding to the joint is 1; if it is the second, the startup sequence value is 2. The startup sequence value is used as the first component of the tuple.

[0031] For the same joint, the corresponding beat interval value is obtained, and multiple preset beat level thresholds are obtained. The number of beat level thresholds is the total number of beat levels minus one. For example, when the total number of beat levels is 4, 3 beat level thresholds are preset. The beat interval value is compared with the multiple beat level thresholds one by one to determine the beat level to which the beat interval value belongs: when the beat interval value is less than or equal to the first level threshold, the beat interval belongs to the first level; when the beat interval value is greater than the first level threshold and less than or equal to the second level threshold, the beat interval belongs to the second level; when the beat interval value is greater than the second level threshold and less than or equal to the third level threshold, the beat interval belongs to the third level; when the beat interval value is greater than the third level threshold, the beat interval belongs to the fourth level. The level number of the beat level is used as the second component of the binary tuple.

[0032] The binary pairs corresponding to each joint are arranged according to the physiological order of each joint in the same finger, starting from the metacarpophalangeal joint, followed by the metacarpophalangeal joint, the proximal interphalangeal joint, and the distal interphalangeal joint. The arranged binary pair sequence is used as the motion pattern encoding sequence corresponding to the same finger. Each element in the motion pattern encoding sequence is a binary pair. The first component of the binary pair represents the activation order of the corresponding joint, and the second component represents the motion beat level of the corresponding joint.

[0033] For example, when the tuple corresponding to the metacarpophalangeal joint of a finger is (1,3), the tuple corresponding to the proximal interphalangeal joint is (2,2), and the tuple corresponding to the distal interphalangeal joint is (3,1), the motion pattern encoding sequence corresponding to that finger is [(1,3),(2,2),(3,1)]. This sequence indicates that the metacarpophalangeal joint of the finger starts first and has a slower beat, the proximal interphalangeal joint starts second and has a medium beat, and the distal interphalangeal joint starts last and has the fastest beat.

[0034] Step S105: Using the motion pattern encoding sequences of all fingers as the processing object, calculate the edit distance between any two finger motion pattern encoding sequences to obtain the inter-finger edit distance matrix.

[0035] In this step, for any two fingers, the first finger and the second finger, the first motion pattern encoding sequence corresponding to the first finger and the second motion pattern encoding sequence corresponding to the second finger are obtained; Using the first motion pattern encoding sequence as the source sequence and the second motion pattern encoding sequence as the target sequence, a dynamic programming matrix is ​​constructed. The number of rows in the dynamic programming matrix is ​​the length of the first motion pattern encoding sequence plus one, and the number of columns is the length of the second motion pattern encoding sequence plus one. The first row and first column of the dynamic programming matrix are initialized, and the first row and first column are filled with the cost of insertion operation and the cost of deletion operation. The cost of insertion operation and the cost of deletion operation are determined according to the positional relationship between the first finger and the second finger in the hand. The cost between adjacent fingers is less than the cost between non-adjacent fingers. Traverse the remaining cells of the dynamic programming matrix. For each cell, calculate the insertion cost of inserting the corresponding element in the second motion pattern encoding sequence from the cell above, the deletion cost of deleting the corresponding element in the first motion pattern encoding sequence from the cell to the left, and the replacement cost of replacing the corresponding element in the first motion pattern encoding sequence with the corresponding element in the second motion pattern encoding sequence from the cell to the top left. Take the minimum value of the three as the edit distance value of the cell. The edit distance value of the bottom right cell of the dynamic programming matrix is ​​used as the edit distance between the first finger and the second finger; By iterating through all finger pairs, we obtain the edit distance matrix between fingers.

[0036] In one specific embodiment, the motion pattern coding sequences corresponding to all fingers obtained in step S104 are first obtained. The number of motion pattern coding sequences is equal to the number of fingers. Each finger corresponds to one motion pattern coding sequence. The length of each motion pattern coding sequence is the same and is equal to the number of joints contained in the corresponding finger.

[0037] For any two fingers, the first and second fingers, obtain the first motion pattern encoding sequence corresponding to the first finger and the second motion pattern encoding sequence corresponding to the second finger. The first motion pattern encoding sequence contains multiple pairs of data, and each pair of data is arranged according to the physiological order of the joints. The second motion pattern encoding sequence also contains multiple pairs of data, and each pair of data is arranged according to the physiological order of the joints. The length of the first motion pattern encoding sequence is the same as the length of the second motion pattern encoding sequence.

[0038] Using the first motion pattern encoding sequence as the source sequence and the second motion pattern encoding sequence as the target sequence, a dynamic programming matrix is ​​constructed. The number of rows in the dynamic programming matrix is ​​the length of the first motion pattern encoding sequence plus one, and the number of columns is the length of the second motion pattern encoding sequence plus one. Each cell in the dynamic programming matrix corresponds to the edit distance between the prefix of the source sequence and the prefix of the target sequence.

[0039] The first row and first column of the dynamic programming matrix are initialized. For the first row, it is filled from left to right. The first cell of the first row has a value of zero, and the value of each subsequent cell is the value of the previous cell plus the cost of the insertion operation. The cost of the insertion operation is the cost required to insert the element at the corresponding position in the target sequence into the source sequence. For the first column, it is filled from top to bottom. The first cell of the first column has a value of zero, and the value of each subsequent cell is the value of the previous cell plus the cost of the deletion operation. The cost of the deletion operation is the cost required to delete the element at the corresponding position in the source sequence from the source sequence. The costs of the insertion operation and the deletion operation are determined based on the positional relationship between the first finger and the second finger in the hand: when the first finger and the second finger are adjacent fingers, the costs of the insertion operation and the deletion operation take a first preset value; when the first finger and the second finger are not adjacent fingers, the costs of the insertion operation and the deletion operation take a second preset value. The first preset value is less than the second preset value. Adjacent fingers are fingers that are adjacent in position in the physiological structure of the hand, and non-adjacent fingers are fingers that are not adjacent in position in the physiological structure of the hand.

[0040] The remaining cells of the dynamic programming matrix are traversed in the following order: starting from the first row and second column of the dynamic programming matrix, traversing row by row from left to right and from top to bottom until the bottom right cell is reached. For each cell, the insertion cost of inserting the corresponding element in the second motion pattern encoding sequence from the cell above, the deletion cost of deleting the corresponding element in the first motion pattern encoding sequence from the cell to the left, and the replacement cost of replacing the corresponding element in the first motion pattern encoding sequence with the corresponding element in the second motion pattern encoding sequence from the cell to the top left are calculated. The replacement cost is determined based on the difference between the corresponding element in the first motion pattern encoding sequence and the corresponding element in the second motion pattern encoding sequence. When the two corresponding elements are completely identical, the replacement cost is zero. When the second components of the two corresponding elements are identical, the replacement cost is a third preset value, which is less than the first preset value. When the first and second components of the two corresponding elements are not identical, the replacement cost is the sum of the cost of the insertion operation and the cost of the deletion operation. The minimum value among the insertion cost, the deletion cost, and the replacement cost is taken as the edit distance value of the cell.

[0041] The edit distance value of the lower right cell of the dynamic programming matrix is ​​used as the edit distance between the first finger and the second finger. The edit distance is the minimum editing operation cost required to transform the motion pattern encoding sequence of the first finger into the motion pattern encoding sequence of the second finger.

[0042] Traverse all finger pairs and perform the above edit distance calculation operation on each finger pair to obtain the edit distance value between all finger pairs. Construct an inter-finger edit distance matrix using all fingers as row and column indices and the edit distance value between each finger pair as matrix elements. The inter-finger edit distance matrix is ​​a symmetric matrix with the number of rows and columns equal to the total number of fingers. The diagonal elements of the matrix are zero, and the off-diagonal elements are the edit distance values ​​between the corresponding finger pairs.

[0043] Step S106: Extract the principal components of the edit distance matrix, use the variance contribution rate of the principal components as the dispersion of the hand movement pattern, and determine whether the dispersion of the hand movement pattern is less than a preset dispersion threshold.

[0044] In this step, the reciprocal of each element in the edit distance matrix is ​​taken to obtain the similarity matrix, and the diagonal elements of the similarity matrix are set to 1; The mean is removed from each column of the similarity matrix to obtain a centered similarity matrix; Calculate the covariance matrix of the centered similarity matrix, perform eigenvalue decomposition on the covariance matrix to obtain multiple eigenvalues ​​and corresponding eigenvectors, and arrange the multiple eigenvalues ​​in descending order; The eigenvalue with the largest value is taken as the first principal component eigenvalue, and the ratio between the first principal component eigenvalue and the plurality of eigenvalues ​​is taken as the variance contribution rate of the principal component. The variance contribution rate is taken as the dispersion of the hand movement pattern.

[0045] Furthermore, after determining whether the dispersion of the hand movement pattern is less than a preset dispersion threshold, when the dispersion of the hand movement pattern is not less than the preset dispersion threshold, the bending angle signal of each finger is mapped to the joint angle command of the corresponding finger of the dexterous hand, and output to the joint controller of the dexterous hand for execution.

[0046] In one specific embodiment, the finger-to-finger edit distance matrix obtained in step S105 is first acquired. The edit distance matrix is ​​a square matrix with the number of rows and columns equal to the total number of fingers. The diagonal elements of the matrix are zero, and the off-diagonal elements are the edit distance values ​​between corresponding finger pairs. The larger the edit distance value, the greater the difference between the motion pattern encoding sequences of the two fingers; the smaller the edit distance value, the smaller the difference between the motion pattern encoding sequences of the two fingers.

[0047] The reciprocal of each off-diagonal element in the edit distance matrix is ​​taken. The reciprocal operation makes the reciprocal smaller when the edit distance value is larger and larger when the edit distance value is smaller, thereby converting the difference measure into a similarity measure. All elements after taking the reciprocal are used as the corresponding position elements of the similarity matrix, and the diagonal elements of the similarity matrix are set to 1, indicating that the similarity between each finger and itself is at its maximum value.

[0048] For each column of the similarity matrix, a mean-removal process is performed. The mean-removal process involves calculating the arithmetic mean of all elements in the column, subtracting the arithmetic mean from each element in the column to obtain the centered column vector corresponding to the column, and traversing all columns to obtain the centered similarity matrix. The mean of each column of the centered similarity matrix is ​​zero.

[0049] Calculate the covariance matrix of the centered similarity matrix. The covariance matrix is ​​a square matrix with the number of rows and columns equal to the total number of fingers. The element in the i-th row and j-th column of the covariance matrix is ​​the covariance between the i-th column and the j-th column of the centered similarity matrix. Perform eigenvalue decomposition on the covariance matrix to obtain multiple eigenvalues ​​and corresponding eigenvectors. The number of eigenvalues ​​is equal to the total number of fingers, and the eigenvalues ​​are arranged in descending order.

[0050] The eigenvalue with the largest value is taken as the eigenvalue of the first principal component. The proportion of the eigenvalue of the first principal component to the sum of all eigenvalues ​​is calculated, and the proportion is taken as the variance contribution rate of the principal component. The variance contribution rate represents the proportion of the first principal component in all principal components, and the value ranges from 0 to 1. The variance contribution rate is taken as the dispersion of the hand movement pattern. The lower the dispersion of the hand movement pattern, the smaller the difference between the movement pattern encoding sequences of each finger, and the more consistent the movement patterns of each finger tend to be. The higher the dispersion of the hand movement pattern, the greater the difference between the movement pattern encoding sequences of each finger, and the more independent the movement patterns of each finger tend to be.

[0051] Obtain a preset dispersion threshold and compare the dispersion of the hand movement pattern with the preset dispersion threshold.

[0052] When the dispersion of the hand movement pattern is less than the preset dispersion threshold, the difference between the movement pattern encoding sequences of each finger is small, the hand is in a coordinated movement state, and the action pattern library matching mapping in step S107 is executed.

[0053] When the dispersion of the hand movement pattern is greater than or equal to the preset dispersion threshold, the difference between the movement pattern encoding sequences of each finger is large, and the hand is in an independent movement state. Perform the following operations: obtain the angle value of the bending angle signal of each joint of each finger at the current moment, obtain the angle range of each joint of the corresponding finger of the dexterous hand, map the angle value of the bending angle signal to the angle command of each joint of the corresponding finger of the dexterous hand based on the proportional relationship between the angle range of each joint and the angle value of the bending angle signal of each joint, and output the mapped angle command to the dexterous hand joint controller for execution.

[0054] Step S107: When the dispersion of the hand movement pattern is less than a preset dispersion threshold, the movement pattern encoding sequence of each finger is used as input, and the preset dexterous hand movement pattern library is used as the matching target. The movement pattern encoding sequence of each finger is matched with each movement pattern encoding template in the movement pattern library. The joint angle trajectory corresponding to the movement pattern encoding template with the highest matching degree is selected as the control command of the dexterous hand and output to the joint controllers of the dexterous hand for execution. In this step, a preset dexterity hand motion pattern library is obtained. The motion pattern library contains multiple motion pattern coding templates. Each motion pattern coding template is formed by splicing the motion pattern coding sequence of each finger in the finger order. Each motion pattern coding template is associated with a predefined joint angle trajectory. The joint angle trajectory contains the angle value sequence of each joint of the dexterity hand. For the motion pattern encoding sequence of each finger at the current moment, the motion pattern encoding sequence of each finger is concatenated in finger order to obtain the current hand encoding sequence; Calculate the sequence similarity between the current hand encoding sequence and each action pattern encoding template in the action pattern library. The sequence similarity is the length of the longest common subsequence between the current hand encoding sequence and the action pattern encoding template divided by the length of the current hand encoding sequence. The action pattern encoding template with the highest sequence similarity is selected as the matching template, and the joint angle trajectory associated with the matching template is used as the control command output of the dexterous hand.

[0055] In one specific embodiment, a preset dexterity hand motion pattern library is first obtained. The motion pattern library contains multiple motion pattern coding templates. Each motion pattern coding template is formed by splicing the motion pattern coding sequence of each finger in the finger order. The finger order is a pre-set fixed order. Each motion pattern coding template is associated with a predefined joint angle trajectory. The joint angle trajectory contains the angle value sequence of each joint of the dexterity hand over a period of time. Each element in the angle value sequence corresponds to the target angle value of each joint at a sampling time. Each motion pattern coding template corresponds to a dexterity hand motion pattern. The motion pattern includes at least a grasping mode, a holding mode, a release mode, and a gesture mode.

[0056] For the motion pattern encoding sequence of each finger at the current moment, the motion pattern encoding sequence of each finger is spliced ​​together according to the same finger order as the action pattern encoding template to obtain a complete sequence, which is used as the current hand encoding sequence. The length of the current hand encoding sequence is equal to the sum of the lengths of the motion pattern encoding sequences of each finger.

[0057] The sequence similarity between the current hand encoding sequence and each action pattern encoding template in the action pattern library is calculated. The sequence similarity is calculated as follows: the length of the longest common subsequence between the current hand encoding sequence and the action pattern encoding template is calculated, the length of the longest common subsequence is divided by the length of the current hand encoding sequence, and the quotient is taken as the sequence similarity. The longest common subsequence is the longest subsequence that appears in both the current hand encoding sequence and the action pattern encoding template and maintains their original order. The value of the sequence similarity ranges from 0 to 1, and the closer the value is to 1, the more similar the two sequences are.

[0058] Traverse all action pattern encoding templates in the action pattern library, calculate the sequence similarity between the current hand encoding sequence and each action pattern encoding template, and obtain the sequence similarity value corresponding to each template.

[0059] The action pattern encoding template corresponding to the sequence with the largest sequence similarity value is selected as the matching template. The joint angle trajectory associated with the matching template is used as the control command of the dexterous hand. The joint angle trajectory contains the angle value sequence of each joint within a preset time length. The angle value corresponding to the first sampling time in the angle value sequence is used as the target angle command of each joint at the current time. The angle values ​​corresponding to subsequent sampling times in the angle value sequence are used as the target angle commands of each joint at subsequent times.

[0060] The control commands are output to the joint controllers of each joint of the dexterous hand for execution, so that the dexterous hand performs movements according to the joint angle trajectory corresponding to the control commands.

[0061] In summary, the method of this application acquires the bending angle signals of each sensor channel of the data glove; detects the start time and movement beat interval of each channel; arranges all start times of the same finger into a start sequence and arranges all movement beat intervals into a beat interval sequence; combines the elements at corresponding positions of the two sequences into a tuple to obtain the finger's movement pattern encoding sequence; calculates the edit distance between any two finger movement pattern encoding sequences to obtain the finger edit distance matrix; extracts the principal component variance contribution rate of the edit distance matrix as the hand movement pattern dispersion; when the dispersion is less than a threshold, the finger movement pattern encoding sequence is matched with a preset dexterous hand movement pattern library, and the joint angle trajectory corresponding to the matching template is selected as the control command output; when the dispersion is not less than the threshold, each finger is mapped independently; and the start sequence and beat interval are used to encode the movement pattern to achieve adaptive mapping from the data glove to the dexterous hand.

[0062] Please see Figure 2 The diagram shows a structural block diagram of a data glove and dexterous hand pose mapping system according to this application.

[0063] like Figure 2 As shown, the data glove and dexterous hand pose mapping system 200 includes an acquisition module 210, a detection module 220, a processing module 230, a sorting module 240, a calculation module 250, a judgment module 260, and an output module 270.

[0064] The acquisition module 210 is configured to acquire the bending angle signals of each sensor channel of the data glove; the detection module 220 is configured to detect the moment when the angle value in the bending angle signal of each channel first exceeds a preset start threshold, as the start time corresponding to each channel, and to detect the time interval between two consecutive adjacent local extreme points in the bending angle signal of the same channel, as the movement beat interval corresponding to each channel; the processing module 230 is configured to take all start times corresponding to the same finger as the processing object, arrange all start times in chronological order to obtain the start sequence sequence corresponding to the same finger, and take all movement beat intervals corresponding to the same finger as the processing object, arrange all movement beat intervals in the corresponding joint order to obtain the beat interval sequence corresponding to the same finger; the sorting module 240 is configured to combine each element in the start sequence sequence with the element at the corresponding position in the beat interval sequence into a binary tuple, and sort all the binary tuples in order. The system obtains the motion pattern encoding sequence corresponding to the same finger; the calculation module 250 is configured to take the motion pattern encoding sequences of all fingers as the processing object, calculate the edit distance between the motion pattern encoding sequences of any two fingers, and obtain the edit distance matrix between fingers; the judgment module 260 is configured to extract the principal components of the edit distance matrix, use the variance contribution rate of the principal components as the hand motion pattern dispersion, and determine whether the hand motion pattern dispersion is less than a preset dispersion threshold; the output module 270 is configured to take the motion pattern encoding sequence of each finger as input, take the preset dexterous hand motion pattern library as the matching target, match the motion pattern encoding sequence of each finger with each motion pattern encoding template in the motion pattern library, select the joint angle trajectory corresponding to the motion pattern encoding template with the highest matching degree as the control command of the dexterous hand, and output it to the dexterous hand joint controller for execution.

[0065] It should be understood that Figure 2 The modules and references described in the document Figure 1 The steps described in the text correspond to those in the method described above. Therefore, the operations, features, and corresponding technical effects described above also apply to the method described in the text. Figure 2 The various modules in the document will not be described in detail here.

[0066] In other embodiments, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the program instructions are executed by a processor, the processor performs the data glove and dexterous hand pose mapping method in any of the above method embodiments. In one embodiment, the computer-readable storage medium of the present invention stores computer-executable instructions, which are configured as follows: Acquire the bending angle signals of each sensor channel of the data glove; The moment when the angle value in the bending angle signal of each channel first exceeds the preset start threshold is taken as the start time of each channel. The time interval between two consecutive adjacent local extreme points in the bending angle signal of the same channel is taken as the motion beat interval of each channel. Taking all the start times corresponding to the same finger as the processing object, the start times are arranged in chronological order to obtain the start sequence sequence corresponding to the same finger. Taking all the movement beat intervals corresponding to the same finger as the processing object, the movement beat intervals are arranged in the corresponding joint order to obtain the beat interval sequence corresponding to the same finger. Each element in the start sequence is combined with the corresponding element in the beat interval sequence to form a tuple. All tuples are arranged in order to obtain the motion pattern encoding sequence corresponding to the same finger. Using the motion pattern coding sequences of all fingers as the processing object, the edit distance between any two motion pattern coding sequences of fingers is calculated to obtain the inter-finger edit distance matrix; Extract the principal components of the edit distance matrix, use the variance contribution rate of the principal components as the dispersion of the hand movement pattern, and determine whether the dispersion of the hand movement pattern is less than a preset dispersion threshold. When the dispersion of the hand movement pattern is less than the preset dispersion threshold, the movement pattern encoding sequence of each finger is used as input, and the preset dexterous hand action pattern library is used as the matching target. The movement pattern encoding sequence of each finger is matched with each action pattern encoding template in the action pattern library. The joint angle trajectory corresponding to the action pattern encoding template with the highest matching degree is selected as the control command of the dexterous hand and output to the dexterous hand joint controller for execution.

[0067] Computer-readable storage media may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required for at least one function; the stored data area may store data created based on the use of the data glove and dexterous hand pose mapping system, etc. Furthermore, the computer-readable storage medium may include high-speed random access memory, and may also include memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the computer-readable storage medium may optionally include memory remotely configured relative to a processor, which can be connected to the data glove and dexterous hand pose mapping system via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0068] Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present invention, such as... Figure 3As shown, the device includes a processor 310 and a memory 320. The electronic device may also include an input device 330 and an output device 340. The processor 310, memory 320, input device 330, and output device 340 can be connected via a bus or other means. Figure 3 Taking a bus connection as an example, the memory 320 is the computer-readable storage medium described above. The processor 310 executes various server functions and data processing by running non-volatile software programs, instructions, and modules stored in the memory 320, thereby implementing the data glove and dexterity hand pose mapping method described in the above embodiment. The input device 330 can receive input digital or character information and generate key signal inputs related to user settings and function control of the data glove and dexterity hand pose mapping system. The output device 340 may include a display screen or other display device.

[0069] The aforementioned electronic device can execute the method provided in the embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the method provided in the embodiments of the present invention.

[0070] In one implementation, the above-described electronic device is applied to a data glove and dexterous hand pose mapping system for a client, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: Acquire the bending angle signals of each sensor channel of the data glove; The moment when the angle value in the bending angle signal of each channel first exceeds the preset start threshold is taken as the start time of each channel. The time interval between two consecutive adjacent local extreme points in the bending angle signal of the same channel is taken as the motion beat interval of each channel. Taking all the start times corresponding to the same finger as the processing object, the start times are arranged in chronological order to obtain the start sequence sequence corresponding to the same finger. Taking all the movement beat intervals corresponding to the same finger as the processing object, the movement beat intervals are arranged in the corresponding joint order to obtain the beat interval sequence corresponding to the same finger. Each element in the start sequence is combined with the corresponding element in the beat interval sequence to form a tuple. All tuples are arranged in order to obtain the motion pattern encoding sequence corresponding to the same finger. Using the motion pattern coding sequences of all fingers as the processing object, the edit distance between any two motion pattern coding sequences of fingers is calculated to obtain the inter-finger edit distance matrix; Extract the principal components of the edit distance matrix, use the variance contribution rate of the principal components as the dispersion of the hand movement pattern, and determine whether the dispersion of the hand movement pattern is less than a preset dispersion threshold. When the dispersion of the hand movement pattern is less than the preset dispersion threshold, the movement pattern encoding sequence of each finger is used as input, and the preset dexterous hand action pattern library is used as the matching target. The movement pattern encoding sequence of each finger is matched with each action pattern encoding template in the action pattern library. The joint angle trajectory corresponding to the action pattern encoding template with the highest matching degree is selected as the control command of the dexterous hand and output to the dexterous hand joint controller for execution.

[0071] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data glove to dexterous hand pose mapping method, characterized by, include: Acquire the bending angle signals of each sensor channel of the data glove; The moment when the angle value in the bending angle signal of each channel first exceeds the preset start threshold is taken as the start time of each channel. The time interval between two consecutive adjacent local extreme points in the bending angle signal of the same channel is taken as the motion beat interval of each channel. Taking all the start times corresponding to the same finger as the processing object, the start times are arranged in chronological order to obtain the start sequence sequence corresponding to the same finger. Taking all the movement beat intervals corresponding to the same finger as the processing object, the movement beat intervals are arranged in the corresponding joint order to obtain the beat interval sequence corresponding to the same finger. Each element in the start sequence is combined with the corresponding element in the beat interval sequence to form a tuple. All tuples are arranged in order to obtain the motion pattern encoding sequence corresponding to the same finger. Using the motion pattern coding sequences of all fingers as the processing object, the edit distance between any two motion pattern coding sequences of fingers is calculated to obtain the inter-finger edit distance matrix; Extract the principal components of the edit distance matrix, use the variance contribution rate of the principal components as the dispersion of the hand movement pattern, and determine whether the dispersion of the hand movement pattern is less than a preset dispersion threshold. When the dispersion of the hand movement pattern is less than the preset dispersion threshold, the movement pattern encoding sequence of each finger is used as input, and the preset dexterous hand action pattern library is used as the matching target. The movement pattern encoding sequence of each finger is matched with each action pattern encoding template in the action pattern library. The joint angle trajectory corresponding to the action pattern encoding template with the highest matching degree is selected as the control command of the dexterous hand and output to the dexterous hand joint controller for execution.

2. The method for mapping the pose of a data glove and a dexterous hand according to claim 1, characterized in that, The step of combining each element in the start sequence with the corresponding element in the beat interval sequence into a tuple, and arranging all the tuples in order to obtain the motion pattern encoding sequence corresponding to the same finger includes: Using the joints corresponding to the same finger as indices, extract the start sequence value corresponding to each joint from the start sequence sequence, and extract the beat interval value corresponding to each joint from the beat interval sequence; For each joint, the start sequence value corresponding to the joint is used as the first component of the tuple, and the beat interval value corresponding to the joint is quantized and used as the second component of the tuple to obtain the tuple corresponding to each joint. The quantization method is to compare the beat interval value with multiple preset beat level thresholds to determine the beat level to which the beat interval belongs, and use the level number of the beat level as the second component of the tuple. Arrange the binary pairs corresponding to each joint according to the physiological order of each joint in the same finger to obtain the motion pattern encoding sequence corresponding to the same finger.

3. The method for mapping the pose of a data glove and a dexterous hand according to claim 1, characterized in that, The step of processing the motion pattern coding sequences of all fingers and calculating the edit distance between any two finger motion pattern coding sequences to obtain the inter-finger edit distance matrix includes: For any two fingers, the first finger and the second finger, obtain the first motion pattern encoding sequence corresponding to the first finger and the second motion pattern encoding sequence corresponding to the second finger; Using the first motion pattern encoding sequence as the source sequence and the second motion pattern encoding sequence as the target sequence, a dynamic programming matrix is ​​constructed. The number of rows in the dynamic programming matrix is ​​the length of the first motion pattern encoding sequence plus one, and the number of columns is the length of the second motion pattern encoding sequence plus one. The first row and first column of the dynamic programming matrix are initialized, and the first row and first column are filled with the cost of insertion operation and the cost of deletion operation. The cost of insertion operation and the cost of deletion operation are determined according to the positional relationship between the first finger and the second finger in the hand. The cost between adjacent fingers is less than the cost between non-adjacent fingers. Traverse the remaining cells of the dynamic programming matrix. For each cell, calculate the insertion cost of inserting the corresponding element in the second motion pattern encoding sequence from the cell above, the deletion cost of deleting the corresponding element in the first motion pattern encoding sequence from the cell to the left, and the replacement cost of replacing the corresponding element in the first motion pattern encoding sequence with the corresponding element in the second motion pattern encoding sequence from the cell to the top left. Take the minimum value of the three as the edit distance value of the cell. The edit distance value of the bottom right cell of the dynamic programming matrix is ​​used as the edit distance between the first finger and the second finger; By iterating through all finger pairs, we obtain the edit distance matrix between fingers.

4. The method for mapping the pose of a data glove and a dexterous hand according to claim 1, characterized in that, The step of extracting the principal components of the edit distance matrix and using the variance contribution rate of the principal components as the dispersion of the hand movement pattern includes: Take the reciprocal of each element in the edit distance matrix to obtain the similarity matrix, and set the diagonal elements of the similarity matrix to 1; The mean is removed from each column of the similarity matrix to obtain a centered similarity matrix; Calculate the covariance matrix of the centered similarity matrix, perform eigenvalue decomposition on the covariance matrix to obtain multiple eigenvalues ​​and corresponding eigenvectors, and arrange the multiple eigenvalues ​​in descending order; The eigenvalue with the largest value is taken as the first principal component eigenvalue, and the ratio between the first principal component eigenvalue and the plurality of eigenvalues ​​is taken as the variance contribution rate of the principal component. The variance contribution rate is taken as the dispersion of the hand movement pattern.

5. The method for mapping the pose of a data glove and a dexterous hand according to claim 1, characterized in that, The process of taking the motion pattern encoding sequence of each finger as input, using a preset dexterity hand motion pattern library as the matching target, matching the motion pattern encoding sequence of each finger with each motion pattern encoding template in the motion pattern library, and selecting the joint angle trajectory corresponding to the motion pattern encoding template with the highest matching degree as the control command of the dexterity hand includes: Obtain a preset dexterity hand motion pattern library. The motion pattern library contains multiple motion pattern coding templates. Each motion pattern coding template is formed by splicing the motion pattern coding sequence of each finger in the order of the fingers. Each motion pattern coding template is associated with a predefined joint angle trajectory. The joint angle trajectory contains the angle value sequence of each joint of the dexterity hand. For the motion pattern encoding sequence of each finger at the current moment, the motion pattern encoding sequence of each finger is concatenated in finger order to obtain the current hand encoding sequence; Calculate the sequence similarity between the current hand encoding sequence and each action pattern encoding template in the action pattern library. The sequence similarity is the length of the longest common subsequence between the current hand encoding sequence and the action pattern encoding template divided by the length of the current hand encoding sequence. The action pattern encoding template with the highest sequence similarity is selected as the matching template, and the joint angle trajectory associated with the matching template is used as the control command output of the dexterous hand.

6. The method for mapping the pose of a data glove and a dexterous hand according to claim 1, characterized in that, After determining whether the dispersion of the hand movement pattern is less than a preset dispersion threshold, the method further includes: When the dispersion of the hand movement pattern is not less than the preset dispersion threshold, the bending angle signal of each finger is mapped to the joint angle command of the corresponding finger of the dexterous hand, and output to the joint controller of the dexterous hand for execution.

7. A data glove and dexterous hand pose mapping system, characterized in that, include: The acquisition module is configured to acquire the bending angle signals of each sensor channel of the data glove; The detection module is configured to detect the moment when the angle value in the bending angle signal of each channel first exceeds the preset start threshold, and use it as the start time of each channel; and to detect the time interval between two consecutive adjacent local extreme points in the bending angle signal of the same channel, and use it as the motion beat interval of each channel. The processing module is configured to take all the start times corresponding to the same finger as the processing object, arrange all the start times in chronological order to obtain the start sequence sequence corresponding to the same finger, and take all the movement beat intervals corresponding to the same finger as the processing object, arrange all the movement beat intervals in the corresponding joint order to obtain the beat interval sequence corresponding to the same finger. The sorting module is configured to combine each element in the start sequence with the corresponding element in the beat interval sequence into a tuple, and arrange all the tuples in order to obtain the motion pattern encoding sequence corresponding to the same finger. The calculation module is configured to take the motion pattern encoding sequences of all fingers as the processing object, calculate the edit distance between any two motion pattern encoding sequences of fingers, and obtain the inter-finger edit distance matrix. The judgment module is configured to extract the principal components of the edit distance matrix, use the variance contribution rate of the principal components as the dispersion of the hand movement pattern, and determine whether the dispersion of the hand movement pattern is less than a preset dispersion threshold. The output module is configured to, when the dispersion of the hand movement pattern is less than a preset dispersion threshold, take the movement pattern encoding sequence of each finger as input, take the preset dexterous hand action pattern library as the matching target, match the movement pattern encoding sequence of each finger with each action pattern encoding template in the action pattern library, select the joint angle trajectory corresponding to the action pattern encoding template with the highest matching degree as the control command of the dexterous hand, and output it to the dexterous hand joint controller for execution.

8. An electronic device, characterized in that, include: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the method described in any one of claims 1 to 6.

Images