Manipulator remote operation method and related equipment

Abstract

The embodiment of the present application discloses a method for remote operation of a manipulator and related equipment for improving the accuracy of remote operation of a manipulator. The method of the embodiment of the present application includes: establishing initial heat map information according to a human hand image; inputting the initial heat map information into a pre-trained global estimation network, and the global estimation network outputs global heat map information and two-dimensional coordinates of each key point in each key area of ​​the human hand in the human hand image; splitting the global heat map information and the two-dimensional coordinates of each key point to obtain local heat map information and two-dimensional coordinates of local key points corresponding to each branch network in the pre-trained local estimation network; inputting the local heat map information and the two-dimensional coordinates of the local key points into the corresponding branch network in the local estimation network, and obtaining the three-dimensional coordinates of each key point in the key area corresponding to the human hand output by each branch network; operating the manipulator according to the three-dimensional coordinates of each key point in each key area of ​​the human hand.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of manipulators, and in particular to a manipulator remote operation method and related equipment. Background Art

[0002] Robot teleoperation technology refers to the technology of mapping human hand movements to the robot through visual data or tactile data of human hand movements to achieve remote operation of the robot. Remote operation can be achieved by using visual data or tactile data.

[0003] The existing remote operation method of a robot arm usually involves obtaining a human hand image, inputting the initial heat map information generated based on the human hand image into a global estimation network, and obtaining the estimated three-dimensional coordinates of each key point of the robot arm. Finally, the robot arm is operated according to the estimated three-dimensional coordinates of each key point of the robot arm.

[0004] However, the existing technical solutions only consider the global biometric features mapped from the human hand to the robot. As a result, after the estimated three-dimensional coordinates of the key points of the robot are used to operate the robot, the hand posture of the robot and the hand posture of the human hand in the human hand image can only have good consistency in general trends, while good mapping cannot be achieved for more detailed hand postures such as each finger and palm, which in turn reduces the accuracy of remote operation of the robot. Summary of the invention

[0005] The embodiments of the present application provide a manipulator remote control method and related equipment for improving the accuracy of manipulator remote control operation.

[0006] A first aspect of an embodiment of the present application provides a manipulator remote operation method, comprising:

[0007] Establish initial heat map information based on the human hand image;

[0008] Inputting the initial heat map information into a pre-trained global estimation network, and the global estimation network outputs the global heat map information and the two-dimensional coordinates of each key point in each key area of ​​the human hand in the human hand image;

[0009] Splitting the global heat map information and the two-dimensional coordinates of each key point to obtain local heat map information and two-dimensional coordinates of local key points corresponding to each branch network in a pre-trained local estimation network, wherein each branch network is used to estimate the three-dimensional coordinates of each key point in each key area of ​​the hand;

[0010] Input the local heat map information and the two-dimensional coordinates of the local key points into the corresponding branch network in the local estimation network, and obtain the three-dimensional coordinates of each key point in the key area corresponding to the human hand output by each branch network;

[0011] The robot is operated according to the three-dimensional coordinates of each key point in each key area of ​​the human hand.

[0012] In a specific implementation, the method further includes:

[0013] Selecting a human hand image from a training database as a training human hand image, iteratively training a global estimation network to be trained and a local estimation network to be trained, and obtaining training three-dimensional coordinates of each key point in each key area corresponding to the training human hand in the training human hand image;

[0014] Calculate, according to the training three-dimensional coordinates and / or the actual three-dimensional coordinates corresponding to the hand image in the training database, at least one global biometric feature and at least one local biometric feature when the training hand is in an actual gesture posture and the training hand is in a training gesture posture;

[0015] Determining each of the global biometric features as a global loss function, and determining each of the local biometric features as a local loss function;

[0016] Calculating a weighted loss function according to preset weights corresponding to each objective loss function, and correcting network parameters of the global estimation network to be trained according to the weighted loss function, and correcting network parameters of each branch network in the local estimation network to be trained according to the weighted loss function;

[0017] If the weighted loss function satisfies a preset convergence condition, the training is terminated and the global estimation network to be trained is determined to be the global estimation network, and the local estimation network to be trained is determined to be the local estimation network.

[0018] In a specific implementation, the global biometric feature includes global bone length loss and / or global bone angle loss, and the local biometric feature includes at least one of local bone length loss, local bone angle loss, lateral movement of the thumb, and palm direction angle.

[0019] In a specific implementation, the global biometric feature includes a global bone length loss, and the calculating of at least one global biometric feature of the training human hand in an actual gesture posture and in a training gesture posture includes:

[0020] Calculating the actual bone length between each adjacent key point of the training hand, and calculating the estimated bone length between each adjacent key point of the training hand;

[0021] The global bone length loss of the training hand in the actual gesture posture and the training gesture posture is calculated according to the following formula:

[0022]

[0023] Among them, L gb is the global bone length loss of the training hand, b i,j is the actual bone length between the i-th key point and the j-th key point of the training hand, The estimated bone length between the i-th key point and the j-th key point of the training hand, i,j∈C indicates that the i-th key point and the j-th key point are any adjacent key points of the training hand.

[0024] In a specific implementation, the local biometric feature includes a local bone angle loss, and the calculating of at least one local biometric feature of the training hand in an actual gesture posture and in a training gesture posture includes:

[0025] The actual angle of each bone length of the training hand is calculated according to the following formula:

[0026]

[0027] Among them, α i,j is the actual angle between the i-th key point and the j-th key point of the training hand, b i,j is the actual bone length between the i-th key point and the j-th key point of the training hand, b j,j+1 is the actual bone length between the jth key point and the j+1th key point of the training hand;

[0028] The estimated angle of each bone length of the training hand is calculated according to the following formula:

[0029]

[0030] in, is the estimated angle between the bone lengths corresponding to the i-th key point and the j-th key point of the training hand, is the estimated bone length between the i-th key point and the j-th key point of the training hand, is the estimated bone length between the jth key point and the j+1th key point of the training hand;

[0031] The local bone angle loss of the training hand in the actual gesture posture and the training gesture posture is calculated according to the following formula:

[0032]

[0033] Among them, L lba is the local bone angle loss of the training hand, h refers to the hth finger of the training hand, i,j∈C h It indicates that the i-th key point and the j-th key point are any adjacent key points corresponding to the h-th finger of the training hand.

[0034] In a specific implementation, the local biometric feature includes lateral movement of the thumb, and the calculating of at least one local biometric feature of the training hand in an actual gesture posture and in a training gesture posture includes:

[0035] The lateral movement of the thumb of the training hand is calculated according to the following formula:

[0036]

[0037] Among them, α 1,5 is the lateral movement of the thumb of the training hand, b 0,1 is the actual bone length corresponding to the wrist key point of the training person's hand and the nearest key point corresponding to the thumb of the training person's hand, b 0,5 It is the actual bone length corresponding to the farthest key point between the wrist key point of the training person's hand and the thumb of the training person's hand.

[0038] In a specific implementation, the local biometric feature includes a palm direction angle, and the calculating of at least one local biometric feature of the training hand in an actual gesture posture and in a training gesture posture includes:

[0039] The palm direction angle of the training hand is calculated according to the following formula:

[0040]

[0041] Among them, L pa is the palm direction angle of the training person's hand, b p1 Indicates the pth 1 The actual 3D coordinates of the key points, b p2 Indicates the pth 2 The actual 3D coordinates of the key points.

[0042] In a specific implementation, the global biometric feature includes a global bone angle loss, and the calculating of at least one global biometric feature of the training human hand in an actual gesture posture and in a training gesture posture includes:

[0043] The actual global bone angle of each finger of the training hand is calculated according to the following formula:

[0044]

[0045] Among them, k is the nearest key point corresponding to any finger of the training hand, l is the farthest key point corresponding to any finger, α k is the actual global bone angle of any finger, b 0,kis the actual bone length between the nearest key point corresponding to any finger and the wrist key point, b k,l is the actual bone length between the nearest key point corresponding to any finger and the farthest key point corresponding to any finger;

[0046] The estimated global bone angle of each finger of the training hand is calculated according to the following formula:

[0047]

[0048] Wherein, k is the nearest key point corresponding to any finger of the training hand, l is the farthest key point corresponding to any finger, is the estimated global bone angle of any finger, is the estimated bone length between the nearest key point corresponding to any finger and the wrist key point, is the estimated bone length between the nearest key point corresponding to any finger and the farthest key point corresponding to any finger;

[0049] The global bone angle loss of the training hand in the actual gesture posture and the training gesture posture is calculated according to the following formula:

[0050]

[0051] Among them, L gba is the global bone angle loss of the training hand, h refers to the hth finger of the training hand, k is the nearest key point corresponding to any finger of the training hand, α k is the actual global bone angle of any finger, is the estimated global bone angle of any finger.

[0052] A second aspect of the present application provides a manipulator remote control device, including:

[0053] An establishing unit, used for establishing initial heat map information according to a human hand image;

[0054] An output unit, inputting the initial heat map information into a pre-trained global estimation network, and the global estimation network outputs the global heat map information and the two-dimensional coordinates of each key point in each key area of ​​the human hand in the human hand image;

[0055] A splitting unit, used to split the global heat map information and the two-dimensional coordinates of each key point, to obtain local heat map information and two-dimensional coordinates of local key points corresponding to each branch network in a pre-trained local estimation network, wherein each branch network is used to estimate the three-dimensional coordinates of each key point in each key area of ​​the hand;

[0056] The output unit is further used to input the local heat map information and the two-dimensional coordinates of the local key points into the corresponding branch network in the local estimation network, and obtain the three-dimensional coordinates of each key point in the key area corresponding to the human hand output by each branch network;

[0057] An operating unit is used to operate the manipulator according to the three-dimensional coordinates of each key point in each key area of ​​the human hand.

[0058] In a specific implementation, the manipulator teleoperation device further includes: a training unit, a calculation unit, a determination unit, and a correction unit;

[0059] The training unit is used to select a human hand image from a training database as a training human hand image, iteratively train a global estimation network to be trained and a local estimation network to be trained, and obtain the training three-dimensional coordinates of each key point in each key area corresponding to the training human hand in the training human hand image;

[0060] The calculation unit is used to calculate at least one global biometric feature and at least one local biometric feature when the training hand is in an actual gesture posture and the training hand is in a training gesture posture according to the training three-dimensional coordinates and / or the actual three-dimensional coordinates corresponding to the hand image in the training database;

[0061] The determining unit is further configured to determine each of the global biometric features as a global loss function, and to determine each of the local biometric features as a local loss function;

[0062] The correction unit is used to calculate the weighted loss function according to the preset weights corresponding to the target loss functions, and correct the network parameters of the global estimation network to be trained according to the weighted loss function, and correct the network parameters of each branch network in the local estimation network to be trained according to the weighted loss function;

[0063] The determining unit is further configured to terminate the training and determine that the global estimation network to be trained is the global estimation network, and the local estimation network to be trained is the local estimation network, if the weighted loss function satisfies a preset convergence condition.

[0064] In a specific implementation, the global biometric feature includes global bone length loss and / or global bone angle loss, and the local biometric feature includes at least one of local bone length loss, local bone angle loss, lateral movement of the thumb, and palm direction angle.

[0065] In a specific implementation, the global biometric feature includes a global bone length loss, and the calculation unit is specifically used to calculate the actual bone length between each adjacent key point of the training human hand, and calculate the estimated bone length between each adjacent key point of the training human hand;

[0066] The global bone length loss of the training hand in the actual gesture posture and the training gesture posture is calculated according to the following formula:

[0067]

[0068] Among them, L gb is the global bone length loss of the training hand, b i,j is the actual bone length between the i-th key point and the j-th key point of the training hand, The estimated bone length between the i-th key point and the j-th key point of the training hand, i,j∈C indicates that the i-th key point and the j-th key point are any adjacent key points of the training hand.

[0069] In a specific implementation, the local biological feature includes a local bone angle loss, and the calculation unit is specifically used to calculate the actual angle of each bone length of the training human hand according to the following formula:

[0070]

[0071] Among them, α i,j is the actual angle between the i-th key point and the j-th key point of the training hand, b i,j is the actual bone length between the i-th key point and the j-th key point of the training hand, b j,j+1 is the actual bone length between the jth key point and the j+1th key point of the training hand;

[0072] The estimated angle of each bone length of the training hand is calculated according to the following formula:

[0073]

[0074] in, is the estimated angle between the bone lengths corresponding to the i-th key point and the j-th key point of the training hand, is the estimated bone length between the i-th key point and the j-th key point of the training hand, is the estimated bone length between the jth key point and the j+1th key point of the training hand;

[0075] The local bone angle loss of the training hand in the actual gesture posture and the training gesture posture is calculated according to the following formula:

[0076]

[0077] Among them, L lba is the local bone angle loss of the training hand, h refers to the hth finger of the training hand, i,j∈C h It indicates that the i-th key point and the j-th key point are any adjacent key points corresponding to the h-th finger of the training hand.

[0078] In a specific implementation, the local biometric feature includes a lateral movement of the thumb, and the calculation unit is specifically configured to calculate the lateral movement of the thumb of the training hand according to the following formula:

[0079]

[0080] Among them, α 1,5 is the lateral movement of the thumb of the training hand, b 0,1 is the actual bone length corresponding to the wrist key point of the training person's hand and the nearest key point corresponding to the thumb of the training person's hand, b 0,5 It is the actual bone length corresponding to the farthest key point between the wrist key point of the training person's hand and the thumb of the training person's hand.

[0081] In a specific implementation, the local biometric feature includes a palm direction angle, and the calculation unit is specifically configured to calculate the palm direction angle of the training hand according to the following formula:

[0082]

[0083] Among them, L pa is the palm direction angle of the training person's hand, b p1 Indicates the pth 1 The actual 3D coordinates of the key points, b p2 Indicates the pth 2 The actual 3D coordinates of the key points.

[0084] In a specific implementation, the global biometric feature includes a global bone angle loss, and the calculation unit is specifically used to calculate the actual global bone angle of each finger of the training hand according to the following formula:

[0085]

[0086] Among them, k is the nearest key point corresponding to any finger of the training hand, l is the farthest key point corresponding to any finger, α k is the actual global bone angle of any finger, b 0,k is the actual bone length between the nearest key point corresponding to any finger and the wrist key point, bk,l is the actual bone length between the nearest key point corresponding to any finger and the farthest key point corresponding to any finger;

[0087] The estimated global bone angle of each finger of the training hand is calculated according to the following formula:

[0088]

[0089] Wherein, k is the nearest key point corresponding to any finger of the training hand, l is the farthest key point corresponding to any finger, is the estimated global bone angle of any finger, is the estimated bone length between the nearest key point corresponding to any finger and the wrist key point, is the estimated bone length between the nearest key point corresponding to any finger and the farthest key point corresponding to any finger;

[0090] The global bone angle loss of the training hand in the actual gesture posture and the training gesture posture is calculated according to the following formula:

[0091]

[0092] Among them, L gba is the global bone angle loss of the training hand, h refers to the hth finger of the training hand, k is the nearest key point corresponding to any finger of the training hand, α k is the actual global bone angle of any finger, is the estimated global bone angle of any finger.

[0093] A third aspect of the present application provides a manipulator remote control device, including:

[0094] CPU, memory and input / output interface;

[0095] The memory is a short-term storage memory or a persistent storage memory;

[0096] The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method described in the first aspect.

[0097] A fourth aspect of the embodiments of the present application provides a computer program product comprising instructions, and when the computer program product is run on a computer, the computer is caused to execute the method described in the first aspect.

[0098] A fifth aspect of an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores instructions, and when the instructions are executed on a computer, the computer executes the method described in the first aspect.

[0099] It can be seen from the above technical scheme that the embodiments of the present application have the following advantages: the generated three-dimensional coordinates of each key point in each key area of ​​the human hand are estimated by the global estimation network and the local estimation network, while taking into account the inconsistency between the actual hand posture of the human hand and the hand posture in the human hand image globally and locally in each key area, which makes up for the deficiency of the existing technical scheme that does not consider the local consistency between the actual hand posture of the human hand and the hand posture in the human hand image, thereby improving the accuracy of remote operation of the manipulator. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] Figure 1a An example diagram of key points of a human hand disclosed in an embodiment of the present application;

[0101] Figure 1b An exemplary diagram of the key points of the manipulator disclosed in the embodiment of the present application;

[0102] Figure 2 A schematic diagram of a flow chart of a manipulator remote operation method disclosed in an embodiment of the present application;

[0103] Figure 3 Another schematic diagram of the process of remote operation of a manipulator disclosed in an embodiment of the present application;

[0104] Figure 4 A schematic diagram of the structure of the manipulator remote control device disclosed in the embodiment of the present application;

[0105] Figure 5 This is another structural schematic diagram of the manipulator remote operation device disclosed in the embodiment of the present application. DETAILED DESCRIPTION

[0106] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0107] The embodiments of the present application provide a manipulator remote control method and related equipment for improving the accuracy of manipulator remote control operation.

[0108] In order to better illustrate the technical solution of the embodiment of the present application, the mapping relationship between the key points of the human hand and the key points of the manipulator in the embodiment of the present application is introduced below. Before the manipulator is remotely operated, the mapping relationship between the key points of the human hand, the key points of the manipulator, and each key point of the human hand and at least one key point of the manipulator is pre-set. Specifically, in order to better achieve a good global and local mapping between the posture of the human hand and the posture of the manipulator, the mapping relationship between each key point of the human hand and the corresponding key point of the manipulator is unique, that is, each key point of the human hand only exists in the mapping relationship between one key point of the manipulator.

[0109] In some specific implementations, the key point settings of the human hand, the key point settings of the robot, and the mapping relationship between the key points of the human hand and the key points of the robot in the embodiment of the present application can be as follows: Figure 1a as well as Figure 1b As shown. Among them, Figure 1a The human hand has 21 key points preset. Specifically, each finger of the human hand includes four human hand key points (1-4, 5-8, 9-12, 13-16 and 17-19) and the human wrist corresponds to a human hand key point 0. The corresponding robot hand also has 21 key points. Figure 1b The key points of the manipulator at the corresponding position of the manipulator correspond to the key points of the human hand at the corresponding position of the human hand ( Figure 1a and Figure 1b The key points with the same label are the corresponding two key points).

[0110] Based on the above description of the mapping method between the key points of the human hand and the key points of the robot in the embodiment of the present application, please refer to Figure 2 The present application provides a method for remote operation of a manipulator, comprising the following steps:

[0111] 201. Establish initial heat map information based on the hand image.

[0112] The human hand image is captured by an image acquisition device, and then the heat map information of the human hand in the captured human hand image is obtained as the initial heat map information. The image acquisition device includes but is not limited to a monocular acquisition device or a multi-camera acquisition device, which is not limited here.

[0113] In some specific implementations, an RGB image of a human hand can be collected by a monocular camera, and then the RGB image of the human hand is input into a stacked hourglass network, and finally the human image information of the RGB image of the human hand is obtained as the initial heat map information.

[0114] 202. Input the initial heat map information into a pre-trained global estimation network, and the global estimation network outputs the global heat map information and the two-dimensional coordinates of each key point in each key area of ​​the human hand in the hand image.

[0115] After obtaining the initial heat map information, the initial heat map information can be input into a pre-trained global estimation network, and the global estimation network can output the global heat map information and the two-dimensional coordinates of each key point in each key area of ​​the hand. The global estimation network can be a residual network or a convolutional neural network.

[0116] 203. Split the global heat map information and the two-dimensional coordinates of each key point to obtain the local heat map information and the two-dimensional coordinates of the local key points corresponding to each branch network in the pre-trained local estimation network, wherein each branch network is used to estimate the three-dimensional coordinates of each key point in each key area of ​​the manipulator.

[0117] Because the local estimation network contains multiple branch networks, each of which is used to estimate the three-dimensional coordinates of each key point in each key area of ​​the manipulator, that is, each branch network is used to estimate the three-dimensional coordinates of each key point in a key area of ​​the manipulator, and each branch network estimates a different key area. Therefore, it is necessary to divide the global heat map information and the two-dimensional coordinates of each key point according to the key areas of the human hand, that is, to find the local heat map information corresponding to each key area of ​​the human hand and the two-dimensional coordinates of each corresponding local key point (referring to the key points corresponding to each key area) from the global heat map information and the two-dimensional coordinates of each key point, so as to provide each branch network with the determination of the estimated three-dimensional coordinates of the corresponding key area. It can be understood that the two-dimensional coordinates of the local key points corresponding to each key area are: the two-dimensional coordinates of each key point in each key area of ​​the human hand output by the global estimation network in step 202, and the two-dimensional coordinates of each key point corresponding to the key area.

[0118] Specifically, the key areas of a human hand include but are not limited to: the thumb, index finger, middle finger, ring finger, little finger and / or palm, but the combination of the key areas of a human hand should be able to obtain a complete human hand.

[0119] 204. Input the local heat map information and the two-dimensional coordinates of some key points into the corresponding branch network in the local estimation network, and obtain the three-dimensional coordinates of each key point in the key area corresponding to the hand output by each branch network.

[0120] After completing the division of the local heat map information of each key area and the two-dimensional coordinates of the local key points in step 203, in this step, the local heat map information corresponding to each key area and the two-dimensional coordinates of the local key points are input into the corresponding branch network for estimating the three-dimensional coordinates of the key points in each key area, and the three-dimensional coordinates of each key point in each key area of ​​the hand are obtained from the output of each branch network.

[0121] 205. Operate the robot according to the three-dimensional coordinates of each key point in each key area of ​​the human hand.

[0122] In actual applications, the movement of each key point of each key area of ​​the manipulator can be controlled by a register. Based on the mapping relationship between each key point of the manipulator and each key point in each key area of ​​the human hand and the three-dimensional coordinates of each key point in each key area of ​​the human hand, the bending angle information of each key area and each key point of the manipulator can be calculated. Based on the bending angle of each key point, the register movement of each key point can be controlled to realize remote operation of the manipulator, ensuring that the hand posture of the manipulator maintains a high degree of consistency with the hand posture of the human hand image obtained in step 201 both locally and globally.

[0123] The above only describes the control of the robot's hand posture based on a human hand image to move to the same state as the hand posture in the human hand image. In practical applications, if multiple continuous hand postures of a human hand, that is, human hand movements (for example, the action of raising a hand includes multiple hand postures), can be collected when the human hand performs multiple consecutive frames of human hand images, and the three-dimensional coordinates of each key point of the robot can be output for each frame of the human hand image. The robot is operated to continuously execute the three-dimensional coordinates of each key point of the robot corresponding to each frame of the human hand image, so that the mapping of human hand movements to robot movements can be realized, and the remote operation of the robot can be realized.

[0124] In the embodiment of the present application, the generated three-dimensional coordinates of each key point in each key area of ​​the human hand are estimated by a global estimation network and a local estimation network, while taking into account the consistency between the hand posture of the human hand and the hand posture in the human hand image globally and locally in each key area, thereby making up for the deficiency of the prior art solution that does not consider the local consistency between the hand posture of the manipulator and the hand posture in the human hand image, thereby improving the accuracy of remote operation of the manipulator.

[0125] It can be known that before using the global estimation network and the local estimation network to estimate the 3D coordinates of each key point in each key area of ​​the hand image based on the hand image, the global estimation network and the local estimation network to be trained need to be trained. Figure 3 In some specific implementations, the training of the global estimation network to be trained and the local estimation network to be trained may include the following steps:

[0126] 301. Select a human hand image from a training database as a training human hand image, iteratively train the global estimation network to be trained and the local estimation network to be trained, and obtain the training three-dimensional coordinates of each key point in each key area corresponding to the training hand in the training human hand image.

[0127] A training database is preset before training, and each hand image in the training database has corresponding actual three-dimensional coordinates. Specifically, a hand image can be selected from the training database as a training hand image to train the network to be trained (including the global estimation network to be trained and the local estimation network to be trained) each time. After each training hand image is input into the network to be trained, the training three-dimensional coordinates of each key point in each key area of ​​the training hand output by the network to be trained can be obtained. Among them, the process of inputting the training hand image to obtain the training three-dimensional coordinates is similar to the process of inputting the hand image to obtain the three-dimensional coordinates in the aforementioned and later embodiments, and will not be repeated here.

[0128] 302. Calculate at least one global biometric feature and at least one local biometric feature of the training hand in the actual gesture posture and the training gesture posture according to the actual three-dimensional coordinates and the training three-dimensional coordinates corresponding to the hand image in the training database.

[0129] Before step 302, the embodiment of the present application presets a calculation formula for each global biometric feature and a calculation formula for each local biometric feature. According to the corresponding biometric calculation formula, the global biometric difference (i.e., global biometric feature) or local biometric difference (i.e., local biometric feature) between the training hand in the actual gesture posture and the training hand posture can be calculated. It should be noted that the training hand in the actual gesture posture means that the three-dimensional coordinates of each key point of the training hand are the actual three-dimensional coordinates corresponding to each key point in the training hand image; similarly. The training hand in the training gesture posture means that the three-dimensional coordinates of each key point of the training hand are the training three-dimensional coordinates of each key point in the training hand image calculated in step 301.

[0130] Among them, global biometric features include but are not limited to: global bone length loss and / or global bone angle loss; local biometric features include but are not limited to local bone length loss (i.e. local bone length loss in each key area of ​​the trained hand), local bone angle loss (i.e. local bone angle loss in each key area of ​​the trained hand), lateral movement of the thumb and / or palm direction angle.

[0131] 303. Determine each global biometric feature as a global loss function, and determine each local biometric feature as a local loss function.

[0132] Each global biometric feature is determined as a global loss function, and each local biometric feature is determined as a local loss function of the branch network that estimates the key area corresponding to the local biometric feature. That is, the lateral movement of the thumb as a local loss function should be used to correct the network parameters of the branch network that estimates the three-dimensional coordinates of the thumb (a key area) of the training person's hand.

[0133] 304. Calculate a weighted loss function according to preset weights corresponding to each target loss function, and correct the network parameters of the global estimation network to be trained according to the weighted loss function, and correct the network parameters of each branch network in the local estimation network to be trained according to the weighted loss function.

[0134] If the weighted loss function does not meet the preset convergence condition (the preset weighted loss function convergence threshold), the network parameters are corrected. Specifically, the network parameters of the global estimation network to be trained and the network parameters of each branch network in the local estimation network are corrected based on the weighted loss function. Among them, the local biometric feature describes the difference between the actual gesture posture and the training gesture posture in a certain key area. Therefore, each local biometric feature corresponds to a key area of ​​the human hand, and the local biometric feature should be used as a correction to estimate the network parameters of the branch network corresponding to the local biometric feature. That is, the local loss function corresponding to each local biometric feature is used to correct the branch network that estimates the key area corresponding to the local biometric feature.

[0135] 305. If the weighted loss function satisfies a preset convergence condition, the training is terminated and the global estimation network to be trained is determined to be the global estimation network, and the local estimation network to be trained is determined to be the local estimation network.

[0136] The preset convergence condition of this step is similar to the convergence condition described in step 304, and will not be repeated here. Here, the weighted loss function satisfies the preset convergence condition, which may mean that the value of the weighted loss function is less than or equal to its corresponding preset weighted loss function convergence threshold, which is not specifically limited here.

[0137] If the preset convergence conditions are met, the global estimation network (network parameters) to be trained when the convergence conditions are met is determined as the global estimation network (network parameters) that can be used (i.e., training is completed), and the local estimation network (network parameters) to be trained when the convergence conditions are met is determined as the local estimation network (network parameters) that can be used (i.e., training is completed).

[0138] In the embodiments of the present application, a specific implementation method for training a global estimation network and a local estimation network is provided, thereby improving the feasibility of the solution.

[0139] Furthermore, each global biometric feature and each local biometric feature may be calculated according to the following formula.

[0140] 1. Global bone length loss L gb

[0141]

[0142] Among them, L gb To train the global bone length loss of the human hand, b i,jTo train the actual bone length between the i-th key point and the j-th key point of the human hand, The estimated bone length between the i-th key point and the j-th key point of the training hand, i,j∈C means that the i-th key point and the j-th key point are any adjacent key points of the training hand. Figure 1a-1b The 21 key points of the hand in the relevant embodiment are Figure 1a Two key points that can be connected by any existing line segment are adjacent key points. Figure 1a Each finger has four pairs of adjacent key points. Taking the thumb as an example, the four pairs of adjacent key points corresponding to the thumb are key point 0 and key point 1, key point 1 and key point 2, key point 2 and key point 3, and key point 3 and key point 4.

[0143] 2. Local bone length loss L lb

[0144]

[0145] The calculation formula for the actual bone length between the i-th key point and the j-th key point of the training hand in the above and the following formulas is: i,j =φ i -φ j , where φ i is the actual 3D coordinate of the i-th key point of the training hand, φ j is the actual three-dimensional coordinate of the jth key point of the training hand; the calculation formula for the estimated bone length between the i-th key point and the j-th key point of the training hand in the above and the following formulas is: in To train the 3D coordinates of the i-th key point of the hand, is the training 3D coordinate of the jth key point of the training hand.

[0146] 3. Global bone angle L gba

[0147]

[0148] in, To train the actual global bone angle of the finger corresponding to the kth key point of the human hand, The estimated global bone angle of the finger corresponding to the kth key point of the training hand.

[0149] Among them, k can be the value of the closest key point corresponding to any finger of the training hand, and l can be the value of the farthest key point corresponding to any finger of the training hand. k and l correspond to the same finger of the training hand. Specifically, the closest key point refers to the key point that can reach the wrist key point through the least key points along the bone direction of each finger among the four key points corresponding to each finger. Figure 1a The closest key points corresponding to the thumb, index finger, middle finger, ring finger and little finger are 1, 5, 9, 13 and 17 respectively; the closest key point refers to the key point that passes through the most key points along the bone direction of each finger among the four key points corresponding to each finger (in Figure 1a The line segments of each finger are drawn in the middle) to reach a key point of the wrist key point. Figure 1a The nearest key points corresponding to the thumb, index finger, middle finger, ring finger and little finger are 4, 8, 12, 16 and 20 respectively.

[0150] 4. Local bone angle loss L lba

[0151]

[0152] in, To train the actual angle of bone length formed by connecting the i-th key point and the j-th key point of the human hand, is the estimated angle of the bone length formed by connecting the i-th key point and the j-th key point of the training hand. h represents the five fingers of a hand. Specifically, the thumb is 1, the index finger is 2, the middle finger is 3, the ring finger is 4, and the little finger is 5.

[0153] 5. Horizontal movement of thumb α 1,5

[0154]

[0155] Combined with the 4 local bone angles, specifically, Figure 1a To train the horizontal movement of the thumb of a person's hand, α 1,5 It can be seen that the horizontal movement of the thumb is α 1,5 Because Figure 1a The farthest key point corresponding to the middle thumb is 5, the closest key point is 1, and the wrist key point is 0. The numbers in the formula are only for better explanation of the calculation method of the lateral movement of the thumb. As long as it is consistent with the physical meaning of each physical quantity, the subscript of the physical quantity of the lateral movement of the thumb depends on the farthest key point and the closest key point corresponding to the thumb.

[0156] 6. Palm direction angle L pa

[0157]

[0158] Considering the problem of hand gesture self-occlusion, the palm direction angle is added as part of the loss function to avoid losing the key points of the hand. Among the six key points of the palm, four groups of triangles are selected as the palm structure K = {(0,1,5), (0,1,9), (0,5,13), (0,9,17)}, that is, the palm is arbitrarily occluded and at least 0 and one other palm key point are retained. The positions of other key points can be obtained through the related paths of the four triangles. Use the vector outer product to restrict the direction of the palm.

[0159] It can be known that the same letters in the formulas in the embodiments of the present application have the same physical meanings, which will not be repeated here.

[0160] In some specific implementations, the dexterous manipulator has 6 degrees of freedom, and the little finger bending angle, ring finger bending angle, middle finger bending angle, index finger bending angle, thumb bending angle and thumb rotation angle are controlled by registers with a value range of 0-1000. According to the relevant embodiments of the global bone angle loss, is the bending angle of the five fingers. When k = 1, the value range is 0-90 degrees, and the rest is 0-180 degrees. Therefore, the bending angles of the ring finger, middle finger, index finger and little finger of the manipulator are mapped as

[0161]

[0162] Consider the lateral movement angle of the thumb in Explanation 8 Its range is 0-90 degrees, which corresponds to the rotation angle of the thumb of the robot. Therefore, the mapping between the bending angle and rotation angle of the thumb of the robot is as follows:

[0163]

[0164] See also Figure 4 The present application also provides a robot teleoperation device, comprising:

[0165] An establishing unit 401 is used to establish initial heat map information according to a human hand image;

[0166] Output unit 402, inputs the initial heat map information into a pre-trained global estimation network, and the global estimation network outputs the global heat map information and the two-dimensional coordinates of each key point in each key area of ​​the human hand in the human hand image;

[0167] A splitting unit 403 is used to split the global heat map information and the two-dimensional coordinates of each key point to obtain the local heat map information and the two-dimensional coordinates of the local key points corresponding to each branch network in the pre-trained local estimation network, wherein each branch network is used to estimate the three-dimensional coordinates of each key point in each key area of ​​the human hand;

[0168] The output unit 402 is further used to input the local heat map information and the two-dimensional coordinates of the local key points into the corresponding branch network in the local estimation network, and obtain the three-dimensional coordinates of each key point in the key area corresponding to the hand output by each branch network;

[0169] The operating unit 404 is used to operate the robot arm according to the three-dimensional coordinates of each key point in each key area of ​​the human hand.

[0170] In a specific implementation, the manipulator teleoperation device further includes: a training unit, a calculation unit, a determination unit, and a correction unit;

[0171] A training unit is used to select a human hand image from a training database as a training human hand image, iteratively train a global estimation network to be trained and a local estimation network to be trained, and obtain the training three-dimensional coordinates of each key point in each key area corresponding to the training human hand in the training human hand image;

[0172] A calculation unit, configured to calculate at least one global biometric feature and at least one local biometric feature when the training hand is in an actual gesture posture and the training hand is in a training gesture posture according to the training three-dimensional coordinates and / or the actual three-dimensional coordinates corresponding to the hand image in the training database;

[0173] The determination unit is further used to determine each global biometric feature as a global loss function and to determine each local biometric feature as a local loss function;

[0174] A correction unit, used to calculate a weighted loss function according to preset weights corresponding to each target loss function, and to correct the network parameters of the global estimation network to be trained according to the weighted loss function, and to correct the network parameters of each branch network in the local estimation network to be trained according to the weighted loss function;

[0175] The determination unit is also used to terminate the training and determine that the global estimation network to be trained is the global estimation network, and the local estimation network to be trained is the local estimation network if the weighted loss function satisfies a preset convergence condition.

[0176] In a specific implementation, the global biometric feature includes global bone length loss and / or global bone angle loss, and the local biometric feature includes at least one of local bone length loss, local bone angle loss, lateral movement of the thumb, and palm direction angle.

[0177] In a specific implementation, the global biometric feature includes a global bone length loss, a calculation unit specifically configured to calculate an actual bone length between each adjacent key point of a training person's hand, and calculate an estimated bone length between each adjacent key point of a training person's hand;

[0178] The global bone length loss of the training hand in the actual gesture posture and the training gesture posture is calculated according to the following formula:

[0179]

[0180] Among them, L gb To train the global bone length loss of the human hand, b i,j To train the actual bone length between the i-th key point and the j-th key point of the human hand, The estimated bone length between the i-th key point and the j-th key point of the training hand, i,j∈C means that the i-th key point and the j-th key point are any adjacent key points of the training hand.

[0181] In a specific implementation, the local biological feature includes a local bone angle loss, and the calculation unit is specifically used to calculate the actual angle of each bone length of the training human hand according to the following formula:

[0182]

[0183] Among them, α i,j is the actual angle between the i-th key point and the j-th key point of the training hand corresponding to the bone length, b i,j is the actual bone length between the i-th key point and the j-th key point of the training hand, b j,j+1 The actual bone length between the jth key point and the j+1th key point of the training hand;

[0184] The estimated angle of each bone length of the training hand is calculated according to the following formula:

[0185]

[0186] in, To train the estimated angle between the bone lengths of the i-th key point and the j-th key point of the human hand, To train the estimated bone length between the i-th key point and the j-th key point of the human hand, To train the estimated bone length between the jth key point and the j+1th key point of the human hand;

[0187] The local bone angle loss of the training hand in the actual gesture posture and the training gesture posture is calculated according to the following formula:

[0188]

[0189] Among them, Llba To train the local bone angle loss of the human hand, h refers to the hth finger of the trained hand, i,j∈C h It means that the i-th key point and the j-th key point are any adjacent key points corresponding to the h-th finger of the training hand.

[0190] In a specific implementation, the local biometric feature includes the lateral movement of the thumb, and the calculation unit is specifically configured to calculate the lateral movement of the thumb of the training hand according to the following formula:

[0191]

[0192] Among them, α 1,5 To train the lateral movement of the thumb of the human hand, b 0,1 is the actual bone length between the wrist key point of the training hand and the nearest key point corresponding to the thumb of the training hand, b 0,5 It is the actual bone length corresponding to the farthest key point corresponding to the wrist key point of the training person's hand and the thumb of the training person's hand.

[0193] In a specific implementation, the local biometric feature includes a palm direction angle, and the calculation unit is specifically used to calculate the palm direction angle of the training hand according to the following formula:

[0194]

[0195] Among them, L pa is the palm direction angle of the training hand, b p1 Indicates the first p in the hands of the trainer 1 The actual 3D coordinates of the key points, b p2 Indicates the training person's first 2 The actual 3D coordinates of the key points.

[0196] In a specific implementation, the global biometric feature includes a global bone angle loss, and the calculation unit is specifically used to calculate the actual global bone angle of each finger of the training hand according to the following formula:

[0197]

[0198] Among them, k is the nearest key point corresponding to any finger of the training hand, l is the farthest key point corresponding to any finger, and α k is the actual global bone angle of any finger, b 0,k is the actual bone length between the nearest key point corresponding to any finger and the wrist key point, b k,l is the actual bone length between the nearest key point corresponding to any finger and the farthest key point corresponding to any finger;

[0199] The estimated global bone angle for each finger of the training hand is calculated according to the following formula:

[0200]

[0201] Among them, k is the closest key point corresponding to any finger of the training hand, l is the farthest key point corresponding to any finger, is the estimated global bone angle of any finger, is the estimated bone length between the nearest keypoint corresponding to any finger and the wrist keypoint, is the estimated bone length between the nearest key point corresponding to any finger and the farthest key point corresponding to any finger;

[0202] The global bone angle loss of the training hand in the actual gesture posture and the training gesture posture is calculated according to the following formula:

[0203]

[0204] Among them, L gba is the global bone angle loss of the training hand, h refers to the hth finger of the training hand, k is the nearest key point corresponding to any finger of the training hand, α k is the actual global bone angle of any finger, is the estimated global bone angle of any finger.

[0205] Figure 5 It is a schematic diagram of the structure of a manipulator teleoperation device provided in an embodiment of the present application. The manipulator teleoperation device 500 may include one or more central processing units (CPU) 501 and a memory 505. The memory 505 stores one or more application programs or data.

[0206] The memory 505 may be a volatile storage or a persistent storage. The program stored in the memory 505 may include one or more modules, each of which may include a series of instruction operations in the manipulator teleoperation device. Furthermore, the central processor 501 may be configured to communicate with the memory 505 and execute a series of instruction operations in the memory 505 on the manipulator teleoperation device 500.

[0207] The manipulator teleoperation device 500 may also include one or more power supplies 502, one or more wired or wireless network interfaces 503, one or more input and output interfaces 504, and / or one or more operating systems, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0208] The CPU 501 can execute the aforementioned Figures 1a to 4 The operations performed by the manipulator teleoperation device in the illustrated embodiment will not be described in detail here.

[0209] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0210] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0211] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0212] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0213] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

[0214] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the robot remote operation method as described above.

Claims

1. A method for remote operation of a manipulator, It is characterized in that include: Establish initial heat map information based on the human hand image; Inputting the initial heat map information into a pre-trained global estimation network, and the global estimation network outputs the global heat map information and the two-dimensional coordinates of each key point in each key area of ​​the human hand in the human hand image; Splitting the global heat map information and the two-dimensional coordinates of each key point to obtain local heat map information and two-dimensional coordinates of local key points corresponding to each branch network in a pre-trained local estimation network, wherein each branch network is used to estimate the three-dimensional coordinates of each key point in each key area of ​​the hand; Input the local heat map information and the two-dimensional coordinates of the local key points into the corresponding branch network in the local estimation network, and obtain the three-dimensional coordinates of each key point in the key area corresponding to the human hand output by each branch network; The robot is operated according to the three-dimensional coordinates of each key point in each key area of ​​the human hand.

2. The method according to claim 1, It is characterized in that The method further comprises: Selecting a human hand image from a training database as a training human hand image, iteratively training a global estimation network to be trained and a local estimation network to be trained, and obtaining training three-dimensional coordinates of each key point in each key area corresponding to the training human hand in the training human hand image; Calculate, according to the training three-dimensional coordinates and / or the actual three-dimensional coordinates corresponding to the hand image in the training database, at least one global biometric feature and at least one local biometric feature when the training hand is in an actual gesture posture and the training hand is in a training gesture posture; Determining each of the global biometric features as a global loss function, and determining each of the local biometric features as a local loss function; Calculating a weighted loss function according to preset weights corresponding to each objective loss function, and correcting network parameters of the global estimation network to be trained according to the weighted loss function, and correcting network parameters of each branch network in the local estimation network to be trained according to the weighted loss function; If the weighted loss function satisfies a preset convergence condition, the training is terminated and the global estimation network to be trained is determined to be the global estimation network, and the local estimation network to be trained is determined to be the local estimation network.

3. The method according to claim 2, It is characterized in that The global biometric feature includes global bone length loss and / or global bone angle loss, and the local biometric feature includes at least one of local bone length loss, local bone angle loss, lateral movement of the thumb, and palm direction angle.

4. The method according to claim 2, It is characterized in that The global biometric feature includes a global bone length loss, and the calculating of at least one global biometric feature of the training human hand in an actual gesture posture and in a training gesture posture includes: Calculating the actual bone length between each adjacent key point of the training hand, and calculating the estimated bone length between each adjacent key point of the training hand; The global bone length loss of the training hand in the actual gesture posture and the training gesture posture is calculated according to the following formula: Among them, L gb is the global bone length loss of the training hand, b i,j is the actual bone length between the i-th key point and the j-th key point of the training hand, The estimated bone length between the i-th key point and the j-th key point of the training hand, i,j∈C indicates that the i-th key point and the j-th key point are any adjacent key points of the training hand.

5. The method according to claim 2, It is characterized in that The local biometric feature includes a local bone angle loss, and the calculating of at least one local biometric feature of the training hand in an actual gesture posture and in a training gesture posture includes: The actual angle of each bone length of the training hand is calculated according to the following formula: Among them, α i,j is the actual angle between the i-th key point and the j-th key point of the training hand, b i,j is the actual bone length between the i-th key point and the j-th key point of the training hand, b j,bj+1 is the actual bone length between the jth key point and the j+1th key point of the training hand; The estimated angle of each bone length of the training hand is calculated according to the following formula: in, is the estimated angle between the bone lengths corresponding to the i-th key point and the j-th key point of the training hand, is the estimated bone length between the i-th key point and the j-th key point of the training hand, is the estimated bone length between the jth key point and the j+1th key point of the training hand; The local bone angle loss of the training hand in the actual gesture posture and the training gesture posture is calculated according to the following formula: Among them, L lba is the local bone angle loss of the training hand, h refers to the hth finger of the training hand, i,j∈C h It indicates that the i-th key point and the j-th key point are any adjacent key points corresponding to the h-th finger of the training hand.

6. The method according to claim 2, It is characterized in that The local biometric feature includes a lateral movement of the thumb, and the calculating of at least one local biometric feature of the training hand in an actual gesture posture and in a training gesture posture includes: The lateral movement of the thumb of the training hand is calculated according to the following formula: Among them, α 1,5 is the lateral movement of the thumb of the training hand, b 0,1 is the actual bone length corresponding to the wrist key point of the training person's hand and the nearest key point corresponding to the thumb of the training person's hand, b 0,5 It is the actual bone length corresponding to the farthest key point between the wrist key point of the training person's hand and the thumb of the training person's hand.

7. The method according to claim 2, It is characterized in that The local biometric feature includes a palm direction angle, and the calculating of at least one local biometric feature of the training hand in an actual gesture posture and in a training gesture posture includes: The palm direction angle of the training hand is calculated according to the following formula: Among them, L pa is the palm direction angle of the training person's hand, b p1 Indicates the pth 1 The actual 3D coordinates of the key points, b p2 Indicates the pth 2 The actual 3D coordinates of the key points.

8. The method according to claim 2, It is characterized in that The global biometric feature includes a global bone angle loss, and the calculating of at least one global biometric feature of the training human hand in an actual gesture posture and in a training gesture posture includes: The actual global bone angle of each finger of the training hand is calculated according to the following formula: Among them, k is the nearest key point corresponding to any finger of the training hand, l is the farthest key point corresponding to any finger, α k is the actual global bone angle of any finger, b 0,k is the actual bone length between the nearest key point corresponding to any finger and the wrist key point, b k,l is the actual bone length between the nearest key point corresponding to any finger and the farthest key point corresponding to any finger; The estimated global bone angle of each finger of the training hand is calculated according to the following formula: Wherein, k is the nearest key point corresponding to any finger of the training hand, l is the farthest key point corresponding to any finger, is the estimated global bone angle of any finger, is the estimated bone length between the nearest key point corresponding to any finger and the wrist key point, is the estimated bone length between the nearest key point corresponding to any finger and the farthest key point corresponding to any finger; The global bone angle loss of the training hand in the actual gesture posture and the training gesture posture is calculated according to the following formula: Among them, L gba is the global bone angle loss of the training hand, h refers to the hth finger of the training hand, k is the nearest key point corresponding to any finger of the training hand, α k is the actual global bone angle of any finger, is the estimated global bone angle of any finger.

9. A manipulator remote control device, It is characterized in that include: An establishing unit, used for establishing initial heat map information according to a human hand image; An output unit, inputting the initial heat map information into a pre-trained global estimation network, and the global estimation network outputs the global heat map information and the two-dimensional coordinates of each key point in each key area of ​​the human hand in the human hand image; A splitting unit, used to split the global heat map information and the two-dimensional coordinates of each key point, to obtain local heat map information and two-dimensional coordinates of local key points corresponding to each branch network in a pre-trained local estimation network, wherein each branch network is used to estimate the three-dimensional coordinates of each key point in each key area of ​​the hand; The output unit is further used to input the local heat map information and the two-dimensional coordinates of the local key points into the corresponding branch network in the local estimation network, and obtain the three-dimensional coordinates of each key point in the key area corresponding to the human hand output by each branch network; An operating unit is used to operate the manipulator according to the three-dimensional coordinates of each key point in each key area of ​​the human hand.

10. A computer storage medium, It is characterized in that The computer storage medium stores instructions, and when the instructions are executed on a computer, the computer is caused to perform the method according to any one of claims 1 to 8.

Images