Humanoid robot intelligent grabbing control system and method, electronic device and storage medium

Abstract

The application discloses a humanoid robot intelligent grabbing control system and method, and relates to the technical field of humanoid robots.The intelligent grabbing control system comprises an electromyographic sensing control device and a humanoid robot device, wherein the electromyographic sensing control device communicates with the humanoid robot device; the electromyographic sensing control device comprises an electromyographic signal sensing module, a gesture posture acquisition module, a motion capture module, a grabbing control module and an AR module; and the humanoid robot device comprises a mechanical hand, a camera mechanism assembly, a mechanical arm, a camera mechanism movement driving module, a target object three-dimensional coordinate and posture calculation module, a target object posture recognition module, a first position calculation module, a second position calculation module, a mechanical hand position verification module and a mechanical hand posture acquisition module.The humanoid robot intelligent grabbing control system and method, electronic equipment and storage medium provided by the application can remotely control the action of a humanoid robot, and improve the accuracy and work efficiency of control.

Description

Technical Field

[0001] This invention belongs to the field of intelligent control technology, and relates to a humanoid robot, and more particularly to an intelligent grasping control system, method, electronic device and storage medium for a humanoid robot. Background Technology

[0002] With the development of technology, humanoid robot technology has advanced rapidly in recent years, and humanoid robot products have been applied in many industries. Among them, some industries require remote operation of robot manipulators. The existing control method usually involves placing a camera next to the robot and then remotely controlling the robot by viewing the camera.

[0003] However, due to the shooting angle, remote operators often cannot determine the exact location of the target object in three-dimensional space based on the camera, which can easily lead to operational errors and lower operational efficiency.

[0004] In view of this, there is an urgent need to design a new intelligent control system for humanoid robots in order to overcome at least some of the aforementioned defects in existing humanoid robot control systems. Summary of the Invention

[0005] This invention provides a humanoid robot intelligent grasping control system, method, electronic device, and storage medium, which can remotely control the humanoid robot's movements, improving control accuracy and work efficiency.

[0006] To solve the above-mentioned technical problems, according to one aspect of the present invention, the following technical solution is adopted:

[0007] A humanoid robot intelligent grasping control system, the intelligent grasping control system comprising: an electromyography (EMG) sensing control device and a humanoid robot device, wherein the EMG sensing control device communicates with the humanoid robot device;

[0008] The electromyography-sensing control device includes:

[0009] The motion capture module is used to capture gesture data;

[0010] The grasping control module is used to control the grasping action of the humanoid robot device based on the gesture action data captured by the motion capture module.

[0011] The AR module is used to display the target object in front of a set part of the human body through stereoscopic projection, thereby simulating the relative positional relationship between the target object and the robotic arm.

[0012] The humanoid robot device includes:

[0013] A robotic arm, used to perform grasping actions;

[0014] A camera frame assembly for acquiring image data of a defined area; the camera frame assembly includes at least two camera frames, which include a first camera frame and a second camera frame.

[0015] A robotic arm is used to position the camera assembly for movement along the robotic arm; the robotic arm includes at least two tracks, including a first track and a second track, wherein the centers of the first track and the second track are not on the same plane; the first camera is positioned on the first track, and the second camera is positioned on the second track;

[0016] A camera mechanism movement drive module is used to drive the robotic arm to move, thereby driving the first camera mechanism and the second camera mechanism to move;

[0017] The target object 3D coordinate and attitude calculation module is used to control the camera mechanism movement drive module to drive the robotic arm to move, thereby driving the camera mechanism assembly to move; each camera mechanism of the camera mechanism assembly acquires image data from at least two different positions during the movement; the target object 3D coordinate and attitude calculation module calculates the target object's 3D coordinates and attitude in a set 3D coordinate system based on the image data acquired by each camera mechanism.

[0018] The robot arm posture acquisition module is used to acquire the robot arm's posture data based on the status of the sensor components and the robot arm's drive mechanism.

[0019] The electromyography (EMG) sensing control device further includes an AR module; the AR module projects the target object's projection based on the acquired three-dimensional coordinates and posture of the target object, the three-dimensional coordinates and posture of the robotic arm, and the three-dimensional coordinates of the human body parts, such that the positional relationship between the target object projection and the set human body parts corresponds to the relative positional relationship between the target object and the robotic arm, and the size relationship between the generated stereoscopic projection of the target object and the set human body parts corresponds to the size relationship between the set human body parts and the robotic arm; or, the AR module generates images of the target object and the set human body parts based on the acquired three-dimensional coordinates and posture of the target object, the three-dimensional coordinates and posture of the robotic arm, and the three-dimensional coordinates of the human body parts, and displays the generated images on a set display module, such that the positional relationship between the target object image and the image of the set human body parts corresponds to the relative positional relationship between the target object and the robotic arm, and the size relationship between the generated stereoscopic image of the target object and the image of the set human body parts corresponds to the size relationship between the set human body parts and the robotic arm.

[0020] In one embodiment of the present invention, the target object 3D coordinate and attitude calculation module acquires image data captured by a set camera mechanism under different 3D coordinates to form an image data set, and simultaneously acquires the 3D coordinates and attitude of the camera mechanism at the time of capture; the image data set is compared with image data sets in a comparison database, the image data sets in the comparison database including image data captured by the camera mechanism under different 3D coordinates, and the 3D coordinates and attitude of the camera mechanism at the time of capture of each image, and also including the 3D coordinates and attitude of the target object; when comparing the image data sets, the image data set with the closest similarity is selected, and the 3D coordinates and attitude of the target object corresponding to the image data value are acquired as the 3D coordinates and attitude of the target object.

[0021] As one embodiment of the present invention, the intelligent grasping control system further includes a target object three-dimensional coordinate and posture mathematical model construction module, which is used to construct the target object three-dimensional coordinate and posture mathematical model.

[0022] The target object 3D coordinate and posture mathematical model construction module preprocesses the data in the training dataset, extracts features from the preprocessed data, and extracts key features. Then, it combines, statistically analyzes, and groups the extracted key features, and standardizes and normalizes the key features to form a key feature combination. The key feature combination includes image data captured by the camera mechanism, the 3D coordinates and posture of the camera mechanism when capturing each image data, and the 3D coordinates and posture of the target object being photographed. Based on the formed key feature combination, a mathematical model of the target object's 3D coordinates and posture is constructed.

[0023] As one embodiment of the present invention, the AR module includes at least one 3D holographic projection device, each 3D holographic projection device being used to project a 3D projection of a target object in a set area.

[0024] As one embodiment of the present invention, the robotic hand includes a first control circuit, a robotic hand body and several fingers, and several pressure sensors and several temperature sensors are distributed on the surface of the robotic hand body and several fingers.

[0025] Each pressure sensor is used to sense the pressure data received in the set area, and each temperature sensor is used to sense the temperature data in the set area; the data sensed by each pressure sensor and each temperature sensor are sent to the first control circuit.

[0026] The electromyography (EMG) sensing control device further includes a second control circuit and a display module. The display module is used to display the pressure data sensed by the set pressure sensor and / or the temperature data sensed by the set temperature sensor, thereby obtaining the force between the robot and the target object and the temperature state of the target object, so as to facilitate the adjustment of the control signals sent to the robot.

[0027] According to another aspect of the present invention, the following technical solution is adopted: a humanoid robot intelligent grasping control method, the intelligent grasping control method comprising:

[0028] A camera module assembly acquires image data of a designated area; the camera module assembly comprises at least two camera modules, including a first camera module and a second camera module; the camera module assembly is mounted on a robotic arm, which can drive the camera module assembly to move;

[0029] The camera mechanism movement drive module drives the robotic arm to move, thereby driving the first camera mechanism and the second camera mechanism to move.

[0030] The target object 3D coordinate and attitude calculation module controls the camera mechanism movement drive module to drive the robotic arm to move, thereby driving the camera mechanism assembly to move; each camera mechanism of the camera mechanism assembly acquires image data from at least two different positions during the movement; the target object 3D coordinate and attitude calculation module calculates the target object's 3D coordinates and attitude in a set 3D coordinate system based on the image data acquired by each camera mechanism.

[0031] The robot arm posture acquisition module acquires the robot arm's posture data based on the status of the sensors and the robot arm's drive mechanism.

[0032] The AR module displays a target object in front of a designated part of the human body using stereoscopic projection, thereby simulating the relative positional relationship between the target object and the robotic arm. The AR module projects the target object's projection based on the acquired 3D coordinates and posture of the target object, the robotic arm, and the human body part. The positional relationship between the target object projection and the designated human body part corresponds to the relative positional relationship between the target object and the robotic arm, and the size relationship between the generated stereoscopic projection of the target object and the designated human body part corresponds to the size relationship between the designated human body part and the robotic arm. Alternatively, the AR module generates images of the target object and the designated human body part based on the acquired 3D coordinates and posture of the target object, the robotic arm, and the human body part, and displays these images on a designated display module. The positional relationship between the target object image and the image of the designated human body part corresponds to the relative positional relationship between the target object and the robotic arm, and the size relationship between the generated stereoscopic image of the target object and the image of the designated human body part corresponds to the size relationship between the designated human body part and the robotic arm.

[0033] The motion capture module captures gesture data;

[0034] The grasping control module controls the grasping action of the humanoid robot device based on the gesture data captured by the motion capture module;

[0035] The robotic arm performs the grasping action according to the control signal generated by the grasping control module.

[0036] In one embodiment of the present invention, the target object 3D coordinate and attitude calculation module acquires image data captured by a set camera mechanism under different 3D coordinates to form an image data set, and simultaneously acquires the 3D coordinates and attitude of the camera mechanism at the time of capture; the image data set is compared with image data sets in a comparison database, the image data sets in the comparison database including image data captured by the camera mechanism under different 3D coordinates, and the 3D coordinates and attitude of the camera mechanism at the time of capture of each image, and also including the 3D coordinates and attitude of the target object; when comparing the image data sets, the image data set with the closest similarity is selected, and the 3D coordinates and attitude of the target object corresponding to the image data value are acquired as the 3D coordinates and attitude of the target object.

[0037] As one embodiment of the present invention, the intelligent grasping control method further includes a step of constructing a mathematical model of the three-dimensional coordinates and attitude of the target object: the mathematical model construction module of the three-dimensional coordinates and attitude of the target object constructs a mathematical model of the three-dimensional coordinates and attitude of the target object;

[0038] The steps for constructing the mathematical model of the target object's three-dimensional coordinates and posture include: preprocessing the data in the training dataset; extracting features from the preprocessed data and setting key features; combining, statistically analyzing, and grouping the extracted key features; standardizing and normalizing the key features to form a key feature combination; the key feature combination includes image data captured by the camera mechanism, the three-dimensional coordinates and posture of the camera mechanism when capturing each image data, and the three-dimensional coordinates and posture of the target object being photographed; and constructing a mathematical model of the target object's three-dimensional coordinates and posture based on the key feature combination.

[0039] As one embodiment of the present invention, the AR module includes at least one 3D holographic projection device, each of which projects a 3D projection of the target object in a set area.

[0040] As one embodiment of the present invention, the robotic hand includes a first control circuit, a robotic hand body and several fingers, and several pressure sensors and several temperature sensors are distributed on the surface of the robotic hand body and several fingers.

[0041] Each pressure sensor senses the pressure data received in the set area, and each temperature sensor senses the temperature data in the set area; the data sensed by each pressure sensor and each temperature sensor are sent to the first control circuit.

[0042] The electromyography (EMG) sensing control device further includes a second control circuit and a display module. The display module displays pressure data sensed by a set pressure sensor and / or temperature data sensed by a set temperature sensor, thereby obtaining the force between the robotic arm and the target object and the temperature state of the target object. Then, the control signal sent to the robotic arm is adjusted according to the force between the robotic arm and the target object and the temperature state of the target object.

[0043] According to another aspect of the present invention, the following technical solution is adopted: an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above method.

[0044] According to another aspect of the present invention, the following technical solution is adopted: a storage medium storing computer program instructions thereon, which, when executed by a processor, implement the steps of the above-described method.

[0045] The beneficial effects of this invention are as follows: The humanoid robot intelligent grasping control system, method, electronic device and storage medium proposed in this invention can remotely control the humanoid robot's movements, thereby improving the accuracy of control and work efficiency. Attached Figure Description

[0046] Figure 1This is a schematic diagram of the composition of a humanoid robot intelligent grasping control system in one embodiment of the present invention.

[0047] Figure 2 This is a flowchart of a humanoid robot intelligent grasping control method in one embodiment of the present invention.

[0048] Figure 3 This is a schematic diagram illustrating a usage scenario of the humanoid robot intelligent grasping control system in one embodiment of the present invention.

[0049] Figure 4 This is a schematic diagram of the composition of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0050] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0051] To further understand the present invention, preferred embodiments of the present invention are described below in conjunction with examples. However, it should be understood that these descriptions are only for further illustrating the features and advantages of the present invention, and not for limiting the scope of the claims of the present invention.

[0052] The description in this section pertains to only a few typical embodiments, and the present invention is not limited to the scope of the embodiments described. Substitution of identical or similar prior art methods with some technical features in the embodiments is also within the scope of the description and protection of this invention.

[0053] The steps described in the various embodiments in the specification are for illustrative purposes only, and the implementation of this application is not limited by the order of the steps.

[0054] The term "connection" in the specification includes both direct and indirect connections, such as connections made through active devices, passive devices, or electrical conduction media; it may also include connections made by other active or passive devices that are known to those skilled in the art and can achieve the same or similar functional purpose, such as connections made through circuits or components such as switches or follower circuits.

[0055] This invention discloses an intelligent grasping control system for a humanoid robot. Figure 1 This is a schematic diagram illustrating the composition of a humanoid robot intelligent grasping control system according to an embodiment of the present invention. Figure 3 This is a schematic diagram illustrating a usage scenario of the humanoid robot intelligent grasping control system according to an embodiment of the present invention; please refer to [link / reference]. Figure 1 , Figure 3 The intelligent grasping control system includes: an electromyography (EMG) sensing control device 1 and a humanoid robot device 2, wherein the EMG sensing control device 1 communicates with the humanoid robot device 2.

[0056] The electromyography control device 1 includes: a motion capture module 103, a grasping control module 104, and an AR module 105.

[0057] The motion capture module 103 is used to capture gesture data. In one embodiment, the motion capture module 103 can be implemented using an existing motion capture system; it can capture human movements through video capture, image recognition, and sensor technology.

[0058] The grasping control module 104 is used to control the grasping action of the humanoid robot device based on the gesture action data captured by the motion capture module 103. Of course, the control action can also include other actions, such as moving the hand, opening the hand, etc.; or each action required in the grasping action can be included as part of the grasping action.

[0059] The AR module 105 is used to generate a target object at a set distance from a set part of the human body based on the three-dimensional coordinates and posture of the target object and the three-dimensional coordinates and posture of the manipulator of the humanoid robot device, so as to facilitate the human body to adjust its position and posture before grasping. In one embodiment, the AR module includes at least one 3D holographic projection device, and each 3D holographic projection device projects a 3D projection of the target object in a set area. Alternatively, the AR module 105 generates images of the target object and the set part of the human body based on the acquired three-dimensional coordinates and posture of the target object, the three-dimensional coordinates and posture of the manipulator, and the three-dimensional coordinates of the human body parts, and displays the generated images on the set display module 106, such that the positional relationship between the target object image and the set part image of the human body corresponds to the relative positional relationship between the target object and the manipulator, and the size relationship between the generated stereoscopic image of the target object and the set part image of the human body corresponds to the size relationship between the set part of the human body and the manipulator.

[0060] In one embodiment of the present invention, the electromyography (EMG) sensing control device 1 further includes an EMG signal sensing module 101 and a gesture posture acquisition module 102. The EMG signal sensing module 101 is used to acquire EMG signals from a designated part 3 of the human body (such as the human arm, or other parts of the human body); the gesture posture acquisition module 102 is used to acquire the human body's gesture posture based on the EMG signals acquired by the EMG signal sensing module 101.

[0061] The motion capture module 103 can capture gesture action data based on the human gesture posture at a set time point within a set time period acquired by the gesture posture acquisition module. For example, it can acquire human actions based on the changes in human gesture posture within a set time period; or it can set the rules for the changes in human gesture posture in a database, and compare the acquired changes in human gesture posture within a set time period with the data in the database to obtain human actions.

[0062] The humanoid robot device 2 includes: a robotic arm 201, a camera mechanism assembly 202, a robotic arm 203, a camera mechanism movement drive module 204, a target object three-dimensional coordinate and attitude calculation module 205, and a robotic arm attitude acquisition module 206. Of course, the target object three-dimensional coordinate and attitude calculation module 205 can also be independent of the humanoid robot device 2.

[0063] The robotic arm 201 is used to perform grasping actions. In one embodiment, the robotic arm may adopt the bionic hand structure described in Chinese Patent Publication No. CN111906809A (other structures may also be used, which will not be elaborated here). The camera assembly 202 is used to acquire image data of a set area; the camera assembly has at least two camera frames, including a first camera frame and a second camera frame.

[0064] The robotic arm 203 is used to position the camera assembly 202, and can drive the camera assembly 202 to move within a defined three-dimensional space. The robotic arm 203 may include two (or more) robotic arm mechanisms, each used to connect the first camera assembly and the second camera assembly. Alternatively, other components can be used to replace the robotic arm, such as a three-dimensional movement track. This track may include at least two tracks, a first track and a second track, where the centers of the first and second tracks are not on the same plane. The first camera assembly is positioned on the first track, and the second camera assembly is positioned on the second track.

[0065] The camera mechanism movement drive module 204 is used to drive the robotic arm to move, thereby driving the first camera mechanism and the second camera mechanism to move.

[0066] The target object 3D coordinate and attitude calculation module 205 is used to control the camera mechanism movement drive module to drive the robotic arm to move, thereby driving the camera mechanism assembly to move; each camera mechanism of the camera mechanism assembly acquires image data at at least two different positions during the movement; the target object 3D coordinate and attitude calculation module calculates the 3D coordinates and attitude of the target object 4 in the set 3D coordinate system based on the image data acquired by each camera mechanism.

[0067] The robot arm posture acquisition module 206 is used to acquire the robot arm's posture data based on the state of the sensing components and the robot arm drive mechanism. In one embodiment, the position of each finger can be determined based on the encoder values ​​of each drive motor in the robot arm (here, the motor encoder can be used as the aforementioned sensing component).

[0068] In one embodiment of the present invention, the humanoid robot device further includes a target object category recognition module 207; the target object state recognition module 207 is used to calculate the category of the target object based on the image data acquired by each camera. The target object may be a book, a cup, a rod, clothing, etc.

[0069] The target object 3D coordinate and attitude calculation module 205 determines the calculation method of the target object's 3D coordinates according to the target object's category. In one embodiment, different calculation methods for 3D coordinates and attitudes can be set for different categories of target objects; for example, for a standard rod, the position of the rod's central axis and the rod's diameter data can be obtained; for irregular objects, the position of the object's central region and the range of the object's coverage area can be calculated, and the target object's grasping area can also be determined.

[0070] In one embodiment of the present invention, the target object three-dimensional coordinate and attitude calculation module 205 acquires image data captured by a set camera mechanism under different three-dimensional coordinates to form an image data set, and simultaneously acquires the three-dimensional coordinates and attitude of the camera mechanism at the time of capture; the image data set is compared with image data sets in a comparison database, the image data sets in the comparison database include image data captured by the camera mechanism under different three-dimensional coordinates, as well as the three-dimensional coordinates and attitude of the camera mechanism at the time of capture of each image, and also include the three-dimensional coordinates and attitude of the target object; when comparing the image data sets, the image data set with the closest similarity is selected, and the target object's three-dimensional coordinates and attitude corresponding to the image data value are acquired as the target object's three-dimensional coordinates and attitude.

[0071] The intelligent grasping and control system may further include a target object three-dimensional coordinate and posture mathematical model construction module 208, which is used to construct a target object three-dimensional coordinate and posture mathematical model. The target object three-dimensional coordinate and posture mathematical model construction module 208 includes: a data preprocessing unit 2081, a feature extraction unit 2082, a feature processing unit 2083, and a mathematical model establishment unit 2084.

[0072] The data preprocessing unit 2081 preprocesses the data in the training dataset; the feature extraction unit 2082 extracts features from the preprocessed data to extract key features. The feature processing unit 2083 combines, statistically analyzes, and groups the key features extracted by the feature extraction unit, standardizing and normalizing them; after processing by the feature processing unit, a key feature combination is formed; the key feature combination includes image data captured by the camera mechanism, the three-dimensional coordinates and pose of the camera mechanism when capturing each image, and the three-dimensional coordinates and pose of the target object. The mathematical model building unit 2084 constructs a mathematical model of the target object's three-dimensional coordinates and pose based on the key feature combination processed by the feature processing unit. The target object's three-dimensional coordinates and pose calculation module 205 inputs the images acquired by the camera mechanism component 202 into the target object's three-dimensional coordinates and pose mathematical model to obtain the target object's three-dimensional coordinates and pose.

[0073] In one embodiment of the present invention, the robotic hand may further include a first control circuit, a robotic hand body, and several fingers. Several pressure sensors and several temperature sensors are distributed on the surfaces of the robotic hand body and the fingers. Each pressure sensor senses pressure data received in a designated area, and each temperature sensor senses temperature data in a designated area; the data sensed by each pressure sensor and each temperature sensor are sent to the first control circuit.

[0074] The electromyography (EMG) sensing control device further includes a second control circuit and a display module. The display module displays pressure data sensed by a set pressure sensor and / or temperature data sensed by a set temperature sensor, thereby obtaining the force between the robotic arm and the target object and the temperature state of the target object. Then, the control signal sent to the robotic arm is adjusted according to the force between the robotic arm and the target object and the temperature state of the target object.

[0075] In one embodiment of the present invention, the intelligent grasping control system may further include a camera device worn by the operator; the camera device is used to sense changes in the operator's line of sight and adjust the viewing angle of the humanoid robot and the target object in the image generated by the AR module accordingly; thereby making it easier for the operator to understand the true relative distance between the humanoid robot and the target object. When a change in the angle of the camera device is detected, the viewing angle of the humanoid robot and the target object in the image generated by the AR module is adjusted accordingly; for example, if the camera device is detected to be adjusted to the left, the viewing angle in the generated image is adjusted to the left accordingly (and the adjustment range corresponds to the change range of the camera device); if the camera device is detected to be adjusted upwards, the viewing angle in the generated image is adjusted upwards accordingly (and the adjustment range corresponds to the change range of the camera device); thereby forming an image with a simulated three-dimensional stereoscopic projection effect.

[0076] In one application scenario of this invention, a human can wear an electromyography (EMG) sensing control device to remotely control a humanoid robot. When sending control signals, the humanoid robot can be visually and three-dimensionally observed in conjunction with the projection or image generated by the AR module, making it easier for the operator to send the correct control signals to the EMG sensing control device, thereby forming a control signal for the humanoid robot to grasp. After receiving the control signal, the humanoid robot performs the corresponding action.

[0077] This invention further discloses an intelligent grasping control method for a humanoid robot. Figure 2 This is a flowchart of a humanoid robot intelligent grasping control method according to an embodiment of the present invention; please refer to [link / reference]. Figure 2 The intelligent grasping control method includes:

[0078]

Step S1

[0079]

Step S2

[0080]

Step S3

[0081]

Step S4

[0082]

Step S5

[0083] The AR module projects the target object's 3D coordinates and posture, the robot's 3D coordinates and posture, and the human body's 3D coordinates based on the acquired target object's 3D coordinates and posture, the robot's 3D coordinates and posture, and the human body's 3D coordinates. The positional relationship between the target object's projection and the designated human body part corresponds to the relative positional relationship between the target object and the robot. The size relationship between the generated 3D projection of the target object and the designated human body part corresponds to the size relationship between the designated human body part and the robot. Alternatively, the AR module generates images of the target object and the designated human body part based on the acquired target object's 3D coordinates and posture, the robot's 3D coordinates and posture, and the human body's 3D coordinates, and displays the generated images on a designated display module. The positional relationship between the target object image and the image of the designated human body part corresponds to the relative positional relationship between the target object and the robot. The size relationship between the generated 3D image of the target object and the image of the designated human body part corresponds to the size relationship between the designated human body part and the robot.

[0084]

Step S6

[0085]

Step S7

[0086] In one embodiment of the present invention, the intelligent grasping control method further includes a target object category recognition step: the target object state recognition module calculates the category of the target object based on the image data acquired by each camera; the target object three-dimensional coordinate and posture calculation module determines the calculation method of the target object's three-dimensional coordinates based on the target object's category.

[0087] In one embodiment of the present invention, the target object 3D coordinate and attitude calculation module acquires image data captured by a set camera mechanism under different 3D coordinates to form an image data set, and simultaneously acquires the 3D coordinates and attitude of the camera mechanism at the time of capture; the image data set is compared with image data sets in a comparison database, the image data sets in the comparison database including image data captured by the camera mechanism under different 3D coordinates, and the 3D coordinates and attitude of the camera mechanism at the time of capture of each image, and also including the 3D coordinates and attitude of the target object; when comparing the image data sets, the image data set with the closest similarity is selected, and the 3D coordinates and attitude of the target object corresponding to the image data value are acquired as the 3D coordinates and attitude of the target object.

[0088] In one embodiment of the present invention, the intelligent grasping control method further includes a step of constructing a mathematical model of the three-dimensional coordinates and attitude of the target object: a mathematical model construction module constructs a mathematical model of the three-dimensional coordinates and attitude of the target object; the step of constructing the mathematical model of the three-dimensional coordinates and attitude of the target object includes:

[0089] Data preprocessing steps: Preprocess the data in the training dataset;

[0090] Feature extraction: Feature extraction is performed on the data preprocessed in the aforementioned data preprocessing steps to extract key features;

[0091] Feature processing: The key features extracted in the feature extraction step are combined, statistically analyzed, grouped and encoded, and the key features are standardized and normalized; after the processing in the feature processing step, a key feature combination is formed; the key feature combination includes image data captured by the camera mechanism, the three-dimensional coordinates and pose of the camera mechanism when capturing each image data, and the three-dimensional coordinates and pose of the target object being photographed.

[0092] Mathematical model establishment steps: Construct a three-dimensional coordinate and posture mathematical model of the target object based on the key feature combination processed by the aforementioned feature processing steps.

[0093] In one embodiment of the present invention, the robotic hand includes a first control circuit, a robotic hand body, and several fingers. Several pressure sensors and several temperature sensors are distributed on the surface of the robotic hand body and the several fingers.

[0094] The intelligent grasping control method further includes:

[0095] Each pressure sensor senses the pressure data received in the set area, and each temperature sensor senses the temperature data in the set area; the data sensed by each pressure sensor and each temperature sensor are sent to the first control circuit.

[0096] The electromyography (EMG) sensing control device further includes a second control circuit and a display module. The display module displays pressure data sensed by a set pressure sensor and / or temperature data sensed by a set temperature sensor, thereby obtaining the force between the robotic arm and the target object and the temperature state of the target object. Then, the control signal sent to the robotic arm is adjusted according to the force between the robotic arm and the target object and the temperature state of the target object.

[0097] This invention also discloses an electronic device, Figure 4 This is a schematic diagram of the composition of an electronic device according to an embodiment of the present invention; please refer to [link / reference]. Figure 4At the hardware level, the electronic device includes a memory, a processor, and at least one communication interface; the processor may be a microprocessor, and the memory may include main memory, such as random access memory (RAM) or non-volatile memory. Of course, the electronic device may also include other hardware as needed.

[0098] The processor, communication interface, and memory can be interconnected via an internal bus. The memory stores programs (including operating system programs and application programs); the programs may include program code, which may include computer operation instructions. The memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0099] In one embodiment, the processor can read the corresponding program from non-volatile memory into memory and then run it; the processor can execute the program stored in memory and specifically perform the following operations (e.g. Figure 2 As shown):

[0100]

Step S1

[0101]

Step S2

[0102]

Step S3

[0103]

Step S4

[0104]

Step S5

[0105] The AR module projects the target object's 3D coordinates and posture, the robot's 3D coordinates and posture, and the human body's 3D coordinates based on the acquired target object's 3D coordinates and posture, the robot's 3D coordinates and posture, and the human body's 3D coordinates. The positional relationship between the target object's projection and the designated human body part corresponds to the relative positional relationship between the target object and the robot. The size relationship between the generated 3D projection of the target object and the designated human body part corresponds to the size relationship between the designated human body part and the robot. Alternatively, the AR module generates images of the target object and the designated human body part based on the acquired target object's 3D coordinates and posture, the robot's 3D coordinates and posture, and the human body's 3D coordinates, and displays the generated images on a designated display module. The positional relationship between the target object image and the image of the designated human body part corresponds to the relative positional relationship between the target object and the robot. The size relationship between the generated 3D image of the target object and the image of the designated human body part corresponds to the size relationship between the designated human body part and the robot.

[0106]

Step S6

[0107]

Step S7

[0108] This invention further discloses a storage medium storing computer program instructions, which, when executed by a processor, implement the following steps of the method of this invention (e.g. Figure 2 As shown):

[0109]

Step S1

[0110]

Step S2

[0111]

Step S3

[0112]

Step S4

[0113]

Step S5

[0114] The AR module projects the target object's 3D coordinates and posture, the robot's 3D coordinates and posture, and the human body's 3D coordinates based on the acquired target object's 3D coordinates and posture, the robot's 3D coordinates and posture, and the human body's 3D coordinates. The positional relationship between the target object's projection and the designated human body part corresponds to the relative positional relationship between the target object and the robot. The size relationship between the generated 3D projection of the target object and the designated human body part corresponds to the size relationship between the designated human body part and the robot. Alternatively, the AR module generates images of the target object and the designated human body part based on the acquired target object's 3D coordinates and posture, the robot's 3D coordinates and posture, and the human body's 3D coordinates, and displays the generated images on a designated display module. The positional relationship between the target object image and the image of the designated human body part corresponds to the relative positional relationship between the target object and the robot. The size relationship between the generated 3D image of the target object and the image of the designated human body part corresponds to the size relationship between the designated human body part and the robot.

[0115]

Step S6

[0116]

Step S7

[0117] In summary, the humanoid robot intelligent grasping control system, method, electronic device, and storage medium proposed in this invention can remotely control the movements of humanoid robots, improving control accuracy and work efficiency.

[0118] It should be noted that this application can be implemented in software and / or a combination of software and hardware; for example, it can be implemented using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In some embodiments, the software program of this application can be executed by a processor to implement the steps or functions described above. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium; for example, RAM memory, magnetic or optical drives, floppy disks, and similar devices. In addition, some steps or functions of this application can be implemented in hardware; for example, as circuitry that cooperates with a processor to perform the various steps or functions.

[0119] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0120] The description and application of the present invention herein are illustrative and not intended to limit the scope of the invention to the embodiments described above. Effects or advantages involved in the embodiments may not be apparent due to various factors, and the description of effects or advantages is not intended to limit the embodiments. Variations and modifications of the embodiments disclosed herein are possible, and various substitutions and equivalents of the components in the embodiments are well known to those skilled in the art. It should be apparent to those skilled in the art that the invention can be implemented in other forms, structures, arrangements, proportions, and with other components, materials, and parts without departing from the spirit or essential characteristics of the invention. Other variations and modifications can be made to the embodiments disclosed herein without departing from the scope and spirit of the invention.

Claims

1. A humanoid robot intelligent grasping control system, characterized in that, The intelligent grasping control system includes: an electromyography (EMG) sensing control device and a humanoid robot device, wherein the EMG sensing control device communicates with the humanoid robot device; The electromyography-sensing control device includes: The motion capture module is used to capture gesture data; The grasping control module is used to control the grasping action of the humanoid robot device based on the gesture action data captured by the motion capture module. The AR module is used to display the target object in front of a set part of the human body through stereoscopic projection, thereby simulating the relative positional relationship between the target object and the robotic arm. The humanoid robot device includes: A robotic arm, used to perform grasping actions; A camera frame assembly for acquiring image data of a defined area; the camera frame assembly includes at least two camera frames, the at least two camera frames including a first camera frame and a second camera frame; A robotic arm is used to set up the camera mechanism assembly and can drive the camera mechanism assembly to move within a set three-dimensional space; A camera mechanism movement drive module is used to drive the robotic arm to move, thereby driving the first camera mechanism and the second camera mechanism to move; The target object 3D coordinate and attitude calculation module is used to control the camera mechanism movement drive module to drive the robotic arm to move, thereby driving the camera mechanism assembly to move; each camera mechanism of the camera mechanism assembly acquires image data from at least two different positions during the movement; the target object 3D coordinate and attitude calculation module calculates the target object's 3D coordinates and attitude in a set 3D coordinate system based on the image data acquired by each camera mechanism. The robot arm posture acquisition module is used to acquire the robot arm's posture data based on the status of the sensor components and the robot arm's drive mechanism. The AR module projects the target object's 3D coordinates and posture, the robotic arm's 3D coordinates and posture, and the human body's 3D coordinates based on the acquired target object's 3D coordinates and posture, the robotic arm's 3D coordinates and posture, and the human body's 3D coordinates. The positional relationship between the target object's projection and the designated human body part corresponds to the relative positional relationship between the target object and the robotic arm, and the size relationship between the generated 3D projection of the target object and the designated human body part corresponds to the size relationship between the designated human body part and the robotic arm. Alternatively, the AR module generates images of the target object and the designated human body part based on the acquired target object's 3D coordinates and posture, the robotic arm's 3D coordinates and posture, and the human body's 3D coordinates, and displays the generated images on a designated display module. The positional relationship between the target object image and the image of the designated human body part corresponds to the relative positional relationship between the target object and the robotic arm, and the size relationship between the generated 3D image of the target object and the image of the designated human body part corresponds to the size relationship between the designated human body part and the robotic arm. The intelligent grasping and control system further includes a target object three-dimensional coordinate and attitude mathematical model construction module, which is used to construct the target object three-dimensional coordinate and attitude mathematical model. The target object 3D coordinate and pose mathematical model construction module preprocesses the data in the training dataset, extracts features from the preprocessed data, and extracts key features. Then, it combines, statistically analyzes, and groups the extracted key features, and standardizes and normalizes the key features to form a key feature combination. The key feature combination includes image data captured by the camera mechanism, the 3D coordinates and pose of the camera mechanism when capturing each image data, and the 3D coordinates and pose of the target object being photographed. Based on the formed key feature combination, a mathematical model of the target object's 3D coordinates and pose is constructed. The target object 3D coordinate and attitude calculation module acquires image data captured by a set camera mechanism under different 3D coordinates to form an image data set, and simultaneously acquires the 3D coordinates and attitude of the camera mechanism at the time of capture. The image data set is compared with image data sets in a comparison database, which includes image data captured by the camera mechanism under different 3D coordinates, as well as the 3D coordinates and attitude of the camera mechanism at the time of capture, and also includes the 3D coordinates and attitude of the target object. When comparing the image data sets, the image data set with the closest similarity is selected, and the corresponding 3D coordinates and attitude of the target object are acquired as the 3D coordinates and attitude of the target object. The AR module includes at least one 3D holographic projection device, each 3D holographic projection device being used to project a 3D projection of a target object into a set area.

2. The humanoid robot intelligent grasping control system according to claim 1, characterized in that: The robotic arm includes a first control circuit, a robotic arm body, and several fingers. Several pressure sensors and several temperature sensors are distributed on the surface of the robotic arm body and the fingers. Each pressure sensor is used to sense the pressure data of the set area, and each temperature sensor is used to sense the temperature data of the set area; each pressure sensor and each temperature sensor send the sensed data to the first control circuit. The electromyography (EMG) sensing control device further includes a second control circuit and a display module. The display module is used to display the pressure data sensed by the set pressure sensor and / or the temperature data sensed by the set temperature sensor, thereby obtaining the force between the robot and the target object and the temperature state of the target object, so as to facilitate the adjustment of the control signals sent to the robot.

3. An intelligent grasping control method for a humanoid robot intelligent grasping control system according to claim 1 or 2, characterized in that, The intelligent grasping control method includes: A camera module assembly acquires image data of a designated area; the camera module assembly includes at least two camera modules, including a first camera module and a second camera module; the camera module assembly is mounted on a robotic arm, and the robotic arm can drive the camera module assembly to move within a designated three-dimensional space; The camera module movement drive module drives the robotic arm to move, thereby driving the first camera module and the second camera module to move; The target object 3D coordinate and attitude calculation module controls the camera mechanism movement drive module to drive the robotic arm to move, thereby driving the camera mechanism assembly to move; each camera mechanism of the camera mechanism assembly acquires image data from at least two different positions during the movement; the target object 3D coordinate and attitude calculation module calculates the target object's 3D coordinates and attitude in a set 3D coordinate system based on the image data acquired by each camera mechanism. The robot arm posture acquisition module acquires the robot arm's posture data based on the status of the sensors and the robot arm's drive mechanism. The AR module displays a target object in front of a designated part of the human body using stereoscopic projection, thereby simulating the relative positional relationship between the target object and the robotic arm. The AR module projects the target object's projection based on the acquired 3D coordinates and posture of the target object, the robotic arm, and the human body part. The positional relationship between the target object's projection and the designated human body part corresponds to the relative positional relationship between the target object and the robotic arm, and the size relationship between the generated stereoscopic projection of the target object and the designated human body part corresponds to the size relationship between the designated human body part and the robotic arm. Alternatively, the AR module generates images of the target object and the designated human body part based on the acquired 3D coordinates and posture of the target object, the robotic arm, and the human body part, and displays these images on a designated display module. The positional relationship between the target object image and the image of the designated human body part corresponds to the relative positional relationship between the target object and the robotic arm, and the size relationship between the generated stereoscopic image of the target object and the image of the designated human body part corresponds to the size relationship between the designated human body part and the robotic arm. The motion capture module captures gesture data; The grasping control module controls the grasping action of the humanoid robot device based on the gesture data captured by the motion capture module; The robotic arm performs the grasping action according to the control signal generated by the grasping control module.

4. The intelligent grasping control method for humanoid robots according to claim 3, characterized in that: The target object 3D coordinate and attitude calculation module acquires image data captured by a set camera mechanism under different 3D coordinates to form an image data set, and simultaneously acquires the 3D coordinates and attitude of the camera mechanism at the time of capture. The image data set is compared with image data sets in a comparison database. The image data sets in the comparison database include image data captured by the camera mechanism under different 3D coordinates, as well as the 3D coordinates and attitude of the camera mechanism at the time of capture of each image, and also include the 3D coordinates and attitude of the target object. When comparing the image data sets, the image data set with the closest similarity is selected, and the 3D coordinates and attitude of the target object corresponding to the image data set are acquired as the 3D coordinates and attitude of the target object.

5. The intelligent grasping control method for humanoid robots according to claim 3, characterized in that: The intelligent grasping control method further includes a step of constructing a mathematical model of the target object's three-dimensional coordinates and attitude: the target object's three-dimensional coordinates and attitude mathematical model construction module constructs a mathematical model of the target object's three-dimensional coordinates and attitude; The steps for constructing the mathematical model of the target object's three-dimensional coordinates and pose include: preprocessing the data in the training dataset; extracting features from the preprocessed data to extract key features; combining, statistically analyzing, and grouping the extracted key features; standardizing and normalizing the key features to form a key feature combination; the key feature combination includes image data captured by the camera mechanism, the three-dimensional coordinates and pose of the camera mechanism when capturing each image data, and the three-dimensional coordinates and pose of the target object being photographed; and constructing a mathematical model of the target object's three-dimensional coordinates and pose based on the key feature combination. The AR module includes at least one 3D holographic projection device, each of which projects a 3D projection of the target object in a set area.

6. The intelligent grasping control method for humanoid robots according to claim 3, characterized in that: The robotic arm includes a first control circuit, a robotic arm body, and several fingers. Several pressure sensors and several temperature sensors are distributed on the surface of the robotic arm body and the fingers. Each pressure sensor senses the pressure data of the set area, and each temperature sensor senses the temperature data of the set area; each pressure sensor and each temperature sensor send the sensed data to the first control circuit. The electromyography (EMG) sensing control device further includes a second control circuit and a display module. The display module displays pressure data sensed by a set pressure sensor and / or temperature data sensed by a set temperature sensor, thereby obtaining the force between the robotic arm and the target object and the temperature state of the target object. Then, the control signal sent to the robotic arm is adjusted according to the force between the robotic arm and the target object and the temperature state of the target object.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 3 to 6.

8. A storage medium storing computer program instructions thereon, characterized in that, When executed by a processor, the computer program instructions implement the steps of the method according to any one of claims 3 to 6.

Images