Humanoid robot, carrying control method and device thereof and readable storage medium

Abstract

The invention relates to the field of humanoid robot control, and discloses a humanoid robot, a carrying control method and device thereof and a readable storage medium, and the method comprises the steps: collecting an image containing a to-be-carried object through a camera, so as to obtain a real-time object pose; according to the object pose, the target pose of the tail ends of the two arms of the robot reaching the object grabbing point is calculated in real time; according to the target pose and the actual pose, fed back in real time, of the tail ends of the two arms, the tail ends of the two arms are controlled to stably move to a grabbing point through a closed-loop feedback controller; in the process that the robot makes contact with an object, the tail ends of the two arms are controlled by the flexible controller to flexibly grab the object and place the object at the designated position. According to the method, the precision and the carrying efficiency of the humanoid robot during object carrying can be improved.

Description

Technical Field

[0001] The present application relates to the field of industrial automation technology, and in particular to a humanoid robot and a handling control method, device and readable storage medium thereof. Background Art

[0002] In the scenario where humans are moving boxes, humans can often subconsciously switch actions according to the actual environment. If they are close to the box, they can reach out to grab it, if they are at a lower distance, they can bend down to move it, and if they are at a farther distance, they can walk over to move it. Although humanoid robots are similar to humans in form, their whole body freedom is far from that of humans. Due to the limitations of operation and control strategies and computing power, in order for humanoid robots to perform certain operations, it is usually necessary to design corresponding methods based on specific operation tasks. Therefore, it is necessary to design a flexible and efficient method to achieve the carrying task. Summary of the invention

[0003] In view of this, the present application provides a humanoid robot and a transport control method, device and readable storage medium thereof, which can improve the accuracy and efficiency of the humanoid robot when transporting objects.

[0004] In a first aspect, an embodiment of the present application provides a humanoid robot transport control method, comprising:

[0005] The camera is used to capture images of the object to be transported to obtain the real-time position and posture of the object;

[0006] According to the object posture, the target posture of the robot's double-arm ends reaching the object grasping point is calculated in real time;

[0007] According to the target posture and the actual posture of the ends of the arms fed back in real time, the ends of the arms are controlled to move smoothly to the grasping points through a closed-loop feedback controller;

[0008] When the robot contacts the object, the flexible controller controls the ends of the dual arms to flexibly grasp the object and place it at a designated position.

[0009] In some embodiments, during the entire process of the robot carrying an object, the process further includes:

[0010] According to the real-time feedback of the actual position and posture of the ends of the two arms and the size information of the object, a kinematic estimation method is used to obtain in real time the distance between the object and the legs of the robot;

[0011] The movement of the robot's waist is controlled according to the distance so that the object does not collide with the robot's legs.

[0012] In some embodiments, controlling the movement of the robot's waist according to the distance includes:

[0013] Control the movement of the robot's waist according to the change in the distance between adjacent control cycles;

[0014] When the change amount decreases, the waist of the robot is controlled to move in a direction approaching the object; when the change amount increases, the waist of the robot is controlled to move in a direction away from the object.

[0015] In some embodiments, the process of performing the operation of placing the specified position further includes:

[0016] Determine whether to perform a stacking operation;

[0017] When it is determined to perform a stacking operation, adjusting the posture of the robot when placing the object so that the placement posture of the object is aligned with the posture of the placed object;

[0018] When it is determined that the stacking operation is not to be performed, the posture of the robot when placing the object is adjusted so that the object is placed parallel to the position of the robot.

[0019] In some embodiments, the step of collecting an image containing the object to be transported by a camera to obtain a real-time position and posture of the object includes:

[0020] Using a camera on the robot to capture an image containing the object to be transported, extracting object features from the image to obtain size information of the object and two-dimensional coordinates of the object in the image;

[0021] Performing coordinate transformation on the two-dimensional coordinates to obtain a first posture and a first three-dimensional coordinate of the object in the camera coordinate system of the robot;

[0022] The conversion matrix between the base coordinate system of the robot and the camera coordinate system is used to respectively calculate the second posture and the second three-dimensional coordinate corresponding to the first posture and the first three-dimensional coordinate when converted to the base coordinate system, so as to obtain the real-time object posture.

[0023] In some embodiments, the object posture includes the coordinate position of one of the vertex angles or the center of the target surface of the object; the object grasping point is selected as the center of two side surfaces of the object perpendicular to the horizontal direction;

[0024] The real-time calculation of the target posture of the robot's double-arm ends reaching the object grasping point according to the object posture comprises:

[0025] According to the coordinate position of the vertex angle or the center of the target surface and the size information of the object, the position of the center of each side surface of the object is calculated respectively to serve as the position of the object grasping point.

[0026] In some embodiments, the step of controlling the end of the double arm to move smoothly to the grasping point through a closed-loop feedback controller according to the target posture and the actual posture of the end of the double arm fed back in real time includes:

[0027] According to the error between the target posture corresponding to the end of each robotic arm and the actual posture fed back in real time, a closed-loop feedback controller corresponding to each robotic arm is used to calculate in real time the first expected speed of the end of each robotic arm;

[0028] Calculate the first driving control amount of the robot's whole body joint motor in real time through the Jacobian matrix according to the first desired speed of each end of the robot arm;

[0029] Each joint motor is driven according to the first drive control amount until the ends of the robot arms move smoothly to the grasping point.

[0030] In some embodiments, the closed-loop feedback controller is expressed as:

[0031]

[0032] Wherein, v represents the first expected velocity of the terminal, and e is the error between the target posture and the actual posture; is the first-order derivative of e; t is the length of a control cycle, K p , K d and K i All are preset coefficients.

[0033] In some embodiments, controlling the ends of the two arms to flexibly grasp the object and place it at a specified position by a flexible controller includes:

[0034] According to the actual force fed back in real time by the end of each robotic arm and the error between the actual posture and the target posture, the desired acceleration of the end of each robotic arm is calculated in real time by using the flexible controller;

[0035] Calculate a corresponding second expected speed according to the expected acceleration at the end of each robotic arm;

[0036] According to the second desired speed at the end of each robot arm, the second driving control amount of the robot's whole body joint motor is calculated in real time through the Jacobian matrix;

[0037] The motors of each joint are driven according to the second driving control amount until the ends of the robot's arms flexibly grasp the object and place it at a designated position.

[0038] In some embodiments, the expression of the flexible controller is:

[0039]

[0040] Wherein, a represents the desired acceleration of the terminal, and e is the error between the target posture and the actual posture; is the first-order derivative of e; F e is the actual force, M is the inertia matrix, B is the damping matrix, and K is the stiffness matrix.

[0041] In a second aspect, an embodiment of the present application further provides a humanoid robot transport control device, comprising:

[0042] A posture acquisition module is used to capture images containing the object to be transported through a camera to obtain the real-time posture of the object;

[0043] A posture conversion module is used to calculate the target posture of the robot's double-arm ends reaching the object grasping point in real time according to the object posture;

[0044] A first control module is used to control the ends of the two arms to move smoothly to the grasping point through a closed-loop feedback controller according to the target posture and the actual posture of the ends of the two arms fed back in real time;

[0045] The second control module is used to control the ends of the two arms to flexibly grasp the object and place it at a designated position through a flexible controller when the robot contacts the object.

[0046] In a third aspect, an embodiment of the present application further provides a humanoid robot, the humanoid robot comprising:

[0047] An execution assembly, including two robotic arms;

[0048] The control system, including the camera and processor;

[0049] Wherein, the camera is used to collect image information of the object in real time;

[0050] The processor is used to obtain the posture of the object to be transported in real time through the image information, and calculate the target posture of the robot's two arms reaching the object grasping point in real time according to the object posture; according to the target posture and the actual posture of the two arms' ends fed back in real time, control the two arms' ends to move smoothly to the grasping point through a closed-loop feedback controller; and in the process of the robot contacting the object, control the two arms' ends to flexibly grasp the object and place it in a designated position through a flexible controller.

[0051] In some embodiments, the control system further comprises a sensor unit, wherein the sensor unit comprises a motion sensor and a six-dimensional force sensor disposed at the end of each robotic arm;

[0052] The motion sensor is used to collect the actual motion state of the terminal in real time;

[0053] The six-dimensional force sensor is used to collect the actual force between the end of the robotic arm and the object.

[0054] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, implements the humanoid robot transport control method.

[0055] The embodiments of the present application have the following advantages:

[0056] The embodiment of the present application proposes a humanoid robot handling control method, which uses a camera to collect images containing objects to be handled to obtain real-time object posture; then, the target posture of the robot's two arms reaching the object grasping point is calculated in real time according to the object posture, so that according to the target posture and the actual posture of the two arms fed back in real time, a closed-loop feedback controller is used to control the two arms to move smoothly to the grasping point; at the same time, in the process from the robot starting to contact the object to the placement, a flexible controller is also used to control the two arms to grasp the object flexibly and place it in a specified position. The method uses visual servoing to correct the motion state of the two arms in real time to track the object grasping point in real time, so that the robot can flexibly and smoothly track the object grasping point according to the distance like a human, and after reaching the grasping point, the robot uses flexible control to contact the object. Similarly, the robot can flexibly adjust the motion state according to the feedback force like a human, effectively avoiding damage to the object or itself, and in the entire handling process, the robot can complete the handling task more accurately and efficiently. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0058] Figure 1 A schematic diagram of the structure of a humanoid robot according to an embodiment of the present application is shown;

[0059] Figure 2 A schematic diagram of the structure of a control system according to an embodiment of the present application is shown;

[0060] Figure 3 A flow chart showing a method for controlling the transport of a humanoid robot according to an embodiment of the present application is shown;

[0061] Figure 4 A flow chart of object posture acquisition in an embodiment of the present application is shown;

[0062] Figure 5 (a) and (b) respectively show schematic diagrams of selecting the center of the upper surface and a vertex angle of the upper surface as the object posture representatives when the transported object is a rectangular box in an embodiment of the present application;

[0063] Figure 6 A flow chart showing the tracking of object grabbing points by the end of the double arm according to an embodiment of the present application is shown;

[0064] Figure 7 A schematic diagram showing a state in which the ends of the dual arms of the robot according to an embodiment of the present application reach a grasping point;

[0065] Figure 8 A flow chart of the compliant control of the two-arm ends according to an embodiment of the present application is shown;

[0066] Fig. 9 A schematic diagram showing waist control based on the distance between the legs and an object according to an embodiment of the present application is shown;

[0067] Fig.10 (a) and (b) respectively show two schematic diagrams of placing objects according to an embodiment of the present application;

[0068] Fig.11 A schematic structural diagram of a humanoid robot transport control device according to an embodiment of the present application is shown.

[0069] Description of main component symbols:

[0070] 100-humanoid robot; 10-head; 20-torso; 30-actuator; 40-control system; 41-processor; 42-memory; 43-sensing unit; 431-camera; 200-humanoid robot handling control device; 210-posture acquisition module; 220-posture conversion module; 230-visual servo module; 240-compliant control module; 250-waist control module; 260-stacking control module. DETAILED DESCRIPTION

[0071] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0072] The components of the embodiments of the present application generally described and shown in the drawings herein may be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0073] Hereinafter, the terms "including", "having" and their cognates, which may be used in various embodiments of the present application, are intended only to indicate specific features, numbers, steps, operations, elements, components, or combinations of the foregoing items, and should not be understood as first excluding the existence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing items or adding the possibility of one or more features, numbers, steps, operations, elements, components, or combinations of the foregoing items.

[0074] Furthermore, the terms “first”, “second”, “third”, etc. are merely used for distinguishing descriptions and are not to be understood as indicating or implying relative importance.

[0075] Unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meanings as those generally understood by those skilled in the art to which the various embodiments of the present application belong. The terms (such as those defined in generally used dictionaries) will be interpreted as having the same meanings as the contextual meanings in the relevant technical field and will not be interpreted as having idealized meanings or overly formal meanings unless clearly defined in the various embodiments of the present application.

[0076] In order to improve the accuracy and efficiency of the humanoid robot 100 in performing the handling task, the humanoid robot handling control method proposed in this application combines visual servo control, compliant control and whole-body obstacle avoidance technologies, so that the humanoid robot 100 can efficiently and safely complete the handling task in a complex environment. It is particularly suitable for occasions such as factories that need to frequently carry heavy objects. It can not only improve work efficiency, but also reduce manual labor intensity and reduce the error rate. It can be understood that the handling task refers to the process of grabbing an object from one position and placing it at another position according to a pre-planned moving path, such as turning a preset angle or walking a preset distance. Among them, the handling task can be to carry only one object, or it can be to carry multiple objects continuously. For the convenience of understanding, this application mainly takes the handling process of a single object as an example for control description.

[0077] The technical solution of the present application will be described in detail below in conjunction with some specific embodiments.

[0078] Figure 1 A structural schematic diagram of a humanoid robot 100 according to an embodiment of the present application is shown. Exemplarily, the humanoid robot 100 may include a head 10, a trunk 20, an actuator 30, and a control system 40, wherein the actuator 30 includes two mechanical arms (referred to as arms) and two mechanical legs (referred to as legs) connected to the trunk 20.

[0079] The control system 40, as the control core of the humanoid robot 100, is similar to the human brain and can be used to drive and control the arms and / or legs to perform corresponding actions to complete various tasks. The trunk 20, as the main structure of the humanoid robot 100, is used to support the weight of the entire robot. In addition, the trunk 20 is also used to accommodate important components such as batteries and controllers in the control system 40. The head 10 usually includes a sensing unit 43 such as a camera, a microphone, a contact sensor, etc., which can be used to perceive the external environment and interact from multiple aspects such as vision, hearing, and touch. Among them, the two mechanical arms of the humanoid robot 100, similar to human hands, can be used to coordinately clamp and move objects. Of course, a single mechanical arm can also be used to grab objects, and the actions or functions that the two mechanical arms can perform are not limited here. The two mechanical legs of the humanoid robot 100 are connected to the trunk 20 through a hip joint drive structure, similar to human legs, and can achieve alternating position changes. In addition, a rotatable drive structure can also be set at the knee joint of each mechanical leg to achieve the knee bending operation of the corresponding mechanical leg. For example, in the handling scenario of the present application, when the robot bends over to grab an object, the two mechanical legs will also perform a corresponding knee bending action, similar to a person bending over and kneeling.

[0080] Exemplarily, the control system 40 mainly includes software and hardware components for implementing the control level, such as Figure 2 As shown, the control system 40 may include, but is not limited to, a processor 41, a memory 42 and a sensing unit 43, wherein the memory stores a computer program, the sensing unit 43 is used to sense its own information and external environment information, and the processor 41 runs the computer program to enable the humanoid robot 100 to execute the humanoid robot transport control method of the embodiment of the present application through the motion component, so that the robot can perform the transport task more accurately and efficiently.

[0081] Among them, the processor 41 can be an integrated circuit chip with signal processing capabilities. The processor 41 can be a general-purpose processor, including a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or at least one of other programmable logic devices, discrete gates or transistor logic devices, and discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., which can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application.

[0082] The memory 42 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electric erasable programmable read-only memory (EEPROM), etc. The memory 42 is used to store a computer program, and the processor 41 may execute the computer program accordingly after receiving an execution instruction.

[0083] The perception unit 43 mainly refers to various sensors provided on the humanoid robot 100, which are used to transmit necessary external information and its own state information to the control system 40 of the humanoid robot 100 (such as the processor 41, of course, for end-to-end control, it can also be some other control modules, etc., which are not limited here), thereby providing the necessary conditions for the decision-making of the humanoid robot 100.

[0084] In some cases, these sensors can be divided into internal sensors and external sensors, wherein the internal sensors are functional elements used to measure the state of the robot itself, and their function is to measure kinematic quantities and mechanical quantities, which can be used to collect the motion state of the robot itself and feed it back to the control system 40, thereby controlling the actuator 30 of the humanoid robot 100 to move according to the specified position, trajectory and speed. The external sensors are mainly used to perceive the mutual information between the robot itself and the environment, such as the camera 431, microphone, contact sensor, etc. It should be understood that the above-mentioned sensors can be selectively set according to the actual application scenario of the robot.

[0085] For example, in the present application, the internal sensors may include motion sensors and force / torque sensors disposed at the end of each robotic arm. The motion sensor is used to collect the motion state of the end in real time, and may include, but is not limited to, acceleration sensors, angular velocity sensors, inertial measurement sensors, etc. The force / torque sensor is used to collect mechanical quantities such as the force or torque between the end of the robotic arm and the external environment (such as an object), and may include, but is not limited to, a one-dimensional force sensor, a three-dimensional force sensor, a six-dimensional force sensor, etc.

[0086] For example, in the present application, the external sensor may include a camera 431, which is used to collect image information of the location of the object to be transported in real time, and adjust the motion state of the ends of the arms of the humanoid robot 100 in real time through visual feedback. As some optional solutions, the camera 431 can be set at the upper side of the head 10 or the torso 20 of the humanoid robot 100. In addition, the integrated form, quantity and type of the camera 431 are not limited here, and can be set according to actual needs.

[0087] Based on the humanoid robot 100 of the above structure, the present application proposes a robot whole-body motion and handling control method suitable for various handling scenarios such as factories and homes, including using visual servo control technology, that is, using a visual device (such as camera 431) to guide and adjust (correct) in real time the position and posture of the robot's execution end when carrying an object, which can improve the accuracy of the humanoid robot 100 in grasping the object being carried. In the process of contacting the object, the arms use a compliant algorithm to adapt to the external vibration of the humanoid robot 100. In addition, collision detection is used to solve the problem that the humanoid robot 100 is prone to collision with the object when bending over to carry an object due to the protrusion of the knees. Through whole-body motion control, the humanoid robot 100 can perform handling tasks in a more flexible and efficient manner, just like humans, and can adjust actions more flexibly according to environmental changes.

[0088] The humanoid robot transport control method proposed in the embodiment of the present application is described in detail below.

[0089] Figure 3 A flow chart of a humanoid robot transport control method according to an embodiment of the present application is shown. Exemplarily, the humanoid robot transport control method comprises the following steps:

[0090] S110, collecting an image containing the object to be transported through the camera 431 to obtain a real-time position and posture of the object.

[0091] The image refers to an image of the object to be transported captured by the robot through the camera 431. By further analyzing the image, the position information of the object can be obtained.

[0092] Among them, the object posture includes the position and posture of the object. It should be understood that the position and posture of the object finally solved in step S110 are for the robot base coordinate system. Among them, the robot base coordinate system is used to describe the position and posture of the robot itself relative to the fixed environment, which can usually be fixed at the bottom of the robot or at a fixed point in the world coordinate system. It can be understood that when the robot grasps an object, it is necessary to convert the object position captured by the camera 431 into coordinates in the robot base coordinate system, so as to express the robot's joint angles, linear movements and other motion instructions in the same coordinate system, so as to control the robot's arms The end can accurately reach the grasping point and grasp the object.

[0093] In other words, from the position of the object to be transported in the captured image to the position and posture of the object in the base coordinate system, two transformations need to be performed. The first is to convert the two-dimensional coordinates in the image into the three-dimensional coordinates in the camera coordinate system; for example, the coordinates of the two-dimensional pixel points in the image can be converted into the coordinates of the three-dimensional space points in the camera coordinate system by pre-calibrating the camera 431 through the intrinsic parameter matrix and the extrinsic parameter matrix. The second is to convert the three-dimensional coordinates in the camera coordinate system into the three-dimensional coordinates in the base coordinate system of the robot; for example, the three-dimensional points in the camera coordinate system can be converted into the robot base coordinate system by pre-calibrating the camera 431 through the extrinsic parameter matrix. Among them, the camera coordinate system is the reference system of the camera 431 installed on the robot. It is used to describe the position and posture of the object in the field of view of the camera 431. It may change due to changes in the installation position of the camera 431, so it is usually necessary to calibrate it regularly to ensure accuracy.

[0094] For example, in one embodiment, Figure 4 As shown, step S110 includes:

[0095] S210, using the camera 431 on the robot to capture an image containing the object to be transported, extracting object features from the image, and obtaining size information of the object and two-dimensional coordinates in the image.

[0096] S220, performing coordinate transformation on the two-dimensional coordinates to obtain a first posture and a first three-dimensional coordinate of the object in the camera coordinate system of the robot.

[0097] Exemplarily, the camera 431 on the robot will take an image of the location of the object at a short interval, and then the control system 40 can identify and extract features such as the position and size of the object to be transported in the image. For example, through edge detection, color segmentation and other operations, the feature information of the object to be transported can be obtained, including but not limited to the specific position, shape, size, etc.

[0098] It can be understood that the size information of the object and the coordinate information of each feature point can be extracted through image processing. For example, taking the transport of a rectangular box as an example, the top corners of the box can be selected as its key feature points; and then through the first transformation from the two-dimensional coordinates in the image to the three-dimensional coordinates in the camera coordinate system, the three-dimensional coordinates and posture of each feature point of the object in the camera coordinate system can be obtained.

[0099] The posture of the object may be described by pitch angle, yaw angle, and roll angle, and may be obtained, for example, by feature point matching or template matching, and is not limited thereto.

[0100] Considering that there are usually multiple feature points of an object in three-dimensional space, as an optional solution, for the convenience of calculation, the position and posture of a target point on the object can be selected to represent the position and posture of the entire object, so as to facilitate the subsequent calculation of the object grasping point, etc. For example, the position of the center of the upper surface of the object, or the position of the center of the object, etc., can be selected, and there is no sole limitation here.

[0101] S230, using the transformation matrix between the base coordinate system of the robot and the camera coordinate system, respectively calculate the second posture and the second three-dimensional coordinate corresponding to the first posture and the first three-dimensional coordinate when converted to the base coordinate system, to obtain a real-time object posture.

[0102] Exemplarily, through the second transformation, the corresponding posture and three-dimensional coordinates of the object in the base coordinate system can be obtained. Among them, the transformation matrix between the robot base coordinate system and the camera coordinate system mainly refers to the camera extrinsic matrix, which is used to transform the points in the camera coordinate system to the robot base coordinate system.

[0103] It can be understood that the “first” in the above-mentioned first posture and first three-dimensional coordinates and the “second” in the second posture and second three-dimensional coordinates are only used to distinguish the posture and three-dimensional coordinates of the object in the camera coordinate system and the base coordinate system.

[0104] In this application, the posture of the object grasping point is used as the target posture that the corresponding robot end needs to reach, that is, the tracking of the end and the grasping point is realized. Since the image of the camera 431 is updated in real time, the posture of the object will be calculated in real time, and the target posture that the end of the double arm needs to reach is also calculated in real time. It can be understood that even if the posture of the object changes at this time, the target posture of the end of the double arm will be adjusted in time accordingly, thereby realizing control based on visual servoing, ensuring that the robot can track the object grasping point more accurately.

[0105] S120, calculating in real time the target posture of the robot's double-arm ends reaching the object grasping point according to the object posture.

[0106] The object grasping points refer to the two contact points set on the surface of the object for the robot's dual arms to grasp the object. For example, taking a rectangular box as an example, the grasping points can be selected as the centers of the two side surfaces of the object perpendicular to the horizontal direction, that is, the center of the left surface and the center of the right surface, such as Figure 4 Of course, in some other implementations, it is not necessarily the center of the side surface, but may be near the upper half of each side surface, or may not be at the same height, which can be selected according to actual needs. However, it should be understood that the specific positions of these grasping points can be calculated by combining the above-mentioned object posture information with the size information of the object.

[0107] It can be understood that if the object posture is represented by selecting the posture of one of the vertex angles or the center of a target surface of the object, then the calculation method of the posture of the object grasping point is related to the posture of the vertex angle or the center of the target surface. Exemplarily, the position of the center of each side surface of the object can be calculated respectively according to the coordinate position of the vertex angle or the center of the target surface obtained in step S110 and the size information of the object, so as to serve as the position of the object grasping point.

[0108] For example, taking the selection center as an example, if the object posture is the coordinate position of the center of the upper surface of the object, then according to the coordinate position of the center of the upper surface and the size information of the object, the coordinate positions of the left surface center and the right surface center of the object are calculated respectively to obtain the coordinate position of the grasping point of the object.

[0109] Take a rectangular box as an example, Figure 5 As shown in (a), for the XYZ axis coordinate system, the right direction on the X axis is defined as the positive direction, and the left direction is the negative direction; the direction close to the robot on the Y axis is defined as the positive direction, and the direction away from the robot is defined as the negative direction; the downward direction on the Z axis is defined as the positive direction, and the upward direction is defined as the negative direction. Assume that the target surface is selected as the center of the upper surface of the object perpendicular to the vertical direction, and its three-dimensional coordinates are marked as (X0, Y0, Z0), and for the size information of the object, its length, width and height are respectively recorded as [lwh], so for the coordinate position of the center of each side surface, the calculation process includes:

[0110] According to the coordinate position of the center of the upper surface, the coordinate value in the Z-axis direction is added with half the height value of the object to obtain the center coordinate position of the object, that is, the center O of the object is (X0, Y0, Z0+h / 2); then, the coordinate value on the X-axis in the center coordinate position is added or subtracted with half the length value of the object to obtain the coordinate positions of the centers of the two side surfaces of the object, that is, the coordinate position of the center of the left surface is (X0-l / 2, Y0, Z0-h / 2), and the coordinate position of the center of the right surface is (X0+l / 2, Y0, Z0-h / 2).

[0111] For another example, take a vertex angle of the upper surface as an example, Figure 5 As shown in (b), assuming that the vertex is the left rear vertex, its three-dimensional coordinates are (X1, Y1, Z1), so for the coordinate position of the center of each side surface, we have:

[0112] According to the coordinate position of the first vertex of the upper surface, the coordinate values ​​on the X-axis, Y-axis and Z-axis are added with half the length, half the width and half the height of the object, respectively, to obtain the center coordinate position of the object, which is (X1+l / 2, Y1+w / 2, Z1+h / 2); further, the coordinate value on the X-axis in the center coordinate position is added or subtracted with half the length of the object, respectively, to obtain the coordinate positions of the centers of the two side surfaces of the object, namely, the coordinate position of the center of the left surface is (X1, Y1+w / 2, Z1+h / 2), and the coordinate position of the center of the right surface is (X1+l, Y1+w / 2, Z1+h / 2). The above-mentioned vertex angle can also be one of the other three vertex angles, which is only an example here.

[0113] It can be understood that when the selected object postures are different, the calculation process of the corresponding target postures will be different, so they will not be listed here.

[0114] S130, according to the target posture and the actual posture of the ends of the arms fed back in real time, controlling the ends of the arms to move smoothly to the grasping points through a closed-loop feedback controller.

[0115] In the present application, the closed-loop feedback control is used to implement the visual servo control of the robot, that is, the position of the object is monitored in real time by the camera 431 on the robot, just like equipping the robot with "eyes". When the robot is ready to grab the box, the "eyes" will tell the end of the robot's arms where to move, and constantly adjust the end position, that is, move with the object to be moved, to ensure that the object grabbing point is accurately reached.

[0116] Generally, when the robot's two arms are ready to grasp an object, the palms of the two arms should face the two side surfaces of the object and the fingers should be arranged horizontally. For example, according to the target posture of reaching the grasping point, it is assumed that the robot's two arms are initially with the fingers drooping toward the ground, such as Figure 1 As shown, in order to grasp an object, the fingers need to be controlled to rotate 90 degrees upward around the palm, with the palm facing the two side surfaces of the object, and then move to the center of the corresponding side surfaces respectively.

[0117] Wherein, each of the two robotic arms is provided with a closed-loop feedback controller, and each closed-loop feedback controller is used to input the motion state of the corresponding robotic arm end, such as displacement, speed, etc., and output the speed control amount of the corresponding robotic arm end. For example, the closed-loop feedback control can be, but is not limited to, a PID (proportional-integral-differential) controller, a PI controller, or a PD controller, etc.

[0118] In one embodiment, in step S130, in order to allow the end of the robot arm to smoothly track the grasping point of the object, exemplary, a corresponding closed-loop feedback controller can be designed to control the end to approach the grasping point of the object by reducing the deviation between the position of the grasping point of the object and the actual position of the end, for example, Figure 6 As shown, the control process includes:

[0119] S310, according to the error between the target posture corresponding to the end of each robotic arm and the actual posture fed back in real time, the closed-loop feedback controller corresponding to each robotic arm is used to calculate the first expected speed of the end of each robotic arm in real time. For example, taking the PID controller as an example, the expression of the closed-loop feedback controller is:

[0120]

[0121] In the formula, v represents the first expected velocity of the terminal, and e is the error between the target posture and the actual posture; is the first-order derivative of e; t is the length of a control cycle, K p , K d and K i All are preset coefficients.

[0122] S320, calculating the first driving control quantity of the robot's whole body joint motors in real time according to the first desired speed of each end of the robot arm through the Jacobian matrix.

[0123] S330, driving each joint motor according to the first drive control amount until the ends of the robot arms move smoothly to the grasping point.

[0124] The Jacobian matrix is ​​used to describe the relationship between the speed of the execution end and the joint speed. In order to control the end motion, the speed control amount of the two robot arms is mapped from the operation space to the joint space through the Jacobian matrix of the robot body, and the driving change of the whole body joint motor in each control cycle can be obtained. Then, each joint is controlled to move accordingly according to the driving change of each joint motor issued, and each control cycle is repeated until the robot's two arms move smoothly to the grasping point of the object, such as Figure 7 shown.

[0125] It can be understood that step S130 is mainly used to realize the tracking control of the object grasping point by the end of the double arms. When the end reaches the grasping point smoothly, the end of the double arms begins to contact the object. Therefore, in the process of contact between the robot arms and the object, in order to reduce the impact during contact and make the robot move more smoothly with a certain flexibility to comply with external forces, a compliant controller can be designed to achieve compliant grasping.

[0126] S140, when the robot contacts the target object, the flexible controller controls the ends of the dual arms to flexibly grasp the object and place it at a designated location.

[0127] Among them, compliant control refers to treating the controlled object and the environment as a unified whole. Its control principle is to equate the contact process between the robot and the external environment to a "spring-mass-damping" model. Therefore, the relationship between the position or speed of the robot end and the force in the contact process can be established as follows:

[0128]

[0129] In the formula, M is the inertia matrix, which is used as the acceleration adjustment coefficient; B is the damping matrix, which is used as the speed adjustment coefficient; K is the stiffness matrix, which is used as the error adjustment coefficient. e is the error between the target posture of the end of the robot manipulator and the actual posture fed back in real time; and They are the first-order derivative and the second-order derivative of e, respectively. It can be understood that their physical meanings represent the velocity and acceleration corresponding to the posture respectively; F e It is the interaction force between the robot end and the external environment.

[0130] Among them, each of the two robotic arms is equipped with a flexible controller, each flexible controller is used to input the actual force between the end of the corresponding robotic arm and the object and the real-time feedback of the motion state of the end, such as posture, speed, etc., output the acceleration control amount of the corresponding robotic arm end, and then calculate the corresponding speed or posture control amount. Then, the Jacobian matrix is ​​used to calculate the drive control amount of the corresponding joint motor.

[0131] For example, in one embodiment, Figure 8 As shown, step S130 includes:

[0132] S410, according to the actual force fed back in real time by the end of each robotic arm and the error between the actual posture and the target posture, the expected acceleration of the end of each robotic arm is calculated in real time by using the flexible controller.

[0133] Exemplarily, the actual force exerted by the external object on the end is obtained, and then brought into the flexible controller to calculate the corresponding expected acceleration, and then the corresponding speed control amount is calculated. For example, the actual force of the real-time feedback can be collected by a six-dimensional force / torque sensor installed at the end of the corresponding mechanical arm, or it can be calculated by the external torque of the joint of the mechanical arm, etc., which is not limited here. It can be understood that the contact force in this embodiment is a broad concept, including three-dimensional force and three-dimensional torque, etc.

[0134] For example, in one embodiment, an admittance controller may be used to construct the flexible controller. Accordingly, the calculation expression for the desired acceleration may be expressed as:

[0135]

[0136] Where a represents the desired acceleration of the corresponding robot end. It can be understood that in addition to the admittance controller, an impedance controller can also be used to achieve compliant control, which is not limited here.

[0137] S420, calculating a corresponding second expected speed according to the expected acceleration of each end of the robot arm. For example, the expected speed can be calculated by time-integrating the expected acceleration.

[0138] S430, calculating the second driving control quantity of the robot's whole body joint motors in real time according to the second desired speed of each end of the robot arm through the Jacobian matrix.

[0139] S440, driving each joint motor according to the second drive control amount until the ends of the robot arms flexibly grasp the object and place it at a designated position.

[0140] Among them, the principles of steps S430 and S440 are the same as those of the above steps S320 and S330, that is, the calculated speed control amount needs to be distributed to each joint space through the robot Jacobian matrix.

[0141] It can be understood that by designing the above-mentioned compliant controller, the robot can slowly and steadily apply the force used to grasp the object when it contacts the object, avoiding damage to the object or the robot itself due to sudden collision. In this application, by combining visual servo control and flexible control, the robot can accurately perform handling tasks in complex environments.

[0142] Optionally, considering the transition stage from the robot just touching the grasping point to the end completely touching the object, as an optional solution, the closed-loop feedback controller in step S130 and the flexible controller in step S140 can simultaneously perform synchronous control on each end of the robot arm. Among them, the second expected speed calculated based on the expected acceleration output by the flexible controller can be directly superimposed on the first expected speed calculated by the closed-loop feedback controller to obtain the final expected speed, and then the joint space mapping is performed through the Jacobian matrix to obtain the drive control amount of the robot's whole body joints in each control cycle, so as to control the robot joints to perform corresponding actions.

[0143] As an optional solution, since the robot has similar movements of bending down and kneeling as a human during the process of carrying and placing the box, in order to effectively prevent the risk of the robot colliding with the surrounding environment, exemplary, during the entire process of the robot carrying the object, the method further includes:

[0144] According to the real-time feedback of the actual posture of the ends of both arms and the size information of the object, the kinematic estimation method is used to obtain in real time the distance between the object and the legs of the robot; then, the movement of the robot's waist is controlled according to the distance to prevent the object from colliding with the legs of the robot.

[0145] Exemplarily, the geometric model of the object is determined based on the length, width and height information of the object. For example, taking a rectangular cargo box as an example, the geometric model is a rectangular frame, and the center of gravity of the object can be determined (here it overlaps with the center O of the object). The actual posture of the ends of the arms can be acquired in real time by the corresponding posture sensors. Then, based on the kinematic principle of the robot, the leg posture of the robot can be estimated. For example, if the posture of the knee is selected, after the posture of the robot's leg knee is acquired, the straight-line distance between the robot's leg knee and the center of gravity of the object can be calculated as the distance between the object and the robot's leg, denoted as d. As an optional scheme, combined with the size information of the object, such as Fig. 9 As shown, the vertical distance between the side of the object close to the robot and the knee of the robot's leg can also be calculated as the distance between the object and the robot's leg.

[0146] It can be understood that the distances obtained by the above two calculation methods can be used to control the waist of the robot. For example, in one embodiment, the movement of the robot waist can be controlled by the change in the distance (Δd). For example, when the change in the distance shows a decreasing trend, it indicates that the robot legs are moving away from the object, and the robot waist can be controlled to move in the direction of approaching the object; conversely, when the change in the distance shows an increasing trend, it indicates that the robot legs are approaching the object, and the robot waist is controlled to move in the direction of moving away from the object.

[0147] It can be understood that, since the Jacobian matrix of the robot's entire body joints is used, the speed control amount of the waist can be directly superimposed on the speed generated based on the Jacobian matrix in the entire control process. In this embodiment, by adjusting the movement speed of the waist back and forth, it can be ensured that the robot can achieve obstacle avoidance movement of the entire body when carrying objects.

[0148] As another optional solution, when there are multiple objects that need to be moved, for example, multiple cargo boxes stacked together or placed side by side, if the robot recognizes that multiple objects need to be moved through the image data returned by the vision, it will generate multiple task queues based on the recognized object posture and overall size data, and then end a task each time an object is moved until all objects are moved.

[0149] In addition, as another optional solution, when multiple objects need to be placed, during the operation of placing the objects at the specified positions by the robot, the following steps are also included:

[0150] Determine whether to perform a stacking operation. The stacking operation is mainly for the scene of object placement. For example, in order to reasonably use space resources, usually, multiple objects are stacked in the height direction to reduce the occupied area on the ground.

[0151] Exemplarily, when it is determined to perform a stacking operation, the posture of the robot when placing the current object can be adjusted so that the placement posture of the object is aligned with the posture of the placed objects, such as Fig.10 Otherwise, when it is determined that the stacking operation is not to be performed, the posture of the robot when placing the current object can be adjusted so that the object is placed parallel to the position of the robot, that is, parallel to the robot and facing it, as shown in FIG. Fig.10 as shown in (b).

[0152] The humanoid robot carrying control method of the embodiment of the present application guides the robot to generate corresponding whole-body movements through the upper-layer carrying and placement strategies, while the lower layer corrects the robot's movement during carrying through real-time visual feedback based on camera 431, and also takes into account the anti-collision problem of the legs. This can not only ensure that the corresponding grasping accuracy is improved during carrying, but also ensure that the robot's carrying movement is adjusted in real time to prevent collisions. Since the control frequency of the lower-layer controllers such as the closed-loop feedback controller and the compliant controller is very high, the efficiency of the robot's real-time adjustment is also guaranteed.

[0153] Fig.11 A schematic structural diagram of a humanoid robot transport control device 200 according to an embodiment of the present application is shown. Exemplarily, the humanoid robot transport control device 200 includes:

[0154] The posture acquisition module 210 is used to capture an image containing the object to be transported through the camera 431 to obtain the real-time posture of the object;

[0155] A posture conversion module 220 is used to calculate the target posture of the robot's double-arm ends reaching the object grasping point in real time according to the object posture;

[0156] A visual servo module 230 is used to control the end of the double arm to move smoothly to the grasping point through a closed-loop feedback controller according to the target posture and the actual posture of the end of the double arm fed back in real time;

[0157] The compliance control module 240 is used to control the ends of the two arms to flexibly grasp the object and place it at a designated position through a compliance controller when the robot is in contact with the object.

[0158] As an optional solution, during the process of the robot carrying an object, the humanoid robot carrying control device 200 further includes:

[0159] The waist control module 250 is used to obtain the distance between the object and the robot's legs in real time by using a kinematic estimation method based on the real-time feedback of the actual position of the ends of the two arms and the size information of the object; and control the movement of the robot's waist based on the distance so that the object does not collide with the robot's legs.

[0160] As an optional solution, during the operation of placing the robot at a designated position, the humanoid robot transport control device 200 further includes:

[0161] The stacking control module 260 is used to determine whether to perform a stacking operation; if it is determined to perform a stacking operation, the posture of the robot when placing the object is adjusted so that the placement posture of the object is aligned with the posture of the placed objects; if it is determined not to perform a stacking operation, the posture of the robot when placing the object is adjusted so that the object is placed parallel to the position of the robot.

[0162] It can be understood that the device of this embodiment corresponds to the humanoid robot transport control method of the above embodiment, and the options in the above embodiment are also applicable to this embodiment, so they will not be described repeatedly here.

[0163] The present application also provides a computer-readable storage medium for storing the computer program used in the humanoid robot 100. For example, the computer-readable storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0164] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and structure diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in an alternative implementation, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the structure diagram and / or the flow diagram, and the combination of boxes in the structure diagram and / or the flow diagram, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0165] In addition, the functional modules or units in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0166] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a smart phone, a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0167] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A humanoid robot handling control method, characterized in that: include: The camera is used to capture images of the object to be transported to obtain the real-time position and posture of the object; According to the object posture, the target posture of the robot's double-arm ends reaching the object grasping point is calculated in real time; According to the target posture and the actual posture of the ends of the arms fed back in real time, the ends of the arms are controlled to move smoothly to the grasping points through a closed-loop feedback controller; When the robot contacts the object, the flexible controller controls the ends of the dual arms to flexibly grasp the object and place it at a designated position.

2. The humanoid robot handling control method according to claim 1, characterized in that: The entire process of the robot carrying objects also includes: According to the real-time feedback of the actual position and posture of the ends of the two arms and the size information of the object, a kinematic estimation method is used to obtain in real time the distance between the object and the legs of the robot; The movement of the robot's waist is controlled according to the distance so that the object does not collide with the robot's legs.

3. The humanoid robot handling control method according to claim 2, characterized in that: The step of controlling the movement of the robot's waist according to the distance comprises: Control the movement of the robot's waist according to the change in the distance between adjacent control cycles; When the change amount decreases, the waist of the robot is controlled to move in a direction approaching the object; when the change amount increases, the waist of the robot is controlled to move in a direction away from the object.

4. The humanoid robot handling control method according to claim 1, characterized in that: The process of performing the operation of placing the specified position also includes: Determine whether to perform a stacking operation; When it is determined to perform a stacking operation, adjusting the posture of the robot when placing the object so that the placement posture of the object is aligned with the posture of the placed object; When it is determined that the stacking operation is not to be performed, the posture of the robot when placing the object is adjusted so that the object is placed parallel to the position of the robot.

5. The humanoid robot handling control method according to claim 1, characterized in that: The method of collecting an image containing the object to be transported by a camera to obtain a real-time position and posture of the object includes: Using a camera on the robot to capture an image containing the object to be transported, extracting object features from the image to obtain size information of the object and two-dimensional coordinates of the object in the image; Performing coordinate transformation on the two-dimensional coordinates to obtain a first posture and a first three-dimensional coordinate of the object in the camera coordinate system of the robot; The conversion matrix between the base coordinate system of the robot and the camera coordinate system is used to respectively calculate the second posture and the second three-dimensional coordinate corresponding to the first posture and the first three-dimensional coordinate when converted to the base coordinate system, so as to obtain the real-time object posture.

6. The humanoid robot transport control method according to claim 1, characterized in that: The object posture includes the coordinate position of one of the vertex angles or the center of the target surface of the object; the object grasping point is selected as the center of the two side surfaces of the object perpendicular to the horizontal direction; The real-time calculation of the target posture of the robot's double-arm ends reaching the object grasping point according to the object posture comprises: According to the coordinate position of the vertex angle or the center of the target surface and the size information of the object, the position of the center of each side surface of the object is calculated respectively to serve as the position of the object grasping point.

7. The humanoid robot handling control method according to claim 1, characterized in that: According to the target posture and the actual posture of the ends of the arms fed back in real time, controlling the ends of the arms to move smoothly to the grasping points through a closed-loop feedback controller includes: According to the error between the target posture corresponding to the end of each robotic arm and the actual posture fed back in real time, a closed-loop feedback controller corresponding to each robotic arm is used to calculate in real time the first expected speed of the end of each robotic arm; Calculate the first driving control amount of the robot's whole body joint motor in real time through the Jacobian matrix according to the first desired speed of each end of the robot arm; Each joint motor is driven according to the first drive control amount until the ends of the robot arms move smoothly to the grasping point.

8. The humanoid robot transport control method according to claim 7, characterized in that: The expression of the closed-loop feedback controller is: Wherein, v represents the first expected velocity of the terminal, and e is the error between the target posture and the actual posture; is the first-order derivative of e; t is the length of a control cycle, K p , K d and K i All are preset coefficients.

9. The humanoid robot handling control method according to claim 1, characterized in that: The method of controlling the ends of the two arms to flexibly grasp the object and place it at a designated position by using a flexible controller includes: According to the actual force fed back in real time by the end of each robotic arm and the error between the actual posture and the target posture, the desired acceleration of the end of each robotic arm is calculated in real time by using the flexible controller; Calculate a corresponding second expected speed according to the expected acceleration at the end of each robotic arm; According to the second desired speed at the end of each robot arm, the second driving control amount of the robot's whole body joint motor is calculated in real time through the Jacobian matrix; The motors of each joint are driven according to the second driving control amount until the ends of the robot's arms flexibly grasp the object and place it at a designated position.

10. The humanoid robot transport control method according to claim 9, characterized in that: The expression of the flexible controller is: Wherein, a represents the desired acceleration of the terminal, and e is the error between the target posture and the actual posture; is the first-order derivative of e; F e is the actual force, M is the inertia matrix, B is the damping matrix, and K is the stiffness matrix.

11. A humanoid robot handling control device, characterized in that: include: A posture acquisition module is used to capture images containing the object to be transported through a camera to obtain the real-time posture of the object; A posture conversion module is used to calculate the target posture of the robot's double-arm ends reaching the object grasping point in real time according to the object posture; A first control module is used to control the ends of the two arms to move smoothly to the grasping point through a closed-loop feedback controller according to the target posture and the actual posture of the ends of the two arms fed back in real time; The second control module is used to control the ends of the two arms to flexibly grasp the object and place it at a designated position through a flexible controller when the robot contacts the object.

12. A humanoid robot, characterized in that: The humanoid robot comprises: An execution assembly, including two robotic arms; The control system, including the camera and processor; Wherein, the camera is used to collect image information of the object to be transported in real time; The processor is used to obtain the posture of the object to be transported in real time through the image information, and calculate the target posture of the robot's two arms reaching the object grasping point in real time according to the object posture; according to the target posture and the actual posture of the two arms' ends fed back in real time, control the two arms' ends to move smoothly to the grasping point through a closed-loop feedback controller; and in the process of the robot contacting the object, control the two arms' ends to flexibly grasp the object and place it in a designated position through a flexible controller.

13. The humanoid robot according to claim 12, characterized in that: The control system further comprises a sensor unit, wherein the sensor unit comprises a motion sensor and a six-dimensional force sensor arranged at the end of each mechanical arm; The motion sensor is used to collect the actual motion state of the terminal in real time; The six-dimensional force sensor is used to collect the actual force between the end of the robotic arm and the object.

14. A computer-readable storage medium, characterized in that: It stores a computer program, and when the computer program is executed, it implements the humanoid robot transport control method according to any one of claims 1-10.

Images