Control method, system and storage medium based on biped robot door operation

Abstract

Disclosed are a control method, system, and storage medium based on bipedal robot door operation, and relate to the technical field of bipedal robot control. The control method includes: a door handle grasping step: an information collection and determination step; a first instruction is generated according to first real-time point cloud information and first real-time body information; a first lower limb balance adjustment step; the information collection and determination step includes: real-time collection of environmental information based on a preset frequency; real-time collection of body information of the bipedal robot based on a preset frequency; the first lower limb balance adjustment step includes: generating a first balance instruction according to first real-time posture information, first real-time point cloud information, and first real-time body information. The control method decouples the upper limb control and lower limb control of the bipedal robot, and is suitable for controlling the bipedal humanoid robot to operate the door.

Description

Technical Field

[0001] The present application relates to the technical field of bipedal robot control, and in particular to a control method, system, and storage medium based on bipedal robot door operation. Background Art

[0002] The current control method for the robot to perform the door opening action can be based on reinforcement learning training, and a database can be established for a wide range of application scenarios such as indoors and outdoors. The door opening operation is performed by a robotic arm with a wheeled mobile platform and a gripper end.

[0003] At present, the control method for the robot to open the door can also be to design a mobile double-arm shaking operation system through robot imitation learning, and the operator can control the whole body of the robot through a fixed base and a shaking movable double arm. During the operation of the robot, the real activity data can be collected through the collection device to predict the robot's action at the next moment.

[0004] However, the inventors of this application have found that since the bipedal robot is a complex high-degree-of-freedom system, the whole-body motion coupling is strong during the movement of the bipedal robot, and the upper limb motion and lower limb motion interfere with each other, which makes it difficult to stably control the bipedal robot during the movement. The current method of controlling the robot to perform the door opening action is not suitable for bipedal robots.

[0005] The contents of the background technology are merely technologies known to the public and do not necessarily represent the prior art in the art. Summary of the invention

[0006] The present application aims to provide a control method, system, and storage medium based on bipedal robot door operation to solve the problem that the current control method for the robot to perform door opening action is not suitable for bipedal robots.

[0007] According to one aspect of the present application, the present application provides a control method based on bipedal robot door operation. The bipedal robot includes upper limbs and lower limbs, and the upper limbs are provided with dexterous hands. The control method includes: a grasping door handle step: an information collection and determination step; through a preset upper limb control strategy, a first instruction is generated according to the first real-time point cloud information and the first real-time body information, so that the dexterous hand grasps the door handle according to the first instruction; a first lower limb balance adjustment step; the information collection and determination step includes: collecting environmental information in real time based on a preset frequency to determine the first real-time posture information of the door and the door handle within a preset period and the first real-time point cloud information of the door and the door handle within a preset period; collecting the body information of the bipedal robot in real time based on a preset frequency to determine the first real-time body information of the bipedal robot within a preset period; the first lower limb balance adjustment step includes: through a preset lower limb control strategy and a preset upper limb control strategy, a first balance instruction is generated according to the first real-time posture information, the first real-time point cloud information and the first real-time body information, so that the lower limb maintains balance according to the first balance instruction.

[0008] According to one aspect of the present application, the present application provides a control system based on bipedal robot door operation. The bipedal robot includes upper limbs and lower limbs, and the upper limbs are provided with dexterous hands. The control system is used to execute the method based on bipedal robot door operation as described above. The control system includes a door perception module, a proprioceptive perception module, an upper limb control module, a lower limb control module and a driving module; the door perception module, the proprioceptive perception module, the upper limb control module, the lower limb control module and the driving module execute the step of grasping the door handle, including: the door perception module and the proprioceptive perception module execute the steps of collecting and determining information; the upper limb control module generates a first instruction according to the first real-time point cloud information and the first real-time proprioceptive information by presetting the upper limb control strategy; the driving module drives the dexterous hand according to the first instruction, so that the dexterous hand grasps the door handle according to the first instruction; the upper limb control module, the lower limb control module and the driving module execute the first lower limb balance adjustment step. The above-mentioned door perception module and body perception module perform the steps of collecting and determining information; including: the door perception module collects environmental information in real time based on a preset frequency to determine the first real-time posture information of the door and the door handle within a preset period and the first real-time point cloud information of the door and the door handle within a preset period; the body perception module collects the body information of the biped robot in real time based on a preset frequency to determine the first real-time body information of the biped robot within a preset period. The above-mentioned upper limb control module, lower limb control module and drive module perform the first lower limb balance adjustment step including: the upper limb control module determines the first target speed and the first target posture of the biped robot according to the first real-time posture information through a preset upper limb control strategy; the lower limb control module generates a first balance instruction according to the first target speed, the first target posture, the first real-time point cloud information and the first real-time body information through a preset lower limb control strategy; the drive module drives the lower limb according to the first balance instruction so that the lower limb maintains balance according to the first balance instruction.

[0009] According to another aspect of the present application, the present application also provides a non-volatile computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the control method based on the bipedal robot door operation as described above can be implemented.

[0010] According to another aspect of the present application, the present application also provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, which, when the one or more programs are executed by one or more processors, enables the one or more processors to implement the control method based on the bipedal robot door operation as described above.

[0011] According to another aspect of the present application, the present application also provides a computer program product, including: a computer program stored on a computer-readable storage medium; the computer program includes program instructions, and when the program instructions are executed by the computer, the computer executes the control method based on the bipedal robot door operation as described above.

[0012] Beneficial Effects

[0013] The control method provided by the present application can generate a balance instruction by processing environmental information and the body information of the bipedal robot during the operation of the bipedal robot grasping the door handle, thereby maintaining the balance of the lower limbs and avoiding the collision between the bipedal robot and the door. The control method provided by the present application decouples the upper limb control and the lower limb control of the bipedal robot, reducing the difficulty of controlling the stability of the lower limbs. The present application can be applied to the process of controlling the bipedal humanoid robot to operate the door. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0015] Figure 1 A schematic diagram showing a flow chart of a control method according to an embodiment of the present application is shown;

[0016] Figure 2 A schematic diagram showing the process of step S230 according to an embodiment of the present application is shown;

[0017] Figure 3 Another schematic diagram of a control method according to an embodiment of the present application is shown;

[0018] Figure 4 A schematic diagram showing the process of step S800 according to an embodiment of the present application is shown;

[0019] Figure 5 A schematic diagram showing the process of step S830 according to an embodiment of the present application is shown;

[0020] Figure 6 A schematic diagram showing the process of step S900 according to an embodiment of the present application is shown;

[0021] Figure 7 Another schematic diagram of a control method according to an embodiment of the present application is shown;

[0022] Figure 8 A schematic diagram showing the process of step S100c according to an embodiment of the present application is shown;

[0023] Fig. 9 A schematic diagram showing the process of step S120c according to an embodiment of the present application is shown;

[0024] Fig.10 A schematic diagram showing the structure of a control system according to an embodiment of the present application is shown;

[0025] Fig.11 A schematic diagram showing the structure of an upper limb control module according to an embodiment of the present application is shown;

[0026] Fig.12 A schematic diagram showing the structure of an upper limb control module according to an embodiment of the present application is shown;

[0027] Fig.13 A schematic diagram of a door opening angle α according to an embodiment of the present application is shown;

[0028] Fig.14 A schematic diagram showing a biped robot grasping a door handle according to an embodiment of the present application;

[0029] Fig.15 Another schematic diagram showing a biped robot grasping a door handle according to an embodiment of the present application;

[0030] Fig.16 A schematic diagram showing a biped robot unlocking a door handle according to an embodiment of the present application;

[0031] Fig.17 Another schematic diagram showing a biped robot unlocking a door handle according to an embodiment of the present application;

[0032] Fig.18 A schematic diagram showing a biped robot opening a door according to an embodiment of the present application;

[0033] Fig.19 A schematic diagram of posture adjustment of a bipedal robot according to an embodiment of the present application is shown.

[0034] Reference numerals:

[0035] Control system 30; door sensing module 31; proprioception module 32; upper limb control module 33; lower limb control module 34; drive module 35; upper limb control strategy unit 331; point cloud processing unit 332; historical information processing unit 333; upper limb controller 334; lower limb control strategy unit 341, periodic feedforward signal generating unit 342; lower limb controller 343; door opening angle α. DETAILED DESCRIPTION

[0036] In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.

[0037] The terms "first", "second" and the like in the specification and claims of this application and the above drawings are used to distinguish different objects rather than to describe a specific order.

[0038] The following is a clear and complete description of the technical solution of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0039] In this application, real-time pose information refers to the real-time pose information of the door and the door handle, that is, the real-time pose information of the door and the real-time pose information of the door handle. In this application, real-time point cloud information refers to the real-time point cloud information of the door and the door handle, that is, the real-time point cloud information of the door and the real-time point cloud information of the door handle. In this application, real-time body information refers to the real-time body information of the bipedal robot.

[0040] The real-time position and posture information in the present application is information describing the relative position and relative posture of the door and the door handle. The real-time position and posture information within a preset period is information describing the relative position and relative posture of the door and the door handle within a preset period. During the movement of the bipedal robot, the real-time position and posture information within a preset period can be updated in real time. For example, the real-time position and posture information within a preset period may include the relative position of the door, the relative posture of the door, the relative position of the door handle, and the relative posture of the door handle within the preset period.

[0041] In the present application, the point cloud information may be data information describing the three-dimensional data of an object. For example, the point cloud information may include three-dimensional coordinate information, color information, and normal vector information, etc. The real-time point cloud information within a preset period may be three-dimensional data information describing the door and the door handle within a preset period. During the movement of the bipedal robot, the real-time point cloud information within a preset period may be updated in real time. For example, the real-time point cloud information within a preset period may include the three-dimensional coordinate information of the door, the color information of the door, the normal vector information of the door, the three-dimensional coordinate information of the door handle, the color information of the door handle, and the normal vector information of the door handle, etc. within the preset period.

[0042] In the present application, the environmental information may be parameter information of the environment in which the bipedal robot is located. For example, the bipedal robot may be in an indoor environment or an outdoor environment. The environmental information may include parameter information of a door and information of a door handle, such as the position of the door, the width of the door, the height of the door, the distance between the door and the bipedal robot, the position of the door handle, the height of the door handle, and the shape of the door handle.

[0043] In the present application, the real-time proprioceptive information within a preset period may be parameter information describing the motion state of the biped robot within the preset period. The real-time proprioceptive information may include real-time joint state information and real-time proprioceptive perception information.

[0044] In the present application, the real-time joint state information may be information describing the motion state of each joint of the biped robot within a preset period. For example, the real-time joint state information may include the motor position information of the joint, the speed information of the joint, and the output torque of the joint. The real-time joint state information may include the real-time state information of the upper limb joints and the real-time state information of the lower limb joints, to respectively describe the motor position information of the upper limb joints and the lower limb joints, the speed information of the joints, and the output torque of the joints.

[0045] In the present application, proprioceptive information may be state information describing the robot body within a preset period. For example, proprioceptive information may include inertial measurement unit information and force sensor information. Inertial unit measurement information may be state information of a biped robot through an inertial measurement unit. For example, inertial unit measurement information may include angular velocity, angular acceleration, angle, acceleration, etc. of the biped robot. Force sensor information may include the force of each joint and the torque of each joint, etc.

[0046] In the present application, the target speed may be the target speed information of the biped robot in the next preset cycle. The target posture may be the target relative position information and target relative posture information of the biped robot in the next preset cycle.

[0047] The biped robot includes an upper limb and a lower limb. The upper limb is provided with a dexterous hand. For example, the structure of the biped robot may include a robot body, a pair of lower limbs and a pair of upper limbs. The robot body may be the torso of the robot. The end of the lower limb is provided with a foot structure. The end of the upper limb is provided with a dexterous hand. The biped robot may be a biped humanoid robot.

[0048] like Fig.10 As shown, the control system 30 may include a door sensing module 31, a proprioceptive sensing module 32, an upper limb control module 33, a lower limb control module 34 and a driving module 35. The control system 30 may control the bipedal robot to operate the door based on the control method of the bipedal robot door operation.

[0049] Combine the following Fig.10 To describe the control method based on bipedal robot door operation provided by the present application.

[0050] For example, the door sensing module 31 may be disposed on the bipedal robot, or in the environment where the bipedal robot is located. The proprioceptive sensing module 32, the upper limb control module 33, the lower limb control module 34 and the driving module 35 may be disposed on the bipedal robot.

[0051] See also Figure 1 , the control method may include grasping the door handle step S200. Figure 1 , step S200 may include steps S210 to S230.

[0052] Step S210 is a step of collecting and determining information. For example, in step S210, the door perception module 31 and the body perception module 32 perform the steps of collecting and determining information. Figure 1 , step S210 may include step S211 and step S212.

[0053] In step S211, environmental information is collected in real time based on a preset frequency to determine first real-time position and posture information within a preset period and first real-time point cloud information within a preset period.

[0054] For example, the preset frequency may be a frequency for collecting environmental information. The preset frequency may be set according to the walking parameters (e.g., step frequency, step length) of the bipedal robot. The preset period may be a time period for collecting environmental information. The preset period may be set according to the walking parameters (e.g., step frequency, step length) of the bipedal robot. For example, the preset period may be set to 2 seconds to 4 seconds. The preset frequency may be set corresponding to the preset period. For example, the preset period may be set to 2 seconds, and the preset frequency may be set to collect environmental information or robot body information once every 2 seconds.

[0055] For example, in step S211, the door sensing module 31 collects environmental information in real time based on a preset frequency to determine the above-mentioned first real-time posture information and first real-time point cloud information.

[0056] The first real-time posture information, the first real-time point cloud information, the first real-time body information, the first real-time joint state information and the first real-time body perception information within the preset period all refer to the corresponding information when the biped robot is in the process of grasping the door handle. The first target speed and the first target posture all refer to the corresponding information when the biped robot is in the process of grasping the door handle.

[0057] For example, the door sensing module 31 can collect environmental information in real time through a sensing device. The sensing device can be a laser scanning device, a depth camera, or a voice acquisition device. The sensing device can be installed on the bipedal robot body or in the environment where the bipedal robot is located. The door sensing module 31 can determine the real-time posture information of a preset period through methods such as PointNet, PointNet++, and YOLO (You Only Look Once) combined with Perspective-n-Point (PnP) algorithm based on one or more of the acquired laser information, visual information, depth information, and voice information of the door and door handle.

[0058] The door sensing module 31 can also be electrically connected to an external sensor. The external sensor can collect real-time posture information of a preset period. The door sensing module 31 can determine the real-time posture information of a preset period by receiving an electrical signal or a voice signal. For example, the door sensing module 31 can collect environmental information in real time through the above-mentioned sensing device, and determine the first real-time posture information of the door and the door handle within the preset period through the above-mentioned method. Alternatively, the door sensing module 31 determines the first real-time posture information by electrically connecting to an external sensor.

[0059] The door sensing module 31 can scan the environment information by a laser scanning device or an optical multi-dimensional scanning device. The door sensing module 31 can determine the real-time point cloud information within a preset period by a visual computing algorithm, etc. For example, the door sensing module 31 determines the first real-time point cloud information within a preset period.

[0060] In step S212, the body information of the biped robot is collected in real time based on a preset frequency to determine first real-time body information within a preset period.

[0061] For example, the preset frequency may also be the frequency of collecting the body information of the bipedal robot. The preset period may be the time period for collecting the body information of the bipedal robot. The body information of the bipedal robot may be parameter information describing the motion state of the bipedal robot.

[0062] The proprioception module 32 can determine the real-time joint state information through the joint encoder. The joint encoder can be set at the upper limb joints and the lower limb joints.

[0063] For example, the proprioceptive perception module 32 may collect inertial measurement unit information through an inertial measurement unit. The proprioceptive perception module 32 may determine force sensor information through a force sensor. The proprioceptive perception module 32 may pre-process the collected proprioceptive information of the biped robot (e.g., outlier processing, data standardization, filtering, etc.) to determine the proprioceptive perception information.

[0064] For example, in step S212, the proprioceptive sensing module 32 collects proprioceptive information of the biped robot in real time based on a preset frequency to determine first real-time proprioceptive information within a preset period.

[0065] For example, the proprioception module 32 may determine the first real-time joint state information through a joint encoder. The proprioception module 32 may determine the first real-time proprioception information through an inertial measurement unit and a force sensor.

[0066] After step S210, step S220 may be performed simultaneously with step S230.

[0067] In step S220, a first instruction is generated according to the first real-time point cloud information and the first real-time body information by presetting the upper limb control strategy, so that the dexterous hand grasps the door handle according to the first instruction.

[0068] For example, the preset upper limb control strategy may be a model algorithm for controlling the motion state of the upper limb joints. For example, the preset upper limb control strategy may be a neural network strategy.

[0069] For example, the first instruction may be instruction information for controlling the dexterous hands of the upper limbs of the bipedal robot to grasp the door handle when the bipedal robot is in the process of grasping the door handle.

[0070] For example, in step S220, the upper limb control module 33 generates a first instruction, and sends the first instruction to the driving module 35. The driving module 35 drives the dexterous hand in response to the first instruction, so that the dexterous hand grasps the door handle according to the first instruction. The driving module 35 may be a driving motor. The driving module 35 may drive the dexterous hand according to the first instruction, and the dexterous hand gradually approaches the door handle and performs a grasping operation on the door handle.

[0071] Step S230 is a first lower limb balance adjustment step. For example, in step S230, the upper limb control module 33 and the lower limb control module 34 perform the first lower limb balance adjustment step.

[0072] See also Figure 2 , step S230 may include step S231.

[0073] In step S231, a first balance instruction is generated according to the first real-time posture information, the first real-time point cloud information and the first real-time body information through a preset lower limb control strategy and a preset upper limb control strategy, so that the lower limb maintains balance according to the first balance instruction.

[0074] The preset lower limb control strategy may be a model algorithm for controlling the motion state of the lower limb joints. For example, the preset lower limb control strategy may be a neural network strategy.

[0075] For example, in step S231, the upper limb control module 33 determines the first target speed and the first target posture of the biped robot according to the first real-time posture information by presetting the upper limb control strategy. The lower limb control module 34 generates a first balance instruction according to the first target speed, the first target posture, the first real-time point cloud information and the first real-time body information by presetting the lower limb control strategy. The lower limb control module 34 sends the first balance instruction to the driving module 35. The driving module 35 drives the lower limb according to the first balance instruction so that the lower limb maintains balance according to the first balance instruction.

[0076] For example, the first balance instruction may be instruction information for controlling the lower limbs of the bipedal robot to maintain balance when the bipedal robot is in the process of grasping a door handle.

[0077] For example, in step S200, the control system 30 may continue to execute the door handle grasping step until the dexterous hand of the biped robot completes the door handle grasping step (eg, the dexterous hand completely grasps the door handle, such as Fig.14 and Fig.15 as shown).

[0078] Through the above embodiments, the control method provided by the present application can generate a balance instruction by processing environmental information and the body information of the bipedal robot during the operation of the bipedal robot grasping the door handle, thereby maintaining the balance of the lower limbs and avoiding the collision between the bipedal robot and the door. The control method provided by the present application decouples the upper limb control and the lower limb control of the bipedal robot, reducing the difficulty of controlling the stability of the lower limbs. The present application can be applied to the process of controlling the bipedal humanoid robot to operate the door.

[0079] Alternatively, see Figure 3 The control method may further include steps S400 to S800.

[0080] After step S200, the biped robot has completed the door handle grasping operation process, and the dexterous hand grasps the door handle.

[0081] The second real-time posture information, the second real-time point cloud information, and the second real-time body information within the preset period all refer to the information corresponding to the situation where the bipedal robot has completed the door handle grasping operation process and the dexterous hand grasps the door handle.

[0082] In step S400, the information collection and determination step is performed again to determine the second real-time point cloud information within the preset period. For example, in step S400, the door sensing module 31 performs the information collection and determination step to determine the second real-time point cloud information within the preset period.

[0083] In step S400, the door perception module 31 also performs the information collection and determination step to determine the second real-time posture information within a preset period. The body perception module 32 can also perform the information collection and determination step to determine the second real-time body information within a preset period.

[0084] In step S600, it is determined whether the door handle needs to be unlocked based on the second real-time point cloud information.

[0085] For example, there are many ways to unlock a door handle. For example, a spherical door handle needs to be rotated to unlock, and a rectangular door handle needs to be pressed to unlock. There are also some door handles that do not need to be unlocked, and these door handles are only for decoration or to provide support points. The upper limb control module 33 can pre-store different types of door handles and corresponding unlocking methods.

[0086] For example, in step S600, the upper limb control module 33 may process the second real-time point cloud information through a point cloud information processor to determine the type of the door handle being grasped and the corresponding unlocking method.

[0087] If the judgment result of step S600 is yes, then step S800 is entered. Step S800 is a step of unlocking the door handle. For example, the door sensing module 31, the body sensing module 32, the upper limb control module 33, the lower limb control module 34 and the driving module 35 perform the step of unlocking the door handle. At this time, the biped robot is in the stage of unlocking the door handle operation process.

[0088] See also Figure 4 , step S800 may include step S810-step S830.

[0089] In step S810, the information collection and determination steps are performed again to determine third real-time pose information, third real-time point cloud information and third real-time body information within a preset period.

[0090] The third real-time posture information, the third real-time point cloud information, the third real-time body information, the third real-time joint state information and the third real-time body perception information within the preset period all refer to the information corresponding to the case where the biped robot is in the unlocking door handle operation stage. The third target speed and the third target posture all refer to the information corresponding to the case where the biped robot is in the unlocking door handle operation stage.

[0091] For example, in step S810, the door perception module 31 and the body perception module 32 perform the steps of collecting and determining information, which will not be repeated here.

[0092] For example, the proprioceptive sensing module 32 collects the proprioceptive information of the biped robot in real time based on a preset frequency to determine the third real-time proprioceptive information within a preset period. The third real-time proprioceptive information may include third real-time joint state information and third real-time proprioceptive sensing information. The proprioceptive sensing module 32 determines the third real-time proprioceptive information in the same manner as the first real-time proprioceptive information, so it is not repeated here.

[0093] After step S810, step S820 may be performed simultaneously with step S830.

[0094] In step S820, a second instruction is generated according to the third real-time point cloud information and the third real-time body information by presetting the upper limb control strategy, so that the dexterous hand unlocks the door handle according to the second instruction.

[0095] For example, the second instruction may be instruction information for controlling the dexterous hands of the upper limbs of the bipedal robot to perform an action of unlocking a door handle when the bipedal robot is in the process of unlocking a door handle.

[0096] For example, in step S820, the upper limb control module 33 generates a second instruction according to the third real-time point cloud information and the third real-time body information by presetting the upper limb control strategy, and sends the second instruction to the driving module 35. The driving module 35 drives the dexterous hand in response to the second instruction so that the dexterous hand unlocks the door handle according to the second instruction.

[0097] Step S830 is a second lower limb balance adjustment step. For example, in step S830, the upper limb control module 33 and the lower limb control module 34 perform the second lower limb balance adjustment step.

[0098] See also Figure 5 , step S830 may include step S831.

[0099] In step S831, a second balance instruction is generated according to the third real-time posture information, the third real-time point cloud information and the third real-time body information through a preset lower limb control strategy and a preset upper limb control strategy, so that the lower limbs maintain balance according to the second balance instruction.

[0100] For example, in step S831, the upper limb control module 33 determines the third target speed and the third target posture of the biped robot according to the third real-time posture information by presetting the upper limb control strategy. The lower limb control module 34 generates a second balance instruction according to the third target speed, the third target posture, the third real-time point cloud information and the third real-time body information by presetting the lower limb control strategy. The lower limb control module 34 sends the second balance instruction to the drive module 35. The drive module 35 drives the lower limb according to the second balance instruction so that the lower limb maintains balance according to the second balance instruction.

[0101] For example, the second balance instruction may be instruction information for controlling the lower limbs of the bipedal robot to maintain balance when the bipedal robot is in the process of unlocking the door handle.

[0102] For example, in step S800, the control system 30 may continue to execute the unlocking door handle step until the dexterous hand of the biped robot completes the unlocking door handle step (for example, the dexterous hand completely unlocks the door handle, such as Fig.16 and Fig.17 As shown, the door reaches a state where it can be opened).

[0103] Through the above embodiments, the control method provided by the present application can generate a balance instruction by processing environmental information and the body information of the bipedal robot during the process of unlocking the door handle, thereby maintaining the balance of the lower limbs and avoiding the collision between the bipedal robot and the door. The control method provided by the present application can be applied to the process of controlling a bipedal humanoid robot to operate a door.

[0104] Alternatively, see Figure 3 The control method may further include step S900. After step S800, step S900 is executed. Step S900 is a door opening step.

[0105] See also Figure 6 , step S900 may include step S910-step S930.

[0106] In step S910, the information collection and determination steps are performed again to determine fourth real-time pose information, fourth real-time point cloud information and fourth real-time body information within a preset period.

[0107] The fourth real-time posture information, the fourth real-time point cloud information, the fourth real-time body information, the fourth real-time joint state information and the fourth real-time body perception information within the preset period all refer to the corresponding information when the biped robot is in the door opening operation process stage. The fourth target speed and the fourth target posture refer to the corresponding information when the biped robot is in the door opening operation process stage.

[0108] For example, in step S910, the door perception module 31 and the body perception module 32 perform the steps of collecting and determining information. The door perception module 31 determines the fourth real-time posture information in the same manner as the first real-time posture information, and the door perception module 31 determines the fourth real-time point cloud information in the same manner as the first real-time point cloud information, so it is not repeated here.

[0109] The proprioceptive sensing module 32 collects proprioceptive information of the biped robot in real time based on a preset frequency to determine fourth real-time proprioceptive information of the biped robot within a preset period. The collection process is the same as the method of determining the first real-time proprioceptive information, which will not be repeated here.

[0110] After step S910, step S920 and step S930 may be performed simultaneously.

[0111] In step S920, a third instruction is generated according to the fourth real-time posture information, the fourth real-time point cloud information and the fourth real-time body information through a preset upper limb control strategy and a preset lower limb control strategy, so that the lower limbs walk according to the third instruction.

[0112] For example, the third instruction may be instruction information for controlling the lower limbs of the biped robot to walk when the biped robot is in the door opening operation process.

[0113] For example, in step S920, the upper limb control module 33 determines the fourth target speed and the fourth target posture of the biped robot according to the fourth real-time posture information by presetting the upper limb control strategy. The lower limb control module 34 generates a third instruction according to the fourth target speed, the fourth target posture and the fourth real-time body information by presetting the lower limb control strategy. The lower limb control module 34 sends the third instruction to the drive module 35, and the drive module 35 drives the lower limb according to the third instruction so that the lower limb walks according to the third instruction. The drive module 35 can drive the lower limb to walk in response to the second instruction, and the lower limb can walk forward, backward or turn.

[0114] In step S930, by presetting the upper limb control strategy, a fourth instruction is generated according to the fourth real-time posture information, the fourth real-time point cloud information and the fourth real-time body information, so that the upper limb performs the door opening action according to the fourth instruction.

[0115] For example, the fourth instruction may be instruction information for controlling the upper limbs of the biped robot to open a door when the biped robot is in the door opening operation process.

[0116] For example, in step S930, the upper limb control module 33 generates a fourth instruction according to the fourth real-time posture information, the fourth real-time point cloud information and the fourth real-time body information by a preset upper limb control strategy. The upper limb control module 33 sends the fourth instruction to the drive module 35. The drive module 35 drives the upper limb according to the fourth instruction so that the upper limb performs the door opening action according to the fourth instruction. The drive module 35 can drive the upper limb to open the door according to the fourth instruction. For example, an upper limb can push or pull the door, or the dexterous hand of an upper limb can push or pull the door handle.

[0117] For example, in step S900, the control system 30 may continue to execute the door opening step until the biped robot opens and passes through the door (e.g. Fig.18 or, until the door opening angle α meets the preset door opening angle condition.

[0118] See also Fig.13 , the door opening angle α may be the angle between the position of the door in the door opening step and the position of the door in the door closing state. The preset door opening angle condition may be the door opening angle at which the bipedal robot can pass through the door body. The preset door opening angle condition may be set according to the volume of the bipedal robot, and this application does not impose any limitation thereto.

[0119] If the judgment result of step S600 is no, that is, the upper limb control module 33 determines that the door handle does not need to be unlocked, step S900 is directly executed and the execution process is not repeated.

[0120] Through the above-mentioned embodiments, the control method provided in the present application decouples the upper limb control and lower limb control of the bipedal robot during the door opening operation of the bipedal robot, thereby reducing the difficulty of controlling the stability of the lower limbs.

[0121] In some scenarios, the relative position and relative posture of the biped robot may not be suitable for door operation. Therefore, it is necessary to determine whether the biped robot is suitable for door operation.

[0122] Alternatively, see Figure 7 The control method may also include steps S100a to S00c before step S200.

[0123] In step S100a, the information collection and determination steps are performed again to determine the initial real-time posture information within a preset period. The initial real-time posture information may be information describing the relative position and relative posture of the door and the door handle within a preset period when the biped robot is about to operate the door.

[0124] For example, in step S100a, the door sensing module 31 collects environmental information in real time based on a preset frequency to determine the initial real-time position information of the door and the door handle within a preset period. The door sensing module 31 determines the initial real-time position information in the same manner as the first real-time position information, so it is not repeated here.

[0125] In step S100b, it is determined whether the bipedal robot meets the preset door opening condition based on the preset position condition and preset posture condition of the bipedal robot and the initial real-time posture information.

[0126] For example, the preset position condition may be a preset position condition for the bipedal robot to perform door operation. For example, the preset position condition may include a relative distance condition between the bipedal robot and the door. The preset position condition may be set according to parameters such as the length of the upper limbs and the length of the lower limbs of the bipedal robot, and the present application does not limit this.

[0127] The preset posture condition may be a posture condition of a door or door handle that the biped robot can operate. For example, the preset posture condition may be that the front of the biped robot faces the door or door handle. For example, when the biped robot is far away from the door and cannot touch the door or door handle, the relative distance needs to be reduced first. Or when the back (or side) of the biped robot faces the door or door handle, the posture of the biped robot needs to be adjusted first.

[0128] The preset door opening condition may be a preset condition for the biped robot to be able to operate the door, and the preset door opening condition may include a relative distance preset condition and a relative posture preset condition between the biped robot and the door. For example, the relative distance preset condition may be a distance at which the dexterous hand of the biped robot can touch the door or the door handle. The relative posture preset condition may be that the biped robot is facing the door or the door handle.

[0129] For example, in step S100b, the upper limb control module 33 can determine the relative distance and relative posture (for example, the bipedal robot faces the door frontally, or faces the door backward, or faces the door sideways) between the bipedal robot and the door (or door handle) based on the initial real-time posture information, and thus determine whether the bipedal robot meets the preset door opening conditions based on the preset position conditions.

[0130] If the judgment result of S100b is yes, then step S200 is executed; if the judgment result is no, then step S100c is executed. Step S100c is a posture adjustment step.

[0131] See also Figure 8 , step S100c may include steps S110c to S120c. At this time, the biped robot is in the posture adjustment operation process stage.

[0132] In step S110c, the information collection and determination steps are performed again to determine fifth real-time pose information, fifth real-time point cloud information and fifth real-time body information within a preset period.

[0133] The fifth real-time posture information, the fifth real-time point cloud information, the fifth real-time body information, the fifth real-time joint state information and the fifth real-time body perception information within the preset period all refer to the information when the biped robot is in the posture adjustment operation process stage. The fifth target speed, the fifth target posture, the fifth periodic feedforward signal and the fifth feedback signal all refer to the information when the biped robot is in the posture adjustment operation process stage.

[0134] For example, in step S110c, the door sensing module 31 collects environmental information in real time based on a preset frequency to determine fifth real-time posture information and fifth real-time point cloud information, which will not be repeated here.

[0135] For example, the proprioceptive sensing module 32 collects proprioceptive information of the biped robot in real time based on a preset frequency to determine fifth real-time proprioceptive information within a preset period, which will not be described in detail here.

[0136] In step S120c, through the preset upper limb control strategy and the preset lower limb control strategy, the fifth instruction is generated according to the fifth real-time posture information, the fifth real-time point cloud information, the fifth real-time body information and the preset kinematic model of the bipedal robot, so that the bipedal robot reaches the preset position and forms the preset posture according to the fifth instruction.

[0137] For example, the fifth instruction may be a control instruction for controlling the posture adjustment of the lower limbs of the bipedal robot when the bipedal robot is in the posture adjustment operation process stage.

[0138] The preset position may be a position where the biped robot can operate the door. For example, the preset position may be a position where the dexterous hand can touch the door handle. The preset posture may be a posture where the biped robot can operate the door. For example, the preset posture may be a posture where the front of the biped robot faces the door.

[0139] For example, the preset kinematic model of the bipedal robot is a kinematic relationship during the movement of the bipedal robot determined according to the structural parameters of the bipedal robot (such as the degree of freedom of the legs, the driving mode, the size of each part and the range of motion, etc.). The preset kinematic model can be preset according to the bipedal robot being controlled.

[0140] For example, in step S120c, the upper limb control module 33 determines the fifth target speed and the fifth target posture of the bipedal robot according to the fifth real-time posture information through the preset upper limb control strategy. The lower limb control module 34 generates a fifth instruction according to the fifth target speed, the fifth target posture, the fifth real-time posture information, the fifth real-time point cloud information, the fifth real-time body information and the preset kinematic model of the bipedal robot through the preset lower limb control strategy. The lower limb control module 34 sends the fifth instruction to the driving module 35. The driving module 35 drives the lower limbs according to the fifth instruction, so that the bipedal robot reaches the preset position and forms a preset posture according to the fifth instruction. The driving module 35 can drive the lower limb shape according to the fifth instruction, and the lower limb drives the bipedal robot to gradually approach the door and the door handle (such as Fig.19 as shown).

[0141] Through the above-mentioned embodiments, the control method provided in the present application can decouple the upper limb control and lower limb control of the bipedal robot when the bipedal robot is in the process of posture adjustment operation (before the door operation is performed), thereby reducing the difficulty of lower limb stability control.

[0142] Alternatively, see Fig.11 The upper limb control module 33 includes an upper limb control strategy unit 331. Fig.12 The lower limb control module 34 includes a lower limb control strategy unit 341, a periodic feedforward signal generating unit 342 and a lower limb controller 343.

[0143] The fifth real-time upper limb joint state information and the fifth real-time lower limb joint state information both refer to information when the biped robot is in the posture adjustment operation process stage.

[0144] See also Fig. 9 , step S120c may include steps S121c to S125c.

[0145] In step S121c, a fifth target speed and a fifth target posture of the biped robot are determined according to the fifth real-time posture information by presetting the upper limb control strategy.

[0146] For example, in step S121c, the upper limb control strategy unit 331 determines the fifth target speed and the fifth target posture according to the fifth real-time posture information by using a preset upper limb control strategy.

[0147] For example, the upper limb control strategy unit 331 is provided with a preset upper limb control strategy. The fifth target speed is not 0, indicating that the biped robot needs to perform a posture adjustment operation. The fifth target speed and the fifth target posture can guide the lower limb control module 34 (lower limb control strategy unit 341) to complete the control of the lower limb movement.

[0148] In step S122c, a fifth periodic feedforward signal is generated according to the fifth real-time posture information, the fifth real-time point cloud information, the fifth target speed, the fifth target posture and the preset kinematic model.

[0149] For example, the periodic feedforward signal may be a signal for instructing the lower limbs to walk, generated based on the real-time posture information, the real-time point cloud information, the target speed, the target posture, and the preset kinematic model of the biped robot. The periodic feedforward signal may reflect information such as whether the lower limbs need to move forward, backward, or stay in place.

[0150] The lower limb walking motion is periodic, so the feedforward signal is also periodic. The period of the periodic feedforward signal can be set according to the lower limb walking parameters (such as step frequency, step length). The periodic feedforward signal generating unit 342 can be a periodic feedforward signal generator. The periodic feedforward signal generator can be set based on a preset kinematic model.

[0151] The fifth periodic feedforward signal may be a signal instructing the lower limbs to walk, generated according to fifth real-time posture information, fifth real-time point cloud information, fifth target speed, fifth target posture and a preset kinematic model when the biped robot is in the posture adjustment operation process.

[0152] The periodic feedforward signal generating unit 342 can calculate the gait characteristics such as the step length and step frequency of the biped robot according to the speed mapping relationship based on the real-time posture information, the real-time point cloud information, the target speed, the target posture and the preset kinematic model. The periodic feedforward signal generating unit 342 can generate a periodic feedforward signal according to the gait characteristics by using Bezier curve planning, polynomial planning and the like to guide the robot to step. For example, the periodic feedforward signal generating unit 342 generates a fifth periodic feedforward signal.

[0153] In step S123c, by presetting the lower limb control strategy, a fifth feedback signal is determined according to the fifth periodic feedforward signal, the fifth target speed, the fifth target posture, the fifth real-time joint state information and the fifth real-time proprioception information.

[0154] The feedback signal is a signal that feeds back the walking state of the lower limbs (for example, the lower limb joint torque and ground contact information, etc.). The feedback signal can reflect the stability of the lower limb walking.

[0155] For example, in step S123c, the lower limb control strategy unit 341 determines the fifth feedback signal. The lower limb control strategy unit 341 is provided with a preset lower limb control strategy. The lower limb control strategy unit 341 determines the fifth feedback signal according to the fifth periodic feedforward signal, the fifth target speed, the fifth target posture, the fifth real-time upper limb joint state information, the fifth real-time lower limb joint state information and the fifth real-time proprioception information through the preset lower limb control strategy.

[0156] The fifth feedback signal may be a signal for feeding back the walking state of the lower limbs when the biped robot is in the posture adjustment operation process stage.

[0157] In step S124c, by presetting the lower limb control strategy, a fifth target position of the lower limb joint is generated according to the fifth periodic feedforward signal and the fifth feedback signal.

[0158] For example, in step S124c, the lower limb control strategy unit 341 generates a fifth target position of the lower limb joint of the lower limb according to the fifth periodic feedforward signal and the fifth feedback signal by presetting the lower limb control strategy. The target position of the lower limb joint can be the target position information of the lower limb joint in the next preset cycle.

[0159] For example, the lower limb control strategy unit 341 can determine the target position of the lower limb joint according to the following formula:

[0160] ;

[0161] in, is the target position of the lower limb joints; is a periodic feedforward signal; is the feedback signal; is the feedback signal coefficient, which can be set based on the experience of training the preset lower limb control strategy, and can usually be set to 0.3 to 0.5.

[0162] The fifth target position of the lower limb joint can be the target position information of the lower limb joint within the next preset cycle when the biped robot is in the posture adjustment operation process stage.

[0163] The fifth target position of the lower limb joint is determined in the same manner as the target position of the lower limb joint described above. The fifth periodic feedforward signal is introduced into , bringing the fifth feedback signal into , and obtain the fifth target position of the lower limb joint.

[0164] In step S125c, a fifth instruction is generated according to the fifth target position of the lower limb joint.

[0165] For example, in step S125c, the lower limb controller 343 generates a fifth instruction according to the fifth target position of the lower limb joint. For example, the lower limb controller 343 may be a proportional-derivative controller (PD controller). The lower limb controller 343 may generate a fifth instruction according to the fifth target position of the lower limb joint by proportional-derivative calculation.

[0166] Alternatively, see Fig.11 The upper limb control module 33 also includes a point cloud processing unit 332 , a historical information processing unit 333 and an upper limb controller 334 .

[0167] The first real-time upper limb joint state information and the first real-time lower limb joint state information both refer to corresponding information when the biped robot is in the stage of grasping the door handle operation process.

[0168] The specific process of step S220 is as follows:

[0169] The point cloud processing unit 332 determines the first grasping point posture of the dexterous hand according to the first real-time point cloud information. The first grasping point posture may be the target position and target posture of the dexterous hand during the process of the dexterous hand grasping the door handle. For example, the point cloud processing unit 332 may be a point cloud processor. The point cloud processor is provided with a point cloud processing algorithm. The point cloud processing algorithm may be a point cloud grasping posture detection network (PointNetGPD), etc.

[0170] The historical information processing unit 333 generates the historical features of the first real-time upper limb joint state information according to the first real-time upper limb joint state information. For example, the historical features of the first real-time upper limb joint state information may be the time dependency features of the first real-time upper limb joint state information determined based on the first real-time upper limb joint state information over a period of time (e.g., the past 2 seconds to 4 seconds).

[0171] For example, the history information processing unit 333 is provided with a history processing network structure. The history processing network structure may be a network structure such as a long short-term memory network (LSTM), a gated recurrent unit (GRU), a convolutional neural network (CNN), or a transformer network.

[0172] The historical information processing unit 333 generates the historical features of the first real-time ontology perception information according to the first real-time ontology perception information. For example, the historical features of the first real-time ontology perception information may be the time dependency features of the first real-time ontology perception information determined according to the first real-time ontology perception information over a period of time (e.g., the past 2 seconds to 4 seconds).

[0173] The upper limb control strategy unit 331 generates a first target position of the upper limb joint of the upper limb by presetting the upper limb control strategy according to the first grasping point posture, the first real-time upper limb joint state information, the first real-time proprioception information, the historical characteristics of the first real-time upper limb joint state information and the historical characteristics of the first real-time proprioception information.

[0174] The target position of the upper limb joint may be the target position information of the upper limb joint in the next preset cycle. The first target position of the upper limb joint may be the target position information of the upper limb joint in the next preset cycle when the biped robot is in the process of grasping the door handle.

[0175] The upper limb controller 334 generates a first instruction according to the first target position of the upper limb joint. For example, the upper limb controller 334 may be a PD controller. The upper limb controller 334 may generate a first instruction according to the first target position of the upper limb joint by proportional differential calculation.

[0176] The third real-time upper limb joint state information and the third real-time lower limb joint state information within the preset period both refer to the information corresponding to the case where the bipedal robot is in the unlocking door handle operation process stage.

[0177] The specific process of step S820 is as follows:

[0178] The historical information processing unit 333 generates the historical features of the third real-time upper limb joint state information according to the third real-time upper limb joint state information. For example, the historical features of the third real-time upper limb joint state information may be the time dependency features of the third real-time upper limb joint state information determined based on the third real-time upper limb joint state information over a period of time (e.g., the past 2 seconds to 4 seconds). The historical information processing unit 333 may generate the historical features of the third real-time upper limb joint state information according to the third real-time upper limb joint state information through a historical processing network structure.

[0179] The historical information processing unit 333 generates the historical features of the third real-time ontology perception information according to the third real-time ontology perception information. For example, the historical features of the third real-time ontology perception information may be the time dependency features of the third real-time ontology perception information determined according to the third real-time ontology perception information over a period of time (e.g., the past 2 seconds to 4 seconds). The historical information processing unit 333 may generate the historical features of the third real-time ontology perception information according to the third real-time ontology perception information through the historical processing network structure.

[0180] The upper limb control strategy unit 331 generates a third target position of the upper limb joint of the upper limb by presetting the upper limb control strategy according to the third real-time point cloud information, the third real-time upper limb joint state information, the third real-time proprioception information, the historical characteristics of the third real-time upper limb joint state information and the historical characteristics of the third real-time proprioception information.

[0181] The third target position of the upper limb joint may be target position information of the upper limb joint within the next preset cycle when the biped robot is in the unlocking door handle operation process.

[0182] The upper limb controller 334 generates a second instruction according to the third target position of the upper limb joint.

[0183] For example, the upper limb controller 334 may generate a second instruction according to the third target position of the upper limb joint through proportional differential calculation.

[0184] The fourth real-time upper limb joint state information and the fourth real-time lower limb joint state information in the present application both refer to the information corresponding to the case where the biped robot is in the door opening operation process stage.

[0185] The fourth periodic feedforward signal may be a signal instructing the lower limbs to walk, generated according to fourth real-time posture information, fourth real-time point cloud information, fourth target speed, fourth target posture and a preset kinematic model when the biped robot is in the door opening operation process.

[0186] The fourth feedback signal may be a signal for feeding back the walking state of the lower limbs when the biped robot is in the door opening operation process.

[0187] The fourth target position of the lower limb joint may be target position information of the lower limb joint within the next preset cycle when the biped robot is in the door opening operation process.

[0188] In step S920, a third instruction is generated according to the fourth real-time posture information, the fourth real-time point cloud information and the fourth real-time body information by using the preset upper limb control strategy and the preset lower limb control strategy.

[0189] The process of generating the third instruction in step S920 is similar to the process of generating the fifth instruction in the above-mentioned step S120c, and will not be repeated here.

[0190] The historical feature of the fourth real-time upper limb joint state information may be a time dependency feature of the fourth real-time upper limb joint state information determined based on the fourth real-time upper limb joint state information over a period of time in the past (eg, the past 2 seconds to 4 seconds).

[0191] The historical feature of the fourth real-time proprioception information may be a time dependency feature of the fourth real-time proprioception information determined based on a period of time in the past (eg, the past 2 seconds to 4 seconds) of the fourth real-time proprioception information.

[0192] The fourth target position of the upper limb joint may be the target position information of the upper limb joint within the next preset cycle when the biped robot is in the door opening operation process.

[0193] In step S930, by presetting the upper limb control strategy, a fourth instruction is generated according to the fourth real-time posture information, the fourth real-time point cloud information and the fourth real-time body information, so that the upper limb performs the door opening action according to the fourth instruction.

[0194] The process of generating the fourth instruction in step S930 is similar to the process of generating the second instruction in step S820 described above, and will not be repeated here.

[0195] In step S231, a first balance instruction is generated according to the first real-time posture information, the first real-time point cloud information and the first real-time body information through a preset lower limb control strategy and a preset upper limb control strategy, so that the lower limb maintains balance according to the first balance instruction.

[0196] The process of generating the first balancing instruction in step S231 is similar to the process of generating the fifth instruction in step S120c described above, and will not be repeated here.

[0197] In step S831, a second balance instruction is generated according to the third real-time posture information, the third real-time point cloud information and the third real-time body information through a preset lower limb control strategy and a preset upper limb control strategy, so that the lower limbs maintain balance according to the second balance instruction.

[0198] The process of generating the second balancing instruction in step S831 is similar to the process of generating the fifth instruction in the above-mentioned step S120c, and will not be repeated here.

[0199] According to another aspect of the present application, the present application also provides a non-volatile computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the control method based on the bipedal robot door operation as described above can be implemented.

[0200] According to another aspect of the present application, the present application also provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, which, when the one or more programs are executed by one or more processors, enables the one or more processors to implement the control method based on the bipedal robot door operation as described above.

[0201] According to another aspect of the present application, the present application also provides a computer program product, including: a computer program stored on a computer-readable storage medium; the computer program includes program instructions, and when the program instructions are executed by the computer, the computer executes the control method based on the bipedal robot door operation as described above.

[0202] Finally, it should be noted that the above is only a preferred embodiment of the present application and is not intended to limit the present application. Although the present application is described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions of the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A control method based on bipedal robot door operation, characterized in that: The biped robot comprises an upper limb and a lower limb, wherein the upper limb is provided with a dexterous hand, and the control method comprises: Steps for grasping a door handle include: The steps to collect and determine information include: Collect environmental information in real time based on a preset frequency to determine first real-time position information of the door and the door handle within a preset period and first real-time point cloud information of the door and the door handle within the preset period; Based on the preset frequency, the proprioceptive information of the biped robot is collected in real time to determine the first real-time proprioceptive information of the biped robot within the preset period, wherein the first real-time proprioceptive information includes the first real-time joint state information within the preset period and the first real-time proprioceptive perception information within the preset period, and the first real-time joint state information includes the first real-time upper limb joint state information within the preset period; By presetting an upper limb control strategy, generating a first instruction according to the first real-time point cloud information and the first real-time body information, so that the dexterous hand grasps the door handle according to the first instruction, including: Determine a first grasping point posture of the dexterous hand according to the first real-time point cloud information; generating historical features of the first real-time upper limb joint state information according to the first real-time upper limb joint state information; generating historical features of the first real-time ontology perception information according to the first real-time ontology perception information; Generate a first target position of an upper limb joint of the upper limb according to the preset upper limb control strategy, the first grasping point posture, the first real-time upper limb joint state information, the first real-time proprioception information, the historical characteristics of the first real-time upper limb joint state information, and the historical characteristics of the first real-time proprioception information; generating the first instruction according to the first target position of the upper limb joint; The first step of lower limb balance adjustment includes: By using a preset lower limb control strategy and the preset upper limb control strategy, a first balance instruction is generated according to the first real-time posture information, the first real-time point cloud information, and the first real-time body information, so that the lower limb maintains balance according to the first balance instruction, including: Determining a first target speed and a first target posture of the biped robot according to the first real-time posture information by using the preset upper limb control strategy; The first balancing instruction is generated through the preset lower limb control strategy according to the first target speed, the first target posture, the first real-time posture information, the first real-time point cloud information and the first real-time body information.

2. The control method according to claim 1, characterized in that: After the step of grasping the door handle, the control method further includes: Executing the steps of collecting and determining information to determine second real-time point cloud information of the door and the door handle within the preset period; Determining whether the door handle needs to be unlocked according to the second real-time point cloud information; If yes, follow the steps to unlock the door handle, including: Execute the steps of collecting and determining information to determine third real-time position information of the door and the door handle within the preset period, third real-time point cloud information of the door and the door handle within the preset period, and third real-time body information of the bipedal robot within the preset period; Generate a second instruction according to the third real-time point cloud information and the third real-time body information through the preset upper limb control strategy, so that the dexterous hand unlocks the door handle according to the second instruction; The second step of lower limb balance adjustment includes: Through the preset lower limb control strategy and the preset upper limb control strategy, a second balance instruction is generated according to the third real-time posture information, the third real-time point cloud information and the third real-time body information, so that the lower limb maintains balance according to the second balance instruction.

3. The control method according to claim 2, characterized in that: After the step of unlocking the door handle or when it is determined according to the second real-time point cloud information that the door handle does not need to be unlocked, the control method further includes: The door opening steps include: Execute the steps of collecting and determining information to determine fourth real-time position information of the door and the door handle within the preset period, fourth real-time point cloud information of the door and the door handle within the preset period, and fourth real-time body information of the biped robot within the preset period; Generate a third instruction according to the fourth real-time posture information, the fourth real-time point cloud information and the fourth real-time body information through the preset upper limb control strategy and the preset lower limb control strategy, so that the lower limb walks according to the third instruction; Through the preset upper limb control strategy, a fourth instruction is generated according to the fourth real-time posture information, the fourth real-time point cloud information and the fourth real-time body information, so that the upper limb performs the door opening action according to the fourth instruction.

4. The control method according to claim 1, characterized in that: Before the step of grasping the door handle, the control method further includes: Executing the steps of collecting and determining information to determine the initial real-time position information of the door and the door handle within the preset period; According to the preset position condition and the preset posture condition of the bipedal robot and the initial real-time posture information, determining whether the bipedal robot meets the preset door opening condition; If yes, then perform the step of grasping the door handle; If not, perform the posture adjustment steps, including: Execute the steps of collecting and determining information to determine fifth real-time position information of the door and the door handle within the preset period, fifth real-time point cloud information of the door and the door handle within the preset period, and fifth real-time body information of the bipedal robot within the preset period; Through the preset upper limb control strategy and the preset lower limb control strategy, a fifth instruction is generated according to the fifth real-time posture information, the fifth real-time point cloud information, the fifth real-time body information and the preset kinematic model of the bipedal robot, so that the bipedal robot reaches the preset position and forms a preset posture according to the fifth instruction.

5. The control method according to claim 4, characterized in that: The fifth real-time proprioceptive information includes the fifth real-time joint state information within the preset period and the fifth real-time proprioceptive perception information within the preset period; The method of generating a fifth instruction according to the fifth real-time posture information, the fifth real-time point cloud information, the fifth real-time body information and the preset kinematic model of the biped robot by using the preset upper limb control strategy and the preset lower limb control strategy includes: Determining a fifth target speed and a fifth target posture of the biped robot according to the fifth real-time posture information by using the preset upper limb control strategy; Generate a fifth periodic feedforward signal according to the fifth real-time posture information, the fifth real-time point cloud information, the fifth target speed, the fifth target posture and the preset kinematic model; Determining a fifth feedback signal through the preset lower limb control strategy according to the fifth periodic feedforward signal, the fifth target speed, the fifth target posture, the fifth real-time joint state information and the fifth real-time proprioception information; Generate a fifth target position of a lower limb joint of the lower limb according to the fifth periodic feedforward signal and the fifth feedback signal by the preset lower limb control strategy; The fifth instruction is generated according to the fifth target position of the lower limb joint.

6. The control method according to claim 2, characterized in that: The third real-time proprioceptive information includes the third real-time joint state information within the preset period and the third real-time proprioceptive perception information within the preset period; The third real-time joint state information includes the third real-time upper limb joint state information within the preset period; The generating a second instruction according to the third real-time point cloud information and the third real-time body information by using the preset upper limb control strategy comprises: Generating historical features of the third real-time upper limb joint state information according to the third real-time upper limb joint state information; generating a historical feature of the third real-time ontology perception information according to the third real-time ontology perception information; Generate a third target position of the upper limb joint of the upper limb through the preset upper limb control strategy according to the third real-time point cloud information, the third real-time upper limb joint state information, the third real-time proprioception information, the historical characteristics of the third real-time upper limb joint state information and the historical characteristics of the third real-time proprioception information; generating the second instruction according to a third target position of the upper limb joint; The method of generating a second balancing instruction according to the third real-time posture information, the third real-time point cloud information and the third real-time body information by using the preset lower limb control strategy and the preset upper limb control strategy includes: By presetting the upper limb control strategy, determining a third target speed and a third target posture of the biped robot according to the third real-time posture information; The second balance instruction is generated through the preset lower limb control strategy according to the third target speed, the third target posture, the third real-time posture information, the third real-time point cloud information and the third real-time body information.

7. The control method according to claim 3, characterized in that: The fourth real-time proprioceptive information includes the fourth real-time joint state information within the preset period and the fourth real-time proprioceptive perception information within the preset period; The fourth real-time joint state information includes the fourth real-time upper limb joint state information within the preset period and the fourth real-time lower limb joint state information within the preset period; The generating of the third instruction according to the fourth real-time posture information, the fourth real-time point cloud information and the fourth real-time body information by the preset upper limb control strategy and the preset lower limb control strategy comprises: Determining a fourth target speed and a fourth target posture of the biped robot according to the fourth real-time posture information by using the preset upper limb control strategy; generating a fourth periodic feedforward signal according to the fourth real-time posture information, the fourth real-time point cloud information, the fourth target speed, the fourth target posture and a preset kinematic model of the biped robot; Determine a fourth feedback signal through the preset lower limb control strategy, according to the fourth periodic feedforward signal, the fourth target speed, the fourth target posture, the fourth real-time upper limb joint state information, the fourth real-time lower limb joint state information and the fourth real-time proprioception information; Generate a fourth target position of a lower limb joint of the lower limb according to the fourth periodic feedforward signal and the fourth feedback signal through the preset lower limb control strategy; generating the third instruction according to a fourth target position of the lower limb joint; The generating of the fourth instruction according to the fourth real-time posture information, the fourth real-time point cloud information and the fourth real-time body information by the preset upper limb control strategy comprises: Generating a historical feature of the fourth real-time upper limb joint state information according to the fourth real-time upper limb joint state information; generating a historical feature of the fourth real-time ontology perception information according to the fourth real-time ontology perception information; Generate a fourth target position of the upper limb joint of the upper limb through the preset upper limb control strategy according to the fourth real-time point cloud information, the fourth real-time upper limb joint state information, the fourth real-time proprioception information, the historical characteristics of the fourth real-time upper limb joint state information and the historical characteristics of the fourth real-time proprioception information; The fourth instruction is generated according to a fourth target position of the upper limb joint.

8. A control system based on bipedal robot door operation, characterized in that: The biped robot comprises an upper limb and a lower limb, wherein the upper limb is provided with a dexterous hand, and the control system is used to execute the control method based on the door operation of the biped robot according to any one of claims 1 to 7, wherein the control system comprises a door sensing module, a proprioceptive sensing module, an upper limb control module, a lower limb control module and a driving module; The door sensing module, the proprioception module, the upper limb control module, the lower limb control module and the driving module perform a step of grasping a door handle, including: The door perception module and the body perception module perform steps of collecting and determining information, including: The door sensing module collects environmental information in real time based on a preset frequency to determine first real-time position information of the door and the door handle within a preset period and first real-time point cloud information of the door and the door handle within the preset period; The proprioceptive perception module collects proprioceptive information of the biped robot in real time based on the preset frequency to determine first real-time proprioceptive information of the biped robot within the preset period, wherein the first real-time proprioceptive information includes first real-time joint state information within the preset period and first real-time proprioceptive perception information within the preset period, and the first real-time joint state information includes first real-time upper limb joint state information within the preset period; The upper limb control module generates a first instruction according to the first real-time point cloud information and the first real-time body information by using a preset upper limb control strategy; The upper limb control module includes an upper limb control strategy unit, a point cloud processing unit, a historical information processing unit and an upper limb controller; The point cloud processing unit determines the first grasping point posture of the dexterous hand according to the first real-time point cloud information; The historical information processing unit generates historical features of the first real-time upper limb joint state information according to the first real-time upper limb joint state information; The historical information processing unit generates historical features of the first real-time ontology perception information according to the first real-time ontology perception information; The upper limb control strategy unit generates a first target position of the upper limb joint of the upper limb according to the preset upper limb control strategy, the first grasping point posture, the first real-time upper limb joint state information, the first real-time proprioception information, the historical characteristics of the first real-time upper limb joint state information, and the historical characteristics of the first real-time proprioception information; The upper limb controller generates the first instruction according to the first target position of the upper limb joint; The driving module drives the dexterous hand according to the first instruction, so that the dexterous hand grasps the door handle according to the first instruction; The upper limb control module, the lower limb control module and the driving module perform a first lower limb balance adjustment step, including: The upper limb control module determines a first target speed and a first target posture of the biped robot according to the first real-time posture information through the preset upper limb control strategy; The lower limb control module generates the first balancing instruction according to the first target speed, the first target posture, the first real-time point cloud information and the first real-time body information through the preset lower limb control strategy; The driving module drives the lower limb according to the first balance instruction so that the lower limb maintains balance according to the first balance instruction.

9. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the control method based on bipedal robot door operation as described in any one of claims 1 to 7 is implemented.

Images