Robot elevator riding method, apparatus, robot, and storage medium

Abstract

The application relates to a robot elevator riding method and device, a robot and a storage medium. The method comprises the following steps: acquiring a target visual image and a target depth image, the target visual image being aligned with the target depth image, the target visual image comprising an elevator control panel region, and the elevator control panel region comprising a target key to be pressed by a robot; determining a target position of an end effector of the robot pressing the target key based on the target visual image and the target depth image; determining a plurality of candidate poses of an arm end of the robot based on the target position; determining a target end pose of the arm end from the plurality of candidate poses; controlling the arm end to run to the target end pose; and controlling the end effector to press the target key. The method can realize autonomous elevator riding of the robot.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, robot, and storage medium for a robot to ride an elevator. Background Technology

[0002] With the development of artificial intelligence technology, robots are widely used in various places, such as commercial buildings, hotels and hospitals. For most service robots, they often need to take elevators to reach designated floors to complete their tasks.

[0003] In related technologies, for elevators that have not been modified, robots face the problem of not being able to ride the elevator autonomously when working across floors, which limits the robot's working range and efficiency. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, and storage medium for robots to ride elevators, enabling robots to ride elevators autonomously.

[0005] In a first aspect, this application provides a method for a robot to ride an elevator, including:

[0006] Acquire a target visual image and a target depth image. Align the target visual image with the target depth image. The target visual image includes the elevator control panel area, and the elevator control panel area includes the target button that the robot needs to press.

[0007] Based on the target visual image and target depth image, determine the target position where the robot's end effector presses the target button;

[0008] Based on the target position, determine multiple candidate poses for the robot's end effector.

[0009] Determine the target end-effector pose from multiple candidate poses;

[0010] The control arm end effector moves to the target end effector position and controls the end effector to press the target button.

[0011] Secondly, this application also provides a robot elevator riding device, comprising:

[0012] The acquisition module is used to acquire the target visual image and the target depth image. The target visual image is aligned with the target depth image. The target visual image includes the elevator control panel area, and the elevator control panel area includes the target button that the robot needs to press.

[0013] The first determining module is used to determine the target position of the robot's end effector pressing the target button based on the target visual image and the target depth image;

[0014] The second determining module is used to determine multiple candidate poses of the robot's end effector based on the target position;

[0015] The selection module is used to determine the target end-effector pose from multiple candidate poses;

[0016] The button module is used to control the end effector of the arm to move to the target end effector position and to control the end effector to press the target button.

[0017] Thirdly, this application also provides a robot including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of any of the methods in the fourth aspect.

[0018] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods in the fourth aspect.

[0019] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the methods in the fourth aspect.

[0020] The aforementioned method, apparatus, robot, and storage medium for a robot to ride an elevator acquire a target visual image and a target depth image, aligning the target visual image and the target depth image. The target visual image includes an elevator control panel area, and the elevator control panel area includes a target button to be pressed by the robot. Based on the target visual image and the target depth image, the target position for the robot's end effector to press the target button is determined. Based on the target position, multiple candidate poses of the robot's arm end effector are determined. The target end effector pose is determined from the multiple candidate poses. The end effector is controlled to move to the target end effector pose, and the end effector is controlled to press the target button. In other words, the robot determines the target position for its end effector to press the target button using the target visual image and the target depth image. Based on the target position, it determines multiple candidate poses for the robot's end effector, selects the target end effector pose from these candidate poses, controls the end effector to move to the target end effector pose, and controls the end effector to press the target button. By autonomously pressing the target button, the robot can ride the elevator without any modifications. Moreover, determining the target end effector pose from multiple candidate poses avoids the problem of the end effector failing to move to the target end effector pose compared to directly determining a target end effector pose based on the target position, thus improving the success rate of the robot autonomously riding the elevator. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a diagram illustrating the application environment of a robot elevator riding method in one embodiment;

[0023] Figure 2 This is a flowchart illustrating a method for a robot to ride an elevator in one embodiment;

[0024] Figure 3 This is a schematic diagram of an end effector in one embodiment;

[0025] Figure 4 This is a schematic diagram of the end effector in another embodiment;

[0026] Figure 5 This is a flowchart illustrating the process of determining the button identifier and center pixel of an elevator button in a target depth image in one embodiment.

[0027] Figure 6 This is a schematic diagram of the target location determination process in one embodiment;

[0028] Figure 7 This is a schematic diagram of the target location determination process in another embodiment;

[0029] Figure 8 This is a schematic diagram illustrating the process of determining target extrinsic parameters in one embodiment;

[0030] Figure 9 This is a schematic diagram of a candidate pose of the arm end in one embodiment;

[0031] Figure 10 This is a flowchart illustrating a method for a robot to ride an elevator in one embodiment;

[0032] Figure 11 This is a structural block diagram of a robot elevator device in one embodiment;

[0033] Figure 12 This is a diagram of the internal structure of a robot in one embodiment. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0035] The robot elevator riding method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with robot 104 via a network. Terminal 102 and robot 104 can work together to execute the robot elevator riding method provided in this embodiment, or robot 104 can be used independently to execute the robot elevator riding method provided in this embodiment. The robot can be any intelligent device with an end effector, including humanoid robots, humanoid robots, and robots with operational capabilities such as cleaning robots, sorting robots, and food delivery robots; no limitations are imposed here. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices, as well as various robots; IoT devices can be smart TVs and smart vehicle devices, and portable wearable devices can be smartwatches, etc.

[0036] In one exemplary embodiment, such as Figure 2 As shown, a method for a robot to ride an elevator is provided, which can be applied to... Figure 1 The following steps are used as an example to illustrate the process of building a robot, including steps 202 to 210. Wherein:

[0037] Step 202: Obtain the target visual image and the target depth image. Align the target visual image with the target depth image. The target visual image includes the elevator control panel area, and the elevator control panel area includes the target button that the robot is to press.

[0038] The target visual image refers to the image of the control panel of the elevator the robot needs to ride. The target visual image can be an RGB (Red-Green-Blue) image or a black and white image; there is no limitation here. The robot can acquire the target visual image using an RGBD (Red-Green-Blue-Depth) camera or an RGB camera. The elevator control panel can be an external control panel or an internal control panel. The elevator control panel includes, but is not limited to, at least one of function buttons and floor buttons. The target visual image includes, but is not limited to, a control panel area and a display area. The control panel area includes, but is not limited to, at least one of a function button area and a floor button area. The function button area includes function buttons, such as open door button, close door button, up button, and down button, etc. The floor button area includes multiple floor buttons, such as 1st floor button, 2nd floor button, and 3rd floor button, etc. The display area refers to the area displaying the elevator's up / down indicators and the current floor indicator; the display area can be a screen area. A depth image is an image that records the distance information between pixels. The pixel value of each pixel in a depth image represents the distance of that pixel to the camera. Robots can acquire depth images using either an RGBD camera or a depth camera. Alignment between the target depth image and the target visual image means that the corresponding pixel positions of points on the same object surface are the same in both the target depth image and the target visual image; that is, there is a one-to-one spatial correspondence between the pixels in the target depth image and the target visual image. A target button is the button that the robot needs to press. The target button can be a function button or a floor button. If the robot is outside the elevator it needs to take, it needs to press either the up or down function button; if the robot is inside the elevator, it needs to press the floor button for the target floor and / or the close button; in this case, the target button is either a floor button or the close button.

[0039] For example, when the robot needs to move from the current floor to the target floor, it uses an RGBD camera to capture a visual image and a depth image of the target on the elevator control panel, including the target button. The current floor refers to the floor the robot is currently on, and the target floor is the floor the robot needs to reach. The following embodiments illustrate this using the robot's acquisition of a visual image and a depth image of the target button to be pressed on the elevator control panel via an RGBD camera.

[0040] It is understood that the RGBD camera is an integrated sensing device, which may include an RGB camera and a depth camera (such as a monocular structured light camera, a binocular camera, etc.). Therefore, in an exemplary embodiment, when the robot needs to move from the current floor to the target floor, it uses the RGB camera to capture an image of the elevator control panel including the target buttons to obtain a target visual image, and uses the depth camera to capture an image of the elevator control panel including the target buttons to obtain an initial depth image. The initial depth image is then spatiotemporally aligned with the target visual image to obtain the target depth image.

[0041] An RGBD camera may include an RGB camera and a depth sensor (such as a ToF sensor). Therefore, in another exemplary embodiment, an elevator control panel including the target buttons can be photographed by an RGB camera to obtain a target visual image, and then the depth information collected by the depth sensor can be fused into the target visual image to obtain a target depth image.

[0042] In an exemplary embodiment, the robot obtains the intrinsic parameter matrix of the RGB camera and the extrinsic parameter matrix of the depth camera relative to the RGB camera. For each depth pixel in the initial depth image, the intrinsic and extrinsic parameter matrices are used to project the depth pixel into the camera coordinate system of the RGB camera, obtaining the projected pixel of the depth pixel in the camera coordinate system of the RGB camera. Based on each projected pixel, the target depth image is obtained. The method for determining the intrinsic parameter matrix of the RGB camera and the extrinsic parameter matrix of the depth camera relative to the RGB camera is not limited here. For example, by determining the four color corner points of the target visual image and the four depth corner points of the target depth image, and determining the mapping relationship between the four depth corner points and the four color corner points, the intrinsic parameter matrix of the RGB camera and the extrinsic parameter matrix of the depth camera relative to the RGB camera can be obtained.

[0043] In an exemplary embodiment, the depth pixels are projected onto the camera coordinate system of the RGB camera using intrinsic and extrinsic parameter matrices to obtain the projected pixels of the depth pixels in the camera coordinate system, which can be achieved using the following formula:

[0044] Formula (1)

[0045] in, This is the intrinsic parameter matrix of the RGB camera; (X,Y,Z) represents the extrinsic parameter matrix of the depth camera relative to the RGB camera; (X,Y,Z) represents the three-dimensional coordinates of the depth pixels in the depth image in the camera coordinate system of the depth camera; and (u,v) represents the two-dimensional coordinates of the projected pixels in the camera coordinate system of the RGB camera.

[0046] Step 204: Based on the target visual image and the target depth image, determine the target position where the robot's end effector presses the target button.

[0047] An end effector is a device or tool installed at the end of a robotic arm that directly interacts with the external environment. For example... Figure 3 As shown, the end effector can be a two-finger gripper, which is installed at the end of the robot arm. The robot can press the elevator button by closing the end point of the two-finger gripper; for example... Figure 4 As shown, the end effector can also be a dexterous hand, which is installed at the end of the robot arm. The dexterous hand can press the elevator button using the tip of its finger (such as the index finger). The target position refers to the position where the end effector presses the target button, i.e., the pressing position of the end effector. The pressing position can be any position of the target button, such as the center position or the edge position. To improve the stability of the end effector pressing the target button, in this embodiment, the center point of the target button is used as the pressing position of the end effector. Therefore, the target position can be understood as the position of the center point of the target button in the robot coordinate system.

[0048] For example, the robot determines the depth information of the center pixel of the target button in the target visual image, corresponding to the depth information in the target depth image. Based on the depth information of the center pixel of the target button, it determines the position of the center pixel of the target button in the camera coordinate system, obtains the transformation relationship from the camera coordinate system to the robot coordinate system, and determines the target position of the center pixel of the target button in the robot coordinate system based on the position and transformation relationship of the center pixel of the target button in the camera coordinate system. Here, depth information refers to the set of information composed of the position and depth of the center pixel in the target depth image; that is, depth information consists of two types of information: position and depth. The camera coordinate system is a coordinate system established with the robot's RGBD camera or RGB camera as the reference system, used to represent the position of the object "seen" by the camera in its own coordinate system. The camera coordinate system can have the optical center of the camera as its origin. The robot coordinate system is a coordinate system established with the robot body as the reference. The robot coordinate system is the basis for robot motion calculations. The robot coordinate system can have the center of the robot's base or the starting point of the joint chain as its origin. The transformation relationship refers to the transformation matrix from the camera coordinate system to the robot coordinate system. The transformation relationship can be a 4*4 homogeneous transformation matrix.

[0049] In an exemplary embodiment, determining the depth information of each button pixel corresponding to a target button in a target visual image in a target depth image includes: inputting the target visual image into a neural network model to obtain the button identifier of each elevator button and the position of the center pixel in the target visual image; obtaining the depth of the pixel in the target depth image that is at the same position as the center pixel in the target visual image, and determining the position of the center pixel in the target visual image and the aforementioned depth as the depth information of the center pixel. This can be understood as follows: since the target depth image is aligned with the target visual image, and the position of the center pixel in the target depth image is the same as the position of the center pixel in the target visual image, the position of the center pixel in the target visual image can be directly determined as the position of the center pixel in the target depth image. Here, the button identifier refers to the identifier representing the button; the button identifier of a function button represents the function of the button, and the button identifier of a floor button represents the floor corresponding to the button.

[0050] In an exemplary embodiment, a flowchart illustrating the process of determining the position of the elevator button's button identifier and center pixel in the target depth image is shown below. Figure 5 As shown, the process includes: inputting the target depth image into an instance segmentation network (such as YOLOv8 instance segment) to obtain the function button area, floor button area, display area, the position of the center pixel of each function button in the function button area and the button label of the function button, and the position of the center pixel of each floor button in the floor button area in the target visual image; inputting the function button area into a switch on / off classification network (such as Mobilenet v3) to obtain the on / off state of the function buttons (the on / off state can be ON or OFF); inputting the floor button area into a switch on / off classification network to obtain the on / off state of the floor buttons; inputting the floor button area into an OCR (Optical Character Recognition) recognition network (such as PPOCR) to obtain the button label of each floor button (i.e., floor label); and inputting the display area into an OCR recognition network to obtain the arrow label in the display panel (the arrow label can be up or down).

[0051] In an exemplary embodiment, the position of the center pixel of the target button in the camera coordinate system is determined based on the depth information of the center pixel of the target button, and can be calculated using the following formula:

[0052] Formula (2)

[0053] Formula (3)

[0054] Formula (4)

[0055] Where (u,v,d) represents the depth information of the center pixel of the target button; (u,v) represents the position of the center pixel of the target button in the target depth image; and d represents the depth of the center pixel of the target button in the target depth image. Let be the equivalent focal length of the camera in the horizontal (X-axis) direction in the image coordinate system. This is the equivalent focal length of the camera in the vertical (Y-axis) direction in the image coordinate system. and These are intrinsic parameters of the camera; (X,Y,Z) represents the optical center of the camera; (X,Y,Z) represents the position of the center pixel of the target button in the camera coordinate system.

[0056] In one exemplary embodiment, the formula for determining the target location is as follows:

[0057] Formula (5)

[0058] in, The depth information of the center pixel of the target button in the target visual image; This represents the position of the center pixel in the target depth image. R represents the depth of the center pixel in the target depth image; R and t are the extrinsic parameters of the robot's camera relative to the robot's base link obtained from hand-eye calibration. R is a 3×3 rotation matrix and t is a 3×1 translation vector.

[0059] Step 206: Based on the target position, determine multiple candidate poses of the robot's end effector.

[0060] As can be understood, a robot's robotic arm includes the arm body and an end effector connected to the arm body. The end effector refers to the end of the robotic arm's arm body that connects to the end effector; in other words, the end effector is the structural part of the robotic arm's arm body that connects to the end effector. A candidate pose refers to the possible pose of the robot's end effector when the target button is pressed.

[0061] For example, the robot determines multiple candidate poses at the end of the robot arm based on the target position and multiple different button poses of the end effector.

[0062] Step 208: Determine the target end-effector pose from multiple candidate poses.

[0063] The target end effector pose refers to the pose that the end effector needs to move to and maintain in order for the end effector to press the target button. The target end effector pose includes the position and angle of the end effector.

[0064] For example, reachable poses are selected from multiple candidate poses; when there is only one reachable pose, the reachable pose is determined as the target end pose; when there are two or more reachable poses, the target end pose is determined from multiple reachable poses.

[0065] In an exemplary embodiment, selecting reachable poses from multiple candidate poses includes: for each candidate pose, the robot calculates the joint angles of each joint using inverse kinematics based on the candidate pose. If the joint angles of each joint are calculated, the candidate pose is determined as a reachable pose; that is, the candidate poses for which the joint angles can be calculated are determined as reachable poses. Here, each joint of the robot includes all joints from the robot's initial link to the end effector. The initial link refers to the starting part of the robot's kinematic chain, typically the origin of the robot's coordinate system. A joint refers to the component connecting any two links from the robot's initial link to the end effector. A joint angle is an angle characterizing the degree of joint rotation. A reachable pose is the pose that the robot's end effector can actually reach from the candidate poses; a pose for which the joint angles can be calculated is considered a reachable pose.

[0066] For example, in a robot structure, 10 rotary joints are arranged in series from the chassis to the end effector. For each candidate pose, the rotation angles of these 10 rotary joints are solved using an inverse kinematics algorithm. If a reasonable set of joint angle values ​​can be calculated, allowing the robot's end effector to accurately reach the candidate pose, then the candidate pose is determined to be an reachable pose for the end effector; otherwise, if a feasible combination of joint angles cannot be found, the candidate pose is considered to be beyond the range of motion capabilities of the end effector and is therefore an unreachable pose.

[0067] In another exemplary embodiment, selecting reachable poses from multiple candidate poses includes: obtaining a set of reachable poses corresponding to the end effector; for each candidate pose, if there is a reachable pose identical to the candidate pose in the reachable pose set, then the candidate pose is determined as a reachable pose. That is, a set of reachable poses is pre-set, which includes multiple reachable poses. This can be done by setting a separate reachable pose set for each button on the elevator control panel, or by setting a single reachable pose set for all buttons on the elevator control panel. If a reachable pose identical to the candidate pose exists in the reachable pose set, then the candidate pose is reachable; otherwise, the candidate pose is unreachable. This process selects reachable poses from multiple candidate poses, ensuring that the determined target end effector pose is reachable by the end effector, avoiding situations where the robot fails to press the target button, and improving the success rate of the robot riding the elevator.

[0068] Step 210: Control the end effector to move to the target end effector position and control the end effector to press the target button.

[0069] For example, the robot arm end effector moves to the target end effector pose and then controls the end effector to press the target button.

[0070] In the aforementioned method for robots to ride elevators, the robot determines the target position for its end effector to press the target button using a target visual image and a target depth image. Based on the target position, it determines multiple candidate poses for the robot's end effector arm. From these candidate poses, it determines the target end effector pose, controls the end effector arm to move to the target end effector pose, and controls the end effector arm to press the target button. By autonomously pressing the target button, the robot can ride the elevator without any modifications to the elevator. Moreover, determining the target end effector pose from multiple candidate poses avoids the problem of the end effector arm failing to reach the target end effector pose compared to directly determining a target end effector pose based on the target position, thereby improving the success rate of the robot autonomously riding the elevator.

[0071] In one exemplary embodiment, such as Figure 6 As shown, based on the target visual image and the target depth image, the target position where the robot's end effector presses the target button is determined, including:

[0072] Step 602: Determine the depth information of each button pixel in the target visual image corresponding to the target button in the target depth image.

[0073] Here, button pixels refer to the pixels that make up the target button in the target visual image.

[0074] For example, the robot inputs the target visual image into an instance segmentation network to obtain a mask image corresponding to the target visual image. The mask image includes the outer envelope of the target button. Based on the outer envelope of the target button, the depth information of each button pixel in the target visual image is determined in the target depth image. The outer envelope of the target button refers to the polygon that surrounds all the button pixels of the target button.

[0075] Step 604: Based on the depth information of the button pixels, determine the position of the button pixels in the robot's camera coordinate system.

[0076] For example, the robot substitutes the depth information of the button pixel into the above formulas (2)-(4) to obtain the position of the button pixel in the robot's camera coordinate system.

[0077] Step 606: Based on the position of the button pixel in the camera coordinate system, determine the position of the center pixel of the target button in the robot coordinate system; the center pixel is the button pixel corresponding to the center point of the target button in the target visual image.

[0078] For example, the robot determines the button pixel corresponding to the center point of the target button as the center pixel. Based on the position of the center pixel in the camera coordinate system, it determines the position of the center pixel in the robot's coordinate system. That is, the position of the button pixel corresponding to the center point of the target button in the camera coordinate system is directly determined as the position of the center pixel in the camera coordinate system.

[0079] Step 608: Determine the position of the center pixel in the robot coordinate system as the target position for the robot's end effector to press the target button.

[0080] For example, the robot determines the position of the center pixel in the robot coordinate system as the target position for the robot's end effector to press the target button.

[0081] In this embodiment, the target position of the robot's end effector pressing the target button is determined based on the target visual image and the target depth image, providing accurate basic data for subsequent determination of candidate poses.

[0082] In an exemplary embodiment, determining the position of the center pixel of the target button in the robot's coordinate system based on the position of the button pixel in the camera coordinate system includes:

[0083] A plane fitting is performed on the positions of multiple button pixels in the camera coordinate system to obtain the fitting plane; based on the position of the center pixel of the target button in the target depth image and the fitting plane, the position of the center pixel in the camera coordinate system is determined; based on the position of the center pixel in the camera coordinate system, the position of the center pixel in the robot's robot coordinate system is determined.

[0084] The fitted plane refers to the plane obtained by fitting the positions of multiple key pixels of the target button in the camera coordinate system. The fitted plane can be represented by a plane equation, for example, A*x+B*y+C*x+D=0, where A, B, C, and D are the coefficients of the fitted plane.

[0085] For example, the robot first performs plane fitting on the positions of multiple button pixels in the camera coordinate system to obtain a fitting plane; secondly, it obtains the position of the center pixel of the target button in the target depth image, and determines the target depth corresponding to the center pixel based on the position of the center pixel of the target button in the target depth image and the fitting plane. It then substitutes the position of the center pixel of the target button in the target depth image and the target depth into the above formulas (2)-(4) to obtain the position of the center pixel in the robot's camera coordinate system; finally, it obtains the target extrinsic parameters, and substitutes the position of the center pixel in the camera coordinate system and the target extrinsic parameters into formula (5) to obtain the position of the center pixel in the robot's robot coordinate system. Here, the target depth refers to the depth determined based on the fitting plane. The target extrinsic parameters refer to the extrinsic parameters of the robot camera relative to the robot coordinate system when the robot's camera acquires the target visual image and target depth.

[0086] In an exemplary embodiment, after performing planar fitting on the positions of multiple button pixels in the camera coordinate system to obtain the fitted plane, the method further includes: determining the normal vector of the fitted plane, and determining the inverse direction vector of the normal vector as the pressing direction of the end effector pressing the target button.

[0087] In an exemplary embodiment, the target depth corresponding to the center pixel is determined based on the position of the center pixel of the target button in the target depth image and the fitting plane. This includes: substituting the position of the center pixel of the target button in the target depth image into formulas (3) and (4) to obtain the depth d as the unknown. , and Substituting X, Y, and Z into the fitting plane A*x+B*y+C*x+D=0, the target depth d is calculated.

[0088] In an exemplary embodiment, the process for determining the target location is illustrated as follows: Figure 7 As shown, the process includes: the robot inputting the target visual image into the instance segmentation network to obtain the mask image corresponding to the target visual image, the mask image including the outer envelope of the target button; determining the depth information of each button pixel in the target visual image corresponding to the target depth image based on the outer envelope of the target button; the robot substituting the depth information of the button pixel into the above formulas (2)-(4) to obtain the position of the button pixel in the robot's camera coordinate system, performing plane fitting on the positions of multiple button pixels in the camera coordinate system to obtain the fitting plane A*x+B*y+C*x+D=0; substituting the position of the center pixel of the target button in the target depth image into formulas (3) and (4) to obtain the depth d as the unknown. , and Substitute X, Y, and Z into the fitting plane A*x+B*y+C*x+D=0 to calculate the target depth. Substitute the position of the center pixel of the target button in the target depth image and the target depth into the above formulas (2)-(4) to obtain the position of the center pixel in the robot's camera coordinate system. The robot obtains the target extrinsic parameters. Substitute the position of the center pixel in the camera coordinate system and the target extrinsic parameters into formula (5) to obtain the position of the center pixel in the robot's robot coordinate system. Determine the position of the center pixel in the robot coordinate system as the target position of the robot's end effector pressing the target button. Determine the opposite direction vector of the normal vector of the fitting plane as the pressing direction of the end effector pressing the target button. Output the target position and pressing direction of the target button.

[0089] In this embodiment, since the elevator control panel contains a large amount of smooth, shiny metal, the depth of pixels in the target depth image may be inaccurate. By performing planar fitting on the positions of multiple button pixels in the target button in the camera coordinate system, a fitting plane is obtained. Based on the position of the center pixel of the target button in the target depth image and the fitting plane, the position of the center pixel in the robot's camera coordinate system is determined. This avoids the problem of low accuracy of the position of the center pixel in the robot's camera coordinate system due to the inaccuracy of pixel depth in the target depth image, thereby improving the accuracy of the position of the center pixel in the robot's camera coordinate system, and thus improving the accuracy of the target position.

[0090] In an exemplary embodiment, determining the position of the center pixel in the robot's coordinate system based on the position of the center pixel in the camera coordinate system includes:

[0091] The robot acquires the target joint angles of each joint when acquiring the target visual image and target depth image; based on the target joint angles of each joint, the target pose of the robot's camera is determined; based on the camera's calibration pose and the target pose, the relative transformation matrix is ​​determined; based on the relative transformation matrix and the calibration extrinsic parameters, the target extrinsic parameters are determined; the position of the center pixel in the camera coordinate system is multiplied by the target extrinsic parameters to obtain the position of the center pixel in the robot's robot coordinate system.

[0092] In this context, the calibration pose refers to the camera's pose relative to the robot's coordinate system during calibration. The target pose refers to the robot's camera's pose when acquiring visual and depth images of the target; the target pose can be calculated using forward kinematics. The relative transformation matrix is ​​the transformation matrix that converts the calibration pose to the target pose. For example, if the robot's camera is located on its head, the robot can change the camera's viewing angle by rotating its head while acquiring images. The camera's pose before rotating its head is the calibration pose, and the camera's pose after rotating its head is the target pose. If the robot cannot change the camera's viewing angle by rotating its head, then there is no target pose.

[0093] For example, the robot obtains the target joint angles of each joint connected to the camera from the first data set when acquiring the target visual image and target depth image. Based on the target joint angles corresponding to each joint, the robot uses forward kinematics to determine the target pose of the robot's camera. The robot then obtains the calibration pose of the camera and determines a relative transformation matrix based on the camera's calibration pose and the target pose. The relative transformation matrix is ​​multiplied by the calibration extrinsic parameters to obtain the target extrinsic parameters. Finally, the position of the center pixel in the camera coordinate system is multiplied by the target extrinsic parameters to obtain the position of the center pixel in the robot's robot coordinate system. The first data set refers to the data set that stores the real-time joint angles of the robot's camera to each joint connected to the initial link.

[0094] In an exemplary embodiment, the flowchart for determining the target extrinsic parameters is shown below. Figure 8As shown, the process includes: the robot acquiring target joint angles from its camera to each joint in the initial link when acquiring target visual and depth images from a first dataset; acquiring calibration joint angles from the robot's camera to each joint in the initial link from URDF (Unified Robot Description Format); determining the target pose P1 of the robot's camera using forward kinematics based on the target joint angles corresponding to each joint; determining the calibration pose P0 of the robot's camera using forward kinematics based on the calibration joint angles corresponding to each joint; and determining the relative transformation matrix M, M = P1 * P0, based on the camera's calibration pose and target pose. -1 Obtain the calibration extrinsic parameter E0, multiply the relative transformation matrix by the calibration extrinsic parameter to obtain the target extrinsic parameter E1, E1=M*E0.

[0095] In this embodiment, the target pose of the robot's camera is determined based on the target joint angles corresponding to each joint. The target pose is the real-time pose of the robot's camera when acquiring the target visual image and target depth image. A relative transformation matrix is ​​determined using the camera's calibration pose and the target pose. Based on the relative transformation matrix and calibration extrinsic parameters, target extrinsic parameters are determined. These target extrinsic parameters are the real-time extrinsic parameters of the camera relative to the robot coordinate system when acquiring the target visual image and target depth image. The position of the center pixel in the camera coordinate system is multiplied by the target extrinsic parameter to obtain the position of the center pixel in the robot's coordinate system. That is, the position of the center pixel in the robot's coordinate system is determined using real-time extrinsic parameters, thereby improving the accuracy of the position of the center pixel in the robot's coordinate system. Moreover, the robot can determine the real-time pose (i.e., target pose) of the robot's camera when acquiring the target visual image and target depth image, indicating that the robot can acquire the target visual image and target depth image through the camera at appropriate positions. Compared to the robot being able to acquire the target visual image and target depth image through the camera only at fixed positions, this expands the camera's observation range.

[0096] In one exemplary embodiment, based on the target position, multiple candidate poses of the robot's end effector are determined, including:

[0097] Multiple candidate button poses of the end effector are obtained, and the candidate button poses are determined based on the preset button poses of the end effector. For each candidate button pose, the corresponding candidate button pose of the end effector is determined based on the candidate button pose and the target position. The candidate pose of the arm end is determined based on the transformation matrix from the end effector to the arm end and the candidate button pose.

[0098] The preset button posture refers to the pre-set posture of the end effector when pressing the target button. The preset button posture can be represented by a 3x3 rotation matrix. For a five-finger dexterity hand as the end effector, the elevator button is typically pressed using the tip of the index finger, with the palm down and the back of the hand up. In this case, the end effector's tip is the index fingertip, and the preset button posture is the palm down and the back of the hand up. For a two-finger parallel gripper, the elevator button is typically pressed using the end of the gripper in the closed position. In this case, the end effector's end is the end of the gripper in the closed position, and the preset button posture is one side of the plane containing the two fingers facing down, and the other side facing up. The candidate button posture refers to the button posture obtained by adjusting the preset button posture. For example, it can be adjusted by rotating left and right around the pressing point at fixed angular intervals (3 degrees or 4 degrees, etc.), or by rotating up and down around the pressing point at fixed rotation angles (30 degrees or 35 degrees, etc.). Candidate button pose refers to the possible pose of the end effector when pressing the target button. The transformation matrix refers to the rotation and translation transformation matrix from the end effector tip to the arm tip. To simplify the calculation, the direction of the end effector tip can be assumed to be the same as the direction of the arm tip, that is, the rotation matrix in the transformation matrix is ​​the identity matrix, and the translation matrix in the transformation matrix can be obtained from URDF.

[0099] For example, the robot acquires a preset button pose of the end effector, determines multiple candidate button poses based on the preset button poses, and for each candidate button pose, determines a corresponding candidate button pose of the end effector based on the candidate button pose and the target position. The robot then multiplies the transformation matrix from the end effector's tip to the arm's end effector with the candidate button pose to obtain the candidate pose of the arm's end effector. For example, a schematic diagram of the candidate pose is shown below. Figure 9 As shown, the three short, perpendicular lines represent the X-axis, Y-axis, and Z-axis in the local coordinate system of the end effector, respectively. The long line is the line connecting the origin of the local coordinate system to the starting link of the robot. There is a candidate pose of the end effector in each local coordinate system. The pressing pose of the center point of the target button is determined by the target position and the preset button pose.

[0100] In an exemplary embodiment, the formula for determining the candidate pose is as follows:

[0101] Formula (6)

[0102] Formula (7)

[0103] in, This represents the candidate pose of the arm's distal end; This is the rotation and translation transformation matrix from the end effector's end effector to the arm's end effector; Candidate button poses for the end effector; Candidate button pose; The target location.

[0104] In this embodiment, the candidate button pose of the end effector is determined by multiple candidate button poses and target positions of the end effector. Each candidate button pose of the end effector determines a candidate pose of the corresponding end arm, laying the foundation for the subsequent determination of the target end pose.

[0105] In one exemplary embodiment, determining the target end-effector pose from a plurality of candidate poses includes:

[0106] The reachable pose is selected from multiple candidate poses; if the number of reachable poses is greater than or equal to two, the total joint torque corresponding to the reachable pose is determined; the total joint torques are compared to obtain the minimum torque; the reachable pose corresponding to the minimum torque is determined as the target end-effector pose of the arm.

[0107] The total joint torque refers to the minimum torque statistical value of each joint of the robot. A reachable pose can determine multiple sets of joint angles. Each set of joint angles determines a set of joint torques. The sum of the joint torques of a set of joints gives a torque statistical value. The minimum value among the torque statistical values ​​corresponding to the joint angles of multiple sets of joints is the total joint torque corresponding to the reachable pose.

[0108] For example, the robot selects reachable poses from multiple candidate poses. If the number of reachable poses is greater than or equal to two, the total joint torque corresponding to each reachable pose is determined. The total joint torques corresponding to each reachable pose are compared to obtain the minimum torque, and the reachable pose corresponding to the minimum torque is determined as the target end-effector pose. In this embodiment, when the number of reachable poses is greater than or equal to two, determining the reachable pose corresponding to the minimum torque as the target end-effector pose can achieve the effects of saving robot energy, reducing robot mechanical losses, and improving robot safety performance.

[0109] In another exemplary embodiment, determining the target pose of the arm's end effector from a plurality of candidate poses to obtain the target end effector pose includes:

[0110] From multiple candidate poses, reachable poses are selected; when the number of reachable poses is greater than or equal to two, the ratio of the actual rotation angle of each joint corresponding to the reachable pose to the maximum allowable angle is determined, and the reachable pose corresponding to the minimum value of the sum of the ratios is determined as the target pose of the arm end effector, thus obtaining the target end effector pose.

[0111] In this embodiment of the application, when the number of reachable poses is greater than or equal to two, the reachable pose corresponding to the minimum value of the sum of the rotation angle ratios is determined as the target end pose. This not only avoids the extreme movement of joints and reduces the risk of overload, but also avoids wear or loss of control of certain joints due to excessive rotation, thereby extending the service life of the robot.

[0112] In one exemplary embodiment, controlling the end effector to move to a target end effector pose and controlling the end effector to press a target button includes:

[0113] Based on the target end-effector pose, generate the running trajectory of the end-effector; control the end-effector to run according to the running trajectory; until the end-effector runs to the target end-effector pose, control the end-effector to press the target button according to the target button pose.

[0114] Here, the motion trajectory refers to the path taken by the end effector from its current pose to the target end effector pose. The robot can generate a collision-free motion trajectory with minimal motion cost based on the target end effector pose and obstacles perceived by the robot in real time. The target button pose refers to the pose of the robot pressing the target button. The target button pose can be a preset button pose or a candidate button pose corresponding to the target end effector pose, i.e., the candidate button pose used to determine the target end effector pose.

[0115] For example, the robot generates the running trajectory of the arm end based on the target end pose. After generating the motion trajectory, the robot controls the arm end to run according to the running trajectory until the arm end reaches the target end pose, and controls the end effector to press the target button according to the target button pose.

[0116] In this embodiment, the robot generates the running trajectory of the end effector based on the target end effector pose, and controls the end effector to run according to the running trajectory until the end effector reaches the target end effector pose, laying the foundation for controlling the end effector to press the target button. On this basis, the robot controls the end effector to press the target button according to the target button pose, thereby realizing the robot's autonomous pressing of the target button.

[0117] In an exemplary embodiment, a flowchart of the robot elevator riding method is shown below. Figure 10 As shown, it includes:

[0118] Step 1002: When the robot needs to move from the current floor to the target floor, the robot uses an RGBD camera to acquire a visual image of the target button and a depth image of the target.

[0119] Step 1004: The robot inputs the target visual image into the instance segmentation network to obtain the mask image corresponding to the target visual image. The mask image includes the outer envelope of the target button. Based on the outer envelope of the target button, the robot determines the depth information of each button pixel in the target visual image corresponding to the target depth image. The robot substitutes the depth information of the button pixels into the above formulas (2)-(4) to obtain the position of the button pixels in the robot's camera coordinate system. The robot performs plane fitting on the positions of multiple button pixels in the camera coordinate system to obtain the fitting plane A*x+B*y+C*x+D=0. The robot substitutes the position of the center pixel of the target button in the target depth image into formulas (3) and (4) to obtain the depth d as the unknown. , and Substitute X, Y and Z into the fitting plane A*x+B*y+C*x+D=0 to calculate the target depth. Substitute the position of the center pixel of the target button in the target depth image and the target depth into the above formulas (2)-(4) to obtain the position of the center pixel in the robot's camera coordinate system.

[0120] Step 1006: The robot obtains the target joint angles from the robot's camera to each joint of the initial link when acquiring the target visual image and target depth image from the first dataset; it also obtains the calibration joint angles from the robot's camera to each joint of the initial link from the URDF (Unified Robot Description Format); based on the target joint angles corresponding to each joint, it uses forward kinematics to determine the target pose P1 of the robot's camera; based on the calibration joint angles corresponding to each joint, it uses forward kinematics to determine the calibration pose P0 of the robot's camera; and based on the camera's calibration pose and target pose, it determines the relative transformation matrix M, M = P1 * P0. -1 Obtain the calibration extrinsic parameter E0, multiply the relative transformation matrix by the calibration extrinsic parameter to obtain the target extrinsic parameter E1, E1=M*E0.

[0121] Step 1008: The robot substitutes the position of the center pixel in the camera coordinate system and the target extrinsic parameters into formula (5) to obtain the position of the center pixel in the robot coordinate system; the position of the center pixel in the robot coordinate system is determined as the target position of the robot's end effector pressing the target button, and the opposite direction vector of the normal vector of the fitting plane is determined as the pressing direction of the end effector pressing the target button.

[0122] Step 1010: The robot obtains the preset button posture of the end effector, determines multiple candidate button postures of the end effector based on the preset button posture, and substitutes the candidate button posture and target position into formula (7) for each candidate button posture to obtain the candidate button posture corresponding to the end effector. The transformation matrix from the end of the end effector to the end of the arm and the candidate button posture are substituted into formula (6) to obtain the candidate posture of the end of the arm.

[0123] Step 1012: Obtain the set of reachable poses corresponding to the end effector. For each candidate pose, if there is a reachable pose in the set that is identical to the candidate pose, then the candidate pose is determined as a reachable pose. If there is only one reachable pose, the reachable pose is determined as the target end effector pose. If there are at least two reachable poses, the total joint torque corresponding to the reachable pose is determined. The total joint torques corresponding to each reachable pose are compared to obtain the minimum torque, and the reachable pose corresponding to the minimum torque is determined as the target end effector pose.

[0124] Step 1014: The robot generates the running trajectory of the arm end based on the target end pose. After generating the motion trajectory, the robot controls the arm end to run according to the running trajectory until the arm end reaches the target end pose and controls the end effector to press the target button according to the preset button posture.

[0125] The robot determines the target position for its end effector to press a target button using a target visual image and a target depth image. Based on the target position, it determines multiple candidate poses for the robot's end effector, selects the target end effector pose from these candidate poses, controls the end effector to move to the target end effector pose, and controls the end effector to press the target button. By autonomously pressing the target button, the robot can ride the elevator without modifying it. Moreover, determining the target end effector pose from multiple candidate poses avoids the problem of the end effector failing to reach the target end effector pose compared to directly determining a target end effector pose based on the target position, thus improving the success rate of the robot autonomously riding the elevator.

[0126] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0127] Based on the same inventive concept, this application also provides a robot elevator-riding device for implementing the robot elevator-riding method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations of one or more robot elevator-riding device embodiments provided below can be found in the limitations of the robot elevator-riding method described above, and will not be repeated here.

[0128] In one exemplary embodiment, such as Figure 11 As shown, a robot elevator riding device is provided, including: an acquisition module 1102, a first determination module 1104, a second determination module 1106, a selection module 1108, and a button module 1110, wherein:

[0129] The acquisition module 1102 is used to acquire a target visual image and a target depth image. The target visual image and the target depth image are aligned. The target visual image includes an elevator control panel area, and the elevator control panel area includes a target button that the robot is to press.

[0130] The first determining module 1104 is used to determine the target position of the robot's end effector pressing the target button based on the target visual image and the target depth image;

[0131] The second determining module 1106 is used to determine multiple candidate poses of the robot's arm end based on the target position;

[0132] Selection module 1108 is used to determine the target end-effector pose from multiple candidate poses;

[0133] The button module 1110 is used to control the end effector of the arm to move to the target end effector position and to control the end effector to press the target button.

[0134] In an exemplary embodiment, the first determining module 1104 is further configured to: determine the depth information of each button pixel corresponding to the target button in the target visual image in the target depth image; determine the position of the button pixel in the robot's camera coordinate system based on the depth information of the button pixel; determine the position of the center pixel of the target button in the robot's robot coordinate system based on the position of the button pixel in the camera coordinate system; the center pixel is the button pixel corresponding to the center point of the target button in the target visual image; and determine the position of the center pixel in the robot coordinate system as the target position for the robot's end effector to press the target button.

[0135] In an exemplary embodiment, the first determining module 1104 is further configured to: perform planar fitting on the positions of multiple button pixels in the camera coordinate system to obtain a fitting plane; determine the position of the center pixel in the camera coordinate system based on the position of the center pixel in the target depth image and the fitting plane; and determine the position of the center pixel in the robot coordinate system based on the position of the center pixel in the camera coordinate system.

[0136] In an exemplary embodiment, the first determining module 1104 is further configured to: acquire the target joint angles of each joint of the robot when the robot acquires the target visual image and the target depth image; determine the target pose of the robot's camera in the robot coordinate system based on the target joint angles corresponding to each joint; determine the relative transformation matrix based on the calibration pose and target pose of the camera in the robot coordinate system; determine the target extrinsic parameter based on the relative transformation matrix and the calibration extrinsic parameter; and multiply the position of the center pixel in the camera coordinate system by the target extrinsic parameter to obtain the position of the center pixel in the robot coordinate system.

[0137] In an exemplary embodiment, the second determining module 1106 is further configured to: acquire multiple candidate button postures of the end effector, wherein the candidate button postures are determined based on preset button postures of the end effector; for each candidate button posture, determine the corresponding candidate button pose of the end effector based on the candidate button posture and the target position; and determine the candidate pose of the arm end based on the transformation matrix from the end of the end effector to the end of the arm and the candidate button pose.

[0138] In an exemplary embodiment, the selection module 1108 is further configured to: filter out reachable poses from multiple candidate poses; determine the total joint torque corresponding to the reachable pose when the number of reachable poses is greater than or equal to two; compare the total joint torques to obtain the minimum torque; and determine the reachable pose corresponding to the minimum torque as the target end-effector pose of the arm.

[0139] In an exemplary embodiment, the button module 1110 is further configured to: generate a running trajectory of the arm end based on the target end pose; control the arm end to run based on the running trajectory until the arm end reaches the target end pose; and control the end effector to press the target button based on the target button pose.

[0140] The various modules in the aforementioned robot elevator device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the robot's processor in hardware form or independent of it, or stored in the robot's memory in software form, so that the processor can call and execute the corresponding operations of each module.

[0141] In one exemplary embodiment, a robot is provided, which may be an end-user device, and its internal structure diagram may be as follows: Figure 12 As shown, the robot includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The robot's processor provides computational and control capabilities. The robot's memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The robot's input / output interface is used for exchanging information between the processor and external devices. The robot's communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a method for the robot to ride an elevator. The robot's display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The robot's input device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the robot's shell, or external keyboards, touchpads, or mice, etc.

[0142] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the robot to which the present application is applied. A specific robot may include more or fewer parts than shown in the figure, or combine certain parts, or have different part arrangements.

[0143] In one embodiment, a robot is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above method embodiments.

[0144] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0145] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0146] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0147] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0148] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0149] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for a robot to ride an elevator, characterized in that, The method includes: Acquire a target visual image and a target depth image, wherein the target visual image is aligned with the target depth image, and the target visual image includes an elevator control panel area, wherein the elevator control panel area includes a target button to be pressed by the robot; Determine the depth information of each button pixel in the target visual image corresponding to the target button in the target depth image; Based on the depth information of the button pixels, the position of the button pixels in the camera coordinate system of the robot is determined; A plane fitting is performed on the positions of the multiple button pixels in the camera coordinate system to obtain a fitting plane; based on the position of the center pixel of the target button in the target depth image and the fitting plane, the position of the center pixel in the camera coordinate system is determined; based on the position of the center pixel in the camera coordinate system, the position of the center pixel in the robot's robot coordinate system is determined; the center pixel is the button pixel corresponding to the center point of the target button in the target visual image; The position of the center pixel in the robot coordinate system is determined as the target position where the robot's end effector presses the target button. Based on the target position, multiple candidate poses of the robot's arm end are determined; Determine the target end-effector pose from a plurality of candidate poses; Control the end effector of the arm to move to the target end effector position, and control the end effector to press the target button.

2. The method according to claim 1, characterized in that, The acquisition of the target visual image and the target depth image includes: When it is necessary to reach the target floor from the current floor, the elevator control panel, including the target buttons, is photographed by an RGB camera to obtain a visual image of the target. An initial depth image is obtained by capturing the elevator control panel, including the target buttons, using a depth camera; The initial depth image is spatiotemporally aligned with the target visual image to obtain the target depth image.

3. The method according to claim 2, characterized in that, The step of spatiotemporally aligning the initial depth image with the target visual image to obtain the target depth image includes: Obtain the intrinsic parameter matrix of the RGB camera and the extrinsic parameter matrix of the depth camera relative to the RGB camera; For each depth pixel in the initial depth image, the intrinsic parameter matrix and the extrinsic parameter matrix are used to project the depth pixel into the camera coordinate system of the RGB camera, so as to obtain the projected pixel of the depth pixel in the camera coordinate system of the RGB camera; Based on each of the projected pixels, a target depth image is obtained.

4. The method according to claim 1, characterized in that, Determining the position of the center pixel in the robot's coordinate system based on its position in the camera coordinate system includes: The target joint angles of each joint of the robot are obtained when the robot acquires the target visual image and the target depth image. Based on the target joint angles corresponding to each of the joints, the target pose of the robot's camera in the robot coordinate system is determined; Based on the camera's calibration pose in the robot's coordinate system and the target pose, determine the relative transformation matrix; Multiply the relative transformation matrix by the calibration extrinsic parameters to obtain the target extrinsic parameters; Multiply the position of the center pixel in the camera coordinate system by the target extrinsic parameter to obtain the position of the center pixel in the robot coordinate system.

5. The method according to claim 1, characterized in that, The step of determining multiple candidate poses of the robot's end effector based on the target position includes: Multiple candidate button postures of the end effector are obtained, and the candidate button postures are determined based on preset button postures of the end effector. For each candidate button pose, the corresponding candidate button pose of the end effector is determined based on the candidate button pose and the target position; Based on the transformation matrix from the end effector to the end of the arm and the candidate button pose, the candidate pose of the end of the arm is determined.

6. The method according to claim 1, characterized in that, Determining the target end-effector pose from a plurality of candidate poses includes: A reachable pose is selected from the multiple candidate poses; When the number of reachable poses is greater than or equal to two, determine the total joint torque corresponding to the reachable poses; The total torque of each joint is compared to obtain the minimum torque; The reachable pose corresponding to the minimum torque is determined as the target end pose of the arm end.

7. The method according to claim 1, characterized in that, The control of the arm end effector to move to the target end effector position and the control of the end effector to press the target button include: Based on the target end-effector pose, the running trajectory of the arm end-effector is generated; Control the end effector of the arm to run according to the running trajectory until the end effector of the arm reaches the target end effector position; control the end effector to press the target button according to the target button position.

8. A robot elevator device, characterized in that, The device includes: An acquisition module is used to acquire a target visual image and a target depth image, wherein the target visual image is aligned with the target depth image, and the target visual image includes an elevator control panel area, wherein the elevator control panel area includes a target button to be pressed by the robot. The first determining module is used to determine the depth information of each button pixel corresponding to the target button in the target visual image in the target depth image; based on the depth information of the button pixels, determine the position of the button pixels in the camera coordinate system of the robot; perform plane fitting on the positions of multiple button pixels in the camera coordinate system to obtain a fitting plane; based on the position of the center pixel of the target button in the target depth image and the fitting plane, determine the position of the center pixel in the camera coordinate system; based on the position of the center pixel in the camera coordinate system, determine the position of the center pixel in the robot coordinate system; the center pixel is the button pixel corresponding to the center point of the target button in the target visual image; and determine the position of the center pixel in the robot coordinate system as the target position for the robot's end effector to press the target button. The second determining module is used to determine multiple candidate poses of the robot's arm end based on the target position; A selection module is used to determine the target end-effector pose of the arm end from a plurality of candidate poses; The button module is used to control the end effector of the arm to move to the target end effector position and to control the end effector to press the target button.

9. A robot comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images