Robot control method, device and equipment and storage medium

Abstract

The embodiment of the invention provides a robot control method and device, equipment and a storage medium, and the method comprises the steps: obtaining a first image which is shot by a first camera and comprises a robot and a target object, and obtaining a second image which is shot by a second camera and comprises the robot and the target object; determining first direction deviation information in the first image and second direction deviation information in the second image; determining third direction deviation information according to the first image and the second image; determining a target motion instruction of the robot according to the first direction deviation information, the second direction deviation information and the third direction deviation information; and controlling the robot to execute the target motion instruction on the target object. Compared with a positioning scheme based on a monocular camera, the embodiment of the invention has the advantages that higher positioning precision can be achieved without maintaining and updating a feature database; compared with a traditional binocular vision positioning scheme, the implementation difficulty of the embodiment of the invention is remarkably reduced, and meanwhile, the requirement on hardware performance is also looser.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a robot control method and device, an electronic device, and a storage medium. Background Art

[0002] In the relevant technical field, intelligent robots typically use two solutions to achieve three-dimensional positioning of target objects: first, depth ranging technology based on a monocular camera; second, parallax ranging technology based on a binocular camera. However, both methods have significant limitations. The monocular camera solution relies on a large sample database for target recognition and distance estimation. This database requires continuous updating and maintenance to ensure recognition rate, and the overall ranging accuracy is relatively low. Although the binocular camera solution can directly achieve ranging through disparity map calculation, its hardware cost is high and the computing performance requirements are strict. It usually requires a dedicated image processing chip to meet real-time requirements.

[0003] Therefore, how to achieve high-precision three-dimensional positioning of target objects by intelligent robots while ensuring low cost and simplicity of methods remains a key technical problem that needs to be solved urgently in this field. Summary of the Invention

[0004] The embodiments of the present application provide a robot control method to solve the problem of how to achieve high-precision three-dimensional positioning of a target object by an intelligent robot while ensuring low cost and simplicity of the method.

[0005] Correspondingly, an embodiment of the present application also provides a robot control device, an electronic device and a storage medium to ensure the implementation and application of the above method.

[0006] To address the above-mentioned issues, an embodiment of the present application discloses a control method for a robot, wherein the robot moves within a target space, a target object is located on a target plane within the target space, a first camera and a second camera are mounted on a plane parallel to the target plane, and the shooting directions of the first camera and the second camera are perpendicular to each other. The method comprises:

[0007] Acquire a first image including the robot and the target object captured by the first camera, and a second image including the robot and the target object captured by the second camera;

[0008] determining first direction deviation information in the first image and second direction deviation information in the second image;

[0009] determining third direction deviation information based on the first image and the second image;

[0010] determining a target motion instruction of the robot according to the first direction deviation information, the second direction deviation information, and the third direction deviation information;

[0011] The robot is controlled to execute the target motion instruction on the target object.

[0012] Optionally, after acquiring the first image captured by the first camera and including the robot and the target object, and the second image captured by the second camera and including the robot and the target object, the method includes:

[0013] A preset target detection algorithm is used to respectively identify the robot and the target object in the first image and the second image, and obtain the position data of the robot in the first image, the position data of the target object in the first image, the position data of the robot in the second image, and the position data of the target object in the second image.

[0014] Optionally, the position data of the robot in the first image includes a first robot identification frame, the position data of the target object in the first image includes a first target object identification frame, and determining the first direction deviation information in the first image includes:

[0015] determining a first distance from a center of the first robot identification frame to a left edge of the first image;

[0016] determining a second distance from the center of the first target object recognition frame to the left edge of the first image;

[0017] The difference between the second distance and the first distance is used as the first direction deviation information.

[0018] Optionally, the position data of the robot in the second image includes a second robot identification frame, the position data of the target object in the second image includes a second target object identification frame, and determining the second direction deviation information in the second image includes:

[0019] determining a third distance from the center of the second robot identification frame to the left edge of the second image;

[0020] determining a fourth distance from a center of the second target object recognition frame to a left edge of the second image;

[0021] The difference between the fourth distance and the third distance is used as the second direction deviation information.

[0022] Optionally, the position data of the robot in the first image has a first confidence level, and the position data of the robot in the second image has a second confidence level, and determining the third direction deviation information based on the first image and the second image includes:

[0023] comparing the first confidence level and the second confidence level to obtain a comparison result;

[0024] determining a target image from the first image and the second image according to the comparison result;

[0025] The third direction deviation information is determined according to the target image.

[0026] Optionally, the target image has a corresponding third robot identification frame and a third target object identification frame, and determining the third direction deviation information according to the target image includes:

[0027] determining a fifth distance from the third robot identification frame to an upper boundary of the target image;

[0028] obtaining a bottom position of the robot according to the height of the third robot identification frame and the fifth distance;

[0029] determining a sixth distance from the third target object recognition frame to an upper boundary of the target image;

[0030] Obtaining a top position of the target object according to the height of the third target object recognition frame and the sixth distance;

[0031] The difference between the top position of the target object and the bottom position of the robot is used as the third direction deviation information.

[0032] Optionally, controlling the robot to execute the target motion instruction on the target object includes:

[0033] controlling the robot to execute the target motion instruction;

[0034] Acquire a third image captured by the first camera and including the robot and the target object, and a fourth image captured by the second camera and including the robot and the target object;

[0035] determining updated first direction deviation information, updated second direction deviation information, and updated third direction deviation information according to the third image and the fourth image;

[0036] If the updated first direction deviation information, the updated second direction deviation information, and the updated third direction deviation information satisfy a preset deviation range, controlling the robot to perform a preset operation on the target object;

[0037] If the updated first direction deviation information, the updated second direction deviation information, and the updated third direction deviation information do not meet the preset deviation range, determining an adjustment motion instruction of the robot according to the updated first direction deviation information, the updated second direction deviation information, and the updated third direction deviation information;

[0038] The robot is controlled to execute the adjustment motion instruction.

[0039] The present application also discloses a control device for a robot, wherein the robot moves in a target space, a target object is located on a target plane in the target space, a first camera and a second camera are mounted on a plane parallel to the target plane, and a shooting direction of the first camera and a shooting direction of the second camera are perpendicular to each other. The device comprises:

[0040] an image acquisition module, configured to acquire a first image captured by the first camera and including the robot and the target object, and a second image captured by the second camera and including the robot and the target object;

[0041] a first information determining module, configured to determine first direction deviation information in the first image and second direction deviation information in the second image;

[0042] a second information determining module, configured to determine third direction deviation information based on the first image and the second image;

[0043] an instruction determination module, configured to determine a target motion instruction of the robot according to the first direction deviation information, the second direction deviation information, and the third direction deviation information;

[0044] A motion control module is used to control the robot to execute the target motion instruction on the target object.

[0045] An embodiment of the present application also discloses an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes the robot control method as described in one or more embodiments of the present application.

[0046] The embodiments of the present application also disclose one or more machine-readable media having executable codes stored thereon. When the executable codes are executed, the processor executes the robot control method as described in one or more of the embodiments of the present application.

[0047] Compared with the prior art, the embodiments of the present application have the following advantages:

[0048] In an embodiment of the present application, a first image captured by a first camera and including a robot and a target object, and a second image captured by a second camera and including the robot and the target object are obtained; first direction deviation information in the first image and second direction deviation information in the second image are determined; third direction deviation information is determined based on the first image and the second image; a target motion instruction of the robot is determined based on the first direction deviation information, the second direction deviation information, and the third direction deviation information; and the robot is controlled to execute the target motion instruction on the target object. In an embodiment of the present application, only a first camera and a second camera with orthogonal shooting directions are installed within the target space where the robot moves and on a plane parallel to the target plane where the target object is placed. The first and second images captured by the first and second cameras, respectively, can be used to obtain the first direction deviation information, the second direction deviation information, and the third direction deviation information, thereby achieving precise positioning of the robot and the target object in the target space. Based on this deviation information, the position and posture of the robot can be adjusted in real time to ensure that it can accurately perform specific operations on the target object. A dual-monocular camera system with a preset orthogonal layout, combined with a simple algorithm calculation, can achieve high-precision three-dimensional positioning of the robot on the target object.

[0049] Compared with the positioning solution based on monocular camera in the prior art, the embodiment of the present application can achieve higher positioning accuracy without maintaining and updating a huge feature database; compared with the traditional binocular vision positioning solution, the calculation method of the embodiment of the present application is simpler and more efficient, the implementation difficulty is significantly reduced, and the requirements for hardware performance are also more relaxed, with better practicality and scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a flowchart of the steps of an embodiment of a robot control method of the present application;

[0051] Figure 2 This is a schematic diagram of the camera layout of an embodiment of a robot control method of the present application;

[0052] Figure 3 This is a schematic diagram of obtaining first direction deviation information in an embodiment of a robot control method of the present application;

[0053] Figure 4This is a schematic diagram of obtaining second direction deviation information in an embodiment of a robot control method of the present application;

[0054] Figure 5 This is a schematic diagram of obtaining third-direction deviation information in an embodiment of a robot control method of the present application;

[0055] Figure 6 This is a structural block diagram of an embodiment of a robot control device of the present application;

[0056] Figure 7 It is a structural diagram of a device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0058] Reference Figure 1 , is a flowchart of an embodiment of a robot control method of the present application, comprising the following steps:

[0059] Step 101: Acquire a first image including the robot and the target object captured by the first camera, and a second image including the robot and the target object captured by the second camera.

[0060] In the embodiments of the present application, the robot may be an embodied intelligent agent. An embodied intelligent agent refers to an intelligent system that has a physical or virtual body and can interact with the environment through perception and action. Its core lies in "embodiment", emphasizing that the cognitive, decision-making and learning abilities of the intelligent agent depend on the real-time interaction between its body and the environment, rather than isolated data processing. The robot can move within a target space, which is the space in which the robot performs a target task. The target task is related to the target object, such as grasping the target object, repairing the target object, etc. The embodiments of the present application do not limit the specific target tasks that the robot needs to perform.

[0061] The target object corresponding to the target task is located on a target plane within the target space. In the embodiment of the present application, a first camera and a second camera are installed on the same plane parallel to the target plane. The first camera and the second camera are both monocular cameras, and the shooting direction of the first camera must be perpendicular to the shooting direction of the second camera. Specifically, with the first camera as the origin, a first ray is drawn along the shooting direction of the first camera, and with the second camera as the origin, a second ray is drawn along the shooting direction of the second camera. The first ray and the second ray are perpendicular to each other and intersect.

[0062] Reference Figure 2, is a schematic diagram of the camera layout of an embodiment of a robot control method of the present application.

[0063] like Figure 2 As shown, the robot is a manipulator in a cube, the cube is the target space where the robot can move, and the target object cylinder is located on the target plane of the target space. In the cube, the x, y, and z directions are extended with vertex A as the origin. When setting the first camera, if the shooting direction of the first camera is along the positive direction of the x direction (i.e., the first camera is on the left side of the target object) or the reverse direction (i.e., the first camera is on the right side of the target object), then when setting the second camera, the shooting direction of the second camera can be along the positive direction of the y direction (i.e., the second camera is in front of the target object) or the reverse direction (i.e., the second camera is behind the target object), so that the first ray and the second ray are perpendicular to each other and intersect.

[0064] In step 101, since the first camera and the second camera are located in the same plane and the shooting directions are vertical, the height of the plane in which the first camera and the second camera are located can be adjusted so that the first camera and the second camera can capture the first image and the second image containing the robot and the target object from the same plane.

[0065] Step 102: Determine first direction deviation information in the first image and second direction deviation information in the second image.

[0066] Because the robot needs to perform a target task on a target object, it is necessary to adjust the robot's position in the target space based on the first and second images so that the robot can complete the target task. In step 102, first direction deviation information is determined from the first image, and second direction deviation information is determined from the second image. The first direction deviation information is used to correct the deviation between the robot and the target object in the first direction, and the second direction deviation information is used to correct the deviation between the robot and the target object in the second direction. Both the first direction and the second direction are parallel to the target plane.

[0067] Step 103: Determine third direction deviation information according to the first image and the second image.

[0068] In step 103, a target image is determined from the first and second images, thereby determining third direction deviation information based on the target image. The third direction deviation information is used to correct the deviation between the robot and the target object in a third direction, which is perpendicular to the target plane.

[0069] Step 104 : Determine a target motion instruction of the robot according to the first direction deviation information, the second direction deviation information, and the third direction deviation information.

[0070] After obtaining the deviation information between the robot and the target object in three directions, in step 104, a target motion instruction for the robot can be determined based on the first direction deviation information, the second direction deviation information, and the third direction deviation information. The target motion instruction includes a motion instruction for the robot to adjust the deviation between its own position and the position of the target object, as well as a motion instruction for the robot to perform a target task on the target object.

[0071] The robot adjusts its own position, which means that the robot needs to adjust its own position based on the first direction deviation information, the second direction deviation information, and the third direction deviation information, thereby reducing the position deviation in the three directions. The robot's movement distance in the three directions can be calculated based on a pre-calibrated scale and the deviation information in the three directions. The pre-calibrated scale reflects the correspondence between the single pixel difference in the image captured by the camera and the actual distance, and the first direction deviation information, the second direction deviation information, and the third direction deviation information are actually the pixel differences between the positions of the robot and the target object in the image in the three directions. Therefore, the direction and distance that the robot needs to adjust can be calculated based on the above data, thereby generating the corresponding motion instructions.

[0072] Step 105: Control the robot to execute the target motion instruction on the target object.

[0073] In step 105, the robot is controlled to adjust its position according to the target motion instruction and then perform the target task on the target object.

[0074] The embodiment of the present application only requires the installation of a first camera and a second camera with orthogonal shooting directions within the target space of the robot's activity, on a plane parallel to the target plane where the target object is placed. Through the first and second images captured by the first and second cameras, respectively, first direction deviation information, second direction deviation information, and third direction deviation information can be obtained, thereby achieving precise positioning of the robot and the target object in the target space. Based on this deviation information, the robot's position and posture can be adjusted in real time to ensure that it can accurately perform specific operations on the target object. Using a dual-monocular camera system with a preset orthogonal layout, combined with simple algorithmic calculations, the robot can achieve high-precision three-dimensional positioning of the target object.

[0075] Compared with the positioning solution based on monocular camera in the prior art, the embodiment of the present application can achieve higher positioning accuracy without maintaining and updating a huge feature database; compared with the traditional binocular vision positioning solution, the calculation method of the embodiment of the present application is simpler and more efficient, the implementation difficulty is significantly reduced, and the requirements for hardware performance are also more relaxed, with better practicality and scalability.

[0076] Optionally, after step 101, the method includes:

[0077] A preset target detection algorithm is used to respectively identify the robot and the target object in the first image and the second image, and obtain the position data of the robot in the first image, the position data of the target object in the first image, the position data of the robot in the second image, and the position data of the target object in the second image.

[0078] In this embodiment, after obtaining the first image and the second image, target recognition is performed on the first image and the second image using a preset target detection algorithm. In one embodiment, the preset target detection algorithm may be a YOLO algorithm (You Only Look Once, a real-time target detection algorithm). The YOLO algorithm may be used to identify the robot and the target object in the first image, as well as the robot and the target object in the second image, and output the position data of the robot in the first image, the position data of the target object in the first image, the position data of the robot in the second image, and the position data of the target object in the second image. In one embodiment, the output position data may be in json (JavaScript Object Notation, a lightweight data exchange format) format.

[0079] The embodiment of the present application adopts a preset target detection algorithm to perform target identification on the robot and the target object in the first image and the second image respectively. The obtained position data can be used to subsequently determine the first direction deviation information, the second direction deviation information and the third direction deviation information.

[0080] Optionally, the position data of the robot in the first image includes a first robot identification frame, and the position data of the target object in the first image includes a first target object identification frame, and step 102 includes:

[0081] determining a first distance from a center of the first robot identification frame to a left edge of the first image;

[0082] determining a second distance from the center of the first target object recognition frame to the left edge of the first image;

[0083] The difference between the second distance and the first distance is used as the first direction deviation information.

[0084] The first direction deviation information is calculated based on the position data of the robot in the first image and the position data of the target object in the first image. Figure 2 If the first camera is arranged on the left side of the target object, the first direction is the y direction.

[0085] Specifically, after the preset target detection algorithm performs target recognition on the first image, the output position data includes a first robot recognition frame corresponding to the robot and a first target object recognition frame corresponding to the target object. A first distance from the center of the first robot recognition frame to the left edge of the first image and a second distance from the center of the first target object recognition frame to the left edge of the first image are calculated. The first distance is subtracted from the second distance to obtain first directional deviation information, i.e., the positional deviation between the robot and the target object in the first direction.

[0086] In the embodiment of the present application, the first direction deviation information can be determined by the position deviation between the first robot identification frame and the first target object identification frame. The calculation method is simple and efficient, and the implementation difficulty is low. The first direction deviation information can be used for subsequent adjustment of the robot position.

[0087] Optionally, the position data of the robot in the second image includes a second robot identification frame, and the position data of the target object in the second image includes a second target object identification frame. Step 102 further includes:

[0088] determining a third distance from the center of the second robot identification frame to the left edge of the second image;

[0089] determining a fourth distance from a center of the second target object recognition frame to a left edge of the second image;

[0090] The difference between the fourth distance and the third distance is used as the second direction deviation information.

[0091] The second direction deviation information is calculated based on the position data of the robot in the second image and the position data of the target object in the second image. Figure 2 If the second camera is arranged in front of the target object, the second direction is the x direction.

[0092] Specifically, after the preset target detection algorithm performs target recognition on the second image, the output position data includes a second robot recognition frame corresponding to the robot and a second target object recognition frame corresponding to the target object. A third distance from the center of the second robot recognition frame to the left edge of the second image and a fourth distance from the center of the second target object recognition frame to the left edge of the second image are calculated. The third distance is subtracted from the fourth distance to obtain the second direction deviation information, i.e., the positional deviation between the robot and the target object in the second direction.

[0093] In the embodiment of the present application, the second direction deviation information can be determined by the position deviation between the second robot identification frame and the second target object identification frame. The calculation method is simple and efficient, and the implementation difficulty is low. The second direction deviation information can be used to subsequently adjust the position of the robot.

[0094] Optionally, the position data of the robot in the first image has a first confidence level, and the position data of the robot in the second image has a second confidence level, and step 103 includes:

[0095] comparing the first confidence level and the second confidence level to obtain a comparison result;

[0096] determining a target image from the first image and the second image according to the comparison result;

[0097] The third direction deviation information is determined according to the target image.

[0098] Since the preset target detection algorithm performs target recognition on the first image and the second image respectively, the output position data also includes the confidence level of the robot position data in the image. Therefore, the embodiment of the present application can compare the first confidence level corresponding to the first image and the second confidence level corresponding to the second image to obtain the image corresponding to the higher confidence level of the first confidence level and the second confidence level. For example, if the first confidence level of the robot position data in the first image is greater than the second confidence level of the robot position data in the second image, the first image can be used as the target image. The third direction deviation information is determined based on the robot position data and the target object position data in the target image.

[0099] The embodiment of the present application selects the image with higher confidence level of robot position data from the first image and the second image as the target image, and then determines the third direction deviation information based on the target image, which can improve the accuracy of the third direction deviation information and thus improve the accuracy of three-dimensional positioning.

[0100] Optionally, the target image has a corresponding third robot identification frame and a third target object identification frame, and determining the third direction deviation information according to the target image includes:

[0101] determining a fifth distance from the third robot identification frame to an upper boundary of the target image;

[0102] obtaining a bottom position of the robot according to the height of the third robot identification frame and the fifth distance;

[0103] determining a sixth distance from the third target object recognition frame to an upper boundary of the target image;

[0104] Obtaining a top position of the target object according to the height of the third target object recognition frame and the sixth distance;

[0105] The difference between the top position of the target object and the bottom position of the robot is used as the third direction deviation information.

[0106] In this embodiment, the target image has a corresponding third robot recognition frame and a third target object recognition frame. If the target image is the first image, the third robot recognition frame is the first robot recognition frame, and the third target object recognition frame is the first target object recognition frame; if the target image is the second image, the third robot recognition frame is the second robot recognition frame, and the third target object recognition frame is the second target object recognition frame.

[0107] Determine the fifth distance from the third robot identification frame to the upper boundary of the target image, and obtain the bottom position of the robot based on the height of the third robot identification frame and the fifth distance. Specifically, calculate half of the height of the third robot identification frame as the first half value, and use the sum of the first half value and the fifth distance as the bottom position of the robot. Determine the sixth distance from the third target object identification frame to the upper boundary of the target image, and obtain the top position of the target object based on the height of the third target object identification frame and the sixth distance. Specifically, calculate half of the height of the third target object identification frame as the second half value, and use the difference between the sixth distance and the second half value as the top position of the target object. Thereafter, use the difference between the top position of the target object and the bottom position of the robot as the third direction deviation information, that is, the position deviation between the robot and the target object in the third direction. Refer to Figure 2 , the third direction is the z direction.

[0108] The embodiment of the present application calculates the top position of the target object and the bottom position of the robot based on the third robot identification frame and the third target object identification frame of the target image, thereby obtaining third direction deviation information. The calculation method is simple and efficient, and the implementation difficulty is low. The third direction deviation information can be used for subsequent adjustment of the robot position.

[0109] Optionally, step 105 includes:

[0110] controlling the robot to execute the target motion instruction;

[0111] Acquire a third image captured by the first camera and including the robot and the target object, and a fourth image captured by the second camera and including the robot and the target object;

[0112] determining updated first direction deviation information, updated second direction deviation information, and updated third direction deviation information according to the third image and the fourth image;

[0113] If the updated first direction deviation information, the updated second direction deviation information, and the updated third direction deviation information satisfy a preset deviation range, controlling the robot to perform a preset operation on the target object;

[0114] If the updated first direction deviation information, the updated second direction deviation information, and the updated third direction deviation information do not meet the preset deviation range, determining an adjustment motion instruction of the robot according to the updated first direction deviation information, the updated second direction deviation information, and the updated third direction deviation information;

[0115] The robot is controlled to execute the adjustment motion instruction.

[0116] In an embodiment of the present application, since some deviations may occur when the robot adjusts its own position when executing motion instructions, the robot has not actually reached the target position for performing the target task. Therefore, it is necessary to repeatedly adjust the position of the robot until the first direction deviation information, the second direction deviation information and the third direction deviation information all meet the preset deviation range.

[0117] Specifically, after the robot is controlled to execute the target motion instruction, a third image captured by the first camera and including the robot and the target object is obtained again, as well as a fourth image captured by the second camera and including the robot and the target object. Updated first direction deviation information, updated second direction deviation information, and updated third direction deviation information are determined based on the third and fourth images. The steps for calculating the updated three direction deviation information have been described in detail above and will not be repeated here. The updated first direction deviation information, updated second direction deviation information, and updated third direction deviation information are compared with a preset deviation range to determine whether the robot's position needs to be adjusted again.

[0118] In one embodiment, the preset deviation range can be determined based on parameters such as the distance between the camera and the target object, or the robot's motion accuracy. When the robot requires the assistance of a component such as a manipulator to perform a target task on the target object, the preset deviation range can also be determined based on the size of the manipulator. For example, if the preset deviation range is determined based on the distance between the camera and the target object, the closer the camera is to the target object, the shorter the actual distance represented by each pixel in the captured image. If the preset deviation range is expressed in pixel distance, the preset deviation range can be set larger, and vice versa for farther distances from the camera to the target object. For example, if the preset deviation range is determined based on the robot's motion accuracy, the preset deviation range can be set larger for higher robot motion accuracy, and vice versa for lower motion accuracy. For example, if the preset deviation range is determined based on the size of the manipulator, a larger manipulator can grasp the target object within a wider range, so the preset deviation range can be set larger, and vice versa for smaller manipulators.

[0119] If the updated first direction deviation information, the updated second direction deviation information, and the updated third direction deviation information satisfy the preset deviation range, that is, the deviation information is all within the preset deviation range, then the robot can be controlled to perform the preset operation on the target object; if the updated first direction deviation information, the updated second direction deviation information, and the updated third direction deviation information do not satisfy the preset deviation range, that is, at least one deviation information is outside the preset deviation range, then the position of the robot needs to be adjusted again, and the adjustment motion instruction of the robot is determined again according to the updated first direction deviation information, the updated second direction deviation information, and the updated third direction deviation information, and the robot is controlled to execute the adjustment motion instruction until the updated first direction deviation information, the updated second direction deviation information, and the updated third direction deviation information satisfy the preset deviation range. Among them, the method of re-determining the adjustment motion instruction of the robot according to the updated first direction deviation information, the updated second direction deviation information, and the updated third direction deviation information is the same as the method of determining the target motion instruction above, and will not be repeated here.

[0120] The embodiment of the present application obtains the third image and the fourth image to determine whether the updated first direction deviation information, the updated second direction deviation information and the updated third direction deviation information between the robot and the target object meet the preset deviation range, thereby repeatedly adjusting the position of the robot to ensure that it can accurately perform specific operations on the target object.

[0121] In order to make those skilled in the art more clearly understand the technical solution of this application, Figures 3 to 5 A robot control method of the present application is exemplified.

[0122] Reference Figure 3 , is a schematic diagram of obtaining first direction deviation information in an embodiment of a robot control method of the present application.

[0123] Reference Figure 2 In the camera layout, when the first camera is installed in front of the target object, the shooting direction is the y direction. Based on the first image captured, the deviation information between the robot and the target object in the x direction can be determined, that is, the first direction deviation information. The YOLO algorithm is used to perform target recognition on the first image, and the following json data can be output:

[0124]

[0125]

[0126] Based on the distance from the center of the robot recognition frame to the left edge of the image and the distance from the center of the target object recognition frame to the left edge of the image in the above data, the first direction deviation information between the robot and the target object is calculated as offsetX = target object "x" - robot "x" = 396 - 604 = -208

[0127] Reference Figure 4 , is a schematic diagram of obtaining the second direction deviation information of an embodiment of a robot control method of the present application.

[0128] Reference Figure 2 In the camera layout, when the second camera is installed on the left side of the target object, the shooting direction is the x-direction. Based on the second image captured, the deviation information between the robot and the target object in the y-direction can be determined, that is, the second direction deviation information. The YOLO algorithm is used to perform target recognition on the second image, and the following json data can be output:

[0129]

[0130]

[0131] Based on the distance from the center of the robot recognition frame to the left edge of the image and the distance from the center of the target object recognition frame to the left edge of the image in the above data, the second direction deviation information between the robot and the target object is calculated as offsetY = target object "x" - robot "x" = 398 - 596 = -198

[0132] Reference Figure 5 , is a schematic diagram of obtaining third-direction deviation information in an embodiment of a robot control method of the present application.

[0133] The image with the highest robot recognition confidence is selected from the first and second images as the image. In this embodiment, the second image is selected as the target image. Therefore, the JSON data of the second image is used to calculate the third direction deviation information, that is, the deviation in the z direction. First, the pixel coordinates of the robot's bottom position are calculated based on the height of the third robot recognition frame and the fifth distance: bottomy = robot "y" + robot "h" / 2 = 570 + 202 / 2 = 671. Then, the pixel coordinates of the target object's top position are calculated based on the height of the third target object recognition frame and the sixth distance: topy = target object "y" - target object "h" / 2 = 932 - 276 / 2 = 794. Finally, the third direction deviation information offsetZ is calculated as topy - bottomy = 794 - 671 = 123.

[0134] After obtaining the deviation information in the three directions, the proportional relationship between the number of image pixels and the actual movement distance of the robot is converted into the adjustment distance of the robot in the x, y, and z directions, and then the position of the robot in the x, y, and z directions is adjusted accordingly.

[0135] The embodiment of the present application only requires the installation of a first camera and a second camera with orthogonal shooting directions within the target space of the robot's activity, on a plane parallel to the target plane where the target object is placed. Through the first and second images captured by the first and second cameras, respectively, first direction deviation information, second direction deviation information, and third direction deviation information can be obtained, thereby achieving precise positioning of the robot and the target object in the target space. Based on this deviation information, the robot's position and posture can be adjusted in real time to ensure that it can accurately perform specific operations on the target object. Using a dual-monocular camera system with a preset orthogonal layout, combined with simple algorithmic calculations, the robot can achieve high-precision three-dimensional positioning of the target object.

[0136] Compared with the positioning solution based on monocular camera in the prior art, the embodiment of the present application can achieve higher positioning accuracy without maintaining and updating a huge feature database; compared with the traditional binocular vision positioning solution, the calculation method of the embodiment of the present application is simpler and more efficient, the implementation difficulty is significantly reduced, and the requirements for hardware performance are also more relaxed, with better practicality and scalability.

[0137] It should be noted that for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the order of the actions described, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0138] On the basis of the above embodiments, this embodiment further provides a robot control device, which is applied to electronic devices such as terminal devices and servers.

[0139] Reference Figure 6 , shows a structural block diagram of an embodiment of a robot control device of the present application, which may specifically include the following modules:

[0140] An image acquisition module 601 is configured to acquire a first image captured by the first camera and including the robot and the target object, and a second image captured by the second camera and including the robot and the target object;

[0141] A first information determining module 602 is configured to determine first direction deviation information in the first image and second direction deviation information in the second image;

[0142] A second information determining module 603 is configured to determine third direction deviation information based on the first image and the second image;

[0143] An instruction determination module 604 is configured to determine a target motion instruction of the robot according to the first direction deviation information, the second direction deviation information, and the third direction deviation information;

[0144] The motion control module 605 is used to control the robot to execute the target motion instruction on the target object.

[0145] Optionally, the device comprises:

[0146] a target detection module, configured to use a preset target detection algorithm to respectively identify the robot and the target object in the first image and the second image, and obtain position data of the robot in the first image, position data of the target object in the first image, position data of the robot in the second image, and position data of the target object in the second image.

[0147] Optionally, the position data of the robot in the first image includes a first robot identification frame, the position data of the target object in the first image includes a first target object identification frame, and the first information determination module 602 includes:

[0148] a first distance determination submodule, configured to determine a first distance from the center of the first robot identification frame to the left edge of the first image;

[0149] a second distance determination submodule, configured to determine a second distance from the center of the first target object recognition frame to the left edge of the first image;

[0150] The first direction deviation information determining submodule is configured to use the difference between the second distance and the first distance as the first direction deviation information.

[0151] Optionally, the position data of the robot in the second image includes a second robot identification frame, the position data of the target object in the second image includes a second target object identification frame, and the first information determination module 602 includes:

[0152] a third distance determination submodule, configured to determine a third distance from the center of the second robot identification frame to the left boundary of the second image;

[0153] a fourth distance determination submodule, configured to determine a fourth distance from the center of the second target object recognition frame to the left edge of the second image;

[0154] The second direction deviation information determining submodule is configured to use the difference between the fourth distance and the third distance as the second direction deviation information.

[0155] Optionally, the position data of the robot in the first image has a first confidence level, and the position data of the robot in the second image has a second confidence level, and the second information determination module 603 includes:

[0156] a confidence determination submodule, configured to compare the first confidence level with the second confidence level to obtain a comparison result;

[0157] a target image determination submodule, configured to determine a target image from the first image and the second image according to the comparison result;

[0158] The third direction deviation information determining submodule is configured to determine the third direction deviation information according to the target image.

[0159] Optionally, the target image has a corresponding third robot recognition frame and a third target object recognition frame, and the third direction deviation information determination submodule is specifically configured to:

[0160] determining a fifth distance from the third robot identification frame to an upper boundary of the target image;

[0161] obtaining a bottom position of the robot according to the height of the third robot identification frame and the fifth distance;

[0162] determining a sixth distance from the third target object recognition frame to an upper boundary of the target image;

[0163] Obtaining a top position of the target object according to the height of the third target object recognition frame and the sixth distance;

[0164] The difference between the top position of the target object and the bottom position of the robot is used as the third direction deviation information.

[0165] Optionally, the motion control module 605 includes:

[0166] A target instruction execution submodule, used to control the robot to execute the target motion instruction;

[0167] an image updating submodule, configured to obtain a third image captured by the first camera and including the robot and the target object, and a fourth image captured by the second camera and including the robot and the target object;

[0168] a deviation information updating submodule, configured to determine updated first direction deviation information, updated second direction deviation information, and updated third direction deviation information based on the third image and the fourth image;

[0169] a preset operation execution submodule, configured to control the robot to perform a preset operation on the target object if the updated first direction deviation information, the updated second direction deviation information, and the updated third direction deviation information satisfy a preset deviation range;

[0170] an adjustment instruction acquisition submodule, configured to determine an adjustment motion instruction for the robot according to the updated first direction deviation information, the updated second direction deviation information, and the updated third direction deviation information if the updated first direction deviation information, the updated second direction deviation information, and the updated third direction deviation information do not satisfy a preset deviation range;

[0171] The adjustment instruction execution submodule is used to control the robot to execute the adjustment motion instruction.

[0172] An embodiment of the present application further provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0173] The present application provides one or more machine-readable media having instructions stored thereon, which, when executed by one or more processors, cause an electronic device to perform one or more of the methods described in the above embodiments. In the present application, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0174] The embodiments of the present disclosure may be implemented as a device configured as desired using any appropriate hardware, firmware, software, or any combination thereof, and the device may include electronic devices such as terminal devices and servers (clusters). Figure 7 An exemplary apparatus 700 that can be used to implement various embodiments described in this application is schematically illustrated.

[0175] For one embodiment, Figure 7An exemplary apparatus 700 is shown having one or more processors 702, a control module (chip set) 704 coupled to at least one of the processor(s) 702, a memory 706 coupled to the control module 704, a non-volatile memory (NVM) / storage device 708 coupled to the control module 704, one or more input / output devices 710 coupled to the control module 704, and a network interface 712 coupled to the control module 704.

[0176] The processor 702 may include one or more single-core or multi-core processors, and the processor 702 may include any combination of general-purpose processors or dedicated processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, the apparatus 700 can serve as a terminal device, server (cluster), or other device described in the embodiments of the present application.

[0177] In some embodiments, the apparatus 700 may include one or more computer-readable media (e.g., memory 706 or NVM / storage 708) having instructions 714 and one or more processors 702 configured in conjunction with the one or more computer-readable media to execute the instructions 714 to implement a module to perform the actions described in the present disclosure.

[0178] For one embodiment, the control module 704 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 702 and / or any suitable device or component in communication with the control module 704 .

[0179] The control module 704 may include a memory controller module to provide an interface to the memory 706. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0180] The memory 706 can be used, for example, to load and store data and / or instructions 714 for the device 700. For one embodiment, the memory 706 can include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the memory 706 can include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0181] For one embodiment, the control module 704 may include one or more input / output controllers to provide interfaces to the NVM / storage device 708 and the input / output device(s) 710 .

[0182] For example, NVM / storage 708 may be used to store data and / or instructions 714. NVM / storage 708 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0183] NVM / storage device 708 may include storage resources that are physically part of the device on which apparatus 700 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 708 may be accessible over a network via input / output device(s) 710.

[0184] (One or more) input / output devices 710 may provide an interface for apparatus 700 to communicate with any other appropriate devices. Input / output devices 710 may include communication components, audio components, sensor components, etc. Network interface 712 may provide an interface for apparatus 700 to communicate via one or more networks. Apparatus 700 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof for wireless communication.

[0185] For one embodiment, at least one of the processor(s) 702 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 704. For one embodiment, at least one of the processor(s) 702 may be packaged together with the logic of one or more controllers of the control module 704 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 702 may be integrated on the same die with the logic of one or more controllers of the control module 704. For one embodiment, at least one of the processor(s) 702 may be integrated on the same die with the logic of one or more controllers of the control module 704 to form a system-on-chip (SoC).

[0186] In various embodiments, the apparatus 700 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the apparatus 700 may have more or fewer components and / or a different architecture. For example, in some embodiments, the apparatus 700 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0187] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in the memory or NVM / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.

[0188] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0189] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0190] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable robot control terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable robot control terminal device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0191] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable robot control terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0192] These computer program instructions may also be loaded onto a computer or other programmable robot control terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0193] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0194] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0195] The above is a detailed introduction to a robot control method and device, an electronic device and a storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A robot control method, characterized in that: The robot moves in a target space, a target object is located on a target plane in the target space, a first camera and a second camera are installed on a plane parallel to the target plane, and a shooting direction of the first camera and a shooting direction of the second camera are perpendicular to each other. The method includes: Acquire a first image including the robot and the target object captured by the first camera, and a second image including the robot and the target object captured by the second camera; determining first direction deviation information in the first image and second direction deviation information in the second image; determining third direction deviation information based on the first image and the second image; determining a target motion instruction of the robot according to the first direction deviation information, the second direction deviation information, and the third direction deviation information; The robot is controlled to execute the target motion instruction on the target object.

2. The method according to claim 1, characterized in that After acquiring a first image including the robot and the target object captured by the first camera, and a second image including the robot and the target object captured by the second camera, the method includes: A preset target detection algorithm is used to respectively identify the robot and the target object in the first image and the second image, and obtain the position data of the robot in the first image, the position data of the target object in the first image, the position data of the robot in the second image, and the position data of the target object in the second image.

3. The method according to claim 2, characterized in that The position data of the robot in the first image includes a first robot identification frame, the position data of the target object in the first image includes a first target object identification frame, and determining the first direction deviation information in the first image includes: determining a first distance from a center of the first robot identification frame to a left edge of the first image; determining a second distance from the center of the first target object recognition frame to the left edge of the first image; The difference between the second distance and the first distance is used as the first direction deviation information.

4. The method according to claim 2, characterized in that The position data of the robot in the second image includes a second robot identification frame, the position data of the target object in the second image includes a second target object identification frame, and determining the second direction deviation information in the second image includes: determining a third distance from the center of the second robot identification frame to the left edge of the second image; determining a fourth distance from a center of the second target object recognition frame to a left edge of the second image; The difference between the fourth distance and the third distance is used as the second direction deviation information.

5. The method according to claim 2, characterized in that The position data of the robot in the first image has a first confidence level, and the position data of the robot in the second image has a second confidence level. The determining of the third direction deviation information based on the first image and the second image includes: comparing the first confidence level and the second confidence level to obtain a comparison result; determining a target image from the first image and the second image according to the comparison result; The third direction deviation information is determined according to the target image.

6. The method according to claim 5, characterized in that The target image has a corresponding third robot identification frame and a third target object identification frame, and determining the third direction deviation information according to the target image includes: determining a fifth distance from the third robot identification frame to an upper boundary of the target image; obtaining a bottom position of the robot according to the height of the third robot identification frame and the fifth distance; determining a sixth distance from the third target object recognition frame to an upper boundary of the target image; Obtaining a top position of the target object according to the height of the third target object recognition frame and the sixth distance; The difference between the top position of the target object and the bottom position of the robot is used as the third direction deviation information.

7. The method according to claim 1, characterized in that The controlling the robot to execute the target motion instruction on the target object includes: controlling the robot to execute the target motion instruction; Acquire a third image captured by the first camera and including the robot and the target object, and a fourth image captured by the second camera and including the robot and the target object; determining updated first direction deviation information, updated second direction deviation information, and updated third direction deviation information according to the third image and the fourth image; If the updated first direction deviation information, the updated second direction deviation information, and the updated third direction deviation information satisfy a preset deviation range, controlling the robot to perform a preset operation on the target object; If the updated first direction deviation information, the updated second direction deviation information, and the updated third direction deviation information do not meet the preset deviation range, determining an adjustment motion instruction of the robot according to the updated first direction deviation information, the updated second direction deviation information, and the updated third direction deviation information; The robot is controlled to execute the adjustment motion instruction.

8. A robot control device, characterized in that: The robot moves in a target space, a target object is located on a target plane in the target space, a first camera and a second camera are installed on a plane parallel to the target plane, and a shooting direction of the first camera and a shooting direction of the second camera are perpendicular to each other, and the device includes: an image acquisition module, configured to acquire a first image captured by the first camera and including the robot and the target object, and a second image captured by the second camera and including the robot and the target object; a first information determining module, configured to determine first direction deviation information in the first image and second direction deviation information in the second image; a second information determining module, configured to determine third direction deviation information based on the first image and the second image; an instruction determination module, configured to determine a target motion instruction of the robot according to the first direction deviation information, the second direction deviation information, and the third direction deviation information; A motion control module is used to control the robot to execute the target motion instruction on the target object.

9. An electronic device, characterized in that: include: processor; and A memory having executable codes stored thereon, which, when executed, causes the processor to execute the robot control method according to one or more of claims 1-7.

10. One or more machine-readable media having executable codes stored thereon, which, when executed, cause a processor to execute the robot control method according to one or more of claims 1-7.

Images