Mechanical arm control method, device and equipment, mechanical arm and robot

Abstract

The invention relates to a mechanical arm control method and device, computer equipment, a computer readable storage medium, a computer program product, a mechanical arm and a robot. The method comprises the following steps: carrying out object detection on a global image of a placement area, and determining respective category and rotation angle of at least one object and two-dimensional projection points corresponding to respective feature points in a placement area coordinate system; for each object, inquiring a reference point cloud of the object according to the category of the object, and correcting the reference point cloud based on the rotation angle of the object to obtain a rotation point cloud; performing projection relation matching on the two-dimensional projection points corresponding to the feature points of the object and the three-dimensional reference points in the rotating point cloud, and determining the pose of the object in the placement area coordinate system; and according to the pose of the at least one object in the placement area coordinate system, a to-be-grabbed object is selected, and the mechanical arm is controlled to grab the to-be-grabbed object. By the adoption of the method, under the condition that the camera assembling condition is limited, the object grabbing accuracy of the mechanical arm can be improved.

Description

Technical Field

[0001] The present application relates to the field of robotics technology, and in particular to a robotic arm control method, apparatus, computer equipment, computer-readable storage medium, and computer program product, as well as a robotic arm and a robot. Background Art

[0002] With the development of robotics technology, robotic arms have been used in many fields, including but not limited to: large robotic arms used in industrial automation scenarios, or small robotic arms used in lightweight collaboration scenarios.

[0003] Generally, during the operation of a robotic arm, it is usually necessary to equip the robotic arm with multiple high-precision cameras so that the robotic arm can be accurately controlled based on the images captured by the cameras. For example, when facing some three-dimensional objects to be grasped, it is necessary to combine the images captured by multiple cameras to identify the posture of the objects to be grasped, so as to control the robotic arm to grasp them accurately at a suitable grasping angle. If the camera assembly conditions are limited, the accuracy of the robotic arm grasping objects will be reduced. Summary of the invention

[0004] Based on this, in response to the above technical problems, the present application at least provides a robotic arm control method, device, computer equipment, computer-readable storage medium and computer program product, as well as a robotic arm and a robot, which can improve the accuracy of the robotic arm in grasping objects when the camera assembly conditions are limited.

[0005] In a first aspect, the present application provides a robotic arm control method, comprising: acquiring a global image of a placement area; placing at least one object in the placement area; performing object detection on the global image to determine the category and rotation angle of at least one object, and the two-dimensional projection points corresponding to the feature points of at least one object in the placement area coordinate system; for each object, according to the category of the object, querying to obtain a reference point cloud of the object in a reference posture, and correcting the reference point cloud based on the rotation angle of the object to obtain a rotated point cloud of the object; the rotated point cloud includes three-dimensional reference points corresponding to the feature points of the object; matching the projection relationship between the two-dimensional projection points corresponding to the feature points of the object and the three-dimensional reference points to determine the position and posture of the object in the placement area coordinate system; selecting an object to be grasped from at least one object according to the position and posture of the at least one object in the placement area coordinate system, and controlling the robotic arm to grasp the object to be grasped.

[0006] In the above-mentioned robot arm control method, when the camera assembly conditions are limited, if the global image of the placement area is obtained, the category and rotation angle of at least one object can be determined by first performing object detection on the global image, as well as the two-dimensional projection points corresponding to the feature points of at least one object in the placement area coordinate system. Further, for each object, the reference point cloud of the object in the reference posture can be obtained according to the category query of the object, and the reference point cloud can be corrected based on the rotation angle of the object to obtain the rotation point cloud of the object, and then the two-dimensional projection points corresponding to the feature points of the object and the three-dimensional reference points can be matched in the projection relationship to determine the position and posture of the object in the placement area coordinate system, so that the robot arm can be controlled to accurately grasp the object at a suitable angle based on the position and posture. That is, in the case of limited camera assembly conditions, the position and posture of the object can be accurately determined by the above-mentioned robot arm control method, so that the object to be grasped can be selected from at least one object according to the position and posture of at least one object in the placement area coordinate system, and the robot arm can be controlled to accurately grasp the object to be grasped, thereby improving the accuracy of the robot arm in grasping the object.

[0007] In one of the embodiments, object detection is performed on a global image to determine the category and rotation angle of at least one object, as well as the two-dimensional projection point corresponding to the feature point of at least one object in the placement area coordinate system, including: feature extraction is performed on the global image to obtain placement area image features and object image features of at least one object; the placement area image features are used to determine the placement area coordinate system; for each object, object detection is performed based on the object image features of the object to determine the category and rotation angle of the object, as well as the feature points of the object in the global image; based on the placement area image features and the feature points of the object in the global image, the two-dimensional projection point corresponding to the feature point of the object in the placement area coordinate system is determined.

[0008] In this embodiment, feature extraction can be performed on the global image to obtain image features carried by the global image, which is beneficial for further accurately detecting effective information in the global image based on the image features, for example, detecting the category and rotation angle of at least one object, and the two-dimensional projection points corresponding to the feature points of each object in the placement area coordinate system, so as to facilitate the subsequent accurate positioning of the object's posture based on the detected effective information, thereby improving the accuracy of the robot arm in grasping the object.

[0009] In a specific example, an object to be grasped is selected from at least one object according to the posture of each object in the placement area coordinate system, and a robotic arm is controlled to grasp the object to be grasped, including: determining the placement state of at least one object according to the posture of each object in the placement area coordinate system; in the case where there is an object with a tilted placement state, the object with a tilted placement state is used as the object to be grasped; according to the posture of the object to be grasped in the placement area coordinate system, the robotic arm is controlled to grasp the object to be grasped to adjust the placement state of the object to be grasped.

[0010] In this embodiment, after determining the position and posture of at least one object in the coordinate system of the placement area, it is possible to first determine whether there is an object in a tipping state based on the position and posture. If there is, the robot arm can be controlled to accurately grasp the object based on the position and posture of the object in the tipping state to adjust the placement state of the object.

[0011] In one of the embodiments, the global image acquisition scene includes a first scene and a second scene; the first scene refers to: the global image is acquired during the execution of the object placement task in the placement area; the second scene refers to: after the object placement task is completed, the global image is acquired during the execution of the object movement task in the placement area. Based on this, in the case where there is an object whose placement state is a tilted state, the object whose placement state is a tilted state is used as an object to be grasped, including: when the global image acquisition scene is the first scene and there is an object whose placement state is a tilted state, the object placement task in the placement area is continued until the object placement task is completed, and the object in the tilted state is used as an object to be grasped whose placement state needs to be adjusted; when the global image acquisition scene is the second scene and there is an object whose placement state is a tilted state, the object movement task in the placement area is suspended, and the object in the tilted state is used as an object to be grasped whose placement state needs to be adjusted.

[0012] In this embodiment, during the process of executing the object placement task in the placement area, if it is determined that there is an object in a tilted state in the placement area, the object placement task in the placement area can be executed first, and after the execution is completed, the object in the tilted state is used as the object to be grasped for grasping and adjusting the posture, so that the global placement of the objects in the placement area can be completed as soon as possible. During the process of executing the object movement task in the placement area, if it is determined that there is an object in a tilted state in the placement area, the object in the tilted state is used as the object to be grasped for grasping and adjusting the posture to avoid affecting the movement of the object.

[0013] In one of the embodiments, according to the posture of the object to be grasped in the placement area coordinate system, the robotic arm is controlled to grasp the object to be grasped, including: obtaining a first conversion relationship between the camera coordinate system and the placement area coordinate system, and a second conversion relationship between the robotic arm coordinate system and the camera coordinate system; according to the first conversion relationship, converting the posture of the object to be grasped in the placement area coordinate system into the posture in the camera coordinate system, and according to the second conversion relationship, converting the posture in the camera coordinate system into the posture in the robotic arm coordinate system; according to the posture of the object to be grasped in the robotic arm coordinate system, the robotic arm is controlled to grasp the object to be grasped.

[0014] In this embodiment, after determining the object to be grasped, multiple coordinate system conversions can be performed through the conversion relationship between different coordinate systems to convert the position and posture of the object to be grasped in the placement area coordinate system into the position and posture in the robotic arm coordinate system, which is conducive to controlling the robotic arm to accurately grasp the object to be grasped according to the position and posture in the robotic arm coordinate system.

[0015] In some possible implementations, after determining the placement state of at least one object according to the posture of at least one object in the placement area coordinate system, the method further includes: when the acquisition scene of the global image is the second scene and there is no object in a tilted state, determining the distribution data of at least one object in the placement area coordinate system according to the posture of at least one object in the placement area coordinate system; performing object movement strategy analysis based on the distribution data to determine the object to be grasped among the at least one object and the target position corresponding to the object to be grasped in the placement area coordinate system; controlling the robotic arm to grasp the object to be grasped according to the posture of the object to be grasped in the placement area coordinate system, and moving the object to be grasped to the target position.

[0016] In this embodiment, after determining that there is no object in a tipped-over state, an object movement strategy analysis can be performed based on the posture of at least one object in the coordinate system of the placement area to determine the object to be grasped and the target position corresponding to the object to be grasped, thereby controlling the robotic arm to accurately grasp the object to be grasped and move it to the target position based on the posture of the object to be grasped.

[0017] In the second aspect, the present application also provides a robotic arm control device, including: a global image acquisition module, used to acquire a global image of a placement area, in which at least one object is placed; a multi-task detection module, used to perform object detection on the global image, determine the category and rotation angle of at least one object, and the two-dimensional projection point corresponding to the feature point of at least one object in the placement area coordinate system; a point cloud acquisition module, used to query and obtain a reference point cloud of the object in a reference posture for each object according to the category of the object, and correct the reference point cloud based on the rotation angle of the object to obtain a rotated point cloud of the object, wherein the rotated point cloud includes three-dimensional reference points corresponding to the feature points of the object; a posture determination module, used to match the projection relationship between the two-dimensional projection points corresponding to the feature points of the object and the three-dimensional reference points, and determine the posture of the object in the placement area coordinate system; an object grasping module, used to select an object to be grasped from at least one object according to the posture of at least one object in the placement area coordinate system, and control the robotic arm to grasp the object to be grasped.

[0018] In a third aspect, the present application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned robot arm control method when executing the computer program.

[0019] In a fourth aspect, the present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned robotic arm control method is implemented.

[0020] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the above-mentioned robotic arm control method when executed by a processor.

[0021] In the sixth aspect, the present application also provides a robotic arm, comprising: a robotic arm body, the robotic arm body comprising an action part and an execution part connected to the end of the action part, the execution part being used to interact with at least one object placed in a placement area; a robotic arm controller, being used to obtain a global image of the placement area, and based on the global image, execute the robotic arm control method provided in the first aspect to control the execution part to interact with at least one object.

[0022] In a seventh aspect, the present application also provides a robot, comprising: an image collector, used to collect a global image of a placement area; and a robotic arm provided in the sixth aspect, used to interact with at least one object placed in the placement area based on the global image.

[0023] The above-mentioned robotic arm control device, computer equipment, computer-readable storage medium, computer program product, robotic arm and robot, when the camera assembly conditions are limited, can obtain a global image of the placement area, and accurately determine the position and posture of the object by performing object detection on the global image, thereby improving the accuracy of the robotic arm in grasping the object.

[0024] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 A diagram showing an application environment of a robot arm control method in an embodiment;

[0028] Figure 2 A schematic diagram of a flow chart of a method for controlling a robotic arm in one embodiment;

[0029] Figure 3 A schematic diagram of adjusting a reference point cloud of a three-dimensional chess piece according to a rotation angle in one embodiment;

[0030] Figure 4 A schematic diagram of a process for determining the position and posture of at least one three-dimensional chess piece on a chessboard based on a preset feature extraction model and a preset image processing model in one embodiment;

[0031] Figure 5 A schematic diagram of a flow chart of a robot arm control method in another embodiment;

[0032] Figure 6 is a structural block diagram of a robot arm control device in one embodiment;

[0033] Figure 7 is an internal structure diagram of a computer device in one embodiment;

[0034] Figure 8 is a schematic diagram of a robot arm body in one embodiment;

[0035] Fig. 9 A top view of a chess-playing robot including a mechanical arm in one embodiment;

[0036] Fig.10 A schematic diagram of the second arm of the chess-playing robot being raised in one embodiment;

[0037] Fig.11 A schematic diagram of lowering the second arm of the chess-playing robot in one embodiment;

[0038] Fig.12 It is a schematic diagram of a chess-playing robot grabbing a three-dimensional chess piece in a tilted state through a mechanical claw at the end of a mechanical arm in one embodiment. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0040] The robot arm control method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the robot arm controller 101 in the robot arm 100 can be connected to the image collector 200 through a network or an electrical connection. The image collector 200 can be used to collect a global image of a placement area, in which at least one object is placed. And the robot arm controller 101 can control the robot arm body 102 in the robot arm 100 to interact with at least one object based on the global image obtained from the image collector 200.

[0041] Specifically, the robot controller 101 can obtain a global image of the placement area, perform object detection on the global image, determine the category and rotation angle of at least one object, and the two-dimensional projection point corresponding to the feature point of at least one object in the placement area coordinate system. Further, for each object, the robot controller 101 can query and obtain the reference point cloud of the object in the reference posture according to the category of the object, and correct the reference point cloud based on the rotation angle of the object to obtain the rotation point cloud of the object, and the rotation point cloud includes the three-dimensional reference points corresponding to the feature points of the object. Further, the robot controller 101 can match the projection relationship between the two-dimensional projection points corresponding to the feature points of the object and the three-dimensional reference points to determine the position and posture of the object in the placement area coordinate system, and then select the object to be grasped from at least one object according to the position and posture of at least one object in the placement area coordinate system, and control the robot body 102 to grasp the object to be grasped.

[0042] Among them, the robot arm 100 includes but is not limited to a large robot arm and a small robot arm. The robot arm controller 101 can be specifically the main control chip of the robot arm 100, and can be deployed with an algorithm or model, and can accurately control the operation of the robot arm body 102 based on the global image collected by the image collector 200. The number of image collectors 200 is at least one. In this embodiment, the image collector 200 can be a monocular camera (capturing two-dimensional image information through a camera). Using the above-mentioned robot arm control method, if a global image of the placement area is collected, the position and posture of at least one object in the placement area can be accurately identified through the global image, and the robot arm can be controlled to accurately grasp.

[0043] Exemplarily, taking the robot arm 100 as a small robot arm for performing chess-playing tasks, if the chess pieces involved in the game are three-dimensional chess pieces of different shapes, the above-mentioned robot arm control method can be used to accurately identify the position and posture of at least one three-dimensional chess piece on the chessboard based on the global image of the chessboard captured by the monocular camera, thereby controlling the robot arm to accurately grasp the three-dimensional chess piece at a suitable angle to perform the game task.

[0044] In an exemplary embodiment, Figure 2 As shown, a method for controlling a robot arm is provided, which can be applied to electronic devices with data processing capabilities, such as Figure 1 The robot arm controller 101 shown in the figure, or other terminal devices, are not particularly limited in the present embodiment of the application. Figure 1 The robot arm controller 101 in FIG. 1 is taken as an example to illustrate, and the steps include the following steps 202 to 210. Among them:

[0045] Step 202, obtaining a global image of a placement area; at least one object is placed in the placement area.

[0046] The global image specifically refers to an image of the placement area acquired by the image collector when the placement area is entirely within the shooting field of view of the image collector.

[0047] Optionally, the robot arm controller may acquire a global image of the placement area from an image collector.

[0048] Exemplarily, the size of the placement area and the size of the object placed in the placement area can be set after testing. When the image collector is in a preset acquisition position, no matter where the object in the placement area is located, at least a part of the object will be exposed in the shooting field of the image collector, so that at least one object in the placement area can be detected separately according to the global image.

[0049] In a possible implementation, taking the placement area as a chessboard as an example, even if there is occlusion between the chess pieces, when the image collector is in a preset acquisition position, the chess pieces in the chessboard, no matter which chessboard square they are in, will be at least partially exposed in the shooting field of view of the image collector.

[0050] Step 204 , performing object detection on the global image, determining the category and rotation angle of at least one object, and the two-dimensional projection point corresponding to the feature point of at least one object in the placement area coordinate system.

[0051] The rotation angle specifically refers to: the rotation angle of the object relative to its own base, which can characterize the change in the object's posture in space. For objects of different shapes, each object can be set with at least one feature point. The feature point can be any point on the central axis of the object, such as the top center point, the bottom center point, etc., or an easily identifiable edge point on the surface of the object. In the global image (two-dimensional image) collected by the image collector, any point in the global image can be regarded as a two-dimensional projection point of a corresponding point on the surface of the object from the perspective of the image collector.

[0052] Optionally, the robot controller can perform multi-task object detection on the global image based on a preset image processing algorithm or a preset image processing model, thereby determining the category and rotation angle of at least one object, and the feature points of at least one object in the global image. Since a placement area mark is provided on the placement area, which can be used to locate the placement area coordinate system, the robot controller can further identify the placement area coordinate system based on the placement area mark in the global image, thereby determining the two-dimensional coordinates of the two-dimensional projection point corresponding to the feature point of at least one object in the placement area coordinate system.

[0053] Alternatively, taking the robot controller performing multi-task object detection on the global image based on a preset image processing model as an example, the preset image processing model may include: a category detection branch, a rotation angle detection branch, and a feature point detection branch, the category detection branch is used to detect the category of the object, the rotation branch is used to detect the rotation angle of the object, and the feature point detection branch is used to detect the feature points of the object, and the two-dimensional coordinates of the two-dimensional projection points corresponding to the feature points in the placement area coordinate system. Among them, the category detection branch, the rotation angle detection branch, and the feature point detection branch can all be obtained based on neural network training, and the neural network includes but is not limited to: CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory).

[0054] In the early model training process, based on the original image and the reference image annotated with the object category, the neural network can be trained to detect the object category based on at least a portion of the edge exposed by the object in the image, thereby obtaining a category detection branch. Furthermore, based on the original image and the reference image annotated with the object rotation angle, the neural network can be trained to detect the object rotation angle based on at least a portion of the edge exposed by the object in the image, thereby obtaining a rotation angle detection branch. Furthermore, based on the original image and the reference image annotated with the object feature points, the neural network can be trained to detect the feature points exposed by at least a portion of the edge exposed by the object in the image, as well as the coordinates of the two-dimensional projection points corresponding to the feature points in the placement area coordinate system, thereby obtaining a feature point detection branch.

[0055] Step 206, for each object, according to the category of the object, query and obtain the reference point cloud of the object in the reference posture, and correct the reference point cloud based on the rotation angle of the object to obtain the rotated point cloud of the object; the rotated point cloud includes the three-dimensional reference points corresponding to the feature points of the object.

[0056] If the object has a front face, the reference pose may refer to the pose when the object is placed upright and presents the front face. A point cloud is a data set of three-dimensional points in a known coordinate system, including the three-dimensional coordinates of the three-dimensional points, which can represent the three-dimensional distribution of points on the surface of an object, that is, a three-dimensional model of the object.

[0057] Optionally, for each object, the robot controller can query the reference point cloud of the object in the reference posture from the stored point cloud data of the object according to the category of the object. Further, the robot controller can correct the reference point cloud of the object based on the rotation angle of the object, for example, according to the rotation angle of the object relative to its own base, adjust the posture of the base or the central axis in the reference point cloud of the object, and then adjust the overall posture of the object in the reference point cloud, so as to obtain the rotation point cloud of the object in the actual posture.

[0058] For example, the object is a three-dimensional chess piece, and the three-dimensional chess piece falls on the chessboard. If the rotation angle of the three-dimensional chess piece indicates that the three-dimensional chess piece is in a state of falling to one side and has not changed its direction, then Figure 3 As shown, a schematic diagram of adjusting the reference point cloud of a three-dimensional chess piece according to the rotation angle is provided. That is, the reference posture represented by the reference point cloud is adjusted to the actual posture represented by the rotation point cloud of the three-dimensional chess piece to represent the tilting posture of the three-dimensional chess piece.

[0059] Step 208 , matching the two-dimensional projection points corresponding to the feature points of the object with the three-dimensional reference points to determine the position and posture of the object in the placement area coordinate system.

[0060] Among them, the posture can specifically include position and attitude, which is a 6D posture with six degrees of freedom, including three degrees of freedom of displacement and three degrees of freedom of spatial rotation.

[0061] Optionally, the robotic arm controller can match the projection relationship between the two-dimensional projection point and the three-dimensional reference point corresponding to the same feature point of the object, for example, matching the two-dimensional projection point corresponding to the top center point of the object with the three-dimensional reference point, and matching the two-dimensional projection point corresponding to the bottom center point of the object with the three-dimensional reference point. Furthermore, the robotic arm controller can derive the back projection matrix from the two-dimensional projection point to the three-dimensional reference point based on the matched two-dimensional projection point (including two-dimensional coordinates) and the three-dimensional reference point (including three-dimensional coordinates), and then restore the actual posture of the object in the placement area coordinate system from the two-dimensional image of the object based on the back projection matrix, that is, the 6D posture of the object in the placement area coordinate system. Among them, back projection specifically refers to: based on the projection principle, when the three-dimensional model of the object (which can be determined by the point cloud of the surface points of the object) is known, and the position of the projection point generated by the projection of the object is known, the actual posture of the object is reversely deduced.

[0062] In an optional implementation, taking the placement area as a chessboard and the objects in the placement area as three-dimensional chess pieces as an example, the robotic arm controller may pre-store a three-dimensional model of each three-dimensional chess piece (stored in the form of a point cloud), including a reference point cloud of each three-dimensional chess piece in a reference posture (the point cloud includes the three-dimensional coordinates of the feature points on the surface of the three-dimensional chess piece). Based on this, for each three-dimensional chess piece in the global image, after obtaining the two-dimensional projection points of the feature points on the surface of the three-dimensional chess piece in the placement area coordinate system and the rotation point cloud of the three-dimensional chess piece, the robotic arm controller may match the projection relationship between the two-dimensional projection points corresponding to the feature points of the three-dimensional chess piece and the three-dimensional reference points to determine the position and posture of the three-dimensional chess piece in the placement area coordinate system, thereby controlling the robotic arm to accurately grasp the three-dimensional chess piece.

[0063] Step 210 , selecting an object to be grasped from at least one object according to the position and posture of each object in the placement area coordinate system, and controlling the robot arm to grasp the object to be grasped.

[0064] Optionally, after obtaining the global image of the placement area, the robot controller can perform multi-task object detection on at least one object in the global image through a preset image processing algorithm or a preset image processing model to obtain the pose of at least one object in the placement area coordinate system. Further, the robot controller can select an object to be grasped from at least one object according to the pose of at least one object in the placement area coordinate system, and after converting the pose of the object to be grasped in the placement area coordinate system into the pose in the robot coordinate system, the robot can be controlled to accurately grasp the object to be grasped.

[0065] For example, taking the placement area as a chessboard and the objects in the placement area as three-dimensional chess pieces, after determining the position and posture of at least one chess piece in the chessboard in the placement area coordinate system (chessboard coordinate system), the robotic arm controller can first filter the chess pieces to be grasped first according to the position and posture of each chess piece in the placement area coordinate system, that is, filter the objects to be grasped, and then control the robotic arm to grasp the chess piece according to the position and posture of the chess piece.

[0066] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0067] In the above-mentioned robot arm control method, when the camera assembly conditions are limited, if the global image of the placement area is obtained, the category and rotation angle of at least one object can be determined by first performing object detection on the global image, as well as the two-dimensional projection points corresponding to the feature points of at least one object in the placement area coordinate system. Further, for each object, the reference point cloud of the object in the reference posture can be obtained according to the category query of the object, and the reference point cloud can be corrected based on the rotation angle of the object to obtain the rotation point cloud of the object, and then the two-dimensional projection points corresponding to the feature points of the object and the three-dimensional reference points can be matched in the projection relationship to determine the position and posture of the object in the placement area coordinate system, so that the robot arm can be controlled to accurately grasp the object at a suitable angle based on the position and posture. That is, in the case of limited camera assembly conditions, the position and posture of the object can be accurately determined by the above-mentioned robot arm control method, so that the object to be grasped can be selected from at least one object according to the position and posture of at least one object in the placement area coordinate system, and the robot arm can be controlled to accurately grasp the object to be grasped, thereby improving the accuracy of the robot arm in grasping the object.

[0068] In one of the embodiments, object detection is performed on a global image to determine the category and rotation angle of at least one object, as well as the two-dimensional projection point corresponding to the feature point of at least one object in the placement area coordinate system, including: feature extraction is performed on the global image to obtain placement area image features and object image features of at least one object; the placement area image features are used to determine the placement area coordinate system; for each object, object detection is performed based on the object image features of the object to determine the category and rotation angle of the object, as well as the feature points of the object in the global image; based on the placement area image features and the feature points of the object in the global image, the two-dimensional projection point corresponding to the feature point of the object in the placement area coordinate system is determined.

[0069] Alternatively, after acquiring the global image captured by the image collector, the robotic arm controller may first perform feature extraction on the global image based on a preset feature extraction algorithm or a preset feature extraction model to obtain the placement area image features and the object image features of at least one object. Furthermore, for each object, the robotic arm controller may input the extracted object image features into a preset image processing model to perform object detection based on the object image features of the object, or perform object detection on the object image features of the object based on a preset image processing algorithm, thereby determining the category and rotation angle of the object, as well as the feature points of the object in the global image. Furthermore, the robotic arm controller may locate the placement area coordinate system in the global image based on the placement area image features, and determine the two-dimensional projection points corresponding to the feature points of the object in the placement area coordinate system based on the positions of the feature points of the object in the global image.

[0070] In some embodiments, a preset feature extraction model may be deployed in the robotic arm controller. The preset feature extraction model may be obtained based on neural network training, such as CNN (Convolutional Neural Network) training, and may comprehensively and accurately extract image features in the global image.

[0071] In a specific application scenario, taking the placement area as a chessboard and the objects in the placement area as three-dimensional chess pieces (such as three-dimensional chess pieces) as an example, the robot arm controller can realize feature extraction and object detection of the global image based on the deployed preset feature extraction model and preset image processing model. Figure 4 As shown, a flow chart of determining the position and posture of at least one three-dimensional chess piece on a chessboard based on a preset feature extraction model and a preset image processing model is provided.

[0072] Specifically, Figure 4 As shown, after acquiring the global image of the chessboard, the robot arm controller can first input the global image into a preset feature extraction model to extract the placement area image features and the object image features of at least one three-dimensional chess piece from the global image. The global image can be an RGB two-dimensional image captured by a monocular camera. The placement area image features can be extracted based on the chessboard grid lines, chessboard logos, and chessboard edges of the chessboard in the global image. For each three-dimensional chess piece, the object image features of the three-dimensional chess piece can be extracted based on the chess piece outline and chess piece pattern revealed by the three-dimensional chess piece in the global image.

[0073] Furthermore, for each three-dimensional chess piece, the robotic arm controller can input the object image features of the three-dimensional chess piece into a preset image processing model to perform multi-task object detection through the category detection branch, rotation angle detection branch and feature point detection branch in the preset image processing model. Specifically, the category detection branch can detect the category of the three-dimensional chess piece based on the object image features of the three-dimensional chess piece; the rotation angle detection branch can detect the rotation angle of the three-dimensional chess piece based on the object image features of the three-dimensional chess piece; the feature point detection branch can detect the feature points of the three-dimensional chess piece based on the object image features of the three-dimensional chess piece, and combine the placement area image features to detect the two-dimensional projection points corresponding to the feature points in the placement area coordinate system. Based on this, the robotic arm controller can determine the position and posture of each three-dimensional chess piece in combination with the above effective information.

[0074] In this embodiment, feature extraction can be performed on the global image to obtain image features carried by the global image, which is beneficial for further accurately detecting effective information in the global image based on the image features, for example, detecting the category and rotation angle of at least one object, and the two-dimensional projection points corresponding to the feature points of each object in the placement area coordinate system, so as to facilitate the subsequent accurate positioning of the object's posture based on the detected effective information, thereby improving the accuracy of the robot arm in grasping the object.

[0075] In a specific example, an object to be grasped is selected from at least one object according to the posture of each object in the placement area coordinate system, and a robotic arm is controlled to grasp the object to be grasped, including: determining the placement state of at least one object according to the posture of each object in the placement area coordinate system; in the case where there is an object with a tilted placement state, the object with a tilted placement state is used as the object to be grasped; according to the posture of the object to be grasped in the placement area coordinate system, the robotic arm is controlled to grasp the object to be grasped to adjust the placement state of the object to be grasped.

[0076] The placement state includes: a tilted state (non-upright placement state) and an upright state.

[0077] Optionally, the robot arm controller may compare the posture of each of the at least one object in the placement area coordinate system with the posture of each object in the upright state, for example, to determine whether there is an angle between the central axis of the object in the current posture and the central axis of the object in the upright state. If so, it can be determined that the object is in a tilted state, otherwise, it is determined that the object is in an upright state. Based on this, the placement state of each of the at least one object can be determined.

[0078] Furthermore, in the case where there is an object in a tilted state, the robot arm controller can treat the object in a tilted state as an object to be grasped whose placement state needs to be adjusted, and control the robot arm to accurately grasp the object to be grasped according to the position of the object to be grasped in the coordinate system of the placement area, so as to adjust the placement state of the object to be grasped. For example, after determining the object to be grasped, the mechanical claw at the end of the robot arm is controlled to grasp the object to be grasped according to the position of the object to be grasped in the coordinate system of the placement area, and the mechanical claw is controlled to adjust the angle so as to place the object to be grasped back to its original position in an upright state.

[0079] It should be noted that, if the mechanical claw cannot be controlled to successfully grasp the object to be grasped, or, after controlling the mechanical claw to adjust the angle, it still cannot place the object to be grasped back to its original position in an upright state, the mechanical arm controller can send an early warning prompt to the connected monitoring end to prompt manual adjustment of the placement state of the object to be grasped. If the mechanical arm is also equipped with a light-emitting device and / or a sound-emitting device, the mechanical arm controller can control the light-emitting device to emit an early warning prompt light, and / or control the sound-emitting device to emit an early warning prompt sound, to prompt manual adjustment of the placement state of the object to be grasped.

[0080] In this embodiment, after determining the position and posture of at least one object in the coordinate system of the placement area, it is possible to first determine whether there is an object in a tipping state based on the position and posture. If there is, the robot arm can be controlled to accurately grasp the object based on the position and posture of the tipping object to adjust the placement state of the object.

[0081] In one of the embodiments, the global image acquisition scene includes a first scene and a second scene; the first scene refers to: the global image is acquired during the execution of the object placement task in the placement area; the second scene refers to: after the object placement task is completed, the global image is acquired during the execution of the object movement task in the placement area. Based on this, in the case where there is an object whose placement state is a tilted state, the object whose placement state is a tilted state is used as an object to be grasped, including: when the global image acquisition scene is the first scene and there is an object whose placement state is a tilted state, the object placement task in the placement area is continued until the object placement task is completed, and the object in the tilted state is used as an object to be grasped whose placement state needs to be adjusted; when the global image acquisition scene is the second scene and there is an object whose placement state is a tilted state, the object movement task in the placement area is suspended, and the object in the tilted state is used as an object to be grasped whose placement state needs to be adjusted.

[0082] Specifically, the first scenario may be: when the robot controller controls the robot body to perform the object placement task in the placement area according to the preset object placement rules, so as to place the object to be placed at the specified position in the placement area, the image collector automatically collects the global image of the placement area and transmits the global image to the robot controller. Specifically, the second scenario may be: after the object placement task is completed, when the robot controller controls the robot body to move the object in the placement area according to the object movement strategy, the image collector automatically collects the global image of the placement area and transmits the global image to the robot controller.

[0083] Alternatively, when the acquisition scene of the global image is the first scene and there are objects in a tilted placement state, the robotic arm controller can control the robotic arm body to continue to perform the object placement task in the placement area until the object placement task is completed, that is, control the robotic arm body to continue to grab the objects to be placed and place them at the designated positions in the placement area until all objects to be placed have been placed in the placement area, and then treat the objects in the tilted placement area as objects to be grabbed whose placement state needs to be adjusted, so as to adjust the placement state of the objects to be grabbed and complete the final placement.

[0084] Optionally, when the acquisition scene of the global image is the second scene and there is an object in a tilted state, the robotic arm controller can suspend the object movement task in the placement area, and first treat the object in the tilted state as an object to be grasped whose placement state needs to be adjusted, so as to adjust the placement state of the object to be grasped and avoid interfering with the movement of objects in the placement area.

[0085] Exemplarily, taking the placement area as a chessboard and the objects in the placement area as three-dimensional chess pieces as an example, the first scenario can specifically be: during the chess piece placement process before the game, the image collector automatically captures the global image of the chessboard and transmits it to the robotic arm controller; the second scenario can specifically be: during the game after the chess piece placement is completed, the image collector automatically captures the global image of the chessboard and transmits it to the robotic arm controller.

[0086] Furthermore, during the chess arrangement process, if it is determined through the global image of the chessboard that there are three-dimensional chess pieces that have been placed on the chessboard and are in a tilted state, the robot arm controller can first control the robot arm body to continue to grab the three-dimensional chess pieces to be placed and place them at the designated position on the chessboard until all the three-dimensional chess pieces to be placed have been placed on the chessboard, and then the three-dimensional chess pieces in the chessboard that are in a tilted state are used as objects to be grabbed, and the objects to be grabbed are grabbed according to their positions, thereby adjusting their placement states and completing the global placement of the three-dimensional chess pieces. During the chess arrangement process, if no three-dimensional chess pieces are in a tilted state, the game process can be directly entered after the chess arrangement is completed.

[0087] Furthermore, during the game, if it is determined through the global image of the chessboard that a placed three-dimensional chess piece on the chessboard is in a tilted state, considering that the three-dimensional chess piece in a tilted state will occupy multiple chessboard squares, affecting the subsequent distribution data analysis of the three-dimensional chess pieces on the chessboard, and then affecting the subsequent object movement strategy analysis, therefore, the robotic arm controller can first control the pause of the game, and use the three-dimensional chess piece in a tilted state on the chessboard as the object to be grasped, so as to grasp the object to be grasped according to the posture of the object to be grasped, thereby adjusting its placement state to avoid the tilted three-dimensional chess piece interfering with the game process.

[0088] In this embodiment, during the process of executing the object placement task in the placement area, if it is determined that there is an object in a tilted state in the placement area, the object placement task in the placement area can be executed first, and after the execution is completed, the object in the tilted state is used as the object to be grasped for grasping and adjusting the posture, so that the global placement of the objects in the placement area can be completed as soon as possible. During the process of executing the object movement task in the placement area, if it is determined that there is an object in a tilted state in the placement area, the object in the tilted state is used as the object to be grasped for grasping and adjusting the posture to avoid affecting the movement of the object.

[0089] In an exemplary embodiment, according to the posture of the object to be grasped in the placement area coordinate system, the robotic arm is controlled to grasp the object to be grasped, including: obtaining a first conversion relationship between the camera coordinate system and the placement area coordinate system, and a second conversion relationship between the robotic arm coordinate system and the camera coordinate system; according to the first conversion relationship, the posture of the object to be grasped in the placement area coordinate system is converted into a posture in the camera coordinate system, and according to the second conversion relationship, the posture in the camera coordinate system is converted into a posture in the robotic arm coordinate system; according to the posture of the object to be grasped in the robotic arm coordinate system, the robotic arm is controlled to grasp the object to be grasped.

[0090] Among them, the placement area coordinate system can be: a known coordinate system constructed based on points selected from the placement area. The camera coordinate system can be: a known coordinate system constructed based on the installation position or fixed position of the image collector. The robotic arm coordinate system can be: a known coordinate system constructed based on the base of the robotic arm, which is the coordinate system used when the robotic arm performs motion control. In actual application scenarios, the image collector is generally installed in a fixed position to maintain a stable relative position with the placement area and the robotic arm. Based on this, the first conversion relationship between the camera coordinate system and the placement area coordinate system, as well as the second conversion relationship between the robotic arm coordinate system and the camera coordinate system can be pre-calculated and stored.

[0091] Alternatively, the robot controller can obtain the first conversion relationship between the camera coordinate system and the placement area coordinate system, and the second conversion relationship between the robot coordinate system and the camera coordinate system from pre-stored data. Based on this, the position and posture of the object to be grasped in the placement area coordinate system can be converted to the position and posture in the camera coordinate system according to the first conversion relationship, and the position and posture in the camera coordinate system can be converted to the position and posture in the robot coordinate system according to the second conversion relationship, so that the robot can be controlled to grasp the object to be grasped according to the position and posture of the object to be grasped in the robot coordinate system.

[0092] In this embodiment, after determining the object to be grasped, multiple coordinate system conversions can be performed through the conversion relationship between different coordinate systems to convert the position and posture of the object to be grasped in the placement area coordinate system into the position and posture in the robotic arm coordinate system, which is conducive to controlling the robotic arm to accurately grasp the object according to the position and posture in the robotic arm coordinate system.

[0093] In some possible implementations, after determining the placement state of at least one object according to the posture of at least one object in the placement area coordinate system, the method further includes: when the acquisition scene of the global image is the second scene and there is no object in a tilted state, determining the distribution data of at least one object in the placement area coordinate system according to the posture of at least one object in the placement area coordinate system; performing object movement strategy analysis based on the distribution data to determine the object to be grasped among the at least one object and the target position corresponding to the object to be grasped in the placement area coordinate system; controlling the robotic arm to grasp the object to be grasped according to the posture of the object to be grasped in the placement area coordinate system, and moving the object to be grasped to the target position.

[0094] There are no objects in a tipped-over state, which may include two situations: no object tipped over, and the tipped-over object has been adjusted to a proper placement state.

[0095] Optionally, in the process of executing the object movement task in the placement area, if there is no object in a dumped state, the robot arm controller can determine the distribution data of at least one object in the placement area coordinate system according to the posture (including position and posture) of at least one object in the placement area coordinate system, and thus perform object movement strategy analysis based on the distribution data, determine the object to be moved among the at least one object, determine the object to be moved as the object to be grasped, and determine the target position corresponding to the object to be grasped in the placement area coordinate system, that is, determine the position to be moved to.

[0096] Furthermore, the robot controller can convert the pose of the object to be grasped in the placement area coordinate system into the pose in the camera coordinate system, and then convert the pose in the camera coordinate system into the pose in the robot coordinate system, and convert the target position corresponding to the placement area coordinate system into the target position in the robot coordinate system. Based on this, the robot controller can control the robot to accurately grasp the object to be grasped according to the pose of the object to be grasped in the robot coordinate system, and accurately move the object to be grasped to the target position.

[0097] For example, taking the placement area as a chessboard and the objects in the placement area as three-dimensional chess pieces, a chess game analysis engine can also be deployed in the robotic arm controller. After determining the distribution data of each three-dimensional chess piece in the chessboard in the coordinate system of the placement area, the robotic arm controller can use the chess game analysis engine to perform object movement strategy analysis on the three-dimensional chess piece, determine the three-dimensional chess piece to be moved, and the target position to which the three-dimensional chess piece needs to be moved, thereby using the three-dimensional chess piece as the object to be grasped, and according to the posture of the object to be grasped, grasp and place the object to be grasped to the target position, thereby completing the chess game process.

[0098] In this embodiment, after determining that there is no object in a tipped-over state, an object movement strategy analysis can be performed based on the posture of at least one object in the coordinate system of the placement area to determine the object to be grasped and the target position corresponding to the object to be grasped, thereby controlling the robotic arm to accurately grasp the object to be grasped and move it to the target position based on the posture of the object to be grasped.

[0099] It should be noted that in the process of executing the object movement task in the placement area, after determining the placement status of at least one object, if there is an object in a tipping state, the robot arm controller can control the robot arm to preferentially adjust the posture of the object in the tipping state to avoid the object in the tipping state interfering with the movement of other objects, and then perform object movement strategy analysis and move the object to the target position. However, when there is sufficient space in the placement area and the change in the position area occupied by the object after the tipping does not affect the distribution data obtained by the analysis, the robot arm controller can also first perform object movement strategy analysis to determine the object to be moved. If the object to be moved is not in a tipping state, the posture of the object in the tipping state can be adjusted after the object to be moved is moved to the target position. If the object to be moved is in a tipping state, the robot arm can be directly controlled to grab the object to be moved, and after adjusting its posture, it can be moved to the target position.

[0100] In a possible implementation, based on the above embodiments, a process of a robot arm performing an object moving task is taken as an example to illustrate a flow chart of another robot arm control method, such as Figure 5 As shown, it mainly includes the following steps:

[0101] Step 502, obtaining a global image of a placement area; at least one object is placed in the placement area.

[0102] Step 504 , extracting features from the global image to obtain image features of the placement area and image features of at least one object; the image features of the placement area are used to determine a coordinate system of the placement area.

[0103] Step 506 : for each object, perform object detection according to the object image features of the object to determine the category and rotation angle of the object, as well as the feature points of the object in the global image.

[0104] Step 508 , determining the two-dimensional projection points corresponding to the feature points of the object in the placement area coordinate system according to the placement area image features and the feature points of the object in the global image.

[0105] Step 510, according to the category of the object, query and obtain the reference point cloud of the object in the reference posture, and correct the reference point cloud based on the rotation angle of the object to obtain the rotated point cloud of the object; the rotated point cloud includes the three-dimensional reference points corresponding to the feature points of the object.

[0106] Step 512, matching the two-dimensional projection points corresponding to the feature points of the object with the three-dimensional reference points to determine the position and posture of the object in the placement area coordinate system.

[0107] Step 514: Determine the placement status of at least one object according to the position and posture of at least one object in the placement area coordinate system.

[0108] In the case where there is an object in a tilted state, execute step 516 to treat the object in a tilted state as the object to be grasped; step 518, convert the posture of the object to be grasped in the placement area coordinate system into the posture in the robot arm coordinate system; step 520, control the robot arm to grasp the object to be grasped according to the posture in the robot arm coordinate system, and adjust the posture of the object to be grasped.

[0109] In the absence of an object in a tilted state, execute step 522 to determine the distribution data of at least one object in the placement area coordinate system according to the posture of at least one object in the placement area coordinate system; step 524 to analyze the object movement strategy according to the distribution data to determine the object to be grasped among the at least one object and the target position of the object to be grasped in the placement area coordinate system; step 526 to convert the posture of the object to be grasped in the placement area coordinate system into the posture in the robot arm coordinate system; step 528 to control the robot arm to grasp the object to be grasped according to the position in the robot arm coordinate system and move the object to be grasped to the target position.

[0110] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0111] Based on the same inventive concept, the embodiment of the present application also provides a robot arm control device for implementing the robot arm control method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more robot arm control device embodiments provided below can refer to the limitations of the robot arm control method above, and will not be repeated here.

[0112] In an exemplary embodiment, Figure 6 As shown, a robotic arm control device is provided, including: a global image acquisition module 602, a multi-task detection module 604, a point cloud acquisition module 606, a posture determination module 608 and an object grasping module 610, wherein:

[0113] A global image acquisition module, used to acquire a global image of a placement area, wherein at least one object is placed in the placement area;

[0114] A multi-task detection module, used to perform object detection on the global image, determine the category and rotation angle of at least one object, and the two-dimensional projection point corresponding to the feature point of at least one object in the placement area coordinate system;

[0115] The point cloud acquisition module is used to query and obtain the reference point cloud of each object in the reference posture according to the category of the object, and to correct the reference point cloud based on the rotation angle of the object to obtain the rotation point cloud of the object, wherein the rotation point cloud includes the three-dimensional reference points corresponding to the feature points of the object;

[0116] A posture determination module is used to match the projection relationship between the two-dimensional projection points corresponding to the feature points of the object and the three-dimensional reference points to determine the posture of the object in the coordinate system of the placement area;

[0117] The object grasping module is used to select an object to be grasped from at least one object according to the position and posture of each object in the placement area coordinate system, and control the robot arm to grasp the object to be grasped.

[0118] The above-mentioned robot arm control device, when the camera assembly conditions are limited, if the global image of the placement area is obtained, can first determine the category and rotation angle of at least one object, and the two-dimensional projection point corresponding to the feature point of at least one object in the placement area coordinate system by performing object detection on the global image. Further, for each object, the reference point cloud of the object in the reference posture can be obtained according to the category query of the object, and the reference point cloud can be corrected based on the rotation angle of the object to obtain the rotation point cloud of the object, and then the two-dimensional projection point corresponding to the feature point of the object and the three-dimensional reference point can be matched to determine the position and posture of the object in the placement area coordinate system, so that the robot arm can be controlled to accurately grasp the object at a suitable angle based on the position and posture. That is, when the camera assembly conditions are limited, the position and posture of the object can be accurately determined by the above-mentioned robot arm control method, so that the object to be grasped can be selected from at least one object according to the position and posture of at least one object in the placement area coordinate system, and the robot arm can be controlled to accurately grasp the object to be grasped, thereby improving the accuracy of the robot arm in grasping the object.

[0119] In one embodiment, the multi-task detection module includes: a feature extraction unit, which is used to extract features from the global image to obtain placement area image features and object image features of at least one object, and the placement area image features are used to determine the placement area coordinate system; a multi-task detection unit, which is used to perform object detection on each object according to the object image features of the object, determine the category and rotation angle of the object, and the feature points of the object in the global image; a projection point detection unit, which is used to determine the two-dimensional projection points corresponding to the feature points of the object in the placement area coordinate system according to the placement area image features and the feature points of the object in the global image.

[0120] In a specific example, the object grasping module includes: a placement state determination unit, used to determine the placement state of at least one object according to the position and posture of the at least one object in the placement area coordinate system; a to-be-grasped object determination unit, used to, when there is an object in a tilted placement state, take the object in a tilted placement state as the object to be grasped; a robotic arm control unit, used to control the robotic arm to grasp the object to be grasped according to the position and posture of the object to be grasped in the placement area coordinate system, so as to adjust the placement state of the object to be grasped.

[0121] In one embodiment, the robotic arm control unit is specifically used to: obtain a first conversion relationship between the camera coordinate system and the placement area coordinate system, and a second conversion relationship between the robotic arm coordinate system and the camera coordinate system; according to the first conversion relationship, convert the posture of the object to be grasped in the placement area coordinate system into the posture in the camera coordinate system, and according to the second conversion relationship, convert the posture in the camera coordinate system into the posture in the robotic arm coordinate system; according to the posture of the object to be grasped in the robotic arm coordinate system, control the robotic arm to grasp the object to be grasped.

[0122] In one of the embodiments, the acquisition scene of the global image includes a first scene and a second scene; the first scene refers to: acquiring the global image in the process of executing the object placement task in the placement area; the second scene refers to: acquiring the global image in the process of executing the object movement task in the placement area after the object placement task is completed. Based on this, the object to be grasped determination unit in the object grasping module is specifically used to: when the acquisition scene of the global image is the first scene, and there is an object whose placement state is a tilted state, continue to execute the object placement task in the placement area until the object placement task is completed, and treat the object in the tilted state as the object to be grasped whose placement state needs to be adjusted; when the acquisition scene of the global image is the second scene, and there is an object whose placement state is a tilted state, suspend the execution of the object movement task in the placement area, and treat the object in the tilted state as the object to be grasped whose placement state needs to be adjusted.

[0123] In some possible implementations, the object to be grasped determination unit in the object grasping module is also used to: when the acquisition scene of the global image is the second scene, and there is no object in a tilted state, determine the distribution data of at least one object in the placement area coordinate system according to the posture of at least one object in the placement area coordinate system; perform object movement strategy analysis based on the distribution data to determine the object to be grasped among the at least one object, and the target position corresponding to the object to be grasped in the placement area coordinate system. The manipulator control unit in the object grasping module is also used to: control the manipulator to grasp the object to be grasped according to the posture of the object to be grasped in the placement area coordinate system, and move the object to be grasped to the target position.

[0124] Each module in the above-mentioned robot arm control device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0125] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 7As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store robot arm control data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a robot arm control method is implemented.

[0126] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0127] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above-mentioned robot arm control method when executing the computer program.

[0128] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned robot arm control method is implemented.

[0129] In one embodiment, a computer program product is provided, comprising a computer program, which implements the above-mentioned robot arm control method when executed by a processor.

[0130] In some embodiments, the present application also provides a robotic arm, comprising: a robotic arm body, the robotic arm body comprising an action part and an execution part connected to the end of the action part, the execution part being used to interact with at least one object placed in a placement area; a robotic arm controller, being used to obtain a global image of the placement area, and based on the global image, execute the above-mentioned robotic arm control method to control the execution part to interact with at least one object.

[0131] Specifically, the action part may include a shoulder structure, a first arm, an elbow structure, and a second arm. The shoulder structure may be mounted on a robot or other device. The first end of the first arm is movably connected to the shoulder structure, the second end of the first arm is movably connected to the elbow structure, and the elbow structure is movably connected to the first end of the second arm. The second end of the second arm (i.e., the end of the action part) may be installed with the execution part, and the connection between the second end of the second arm and the execution part has an additional degree of freedom that can be lifted up and down. In addition, the execution part may have one degree of freedom, which may be a mechanical claw that can grab and place objects by opening and closing. Based on this, the robotic arm in this embodiment may add two more degrees of freedom at the end of the second arm on the basis of a traditional two-degree-of-freedom robotic arm to achieve four degrees of freedom. Combined with the above-mentioned robotic arm control method, it can flexibly and accurately grab objects.

[0132] It should be noted that the degree of freedom added at the connection between the second arm and the actuator to lift up and down enables the robot arm to grasp the object vertically instead of in an arc when grasping the object, thereby avoiding accidental contact with other objects, which is conducive to improving the accuracy of the robot arm grasping the object. Specifically, when the second end of the second arm is raised or lowered, the first arm and the second arm will move in the horizontal direction accordingly, thereby ensuring that the end of the second arm can grasp the object vertically.

[0133] Alternatively, if Figure 8 As shown, a schematic diagram of a mechanical arm body is provided, taking the actuator as a mechanical claw as an example, wherein the shoulder structure 801 is movably connected to the first end of the first arm 802, the second end of the first arm 802 is movably connected to the elbow structure 803, and the elbow structure 803 is movably connected to the first end of the second arm 804, and the actuator 805 can be installed on the second end of the second arm 804.

[0134] In this embodiment, compared with purely mechanical control of object grasping, the size of the equipment can be simplified and the equipment cost can be reduced through a four-degree-of-freedom robotic arm and the above-mentioned robotic arm control method. When using a lightweight image acquisition device, such as a monocular camera, the robotic arm can also be controlled to grasp objects with high precision based on the global image captured by the image acquisition device.

[0135] In some possible implementations, the present application also provides a robot, including: an image collector, used to collect a global image of a placement area; and the above-mentioned robotic arm, used to interact with at least one object placed in the placement area based on the global image.

[0136] Among them, the robot arm may include a robot arm body and a robot arm controller. The robot arm body includes an action part and an execution part connected to the end of the action part, and the execution part is used to interact with at least one object placed in the placement area. The robot arm controller is used to obtain a global image of the placement area, and based on the global image, execute the above-mentioned robot arm control method to control the execution part to interact with at least one object. The robot arm body may specifically include: a shoulder structure, a first arm rod, an elbow structure, a second arm rod, and an execution part, and the connection between the second arm rod and the execution part has an additional degree of freedom that can be lifted up and down, and the execution part can also grab and place objects by opening and closing.

[0137] Optionally, during the operation of the robot, the image collector can transmit the collected global image to the robotic arm controller in the robotic arm in a scheduled or real-time manner. The robotic arm controller can execute the above-mentioned robotic arm control method based on the global image to control the robotic arm body to grasp objects accurately and flexibly.

[0138] For example, the robot is a chess-playing robot, the placement area is a chessboard, and the objects in the placement area are three-dimensional chess pieces. Fig. 9 As shown, a top view of a chess-playing robot including the above-mentioned mechanical arm is provided. The image collector installed on the chess-playing robot can capture the entire chessboard from top to bottom, thereby acquiring a global image of the chessboard. Fig.10 As shown, a schematic diagram of the second arm of the chess robot is provided, such as Fig.11 As shown in FIG. 1 , a schematic diagram of lowering the second arm of the chess-playing robot is provided. That is, when the chess-playing robot grabs chess pieces during the chess-playing task, it can realize vertical grabbing instead of arc grabbing. Furthermore, as Fig.12 As shown, a schematic diagram of a chess-playing robot grabbing a three-dimensional chess piece in a tilted state through a mechanical claw at the end of a mechanical arm is provided. During the entire chess-playing process, the image collector of the chess-playing robot can collect a global image, and the mechanical arm controller controls the mechanical arm body to grab the chess piece with high precision based on the global image to perform the chess-playing task.

[0139] It should be noted that the information (including but not limited to device information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0140] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0141] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present application. The above embodiments only express several implementation methods of the present application, and the description is relatively specific and detailed, but it cannot be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present application, several deformations and improvements can be made, which all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the attached claims.

Claims

1. A robot arm control method, characterized in that: The method comprises: Acquire a global image of a placement area; at least one object is placed in the placement area; Performing object detection on the global image to determine the category and rotation angle of each of the at least one object, and the two-dimensional projection points corresponding to the feature points of each of the at least one object in the placement area coordinate system; For each of the objects, according to the category of the object, a reference point cloud of the object in a reference posture is queried, and the reference point cloud is corrected based on the rotation angle of the object to obtain a rotation point cloud of the object; the rotation point cloud includes three-dimensional reference points corresponding to the feature points of the object; Matching the projection relationship between the two-dimensional projection points and the three-dimensional reference points corresponding to the feature points of the object to determine the position and posture of the object in the coordinate system of the placement area; According to the posture of each of the at least one object in the coordinate system of the placement area, an object to be grasped is selected from the at least one object, and the robot arm is controlled to grasp the object to be grasped.

2. The method according to claim 1, characterized in that The performing object detection on the global image to determine the category and rotation angle of each of the at least one object, and the two-dimensional projection points corresponding to the feature points of each of the at least one object in the placement area coordinate system, includes: Extracting features from the global image to obtain image features of the placement area and image features of each of the at least one object; the image features of the placement area are used to determine a coordinate system of the placement area; For each of the objects, perform object detection according to the object image features of the object, and determine the category and rotation angle of the object, as well as feature points of the object in the global image; According to the placement area image features and the feature points of the object in the global image, a two-dimensional projection point corresponding to the feature point of the object in the placement area coordinate system is determined.

3. The method according to claim 1, characterized in that The selecting an object to be grasped from the at least one object according to the position and posture of each of the at least one object in the coordinate system of the placement area, and controlling the robot arm to grasp the object to be grasped includes: Determining a placement state of each of the at least one object according to a position of each of the at least one object in the coordinate system of the placement area; In the case where there is an object in a tilted state, the object in a tilted state is used as an object to be grasped; According to the posture of the object to be grasped in the coordinate system of the placement area, the mechanical arm is controlled to grasp the object to be grasped, so as to adjust the placement state of the object to be grasped.

4. The method according to claim 3, characterized in that: The acquisition scene of the global image includes a first scene and a second scene; the first scene refers to: the global image is acquired during the process of performing the object placement task of the placement area; the second scene refers to: after the object placement task is completed, the global image is acquired during the process of performing the object movement task of the placement area; In the case where there is an object in a collapsed state, taking the object in a collapsed state as an object to be grasped includes: When the acquisition scene of the global image is the first scene and there is an object whose placement state is a tilted state, continue to execute the object placement task of the placement area until the object placement task is completed, and treat the object in the tilted state as an object to be grasped whose placement state needs to be adjusted; When the acquisition scene of the global image is the second scene and there is an object in a tilted state, the object movement task in the placement area is suspended, and the object in the tilted state is used as an object to be grasped whose placement state needs to be adjusted.

5. The method according to claim 3, characterized in that: The controlling the robot arm to grasp the object to be grasped according to the posture of the object to be grasped in the coordinate system of the placement area includes: Acquire a first conversion relationship between a camera coordinate system and the placement area coordinate system, and a second conversion relationship between a robot arm coordinate system and the camera coordinate system; According to the first conversion relationship, convert the posture of the object to be grasped in the placement area coordinate system into the posture in the camera coordinate system, and according to the second conversion relationship, convert the posture in the camera coordinate system into the posture in the robot arm coordinate system; According to the position and posture of the object to be grasped in the robotic arm coordinate system, the robotic arm is controlled to grasp the object to be grasped.

6. The method according to claim 4, characterized in that After determining the placement state of each of the at least one object according to the position and posture of each of the at least one object in the placement area coordinate system, the method further includes: When the acquisition scene of the global image is the second scene and there is no object in a tilted state, determining distribution data of the at least one object in the placement area coordinate system according to the posture of each of the at least one object in the placement area coordinate system; Performing object movement strategy analysis according to the distribution data to determine an object to be grasped among the at least one object and a target position corresponding to the object to be grasped in the coordinate system of the placement area; According to the position and posture of the object to be grasped in the coordinate system of the placement area, the robotic arm is controlled to grasp the object to be grasped, and the object to be grasped is moved to the target position.

7. A robotic arm, characterized in that: The robotic arm comprises: A robot body, the robot body comprising an action part and an execution part connected to a terminal end of the action part, the execution part being used to interact with at least one object placed in the placement area; A robot arm controller is used to obtain a global image of the placement area and, based on the global image, execute the robot arm control method according to any one of claims 1 to 6 to control the execution unit to interact with the at least one object.

8. A robot, characterized in that: The robot comprises: An image collector, used for collecting a global image of the placement area; The robotic arm of claim 7, configured to interact with at least one object placed in the placement area based on the global image.

9. A robot arm control device, characterized in that: The device comprises: A global image acquisition module, used to acquire a global image of a placement area; at least one object is placed in the placement area; A multi-task detection module, used to perform object detection on the global image, determine the category and rotation angle of each of the at least one object, and the two-dimensional projection point corresponding to the feature point of each of the at least one object in the placement area coordinate system; A point cloud acquisition module, for querying and obtaining a reference point cloud of the object in a reference posture according to the category of the object for each of the objects, and correcting the reference point cloud based on the rotation angle of the object to obtain a rotation point cloud of the object; the rotation point cloud includes three-dimensional reference points corresponding to the feature points of the object; A posture determination module, used for matching the projection relationship between the two-dimensional projection points and the three-dimensional reference points corresponding to the feature points of the object, and determining the posture of the object in the coordinate system of the placement area; The object grasping module is used to select an object to be grasped from the at least one object according to the position and posture of each of the at least one object in the coordinate system of the placement area, and control the robot arm to grasp the object to be grasped.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

Images