Mechanical arm control method, device and equipment, mechanical arm and robot

Abstract

The application relates to a mechanical arm control method and device, computer equipment, a computer readable storage medium and a computer program product, and a mechanical arm and a robot. The method comprises the following steps: performing object detection on a global image of a placement area, determining the respective categories and rotation angles of at least one object, and the respective two-dimensional projection points of feature points of the object in a placement area coordinate system; for each object, querying a reference point cloud of the object according to the category of the object, correcting the reference point cloud based on the rotation angle of the object to obtain a rotated point cloud; matching the two-dimensional projection points of the feature points of the object with three-dimensional reference points in the rotated point cloud according to a projection relationship to determine the pose of the object in the placement area coordinate system; and selecting a to-be-grasped object according to the respective poses of the at least one object in the placement area coordinate system, and controlling the mechanical arm to grasp the to-be-grasped object. By using the method, the accuracy of the mechanical arm in grasping the object can be improved even if the camera assembly condition is limited.

Description

Technical Field

[0001] This application relates to the field of robotics technology, specifically to a robotic arm control method, apparatus, computer equipment, computer-readable storage medium and computer program product, as well as a robotic arm and a robot. Background Technology

[0002] With the development of robotics technology, robotic arms have been applied in many fields, including but not limited to: large robotic arms used in industrial automation scenarios, or small robotic arms used in lightweight collaborative scenarios.

[0003] Generally, during the operation of a robotic arm, it is necessary to equip it with multiple high-precision cameras to accurately control its work based on the images captured by the cameras. For example, when dealing with three-dimensional objects to be grasped, it is necessary to combine images from multiple cameras to identify the object's pose, thereby facilitating the control of the robotic arm to grasp it accurately at a suitable angle. If the camera installation conditions are limited, the accuracy of the robotic arm in grasping objects will be reduced. Summary of the Invention

[0004] Based on this, and in response to the aforementioned technical problems, this application provides at least one robotic arm control method, device, computer equipment, computer-readable storage medium, and computer program product, as well as a robotic arm and robot, which can improve the accuracy of the robotic arm in grasping objects under limited camera assembly conditions.

[0005] In a first aspect, this application provides a robotic arm control method, comprising: acquiring a global image of a placement area; placing at least one object in the placement area; performing object detection on the global image to determine the category and rotation angle of each of the at least one object, as well as the two-dimensional projection points of the feature points of each of the at least one object in the coordinate system of the placement area; for each object, querying a reference point cloud of the object in a reference pose according to the object's category, and correcting the reference point cloud based on the object's rotation angle to obtain a rotating point cloud of the object; the rotating point cloud includes three-dimensional reference points corresponding to the feature points of the object; matching the projection relationship between the two-dimensional projection points and the three-dimensional reference points corresponding to the feature points of the object to determine the pose of the object in the coordinate system of the placement area; selecting an object to be grasped from the at least one object according to the pose of each of the at least one object in the coordinate system of the placement area, and controlling the robotic arm to grasp the object to be grasped.

[0006] The aforementioned robotic arm control method, under limited camera assembly conditions, can first perform object detection on the global image of the placement area to determine the category and rotation angle of at least one object, as well as the two-dimensional projection points of the feature points of at least one object in the placement area coordinate system, if a global image of the placement area is obtained. Further, for each object, a reference point cloud in the reference pose can be obtained based on the object's category. This reference point cloud is then corrected based on the object's rotation angle to obtain the object's rotation point cloud. Subsequently, by matching the projection relationship between the two-dimensional projection points corresponding to the object's feature points and the three-dimensional reference points, the pose of the object in the placement area coordinate system is determined. This allows the robotic arm to be controlled to accurately grasp the object at a suitable angle based on the pose. In other words, even with limited camera assembly conditions, the above robotic arm control method can accurately determine the pose of the object. Based on the pose of at least one object in the placement area coordinate system, an object to be grasped can be selected from at least one object, and the robotic arm can be controlled to accurately grasp the object, improving the accuracy of the robotic arm's object grasping.

[0007] In one embodiment, object detection is performed on the global image to determine the category and rotation angle of at least one object, as well as the two-dimensional projection points of the feature points of at least one object in the placement area coordinate system. This includes: extracting features from the global image to obtain placement area image features and object image features of at least one object; using the placement area image features to determine the placement area coordinate system; for each object, performing object detection based on the object image features to determine the object's category and rotation angle, as well as the object's feature points in the global image; and determining the two-dimensional projection points of the object's feature points in the placement area coordinate system based on the placement area image features and the object's feature points in the global image.

[0008] In this embodiment, feature extraction can be performed on the global image to obtain the image features carried by the global image. This is beneficial for further accurate detection of effective information in the global image based on the image features. For example, at least one object's category and rotation angle, as well as the two-dimensional projection points of the object's feature points in the placement area coordinate system, can be detected. This facilitates accurate positioning of the object's pose based on the detected effective information, thereby improving the accuracy of the robotic arm in grasping the object.

[0009] In a specific example, selecting an object to be grasped from at least one object based on the pose of each of the at least one objects in the coordinate system of the placement area, and controlling the robotic arm to grasp the object to be grasped, includes: determining the placement state of each of the at least one object based on the pose of each of the at least one object in the coordinate system of the placement area; if there is an object in the tilted state, taking the object in the tilted state as the object to be grasped; and controlling the robotic arm to grasp the object to be grasped based on the pose of the object to be grasped in the coordinate system of the placement area, so as to adjust the placement state of the object to be grasped.

[0010] In this embodiment, after determining the pose of at least one object in the coordinate system of the placement area, it can be first determined whether any object is in a tilted state based on the pose. If so, the robotic arm can be controlled to precisely grasp the object based on its pose, thereby adjusting the object's placement state.

[0011] In one embodiment, the global image acquisition scenario includes a first scenario and a second scenario. The first scenario refers to acquiring the global image during the object placement task in the placement area. The second scenario refers to acquiring the global image during the object movement task in the placement area after the object placement task has been completed. Based on this, when there is an object in a tilted state, the object in the tilted state is taken as the object to be grasped, including: when the global image acquisition scenario is the first scenario and there is an object in a tilted state, continuing to execute the object placement task in the placement area until the object placement task is completed, and taking the tilted object as the object to be grasped whose placement state needs to be adjusted; when the global image acquisition scenario is the second scenario and there is an object in a tilted state, pausing the object movement task in the placement area, and taking the tilted object as the object to be grasped whose placement state needs to be adjusted.

[0012] In this embodiment, during the object placement task in the placement area, if it is determined that there is an object in a tilted state within the placement area, the object placement task in the placement area can be performed first. After the placement task is completed, the tilted object can be used as a grabbable object for grabbing and adjusting its pose, thereby completing the global placement of objects in the placement area as quickly as possible. During the object movement task in the placement area, if it is determined that there is an object in a tilted state within the placement area, the tilted object is prioritized as a grabbable object for grabbing and adjusting its pose, avoiding interference with object movement.

[0013] In one embodiment, controlling the robotic arm to grasp the object to be grasped based on the pose of the object to be grasped in the coordinate system of the placement area includes: acquiring a first transformation relationship between the camera coordinate system and the coordinate system of the placement area, and a second transformation relationship between the robotic arm coordinate system and the camera coordinate system; converting the pose of the object to be grasped in the coordinate system of the placement area to the pose in the camera coordinate system based on the first transformation relationship, and converting the pose in the camera coordinate system to the pose in the robotic arm coordinate system based on the second transformation relationship; and controlling the robotic arm to grasp the object to be grasped based on the pose of the object to be grasped in the robotic arm coordinate system.

[0014] In this embodiment, after determining the object to be grasped, multiple coordinate system transformations can be performed through the transformation relationship between different coordinate systems to convert the pose of the object to be grasped in the coordinate system of the placement area into the pose in the coordinate system of the robotic arm. This is beneficial for controlling the robotic arm to accurately grasp the object according to the pose in the coordinate system of the robotic arm.

[0015] In some possible implementations, after determining the placement state of at least one object based on its pose in the placement area coordinate system, the method further includes: if the global image acquisition scene is the second scene and there are no objects in the tilted state, determining the distribution data of at least one object in the placement area coordinate system based on its pose; performing object movement strategy analysis based on the distribution data to determine the object to be grasped among the at least one objects and the target position of the object to be grasped in the placement area coordinate system; and controlling the robotic arm to grasp the object to be grasped and move it to the target position based on its pose in the placement area coordinate system.

[0016] In this embodiment, after determining that no object is in a tilted state, an object movement strategy analysis can be performed based on the pose of at least one object in its respective placement area coordinate system to determine the object to be grasped and its corresponding target position. Then, based on the pose of the object to be grasped, the robotic arm can be controlled to accurately grasp the object and move it to the target position.

[0017] Secondly, this application also provides a robotic arm control device, comprising: a global image acquisition module for acquiring a global image of a placement area, wherein at least one object is placed in the placement area; a multi-task detection module for performing object detection on the global image, determining the category and rotation angle of each of the at least one object, and the two-dimensional projection points of the feature points of each of the at least one object in the coordinate system of the placement area; a point cloud acquisition module for obtaining a reference point cloud of the object in a reference pose according to the category of the object, and correcting the reference point cloud based on the rotation angle of the object to obtain a rotating point cloud of the object, wherein the rotating point cloud includes three-dimensional reference points corresponding to the feature points of the object; a pose determination module for matching the projection relationship between the two-dimensional projection points and the three-dimensional reference points corresponding to the feature points of the object to determine the pose of the object in the coordinate system of the placement area; and an object grasping module for selecting an object to be grasped from the at least one object according to the pose of each of the at least one object in the coordinate system of the placement area, and controlling the robotic arm to grasp the object to be grasped.

[0018] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-mentioned robotic arm control method.

[0019] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described robotic arm control method.

[0020] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described robotic arm control method.

[0021] In a sixth aspect, this application also provides a robotic arm, comprising: a robotic arm body, the robotic arm body including a motion part and an execution part connected to the end of the motion part, the execution part being used to interact with at least one object placed in a placement area; and a robotic arm controller, used to acquire a global image of the placement area and, based on the global image, execute the robotic arm control method provided in the first aspect to control the execution part to interact with at least one object.

[0022] In a seventh aspect, this application also provides a robot, comprising: an image acquisition device for acquiring a global image of a placement area; and a robotic arm, as provided in the sixth aspect, for interacting with at least one object placed within the placement area based on the global image.

[0023] The aforementioned robotic arm control device, computer equipment, computer-readable storage medium, computer program product, robotic arm, and robot, under limited camera assembly conditions, can accurately determine the pose of objects by performing object detection on the global image if they acquire a global image of the placement area, thereby improving the accuracy of the robotic arm in grasping objects.

[0024] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is an application environment diagram of the robotic arm control method in one embodiment;

[0028] Figure 2 This is a flowchart illustrating a robotic arm control method in one embodiment;

[0029] Figure 3 This is a schematic diagram of adjusting the reference point cloud of a 3D chess piece according to a rotation angle in one embodiment;

[0030] Figure 4 This is a flowchart illustrating the process of determining the pose of at least one 3D chess piece on a chessboard based on a preset feature extraction model and a preset image processing model in one embodiment.

[0031] Figure 5 This is a flowchart illustrating the robotic arm control method in another embodiment;

[0032] Figure 6 This is a structural block diagram of a robotic arm control device in one embodiment;

[0033] Figure 7 This is an internal structural diagram of a computer device in one embodiment;

[0034] Figure 8 This is a schematic diagram of the robotic arm body in one embodiment;

[0035] Figure 9 A top view of a chess-playing robot including a robotic arm in one embodiment;

[0036] Figure 10 This is a schematic diagram of the second arm of the chess-playing robot being raised in one embodiment;

[0037] Figure 11 This is a schematic diagram of the second arm of the chess-playing robot being lowered in one embodiment;

[0038] Figure 12 This is a schematic diagram of a chess-playing robot grasping a three-dimensional chess piece that is tilted in one embodiment using a mechanical claw at the end of its robotic arm. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0040] The robotic arm control method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, the robotic arm controller 101 in the robotic arm 100 can be connected to the image acquisition unit 200 via a network or electrical connection. The image acquisition unit 200 can be used to acquire a global image of the placement area, in which at least one object is placed. Based on the global image acquired from the image acquisition unit 200, the robotic arm controller 101 can control the robotic arm body 102 in the robotic arm 100 to interact with at least one object.

[0041] Specifically, the robotic arm controller 101 can acquire a global image of the placement area, perform object detection on the global image, determine the category and rotation angle of at least one object, and the two-dimensional projection points of the feature points of at least one object in the coordinate system of the placement area. Further, for each object, the robotic arm controller 101 can query the reference point cloud of the object in its reference pose based on the object's category, and correct the reference point cloud based on the object's rotation angle to obtain a rotating point cloud of the object, which includes the three-dimensional reference points corresponding to the object's feature points. Even further, the robotic arm controller 101 can perform projection relationship matching between the two-dimensional projection points and the three-dimensional reference points corresponding to the object's feature points to determine the object's pose in the coordinate system of the placement area. Then, based on the poses of at least one object in the coordinate system of the placement area, it selects the object to be grasped from the at least one object and controls the robotic arm body 102 to grasp the object to be grasped.

[0042] The robotic arm 100 includes, but is not limited to, large and small robotic arms. The robotic arm controller 101 can specifically be the main control chip of the robotic arm 100, which can be deployed with algorithms or models and can precisely control the operation of the robotic arm body 102 based on the global image acquired by the image acquisition device 200. There is at least one image acquisition device 200; in this embodiment, the image acquisition device 200 can be a monocular camera (capturing two-dimensional image information through a single camera). Using the above-described robotic arm control method, if a global image of the placement area is acquired, the pose of at least one object within the placement area can be accurately identified using this global image, and the robotic arm can be controlled to perform precise grasping.

[0043] For example, taking the robotic arm 100 as a small robotic arm used to perform a game task, if the game involves three-dimensional pieces of different shapes, the above-mentioned robotic arm control method can accurately identify the pose of at least one three-dimensional piece on the chessboard based on the global image of the chessboard captured by the monocular camera, thereby controlling the robotic arm to accurately grasp the three-dimensional piece at a suitable angle to perform the game task.

[0044] In one exemplary embodiment, such as Figure 2 As shown, a robotic arm control method is provided, which can be applied to electronic devices with data processing capabilities, such as... Figure 1 The robotic arm controller 101 shown, or other terminal devices, are not specifically limited in this embodiment. This method is applied to... Figure 1 The following description uses the robotic arm controller 101 as an example, including steps 202 to 210. Wherein:

[0045] Step 202: Obtain the global image of the placement area; at least one object is placed in the placement area.

[0046] Specifically, the global image refers to the image of the placement area captured by the image acquisition device when the entire placement area is within the field of view of the image acquisition device.

[0047] Alternatively, the robotic arm controller can acquire a global image of the placement area from an image acquisition device.

[0048] For example, the size of the placement area and the size of the object placed within the placement area can be set after testing. When the image acquisition device is in the preset acquisition position, regardless of the location of the object in the placement area, at least a portion of the object will be exposed within the field of view of the image acquisition device, so that object detection can be performed on at least one object in the placement area based on the global image.

[0049] In one possible implementation, taking a chessboard as an example, even if there is obstruction between the pieces, when the image acquisition device is in the preset acquisition position, at least a part of the pieces in the chessboard will be exposed in the field of view of the image acquisition device, regardless of which chessboard square they are in.

[0050] Step 204: Perform object detection on the global image to determine the category and rotation angle of at least one object, as well as the two-dimensional projection points of the feature points of at least one object in the placement area coordinate system.

[0051] Specifically, the rotation angle refers to the angle of rotation of an object relative to its base, which characterizes the change in the object's posture in space. For objects of different shapes, each object can be assigned at least one feature point. A feature point can be any point on the object's central axis, such as the top center point or bottom center point, or an easily identifiable edge point on the object's surface. In the global image (two-dimensional image) acquired by the image acquisition device, any point in the global image can be considered as a two-dimensional projection of a corresponding point on the object's surface from the image acquisition device's perspective.

[0052] Optionally, the robotic arm controller can perform multi-task object detection on the global image based on a preset image processing algorithm or model, thereby determining the category and rotation angle of at least one object, as well as the feature points of at least one object in the global image. Since the placement area is marked with a placement area identifier, which can be used to locate the placement area coordinate system, the robotic arm controller can further identify the placement area coordinate system based on the placement area identifier in the global image, thereby determining the two-dimensional coordinates of the two-dimensional projection points corresponding to the feature points of at least one object in the placement area coordinate system.

[0053] Alternatively, taking the example of a robotic arm controller performing multi-task object detection on a global image based on a preset image processing model, the preset image processing model may include: a category detection branch, a rotation angle detection branch, and a feature point detection branch. The category detection branch is used to detect the category of the object, the rotation branch is used to detect the rotation angle of the object, and the feature point detection branch is used to detect the feature points of the object and the two-dimensional coordinates of the corresponding two-dimensional projection points in the placement area coordinate system. The category detection branch, rotation angle detection branch, and feature point detection branch can all be trained based on neural networks, including but not limited to: CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), and LSTM (Long Short-Term Memory).

[0054] In the initial model training process, based on the original image and a reference image labeled with object categories, the neural network can be trained to detect object categories based on at least a portion of the exposed edges of the object in the image, thus obtaining a category detection branch. Furthermore, based on the original image and a reference image labeled with object rotation angles, the neural network can be trained to detect object rotation angles based on at least a portion of the exposed edges of the object in the image, thus obtaining a rotation angle detection branch. Even further, based on the original image and a reference image labeled with object feature points, the neural network can be trained to detect exposed feature points of the object through at least a portion of the exposed edges of the object in the image, and to detect the coordinates of the corresponding two-dimensional projection points of the feature points in the placement area coordinate system, thus obtaining a feature point detection branch.

[0055] Step 206: For each object, according to the object category, query the reference point cloud of the object in the reference pose, and correct the reference point cloud based on the rotation angle of the object to obtain the rotating point cloud of the object; the rotating point cloud includes the three-dimensional reference points corresponding to the feature points of the object.

[0056] In this context, if an object has a frontal view, the reference pose can refer to the orientation of the object when it is placed upright and facing forward. A point cloud is a dataset of three-dimensional points in a known coordinate system, including the three-dimensional coordinates of the points. It can represent the three-dimensional distribution of points on the surface of an object, that is, it represents the three-dimensional model of the object.

[0057] Optionally, for each object, the robotic arm controller can retrieve the reference point cloud of the object in its reference posture from the stored point cloud data of the object, based on the object's category. Furthermore, the robotic arm controller can correct the reference point cloud of the object based on its rotation angle. For example, based on the rotation angle of the object relative to its base, the controller can adjust the posture of the base or central axis in the reference point cloud, and consequently adjust the overall posture of the object in the reference point cloud, thereby obtaining the rotation point cloud of the object in its actual posture.

[0058] For example, taking a three-dimensional chess piece as an example, and the three-dimensional chess piece tipping over on the chessboard, if the rotation angle of the three-dimensional chess piece represents: the three-dimensional chess piece is in a tipped state facing one side, and there is no change in orientation, then as follows: Figure 3 The diagram illustrates a method for adjusting the reference point cloud of a 3D chess piece based on its rotation angle. Specifically, the reference pose represented by the reference point cloud is adjusted to the actual pose represented by the rotation point cloud of the 3D chess piece, thus representing the tilting pose of the piece.

[0059] Step 208: Match the projection relationship between the two-dimensional projection points and the three-dimensional reference points corresponding to the feature points of the object to determine the pose of the object in the coordinate system of the placement area.

[0060] Specifically, pose can include position and orientation, and is a 6D pose with six degrees of freedom, including three degrees of freedom of displacement and three degrees of freedom of spatial rotation.

[0061] Optionally, the robotic arm controller can match the projection relationships between the 2D projection points and 3D reference points corresponding to the same feature point of the object. For example, it can match the 2D projection point corresponding to the top center point of the object with the 3D reference point, and the 2D projection point corresponding to the bottom center point of the object with the 3D reference point. Further, the robotic arm controller can derive the back-projection matrix from the 2D projection points (including 2D coordinates) to the 3D reference points (including 3D coordinates) based on the matched 2D projection points (including 3D coordinates). Then, based on the back-projection matrix, it can recover the actual pose of the object in the placement area coordinate system from the 2D image of the object, i.e., the 6D pose of the object in the placement area coordinate system. Specifically, back-projection refers to: based on the principle of projection, given that the 3D model of the object (which can be determined by the point cloud of points on the object's surface) is known, and the positions of the projection points generated by the projection are known, the actual pose of the object is deduced in reverse.

[0062] In one optional implementation, taking a chessboard as the placement area and 3D chess pieces as the objects within it, the robotic arm controller can pre-store a 3D model (stored in point cloud form) of each 3D chess piece. This includes a reference point cloud of each 3D chess piece in a reference pose (the point cloud includes the 3D coordinates of the feature points on the surface of the 3D chess piece). Based on this, for each 3D chess piece in the global image, after obtaining the 2D projection points of the feature points on the surface of the 3D chess piece in the placement area coordinate system, as well as the rotated point cloud of the 3D chess piece, the robotic arm controller can match the projection relationship between the 2D projection points and the 3D reference points corresponding to the feature points of the 3D chess piece to determine the pose of the 3D chess piece in the placement area coordinate system, thereby controlling the robotic arm to accurately grasp the 3D chess piece.

[0063] Step 210: Based on the pose of each of the at least one object in the coordinate system of the placement area, select the object to be grasped from the at least one object, and control the robotic arm to grasp the object to be grasped.

[0064] Optionally, after obtaining the global image of the placement area, the robotic arm controller can perform multi-task object detection on at least one object in the global image using a preset image processing algorithm or a preset image processing model to obtain the pose of each of the at least one object in the placement area coordinate system. Further, based on the pose of each of the at least one object in the placement area coordinate system, the robotic arm controller can select the object to be grasped from the at least one object. After converting the pose of the object to be grasped in the placement area coordinate system to the pose in the robotic arm coordinate system, the controller can control the robotic arm to accurately grasp the object.

[0065] For example, taking a chessboard as the placement area and three-dimensional chess pieces as the objects in the placement area, after determining the pose of at least one chess piece in the chessboard in the placement area coordinate system (chessboard coordinate system), the robotic arm controller can first filter the chess pieces that need to be grasped first according to the pose of each chess piece in the placement area coordinate system, that is, filter the objects to be grasped, and then control the robotic arm to grasp the chess piece according to the pose of the chess piece.

[0066] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0067] The aforementioned robotic arm control method, under limited camera assembly conditions, can first perform object detection on the global image of the placement area to determine the category and rotation angle of at least one object, as well as the two-dimensional projection points of the feature points of at least one object in the placement area coordinate system, if a global image of the placement area is obtained. Further, for each object, a reference point cloud in the reference pose can be obtained based on the object's category. This reference point cloud is then corrected based on the object's rotation angle to obtain the object's rotation point cloud. Subsequently, by matching the projection relationship between the two-dimensional projection points corresponding to the object's feature points and the three-dimensional reference points, the pose of the object in the placement area coordinate system is determined. This allows the robotic arm to be controlled to accurately grasp the object at a suitable angle based on the pose. In other words, even with limited camera assembly conditions, the above robotic arm control method can accurately determine the pose of the object. Based on the pose of at least one object in the placement area coordinate system, an object to be grasped can be selected from at least one object, and the robotic arm can be controlled to accurately grasp the object, improving the accuracy of the robotic arm's object grasping.

[0068] In one embodiment, object detection is performed on the global image to determine the category and rotation angle of at least one object, as well as the two-dimensional projection points of the feature points of at least one object in the placement area coordinate system. This includes: extracting features from the global image to obtain placement area image features and object image features of at least one object; using the placement area image features to determine the placement area coordinate system; for each object, performing object detection based on the object image features to determine the object's category and rotation angle, as well as the object's feature points in the global image; and determining the two-dimensional projection points of the object's feature points in the placement area coordinate system based on the placement area image features and the object's feature points in the global image.

[0069] Alternatively, after acquiring the global image from the image acquisition device, the robotic arm controller can first extract features from the global image based on a preset feature extraction algorithm or model to obtain the image features of the placement area and the object image features of at least one object. Further, for each object, the robotic arm controller can input the extracted object image features into a preset image processing model to perform object detection based on the object's image features, or perform object detection based on the object's image features using a preset image processing algorithm, thereby determining the object's category, rotation angle, and feature points in the global image. Even further, the robotic arm controller can locate the placement area coordinate system in the global image based on the placement area image features, and determine the two-dimensional projection points of the object's feature points in the placement area coordinate system based on the positions of the object's feature points in the global image.

[0070] In some embodiments, a preset feature extraction model may be deployed in the robotic arm controller. This preset feature extraction model can be trained based on a neural network, such as a CNN (Convolutional Neural Network), which can comprehensively and accurately extract image features from the global image.

[0071] In a specific application scenario, taking a chessboard as the placement area and three-dimensional chess pieces (such as three-dimensional chess pieces) as an example, the robotic arm controller can achieve feature extraction and object detection of the entire image based on the deployed preset feature extraction model and preset image processing model. For example... Figure 4 As shown, a flowchart illustrates a process for determining the pose of at least one 3D chess piece on a chessboard based on a preset feature extraction model and a preset image processing model.

[0072] Specifically, such as Figure 4 As shown, after acquiring the global image of the chessboard, the robotic arm controller can first input the global image into a preset feature extraction model to extract the placement area image features and the object image features of at least one 3D chess piece from the global image. The global image can be an RGB 2D image captured by a monocular camera. The placement area image features can be extracted based on the chessboard grid lines, chessboard markers, and chessboard edges in the global image. For each 3D chess piece, the object image features can be extracted based on the piece's outline and pattern as revealed in the global image.

[0073] Furthermore, for each 3D chess piece, the robotic arm controller can input the object image features of the 3D chess piece into a preset image processing model. This model then performs multi-task object detection using its category detection branch, rotation angle detection branch, and feature point detection branch. Specifically, the category detection branch can detect the category of the 3D chess piece based on its object image features; the rotation angle detection branch can detect the rotation angle of the 3D chess piece based on its object image features; and the feature point detection branch can detect the feature points of the 3D chess piece based on its object image features, and, combined with the image features of the placement area, detect the two-dimensional projection points of these feature points in the placement area coordinate system. Based on this, the robotic arm controller can determine the pose of each 3D chess piece by combining the above effective information.

[0074] In this embodiment, feature extraction can be performed on the global image to obtain the image features carried by the global image. This is beneficial for further accurate detection of effective information in the global image based on the image features. For example, at least one object's category and rotation angle, as well as the two-dimensional projection points of the object's feature points in the placement area coordinate system, can be detected. This facilitates accurate positioning of the object's pose based on the detected effective information, thereby improving the accuracy of the robotic arm in grasping the object.

[0075] In a specific example, selecting an object to be grasped from at least one object based on the pose of each of the at least one objects in the coordinate system of the placement area, and controlling the robotic arm to grasp the object to be grasped, includes: determining the placement state of each of the at least one object based on the pose of each of the at least one object in the coordinate system of the placement area; if there is an object in the tilted state, taking the object in the tilted state as the object to be grasped; and controlling the robotic arm to grasp the object to be grasped based on the pose of the object to be grasped in the coordinate system of the placement area, so as to adjust the placement state of the object to be grasped.

[0076] The placement states include: tilted state (non-upright placement state) and upright state.

[0077] Optionally, the robotic arm controller can compare the pose of at least one object in the placement area coordinate system with its pose in an upright state. For example, it can determine whether there is an angle between the central axis of the object in its current pose and the central axis of the object in its upright state. If an angle exists, the object is determined to be tilted; otherwise, the object is determined to be upright. Based on this, the placement state of at least one object can be determined.

[0078] Furthermore, when an object is in a tilted position, the robotic arm controller can identify it as the object to be grasped, requiring adjustment of its position. Based on the object's pose in the placement area coordinate system, the controller precisely grasps the object to adjust its position. For example, after identifying the object, the controller uses its position in the placement area coordinate system to control the robotic arm's end effector to grasp it and adjust its angle to place the object back into its original upright position.

[0079] It should be noted that if the robotic gripper cannot successfully grasp the object to be grasped, or if adjusting the angle of the gripper still fails to return the object to its original upright position, the robotic arm controller can send a warning message to the connected monitoring terminal to prompt manual adjustment of the object's placement. If the robotic arm is also equipped with a light-emitting device and / or a sound-emitting device, the robotic arm controller can control the light-emitting device to emit a warning light, and / or control the sound-emitting device to emit a warning sound, to prompt manual adjustment of the object's placement.

[0080] In this embodiment, after determining the pose of at least one object in the coordinate system of the placement area, it can be first determined whether any object is in a tilted state based on the pose. If so, the robotic arm can be controlled to precisely grasp the object based on its pose, thereby adjusting the object's placement state.

[0081] In one embodiment, the global image acquisition scenario includes a first scenario and a second scenario. The first scenario refers to acquiring the global image during the object placement task in the placement area. The second scenario refers to acquiring the global image during the object movement task in the placement area after the object placement task has been completed. Based on this, when there is an object in a tilted state, the object in the tilted state is taken as the object to be grasped, including: when the global image acquisition scenario is the first scenario and there is an object in a tilted state, continuing to execute the object placement task in the placement area until the object placement task is completed, and taking the tilted object as the object to be grasped whose placement state needs to be adjusted; when the global image acquisition scenario is the second scenario and there is an object in a tilted state, pausing the object movement task in the placement area, and taking the tilted object as the object to be grasped whose placement state needs to be adjusted.

[0082] The first scenario can be as follows: During the process of the robotic arm controller placing objects in a designated location within the placement area according to preset object placement rules, the image acquisition unit automatically captures a global image of the placement area and transmits it to the robotic arm controller. The second scenario can be as follows: After the object placement task is completed, during the process of the robotic arm controller moving objects within the placement area according to an object movement strategy, the image acquisition unit automatically captures a global image of the placement area and transmits it to the robotic arm controller.

[0083] Alternatively, if the global image acquisition scene is the first scene and there are objects in a tilted state, the robotic arm controller can control the robotic arm to continue performing the object placement task in the placement area until the object placement task is completed. That is, the robotic arm can continue to grab the objects to be placed and place them in the designated positions in the placement area until all the objects to be placed have been placed in the placement area. Then, the objects in the tilted state in the placement area are taken as objects to be grabbed with their placement state adjusted to complete the final placement.

[0084] Optionally, if the global image acquisition scenario is the second scenario and there is an object in a tilted state, the robotic arm controller can pause the object movement task in the placement area and first treat the tilted object as the object to be grasped whose placement state needs to be adjusted, so as to adjust the placement state of the object to be grasped and avoid interfering with the movement of objects in the placement area.

[0085] For example, taking a chessboard as the placement area and three-dimensional chess pieces as the objects in the placement area, the first scenario can be: during the chessboard placement process before the game, the image acquisition device automatically acquires a global image of the chessboard and transmits it to the robotic arm controller. The second scenario can be: during the game after the chessboard placement is completed, the image acquisition device automatically acquires a global image of the chessboard and transmits it to the robotic arm controller.

[0086] Furthermore, during the game setup process, if the global image of the chessboard indicates that any placed 3D pieces are tilted, the robotic arm controller can first control the robotic arm to continue grabbing the 3D pieces to be placed and placing them at designated positions on the chessboard. This process continues until all 3D pieces have been placed. Then, the tilted pieces on the chessboard are treated as objects to be grabbed, and their positions are adjusted based on their posture, thus completing the global placement of the 3D pieces. If no 3D pieces are tilted during the setup process, the game can proceed directly after the setup is complete.

[0087] Furthermore, during the game, if the global image of the chessboard reveals that some placed 3D pieces are tilted, considering that these tilted pieces occupy multiple squares and affect subsequent distribution data analysis, which in turn affects object movement strategy analysis, the robotic arm controller can first pause the game and treat the tilted 3D pieces as objects to be grasped. The controller can then grasp these objects based on their pose, adjusting their placement and preventing them from interfering with the game.

[0088] In this embodiment, during the object placement task in the placement area, if it is determined that there is an object in a tilted state within the placement area, the object placement task in the placement area can be performed first. After the placement task is completed, the tilted object can be used as a grabbable object for grabbing and adjusting its pose, thereby completing the global placement of objects in the placement area as quickly as possible. During the object movement task in the placement area, if it is determined that there is an object in a tilted state within the placement area, the tilted object is prioritized as a grabbable object for grabbing and adjusting its pose, avoiding interference with object movement.

[0089] In an exemplary embodiment, controlling a robotic arm to grasp an object based on its pose in the placement area coordinate system includes: acquiring a first transformation relationship between the camera coordinate system and the placement area coordinate system, and a second transformation relationship between the robotic arm coordinate system and the camera coordinate system; converting the pose of the object in the placement area coordinate system to the pose in the camera coordinate system based on the first transformation relationship, and converting the pose in the camera coordinate system to the pose in the robotic arm coordinate system based on the second transformation relationship; and controlling the robotic arm to grasp the object based on its pose in the robotic arm coordinate system.

[0090] The coordinate system of the placement area can be a known coordinate system constructed based on points selected from the placement area. The coordinate system of the camera can be a known coordinate system constructed based on the installation position or fixed position of the image acquisition device. The coordinate system of the robotic arm can be a known coordinate system constructed based on the base of the robotic arm, which is the coordinate system used for motion control of the robotic arm. In practical applications, the image acquisition device is generally installed in a fixed position, maintaining a stable relative position with the placement area and the robotic arm. Based on this, the first transformation relationship between the camera coordinate system and the placement area coordinate system, and the second transformation relationship between the robotic arm coordinate system and the camera coordinate system can be pre-calculated and stored.

[0091] Alternatively, the robotic arm controller can obtain a first transformation relationship between the camera coordinate system and the placement area coordinate system, and a second transformation relationship between the robotic arm coordinate system and the camera coordinate system, from pre-stored data. Based on this, the pose of the object to be grasped in the placement area coordinate system can be converted to the pose in the camera coordinate system according to the first transformation relationship, and the pose in the camera coordinate system can be converted to the pose in the robotic arm coordinate system according to the second transformation relationship, so that the robotic arm can be controlled to grasp the object according to the pose of the object to be grasped in the robotic arm coordinate system.

[0092] In this embodiment, after determining the object to be grasped, multiple coordinate system transformations can be performed through the transformation relationship between different coordinate systems to convert the pose of the object to be grasped in the coordinate system of the placement area into the pose in the coordinate system of the robotic arm. This is beneficial for controlling the robotic arm to accurately grasp the object according to the pose in the coordinate system of the robotic arm.

[0093] In some possible implementations, after determining the placement state of at least one object based on its pose in the placement area coordinate system, the method further includes: if the global image acquisition scene is the second scene and there are no objects in the tilted state, determining the distribution data of at least one object in the placement area coordinate system based on its pose; performing object movement strategy analysis based on the distribution data to determine the object to be grasped among the at least one objects and the target position of the object to be grasped in the placement area coordinate system; and controlling the robotic arm to grasp the object to be grasped and move it to the target position based on its pose in the placement area coordinate system.

[0094] The absence of an object in a tilted state can include two situations: no object has tilted, or an object that has tilted has been properly positioned.

[0095] Optionally, during the execution of the object movement task in the placement area, if there is no object in the tilted state, the robotic arm controller can determine the distribution data of at least one object in the placement area coordinate system based on the pose (including position and orientation) of each object in the placement area coordinate system. Based on the distribution data, the controller can perform object movement strategy analysis, identify the object to be moved among the at least one objects, identify the object to be moved as the object to be grasped, and determine the target position of the object to be grasped in the placement area coordinate system, that is, determine the position to be moved to.

[0096] Furthermore, the robotic arm controller can convert the pose of the object to be grasped in the placement area coordinate system to the pose in the camera coordinate system, then convert the pose in the camera coordinate system to the pose in the robotic arm coordinate system, and finally convert the target position in the placement area coordinate system to the target position in the robotic arm coordinate system. Based on this, the robotic arm controller can control the robotic arm to accurately grasp the object to be grasped according to its pose in the robotic arm coordinate system, and precisely move the object to the target position.

[0097] For example, taking a chessboard as the placement area and three-dimensional chess pieces as the objects in the placement area, the robotic arm controller can also be equipped with a chess game analysis engine. After determining the distribution data of each three-dimensional chess piece in the chessboard under the coordinate system of the placement area, the robotic arm controller can use the chess game analysis engine to analyze the object movement strategy of the three-dimensional chess pieces, determine the three-dimensional chess piece to be moved and the target position to which the three-dimensional chess piece needs to be moved, and thus take the three-dimensional chess piece as the object to be grasped, and grasp and place the object to be grasped to the target position according to the pose of the object to be grasped, thereby completing the move of the game process.

[0098] In this embodiment, after determining that no object is in a tilted state, an object movement strategy analysis can be performed based on the pose of at least one object in its respective placement area coordinate system to determine the object to be grasped and its corresponding target position. Then, based on the pose of the object to be grasped, the robotic arm can be controlled to accurately grasp the object and move it to the target position.

[0099] It should be noted that during the object movement task in the placement area, after determining the placement state of at least one object, if any object is tilted, the robotic arm controller can control the robotic arm to prioritize adjusting the pose of the tilted object to prevent it from interfering with the movement of other objects. Then, object movement strategy analysis is performed, and the object is moved to the target position. However, if the placement area has ample space, and the change in the area occupied by a tilted object does not affect the analyzed distribution data, the robotic arm controller can also first perform object movement strategy analysis to determine the object to be moved. If the object to be moved is not tilted, its pose can be adjusted after moving it to the target position. If the object to be moved is tilted, the robotic arm can be directly controlled to grasp the object, adjust its pose, and move it to the target position.

[0100] In one possible implementation, based on the above embodiments, taking the process of a robotic arm performing an object movement task as an example, a flowchart of another robotic arm control method is provided, such as... Figure 5 As shown, the main steps include:

[0101] Step 502: Obtain the global image of the placement area; at least one object is placed in the placement area.

[0102] Step 504: Extract features from the global image to obtain the placement area image features and the object image features of at least one object; the placement area image features are used to determine the placement area coordinate system.

[0103] Step 506: For each object, perform object detection based on the object's image features to determine the object's category, rotation angle, and feature points of the object in the global image.

[0104] Step 508: Based on the features of the placement area image and the feature points of the object in the global image, determine the two-dimensional projection points of the object's feature points in the placement area coordinate system.

[0105] Step 510: Based on the object's category, retrieve the reference point cloud of the object in its reference pose, and correct the reference point cloud based on the object's rotation angle to obtain the object's rotation point cloud; the rotation point cloud includes the three-dimensional reference points corresponding to the object's feature points.

[0106] Step 512: Match the projection relationship between the two-dimensional projection points and the three-dimensional reference points corresponding to the feature points of the object to determine the pose of the object in the coordinate system of the placement area.

[0107] Step 514: Determine the placement state of at least one object based on its pose in the coordinate system of the placement area.

[0108] If there is an object in a tilted position, proceed to step 516 to use the tilted object as the object to be grasped; step 518 to convert the pose of the object to be grasped in the coordinate system of the placement area to the pose in the coordinate system of the robotic arm; step 520 to control the robotic arm to grasp the object to be grasped according to the pose in the coordinate system of the robotic arm and adjust the pose of the object to be grasped.

[0109] If no object is in a tilted position, proceed to step 522: determine the distribution data of at least one object in the placement area coordinate system based on the pose of each object in the placement area coordinate system; step 524: perform object movement strategy analysis based on the distribution data to determine the object to be grasped among the at least one objects, and the target position of the object to be grasped in the placement area coordinate system; step 526: convert the pose of the object to be grasped in the placement area coordinate system into the pose in the robotic arm coordinate system; step 528: control the robotic arm to grasp the object to be grasped according to its position in the robotic arm coordinate system, and move the object to be grasped to the target position.

[0110] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0111] Based on the same inventive concept, this application also provides a robotic arm control device for implementing the robotic arm control method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more embodiments of the robotic arm control device provided below can be found in the limitations of the robotic arm control method described above, and will not be repeated here.

[0112] In one exemplary embodiment, such as Figure 6 As shown, a robotic arm control device is provided, including: a global image acquisition module 602, a multi-task detection module 604, a point cloud acquisition module 606, a pose determination module 608, and an object grasping module 610, wherein:

[0113] The global image acquisition module is used to acquire the global image of the placement area, where at least one object is placed.

[0114] The multi-task detection module is used to perform object detection on the global image, determine the category and rotation angle of at least one object, and the two-dimensional projection points of the feature points of at least one object in the placement area coordinate system.

[0115] The point cloud acquisition module is used to query the reference point cloud of each object in the reference pose according to the object category, and correct the reference point cloud based on the rotation angle of the object to obtain the rotating point cloud of the object. The rotating point cloud includes the three-dimensional reference points corresponding to the feature points of the object.

[0116] The pose determination module is used to match the projection relationship between the two-dimensional projection points and the three-dimensional reference points corresponding to the feature points of the object, and determine the pose of the object in the coordinate system of the placement area.

[0117] The object grasping module is used to select an object to be grasped from at least one object based on the pose of each object in the coordinate system of the placement area, and to control the robotic arm to grasp the object to be grasped.

[0118] The aforementioned robotic arm control device, under limited camera assembly conditions, can first perform object detection on the global image of the placement area to determine the category and rotation angle of at least one object, as well as the two-dimensional projection points of the feature points of at least one object in the placement area coordinate system, if a global image of the placement area is obtained. Further, for each object, a reference point cloud in the reference pose can be obtained based on the object's category. This reference point cloud is then corrected based on the object's rotation angle to obtain the object's rotation point cloud. Subsequently, by matching the projection relationship between the two-dimensional projection points corresponding to the object's feature points and the three-dimensional reference points, the pose of the object in the placement area coordinate system is determined. This allows the robotic arm to be controlled to accurately grasp the object at a suitable angle based on the pose. In other words, even with limited camera assembly conditions, the above robotic arm control method can accurately determine the pose of objects. Based on the pose of at least one object in the placement area coordinate system, an object to be grasped can be selected from at least one object, and the robotic arm can be controlled to accurately grasp the object, improving the accuracy of the robotic arm's object grasping.

[0119] In one embodiment, the multi-task detection module includes: a feature extraction unit, used to extract features from the global image to obtain placement area image features and object image features of at least one object, the placement area image features being used to determine the placement area coordinate system; a multi-task detection unit, used to perform object detection for each object based on the object image features of the object, to determine the object's category and rotation angle, as well as the object's feature points in the global image; and a projection point detection unit, used to determine the two-dimensional projection points of the object's feature points in the placement area coordinate system based on the placement area image features and the object's feature points in the global image.

[0120] In a specific example, the object grasping module includes: a placement state determination unit, used to determine the placement state of at least one object based on the pose of each object in the placement area coordinate system; a graspable object determination unit, used to identify an object in the tilted state as the graspable object if an object in the tilted state exists; and a robotic arm control unit, used to control the robotic arm to grasp the graspable object based on the pose of the graspable object in the placement area coordinate system, so as to adjust the placement state of the graspable object.

[0121] In one embodiment, the robotic arm control unit is specifically configured to: acquire a first transformation relationship between the camera coordinate system and the placement area coordinate system, and a second transformation relationship between the robotic arm coordinate system and the camera coordinate system; convert the pose of the object to be grasped in the placement area coordinate system to the pose in the camera coordinate system according to the first transformation relationship, and convert the pose in the camera coordinate system to the pose in the robotic arm coordinate system according to the second transformation relationship; and control the robotic arm to grasp the object to be grasped according to the pose of the object to be grasped in the robotic arm coordinate system.

[0122] In one embodiment, the global image acquisition scenario includes a first scenario and a second scenario. The first scenario refers to acquiring the global image during the object placement task in the placement area. The second scenario refers to acquiring the global image during the object movement task in the placement area after the object placement task has been completed. Based on this, the object to be grasped determination unit in the object grasping module is specifically used to: when the global image acquisition scenario is the first scenario and there is an object in a tilted state, continue to execute the object placement task in the placement area until the object placement task is completed, and regard the tilted object as the object to be grasped whose placement state needs to be adjusted; when the global image acquisition scenario is the second scenario and there is an object in a tilted state, pause the object movement task in the placement area, and regard the tilted object as the object to be grasped whose placement state needs to be adjusted.

[0123] In some possible implementations, the object-to-be-grabbed unit in the object grasping module is further configured to: determine the distribution data of at least one object in the placement area coordinate system based on the pose of each object in the placement area coordinate system, provided that the global image acquisition scene is the second scene and there are no objects in the tilted state; perform object movement strategy analysis based on the distribution data to determine the object to be grasped among the at least one objects, and the target position of the object to be grasped in the placement area coordinate system. The robotic arm control unit in the object grasping module is further configured to: control the robotic arm to grasp the object to be grasped based on its pose in the placement area coordinate system, and move the object to be grasped to the target position.

[0124] Each module in the aforementioned robotic arm control device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0125] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores robotic arm control data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a robotic arm control method.

[0126] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0127] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described robotic arm control method.

[0128] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above-described robotic arm control method.

[0129] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described robotic arm control method.

[0130] In some embodiments, this application also provides a robotic arm, including: a robotic arm body, the robotic arm body including a motion part and an execution part connected to the end of the motion part, the execution part being used to interact with at least one object placed in a placement area; and a robotic arm controller, used to acquire a global image of the placement area and, based on the global image, execute the above-described robotic arm control method to control the execution part to interact with at least one object.

[0131] Specifically, the motion unit may include a shoulder structure, a first arm, an elbow structure, and a second arm. The shoulder structure can be mounted on the robot or other device. The first end of the first arm is movably connected to the shoulder structure, and the second end of the first arm is movably connected to the elbow structure. The elbow structure is in turn movably connected to the first end of the second arm. The second end of the second arm (i.e., the end of the motion unit) can be fitted with an actuator, and an additional degree of freedom—the ability to lift up and down—is added at the connection between the second end of the second arm and the actuator. Furthermore, the actuator may have one degree of freedom, specifically a mechanical gripper, capable of grasping and placing objects through opening and closing. Based on this, the robotic arm in this embodiment can achieve four degrees of freedom by adding two more degrees of freedom at the end of the second arm, building upon the traditional two-degree-of-freedom robotic arm. Combined with the aforementioned robotic arm control method, it can grasp objects flexibly and precisely.

[0132] It should be noted that the added vertical lifting freedom at the connection between the second arm and the actuator allows the robotic arm to grasp objects vertically, rather than in an arc, thus avoiding accidental collisions with other objects and improving the accuracy of object grasping. Specifically, when the second end of the second arm is raised or lowered, both the first and second arms move horizontally accordingly, ensuring that the end of the second arm can grasp the object vertically.

[0133] Alternatively, such as Figure 8 As shown, taking a mechanical gripper as the actuator, a schematic diagram of the robotic arm body is provided. The shoulder structure 801 is movably connected to the first end of the first arm 802, the second end of the first arm 802 is movably connected to the elbow structure 803, and the elbow structure 803 is movably connected to the first end of the second arm 804. The second end of the second arm 804 can be fitted with the actuator 805.

[0134] In this embodiment, compared with purely mechanical control of object grasping, the size of the equipment and the cost of the equipment can be simplified and reduced by using a four-degree-of-freedom robotic arm and the above-mentioned robotic arm control method. When using a lightweight image acquisition device, such as a monocular camera, the robotic arm can also be controlled to grasp objects with high precision based on the global image acquired by the image acquisition device.

[0135] In some possible implementations, this application also provides a robot, comprising: an image acquisition device for acquiring a global image of a placement area; and the aforementioned robotic arm for interacting with at least one object placed within the placement area based on the global image.

[0136] The robotic arm may include a robotic arm body and a robotic arm controller. The robotic arm body includes a motion unit and an actuator connected to the end of the motion unit. The actuator is used to interact with at least one object placed in a placement area. The robotic arm controller is used to acquire a global image of the placement area and, based on the global image, execute the aforementioned robotic arm control method to control the actuator to interact with at least one object. Specifically, the robotic arm body may include a shoulder structure, a first arm, an elbow structure, a second arm, and an actuator. The connection between the second arm and the actuator adds a degree of freedom for vertical lifting. The actuator can also grasp and place objects by opening and closing.

[0137] Optionally, during the robot's operation, the image acquisition device can transmit the acquired global images to the robotic arm controller in the robotic arm at regular intervals or in real time. The robotic arm controller can then execute the aforementioned robotic arm control methods based on the global images to control the robotic arm body to grasp objects accurately and flexibly.

[0138] For example, taking a game of chess as an example, the placement area is a chessboard, and the objects in the placement area are three-dimensional chess pieces, such as... Figure 9 As shown, a top view of a chess-playing robot including the aforementioned robotic arm is provided. The image acquisition device mounted on the robot can capture a global image of the entire chessboard from top to bottom. Furthermore, as... Figure 10 As shown, a schematic diagram of the second arm of the chess-playing robot being raised is provided, such as... Figure 11 The diagram shows the second arm of the chess-playing robot in motion. That is, during the process of grasping pieces to perform a chess-playing task, the robot can grasp vertically, rather than in an arc. Furthermore, as... Figure 12 The diagram illustrates a chess-playing robot using a robotic gripper at the end of its arm to grasp a tilted 3D chess piece. Throughout the game, the robot's image acquisition unit captures a global image, which the robotic arm controller uses to precisely grasp the pieces and execute the game task.

[0139] It should be noted that the information (including but not limited to device information) and data (including but not limited to data used for analysis, data stored, data displayed) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0140] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0141] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this application. The above embodiments only illustrate several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of this application. It should be noted that for those skilled in the art, several modifications and improvements can be made without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A robotic arm control method, characterized in that, The method includes: Obtain a global image of the placement area; at least one object is placed within the placement area. Object detection is performed on the global image to determine the category and rotation angle of each of the at least one object, as well as the two-dimensional projection points of the feature points of each of the at least one object in the placement area coordinate system. For each object, a reference point cloud of the object in a reference pose is obtained according to the object's category. The reference point cloud is then corrected based on the object's rotation angle to obtain a rotated point cloud of the object. The rotated point cloud includes three-dimensional reference points corresponding to the object's feature points. The two-dimensional projection points and three-dimensional reference points corresponding to the feature points of the object are matched by projection relationship to determine the pose of the object in the coordinate system of the placement area. Based on the pose of each of the at least one object in the coordinate system of the placement area, an object to be grasped is selected from the at least one object, and the robotic arm is controlled to grasp the object to be grasped.

2. The method according to claim 1, characterized in that, The step of performing object detection on the global image to determine the category and rotation angle of each of the at least one object, and the two-dimensional projection points of the feature points of each of the at least one object in the placement area coordinate system, includes: Feature extraction is performed on the global image to obtain the placement area image features and the object image features of each of the at least one object; the placement area image features are used to determine the placement area coordinate system; For each object, object detection is performed based on the object image features to determine the object's category, rotation angle, and feature points of the object in the global image; Based on the image features of the placement area and the feature points of the object in the global image, determine the two-dimensional projection points of the object's feature points in the coordinate system of the placement area.

3. The method according to claim 1, characterized in that, The step of selecting an object to be grasped from the at least one object based on the pose of each of the at least one object in the coordinate system of the placement area, and controlling the robotic arm to grasp the object to be grasped, includes: The placement state of each of the at least one object is determined based on its pose in the coordinate system of the placement area. If there is an object in the tilted state, the object in the tilted state will be taken as the object to be grabbed. Based on the pose of the object to be grasped in the coordinate system of the placement area, the robotic arm is controlled to grasp the object to be grasped, so as to adjust the placement state of the object to be grasped.

4. The method according to claim 3, characterized in that, The global image acquisition scenarios include a first scenario and a second scenario; the first scenario refers to acquiring the global image during the execution of the object placement task in the placement area; the second scenario refers to acquiring the global image during the execution of the object movement task in the placement area after the object placement task has been completed. The step of using an object in a tilted state as the object to be grasped when such an object exists includes: If the global image acquisition scene is the first scene and there is an object in the tilted state, continue to execute the object placement task in the placement area until the object placement task is completed, and take the tilted object as the object to be grabbed that needs to have its placement state adjusted. If the global image acquisition scenario is the second scenario and there is an object in a tilted state, the object movement task in the placement area is paused, and the tilted object is taken as the object to be grabbed whose placement state needs to be adjusted.

5. The method according to claim 3, characterized in that, The step of controlling the robotic arm to grasp the object according to its pose in the coordinate system of the placement area includes: Obtain the first transformation relationship between the camera coordinate system and the placement area coordinate system, and the second transformation relationship between the robot arm coordinate system and the camera coordinate system; According to the first transformation relationship, the pose of the object to be grasped in the coordinate system of the placement area is converted into the pose in the camera coordinate system, and according to the second transformation relationship, the pose in the camera coordinate system is converted into the pose in the coordinate system of the robotic arm. Based on the pose of the object to be grasped in the coordinate system of the robotic arm, the robotic arm is controlled to grasp the object.

6. The method according to claim 4, characterized in that, After determining the placement state of the at least one object based on its pose in the coordinate system of the placement area, the method further includes: When the global image acquisition scene is the second scene and there are no objects in the tilted state, the distribution data of the at least one object in the placement area coordinate system is determined according to the pose of the at least one object in the placement area coordinate system. Based on the distribution data, an object movement strategy analysis is performed to determine the object to be grabbed among the at least one objects, and the target position of the object to be grabbed in the coordinate system of the placement area. Based on the pose of the object to be grasped in the coordinate system of the placement area, the robotic arm is controlled to grasp the object and move it to the target position.

7. A robotic arm, characterized in that, The robotic arm includes: A robotic arm body, the robotic arm body including a motion part and an execution part connected to the end of the motion part, the execution part being used to interact with at least one object placed in the placement area; A robotic arm controller is configured to acquire a global image of the placement area and, based on the global image, execute the robotic arm control method according to any one of claims 1 to 6 to control the actuator to interact with the at least one object.

8. A robot, characterized in that, The robot includes: Image acquisition device, used to acquire a global image of the placement area; The robotic arm of claim 7 is used to interact with at least one object placed within the placement area based on the global image.

9. A robotic arm control device, characterized in that, The device includes: A global image acquisition module is used to acquire a global image of the placement area; at least one object is placed in the placement area. A multi-task detection module is used to perform object detection on the global image, determine the category and rotation angle of each of the at least one object, and the two-dimensional projection points of the feature points of each of the at least one object in the placement area coordinate system. The point cloud acquisition module is used to query the reference point cloud of each object in a reference pose according to the object's category, and correct the reference point cloud based on the object's rotation angle to obtain the object's rotation point cloud; the rotation point cloud includes three-dimensional reference points corresponding to the object's feature points. The pose determination module is used to match the projection relationship between the two-dimensional projection points and the three-dimensional reference points corresponding to the feature points of the object, and to determine the pose of the object in the coordinate system of the placement area. The object grasping module is used to select an object to be grasped from the at least one object according to the pose of each of the at least one object in the coordinate system of the placement area, and to control the robotic arm to grasp the object to be grasped.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

Images