Method and device for controlling a robot, robot, medium and product

Abstract

The present disclosure provides a kind of control method, device, manipulator, robot, medium and product of manipulator, it is related to control technical field.The method comprises the posture parameter of target object is obtained in response to the grasping task of manipulator;Target object is located at first target position, and target object is obtained according to the analysis of scene image comprising multiple objects;The posture of target object is adjusted by controlling manipulator, until the posture parameter meets preset posture condition;Control manipulator to move the target object after adjustment to second target position.Through the control strategy of adjusting the posture of target object first and then moving target object, target object is transferred with suitable posture, and the stability and reliability of manipulator moving target object are improved.

Description

Technical Field

[0001] This disclosure relates to the field of control technology, and in particular to a control method, device, manipulator, robot, medium, and product for a robotic arm. Background Technology

[0002] With the continuous development of robotic arm technology, the control technology of robotic arms has become more and more mature, and robotic arms can perform some simple tasks (such as moving objects), making the application scenarios of robotic arms more and more widespread. However, robotic arms have the problem of poor performance in moving objects. Summary of the Invention

[0003] This disclosure provides a control method, device, robot, medium, and product for a robotic arm to solve problems in the related art.

[0004] A first aspect of this disclosure provides a control method for a robotic arm, the method comprising: In response to the robotic arm's grasping task, the orientation parameters of the target object are acquired; the target object is located at the first target position, and the target object is obtained based on scene image analysis including multiple objects; The robot arm is controlled to adjust the posture of the target object until the posture parameters meet the preset posture conditions. The robotic arm is controlled to move the adjusted target object to the second target position.

[0005] A second aspect of this disclosure provides a control device for a robotic arm, comprising: The acquisition unit is used to acquire the posture parameters of the target object in response to the grasping task of the robotic arm; the target object is located at the first target position, and the target object is obtained by analyzing a scene image including multiple objects; The first control unit is used to control the robot arm to adjust the posture of the target object until the posture parameters meet the preset posture conditions. The second control unit is used to control the robotic arm to move the adjusted target object to the second target position.

[0006] A third aspect of this disclosure provides a robotic arm, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the methods described in the first aspect of this disclosure.

[0007] A fourth aspect of this disclosure provides a robot for performing the methods described in the first aspect of this disclosure, or including the robotic arm described in the third aspect of this disclosure.

[0008] A fifth aspect of this disclosure provides a non-transitory computer-readable storage medium that, when instructions in the storage medium are executed by a processor of a mobile terminal, enables the mobile terminal to perform the methods described in the first aspect of this disclosure.

[0009] According to a sixth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods described in the embodiments of the first aspect of this disclosure.

[0010] In summary, the control method for the robotic arm proposed in this disclosure includes: in response to the robotic arm's grasping task, acquiring the posture parameters of a target object; the target object being located at a first target position, the target object being obtained based on scene image analysis including multiple objects; controlling the robotic arm to adjust the posture of the target object until the posture parameters meet preset posture conditions; and controlling the robotic arm to move the adjusted target object to a second target position. By first adjusting the posture of the target object and then moving it, the target object is transferred in a suitable posture, improving the stability and reliability of the robotic arm's movement of the target object.

[0011] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0013] Figure 1 A flowchart of a control method for a robotic arm provided in this embodiment of the present disclosure; Figure 2 A flowchart illustrating another control method for a robotic arm provided in this embodiment of the present disclosure; Figure 3 A flowchart illustrating another control method for a robotic arm provided in this embodiment of the present disclosure; Figure 4 This is a diagram illustrating a grasping coordinate system provided in an embodiment of this disclosure; Figure 5 This diagram illustrates a capture method provided in an embodiment of the present disclosure. Figure 6 This diagram illustrates another scraping method provided in this embodiment of the disclosure. Figure 7 A flowchart illustrating another control method for a robotic arm provided in this embodiment of the present disclosure; Figure 8A flowchart illustrating another control method for a robotic arm provided in this embodiment of the present disclosure; Figure 9 This is a schematic diagram of the structure of a control device for a robotic arm provided in an embodiment of the present disclosure; Figure 10 A schematic diagram of the structure of another control device for a robotic arm provided in an embodiment of this disclosure; Figure 11 This is a schematic diagram of the structure of a robotic arm provided in an embodiment of the present disclosure; Figure 12 This is a schematic diagram of the structure of a chip provided in an embodiment of the present disclosure. Detailed Implementation

[0014] Some embodiments of this disclosure will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. Various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but can be changed as will become apparent upon understanding this disclosure, except for operations that must be performed in a particular order. Furthermore, for clarity and brevity, descriptions of features known in the art may be omitted.

[0015] The embodiments described in the following examples of this disclosure are not representative of all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0016] With the continuous development of robotic arm technology, the control technology of robotic arms has become more and more mature, and robotic arms can perform some simple tasks (such as moving objects), making the application scenarios of robotic arms more and more widespread. However, robotic arms have the problem of poor performance in moving objects.

[0017] Therefore, to address the problems existing in related technologies, this disclosure proposes a control method for a robotic arm. The method includes: in response to a grasping task by the robotic arm, acquiring the posture parameters of a target object; the target object being located at a first target position, the target object being obtained based on scene image analysis including multiple objects; controlling the robotic arm to adjust the posture of the target object until the posture parameters meet preset posture conditions; and controlling the robotic arm to move the adjusted target object to a second target position. By employing a control strategy of first adjusting the posture of the target object and then moving it, the target object is transferred in a suitable posture, improving the stability and reliability of the robotic arm in moving the target object.

[0018] This disclosure is not exhaustive, but merely illustrative of some embodiments, and is not intended to limit the scope of protection of this disclosure. Unless otherwise specified, each step in a particular embodiment can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a particular embodiment can also be implemented as an independent embodiment, and the order of the steps in a particular embodiment can be arbitrarily interchanged. Furthermore, the optional implementation methods in a particular embodiment can be arbitrarily combined; moreover, the embodiments can be arbitrarily combined, for example, some or all steps of different embodiments can be arbitrarily combined, and a particular embodiment can be arbitrarily combined with the optional implementation methods of other embodiments.

[0019] In each of the disclosed embodiments, unless otherwise specified or in case of logical conflict, the terminology and / or descriptions of the embodiments are consistent and can be referenced by each other. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0020] The terminology used in the embodiments of this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure.

[0021] In this embodiment of the disclosure, unless otherwise stated, elements expressed in the singular form, such as "a," "an," "the," "the aforementioned," "the," "this," etc., can mean "one and only one," or "one or more," "at least one," etc. For example, when using articles such as "a," "an," "the," etc. in translation, the noun following the article can be understood as either a singular or a plural expression.

[0022] In some embodiments, the terms “in response to…”, “in response to determining…”, “in the case of…”, “when…”, “if…”, “if…”, etc., can be used interchangeably.

[0023] In some embodiments, the terms “greater than,” “greater than or equal to,” “not less than,” “more than,” “more than or equal to,” “not less than,” “higher than,” “higher than or equal to,” “not lower than,” and “above” can be used interchangeably, as can the terms “less than,” “less than or equal to,” “not greater than,” “less than,” “less than or equal to,” “not more than,” “lower than,” “lower than or equal to,” “not higher than,” and “below”.

[0024] The prefixes such as "first" and "second" in the embodiments of this disclosure are only for distinguishing different descriptive objects and do not constitute restrictions on the position, order, priority, number or content of the descriptive objects. For the description of the descriptive objects, please refer to the description in the claims or the context of the embodiments. The use of prefixes should not constitute unnecessary restrictions.

[0025] In the embodiments disclosed herein, "multiple" refers to two or more.

[0026] In the embodiments disclosed herein, terms such as “import”, “input”, and “read in” can be used interchangeably.

[0027] In some embodiments, devices, etc., can be interpreted as physical or virtual, and their names are not limited to the names recorded in the embodiments. Terms such as “device”, “equipment”, “circuit”, “network element”, “node”, “function”, “unit”, “section”, “system”, “network”, “chip”, “chip system”, “entity”, and “subject” can be used interchangeably.

[0028] Figure 1 This is a flowchart illustrating a control method for a robotic arm provided in an embodiment of this disclosure. This method can be applied to application scenarios such as smart terminals, and can be executed by a robotic arm or a robot including a robotic arm, or by a terminal integrating robotic arm control functions or a processor within a terminal, or by other devices suitable for controlling the robotic arm; this disclosure does not limit the scope of the application. Figure 1 As shown, the control method of the robotic arm includes steps S101-S103.

[0029] Step S101: In response to the grasping task of the robotic arm, the posture parameters of the target object are obtained; the target object is located at the first target position, and the target object is obtained by analyzing the scene image including multiple objects.

[0030] In the embodiments of this disclosure, the target object can be an object to be grasped, an object to be moved, an object to be assembled, an object to be mated, etc. The type of the target object depends on the application scenario or the task to be performed, and this disclosure does not limit the type of the target object. Posture parameters include, but are not limited to, the tilt angle and rotation angle between any axis of the target object and the horizontal plane or between any axis of the target object and the vertical plane. The first target position can be a preset position or a position determined according to the grasping scenario. The robotic arm can be mounted on a robot, and the scene image can be an image obtained by the robotic arm's sensors from a scene containing multiple objects. For example, the sensor at the top of the robotic arm can collect an image of a scene containing multiple objects. The multiple objects can be objects of the same type, and the scene image can be an RGB image or a spatial data image. This disclosure does not limit the specific type of the target object, nor does it limit the specific position of the first target position. The grasping task can be triggered by issuing a grasping command to the robotic arm, or it can be automatically triggered by the robotic arm based on the analysis results of the scene image. This disclosure does not limit the triggering method of the grasping task. According to the grasping task, the robotic arm can grasp the target object in the first target position.

[0031] By using the gripping task of the robotic arm as the trigger condition for acquiring posture parameters, the deviation in posture parameter acquisition caused by failure to grip or unstable gripping is reduced, thereby improving the accuracy of posture parameter acquisition.

[0032] Step S102: Control the robotic arm to adjust the posture of the target object until the posture parameters meet the preset posture conditions.

[0033] In the embodiments of this disclosure, the preset attitude condition can be a pre-set attitude parameter threshold corresponding to the standard attitude that the target object needs to achieve. The embodiments of this disclosure do not limit the specific value of the attitude parameter threshold.

[0034] The preset posture conditions can be flexibly set according to the actual application scenario, and can be adjusted according to target objects of different shapes and different movement requirements, which improves the flexibility of the robot arm in adjusting the posture of the target object.

[0035] Step S103: Control the robotic arm to move the adjusted target object to the second target position.

[0036] In the embodiments of this disclosure, the second target position can be a pre-set final placement position of the target object, and this disclosure does not limit the specific location of the second target position.

[0037] By adjusting the attitude of the target object before moving it, the target object can maintain a stable attitude during the movement. This also ensures that the adjusted target object meets the requirements of subsequent processes, reducing positioning deviations caused by the target object's attitude not meeting process requirements during the movement.

[0038] According to the control method for the robotic arm proposed in this disclosure, the method includes: in response to the grasping task of the robotic arm, acquiring the posture parameters of a target object; the target object being located at a first target position, the target object being obtained based on scene image analysis including multiple objects; controlling the robotic arm to adjust the posture of the target object until the posture parameters meet preset posture conditions; and controlling the robotic arm to move the adjusted target object to a second target position. By first adjusting the posture of the target object and then moving it, the target object is transferred in a suitable posture, improving the stability and reliability of the robotic arm in moving the target object.

[0039] For example, when the robot arm is controlling the adjustment of the orientation of the target object until the orientation parameters meet the preset orientation conditions, the following methods can be used, but are not limited to: determining the tilt angle parameter in the orientation parameters; obtaining the reference tilt angle parameter corresponding to the tilt angle parameter; determining the tilt angle error of the target object based on the tilt angle parameter and the reference tilt angle parameter; and controlling the robot arm to adjust the orientation of the target object until the tilt angle error is less than the preset error threshold.

[0040] In the embodiments of this disclosure, the tilt angle parameter can reflect the tilt angle of the target object, and the reference tilt angle parameter can be a preset tilt angle parameter of the target object that meets the movement requirements. The embodiments of this disclosure do not limit the specific value of the preset tilt angle parameter. The embodiments of this disclosure also do not limit the specific value of the preset error threshold.

[0041] In an embodiment of this disclosure, an example is provided of a robotic arm adjusting the posture of a target object. Assuming the tilt angle parameter is 180 degrees, the reference tilt angle parameter is 90 degrees, and the preset error threshold is 5 degrees, the robotic arm is controlled to adjust the posture of the target object, adjusting the tilt angle parameter towards 90 degrees, until the tilt angle parameter is adjusted to any tilt angle between 85 degrees and 95 degrees.

[0042] By determining the tilt angle error based on the tilt angle parameters and reference tilt angle parameters, the difference between the current attitude and the target attitude of the target object is quantified into a numerical value, thereby improving the accuracy of target object attitude monitoring.

[0043] For example, when determining the tilt angle error of a target object based on tilt angle parameters and reference tilt angle parameters, it can be implemented in the following ways, but is not limited to: determining the visual tilt angle and tactile tilt angle in the tilt angle parameters; the visual tilt angle is obtained from the visual image analysis of the target object, and the tactile tilt angle is obtained from the tactile image analysis of the target object; determining the reference visual tilt angle and reference tactile tilt angle in the reference tilt angle parameters; determining the multi-tilt angle error of the target object based on the visual tilt angle, tactile tilt angle, reference visual tilt angle, and reference tactile tilt angle; the tilt angle error is a multi-tilt angle error, which is obtained from the error analysis between the visual tilt angle and the reference visual tilt angle and the error between the tactile tilt angle and the reference tactile tilt angle.

[0044] In embodiments of this disclosure, the visual image can be an image acquired by the visual sensor of the robotic arm of a target object. For example, the visual image can be acquired by the wrist sensor of the robotic arm. The tactile image can be a pressure image acquired by the tactile sensor of the robotic arm's fingers, which reflects the pressure distribution in the contact area between the target object's surface and the tactile sensor. The visual tilt angle can be the tilt angle of any axis of the target object relative to a reference plane, calculated from the visual image. The tactile tilt angle can be the tilt angle of the target object relative to a reference plane, calculated from the pressure distribution in the tactile image. The reference plane can be a horizontal plane or a vertical plane.

[0045] In the embodiments of this disclosure, the reference visual tilt angle can be a preset visual tilt angle target value, and the reference tactile tilt angle can be a preset tactile tilt angle target value. The embodiments of this disclosure do not limit the specific values ​​of the reference visual tilt angle and the reference tactile tilt angle.

[0046] In the embodiments of this disclosure, the multi-tilt error can be a sum of error indices obtained by fusing visual tilt and tactile tilt errors, which can be used to characterize the degree of deviation between the current posture of the target object and the reference posture.

[0047] By simultaneously utilizing both visual and tactile tilt angles for posture analysis, more posture information can be obtained compared to posture detection methods that rely on a single posture parameter, thereby improving the accuracy of posture detection.

[0048] For example, when determining the multi-tilt error of a target object based on visual tilt angle, tactile tilt angle, reference visual tilt angle, and reference tactile tilt angle, it can be implemented in the following ways, but is not limited to: determining a first weight corresponding to the visual tilt angle and a second weight corresponding to the tactile tilt angle; determining the multi-tilt error based on a first difference between the visual tilt angle and the reference visual tilt angle, a second difference between the tactile tilt angle and the reference tactile tilt angle, the first weight, and the second weight.

[0049] In embodiments of this disclosure, the multi-tilt angle error can be achieved using the following formula:

[0050] in, For multiple tilt angle errors, As the first weight, For reference visual tilt angle, For visual tilt angle, As the second weight, For reference tactile tilt angle, For tactile tilt angle.

[0051] By setting different first and second weights, the influence of the two detection dimensions on multi-tilt error can be adjusted according to the accuracy differences or timing requirements of visual tilt detection and tactile tilt detection, making the multi-tilt error adaptable to the actual application scenario and improving the accuracy of multi-tilt error calculation.

[0052] For example, when determining the first weight corresponding to the visual tilt angle and the second weight corresponding to the tactile tilt angle, the following methods can be used, but are not limited to: obtaining a first variance estimate of the visual tilt angle and a second variance estimate of the tactile tilt angle; the first variance estimate is obtained by analyzing the detection results of historical visual tilt angles, and the second variance estimate is obtained by analyzing the detection results of historical tactile tilt angles; and the first weight and the second weight are determined based on the first variance estimate and the second variance estimate.

[0053] In embodiments of this disclosure, the first weight and the second weight can be complementary.

[0054] In embodiments of this disclosure, the calculation of the first weight and the second weight can be achieved using the following formula:

[0055] in, As the first weight, This is the first variance estimate. This is the second variance estimate. It is the second weight.

[0056] The variance estimate reflects the dispersion of the detection results. The greater the dispersion, the lower the reliability of the sensor. By allocating weights based on the variance estimate, the weight allocation becomes more reasonable, thereby improving the accuracy of multi-tilt error calculation.

[0057] For example, when executing the control of the robotic arm to move the adjusted target object to the second target position, it can be implemented in the following ways, but not limited to: obtaining the first pose of the second target position relative to the first sensor of the robotic arm, the second pose of the first sensor relative to the base coordinate system of the robotic arm, and obtaining the third pose of the target object relative to the second sensor of the robotic arm; the base coordinate system is a coordinate system with the robotic arm as the reference, and the installation position of the first sensor is higher than the installation position of the second sensor; based on the first pose, the second pose, and the third pose, determining the amount of motion of the robotic arm to move the target object; and based on the amount of motion, controlling the robotic arm to move the adjusted target object to the second target position.

[0058] In the embodiments of this disclosure, the first pose, second pose, and third pose can be 6-dimensional poses, including three-dimensional translation and three-dimensional rotation. The first sensor can be mounted on the top of the robotic arm, and the second sensor can be mounted on the wrist of the robotic arm. However, it should be clarified that this is not intended to limit the mounting positions of the first and second sensors; it is only necessary that the mounting position of the first sensor is higher than that of the second sensor. This disclosure does not limit the specific mounting positions of the first and second sensors, and both the first and second sensors can be vision sensors. The XY plane of the base coordinate system can be parallel to the horizontal plane, the X-axis of the base coordinate system can point towards the positive direction of the robotic arm, the Y-axis of the base coordinate system can be along the line connecting the thumb and little finger of the robotic arm, and the Z-axis of the base coordinate system can be perpendicular to the ground and upwards. This disclosure does not limit the position of the origin of the base coordinate system.

[0059] The first pose reflects the positional relationship between the second target position and the first sensor, the second pose reflects the positional relationship between the first sensor and the base coordinate system, and the third pose reflects the positional relationship between the target object and the second sensor. Based on the first pose, the second pose, and the third pose, the spatial association between the second target position, the first sensor, the first sensor, and the target object can be realized, thereby improving the accuracy of motion determination.

[0060] For example, when executing the action of controlling the robotic arm to move the adjusted target object to the second target position based on the amount of motion, it can be implemented in the following ways, but not limited to: obtaining the real-time position of the target object during the movement process; determining the control force of the robotic arm based on the real-time position and the amount of motion; and controlling the robotic arm to move the adjusted target object to the second target position based on the control force and the amount of motion.

[0061] In the embodiments of this disclosure, the target object may include, but is not limited to, bolts, gears, brackets, castings and connectors. However, it should be clear that this statement is not intended to limit the specific type of the target object. The target object may also be other objects that can be grasped and moved by a robotic arm.

[0062] In the embodiments of this disclosure, the scenario of loading vehicle body bolts is used as an example. The target object is the vehicle body bolt, and the second target position can be a support column on a support platform. The vehicle chassis moves from top to bottom to the support platform via a translational guide rail. The holes on the vehicle chassis align with and enter the support column, which can rotate automatically, thereby completing the bolt connection of the vehicle chassis. However, it should be clarified that this statement is not intended to limit the method of the embodiments of this disclosure to the scenario of loading and unloading vehicle body bolts. It can also be applied to other scenarios that require object movement. The embodiments of this disclosure do not limit the application scenario of the control method of the robot arm.

[0063] In embodiments of this disclosure, the control force can be the magnitude of the pressure applied by the robotic arm to the target object during the movement of the target object.

[0064] In embodiments of this disclosure, the control force can be calculated using the following formula:

[0065] in, To control force, For stiffness coefficient, For reference position, For real-time location, The damping coefficient is... For reference purposes, This refers to the amount of exercise.

[0066] The control force is dynamically adjusted according to the real-time position and motion of the target object, so that the control force adapts to the actual movement process, making the robot arm move the target object more smoothly.

[0067] For example, the target object can be obtained from scene image analysis including multiple objects, and can be implemented in ways that are not limited to: obtaining a first orientation of the first target position in the scene image; determining a second orientation of each of the multiple objects in the scene image; and determining the target object from the multiple objects based on the similarity between the first orientation and the second orientation.

[0068] In embodiments of this disclosure, the first orientation can be the orientation of the first target location relative to the center point of the scene image, and the second orientation can be the orientation of each of the multiple objects relative to the center point of the scene image. This disclosure does not limit the method for calculating the similarity between the first and second orientations. The object in the second orientation corresponding to the maximum similarity can be determined as the target object. The greater the similarity, the more similar the second orientation is to the first orientation, and the greater the probability that the object in the second orientation is placed at the first target location.

[0069] By calculating the similarity between the orientation of each of the multiple objects and the orientation of the first target location, the quantitative selection of the target object is achieved. The similarity converts the spatial proximity into a comparable value, so that the selection result reflects the spatial relationship between the object and the first target location.

[0070] For example, the robotic arm can grasp a target object in the following ways, but is not limited to: acquiring scene space data of a stacked object group collected by the first sensor of the robotic arm; the stacked object group is a group of multiple objects stacked together before the target object is moved to the first target position; determining the object to be moved in the stacked object group based on the scene space data; controlling the robotic arm to move the object to be moved to the first target position; the target object is the object to be moved in the first target position; and controlling the robotic arm to grasp the target object in response to the grasping task of the robotic arm.

[0071] Scene spatial data can be scene point clouds, but it should be clarified that this statement is not intended to limit scene spatial data to scene point clouds; it can also be other data that can reflect the spatial characteristics of objects. Scene spatial data reflects the three-dimensional geometric information of stacked object groups. Compared with two-dimensional images, scene spatial data contains the spatial depth relationship between objects, can distinguish the layers of stacked objects, and improves the recognition rate of stacked objects.

[0072] For example, when performing the task of determining the object to be moved in a stacked object group based on scene spatial data, the following methods may be used, but are not limited to: identifying the spatial data set of each object from the stacked object group in which identifiable spatial data exists in the scene spatial data; determining multiple candidate objects in the stacked object group based on the spatial data set of each object; and determining the object to be moved from the multiple candidate objects.

[0073] A spatial dataset can be any set of spatial data, but it should be clarified that this statement is not intended to limit it to only spatial datasets. It can also be any set of data that reflects the spatial characteristics of an object. A spatial dataset includes multiple spatial data features of the object, such as multiple point clouds. By identifying the spatial dataset of each object in the scene's spatial data that contains identifiable spatial data, the separation and filtering of objects in a stacked object group can be achieved. Spatial data reflects the geometric information and spatial position of objects. Identifying objects based on spatial data can improve the accuracy of object recognition.

[0074] For example, when performing the task of determining multiple candidate objects in a stacked object group based on the spatial data set of each object, the following methods may be used, but are not limited to: filtering the spatial data set of each object based on the number of spatial data in the spatial data set to obtain a filtered spatial data set; obtaining the target depth value of the filtered spatial data set; and determining multiple candidate objects from the objects corresponding to the filtered spatial data set based on the target depth value.

[0075] In the embodiments of this disclosure, a preset quantity threshold can be set to filter out spatial data sets whose quantity is greater than or equal to the preset quantity threshold. The specific value of the preset quantity threshold is not limited in the embodiments of this disclosure. Through dual filtering of spatial data quantity and depth value, objects with incomplete spatial data or whose depth does not meet the requirements can be excluded, improving the efficiency of identifying multiple candidate objects.

[0076] For example, when performing the acquisition of the target depth value of the filtered spatial data set, it can be implemented in the following ways, but not limited to: acquiring the initial depth value of the filtered spatial data set collected by the first sensor, and acquiring the second pose of the first sensor relative to the base coordinate system of the manipulator; the base coordinate system is a coordinate system based on the manipulator; and adjusting the initial depth value according to the second pose to obtain the target depth value.

[0077] In the embodiments of this disclosure, since the first sensor and the stacked object group are obliquely aligned, the depth value stored is also obliquely aligned, which cannot reflect the true vertical relationship of each object in the stacked object group. The Y-axis of the base coordinate system is parallel to the vertical plane. Based on the second pose of the first sensor relative to the base coordinate system, the initial depth value can be adjusted to the target depth value in the vertical direction, so that the target depth value better reflects the true vertical relationship of each object in the stacked object group.

[0078] For example, when determining the object to be moved from multiple candidate objects, the following methods can be used, but are not limited to: obtaining the simulated grasping pose of the robot arm simulating grasping each candidate object, and obtaining the number of collision points between the robot arm model in the simulated grasping pose and the scene space data; and determining the object to be moved based on the simulated grasping pose and the number of collision points.

[0079] In the embodiments of this disclosure, the simulated grasping pose can be the pose obtained by the robotic arm simulating a grasping operation on a candidate object in a simulation environment. The robotic arm model can be a digital model of the robotic arm, which describes the geometry of the robotic arm in the form of a mesh. The number of collision points can be the number of points in space where the robotic arm model and the scene space data overlap.

[0080] By introducing simulated grasping postures for virtual grasping verification, no physical movement of the robotic arm is required, reducing the risk of collision or grasping failure due to improper grasping posture and lowering the risk of damage to the robotic arm.

[0081] For example, when performing the simulation of obtaining the gripping pose of the robot arm to grasp each candidate object, it can be implemented in the following ways, but not limited to: determining the target placement state of each candidate object; the target placement state is a vertical placement state or a horizontal placement state; in response to the target placement state being a vertical placement state, randomly selecting a first preset number of simulated gripping poses around the axis of the candidate object in the vertical placement state.

[0082] In the embodiments of this disclosure, the axis can be the principal axis of symmetry or the length direction of the object, and the specific value of the first preset quantity is not limited in the embodiments of this disclosure. The grasping methods for vertically placed objects and horizontally placed objects differ. Distinguishing the placement state of an object can improve the matching degree between the simulated grasping pose and the actual geometric conditions. By distinguishing between vertical and horizontal placement states, differentiated grasping strategies for different object postures are achieved.

[0083] For example, when performing the simulation of obtaining the gripping pose of the robotic arm to grasp each candidate object, it can be implemented in the following ways, but is not limited to: in response to the target placement state being a horizontal placement state, obtaining the length of the candidate object in the horizontal placement state, the finger width of the robotic arm in the gripping state, and obtaining a second preset number of gripping angles of the candidate object in the horizontal placement state; determining the number of gripping points along the axis of the candidate object in the horizontal placement state based on the length and finger width; determining a target number of simulated gripping poses of the candidate object in the horizontal placement state based on the gripping points and gripping angles; the target number is obtained by analyzing the number of gripping points and the second preset number.

[0084] In the embodiments of this disclosure, the specific value of the second preset quantity is not limited.

[0085] In embodiments of this disclosure, the number of grab points can be calculated according to the following formula:

[0086] in, This indicates rounding down to zero. To the number of points to be captured, The length of the candidate object, The width is the width of a finger.

[0087] By calculating the number of gripping points based on the object's length and the finger's width, an adaptive distribution of gripping positions is achieved. The adaptive mechanism matches the gripping point layout with the object's size, improving gripping stability and flexibility.

[0088] For example, when performing the process of obtaining the number of collision points between the robotic arm model and the scene space data in the simulated grasping pose, the following methods can be used, but are not limited to: obtaining the minimum and maximum coordinates of the minimum bounding box of the candidate object in the object coordinate system of the candidate object, and obtaining the magnification factor of the minimum bounding box; determining the first magnified coordinate of the minimum coordinate and the second magnified coordinate of the maximum coordinate based on the magnification factor; filtering the scene space data based on the first and second magnified coordinates to obtain the filtered scene space data; and determining the number of collision points based on the robotic arm model and the filtered scene space data.

[0089] In the embodiments of this disclosure, the minimum bounding box can be the minimum rectangular bounding box, but it should be clear that this statement is not intended to limit the shape of the minimum bounding box to only rectangles, but can also be other shapes.

[0090] The first and second magnified coordinates can be calculated using the following formulas:

[0091] in, As the first magnified coordinate, The coordinates of the center point of the minimum bounding box. This is the magnification factor. For the maximum coordinates, For the minimum coordinates, This is the second magnified coordinate.

[0092] By introducing a minimum bounding box and a magnification factor, the collision detection area is adaptively adjusted. The screening area generated by combining the minimum bounding box and the magnification factor includes both the object itself and the safety space required for the robot arm operation.

[0093] For example, when performing the filtering of scene space data based on the first magnified coordinates and the second magnified coordinates to obtain filtered scene space data, the following methods can be used, but are not limited to: obtaining the fourth pose of the first sensor relative to the candidate object; converting the scene space data from the first coordinate system of the first sensor to the object coordinate system of the candidate object based on the fourth pose; and filtering the converted scene space data based on the first magnified coordinates and the second magnified coordinates to obtain filtered scene space data.

[0094] In the embodiments of this disclosure, the first coordinate system can be a coordinate system established based on the first sensor, and the specific method of establishing the first coordinate system is not limited in the embodiments of this disclosure. The Y-axis of the object coordinate system can be the direction of the major axis of the object, the X-axis of the object coordinate system is perpendicular to the Y-axis of the object coordinate system and parallel to the world coordinate system, and the Z-axis of the object coordinate system is obtained by the cross product of the X-axis and Y-axis of the object coordinate system.

[0095] In embodiments of this disclosure, the transformation of scene spatial data from a first coordinate system to the object coordinate system of the candidate object can be achieved using the following formula:

[0096] in, For the converted scene space data, The scene space data before conversion, Let be the rotation matrix of the first sensor relative to the object's coordinate system. Let be the translation vector of the first sensor relative to the object's coordinate system.

[0097] The fourth pose can be represented as follows:

[0098] in, This is the fourth pose. Let be the rotation matrix of the first sensor relative to the object's coordinate system. Let be the translation vector of the first sensor relative to the object's coordinate system. The pose of the candidate object relative to the first sensor The inverse matrix.

[0099] By introducing a fourth pose for coordinate system transformation, the unification of scene spatial data and object local space is achieved.

[0100] For example, when performing the determination of the first magnified coordinates of the minimum coordinates and the second magnified coordinates of the maximum coordinates based on the magnification factor, the following methods can be used, but are not limited to: determining the center point coordinates of the minimum rectangular bounding box based on the minimum coordinates and the maximum coordinates; and determining the first magnified coordinates and the second magnified coordinates based on the center point coordinates, the minimum coordinates, the maximum coordinates, and the magnification factor.

[0101] In the embodiments of this disclosure, the coordinates of the center point can be calculated according to the following formula:

[0102] in, The coordinates of the center point, For the minimum coordinates, The maximum coordinates.

[0103] The first and second magnification coordinates are determined based on the center point coordinates, minimum coordinates, maximum coordinates and magnification factor, which realizes symmetrical magnification with the center point as the reference, so that the bounding box is magnified in all directions at the same ratio, and the shape ratio and center position of the bounding box remain unchanged.

[0104] For example, when determining the number of collision points based on the robot model and the filtered scene space data, the following methods can be used, but are not limited to: obtaining the fifth pose of the gripping coordinate system of the simulated gripping pose relative to the second coordinate system of the second sensor of the robot; the installation position of the first sensor is higher than the installation position of the second sensor; transforming the robot model into the second coordinate system according to the fifth pose; and determining the number of collision points based on the transformed robot model and the filtered scene space data.

[0105] In the embodiments of this disclosure, the second coordinate system can be a coordinate system established based on the second sensor, and the specific method of establishing the second coordinate system is not limited in the embodiments of this disclosure. The X-axis of the grasping coordinate system can be the point from the tip of the thumb of the robotic hand to the midpoint between the index and middle fingers, the Y-axis can be a vector parallel to the point from the tip of the middle finger to the tip of the index finger, and the distances of the thumb, middle finger, and index finger from the Y-axis are equal, and the Z-axis is obtained by the cross product of the X-axis and the Y-axis. Obtaining the fifth pose and performing coordinate transformation on the robotic hand model makes the coordinate reference of the robotic hand model consistent with the filtered scene space data, thereby improving the accuracy of collision detection.

[0106] For example, when determining the object to be moved based on the simulated grasping pose and the number of collision points, the following methods can be used, but are not limited to: in response to the minimum number of collision points being one, determining the candidate object of the simulated grasping pose corresponding to the minimum value as the object to be moved; or, in response to the minimum number of collision points being multiple, determining the candidate object with the largest component among the simulated grasping poses corresponding to multiple minimum values ​​as the object to be moved; any component is the pose of the simulated grasping pose on any axis in the grasping coordinate system of the simulated grasping pose.

[0107] By selecting the pose with the fewest collision points, the robotic arm minimizes contact with environmental objects, reducing the risk of collision.

[0108] For example, when performing the acquisition of the grasping coordinate system for the simulated grasping pose, it can be implemented in the following ways, but not limited to: acquiring the third coordinate system of the robot and acquiring the chessboard coordinate system; and determining the grasping coordinate system based on the third coordinate system and the chessboard coordinate system.

[0109] In the embodiments of this disclosure, the checkerboard coordinate system can be a coordinate system established based on a preset checkerboard calibration plate. The robotic arm can include a hand area and a robotic arm area, and the third coordinate system can be a coordinate system established based on the hand area of ​​the robotic arm. The embodiments of this disclosure do not limit the construction method of the third coordinate system. Using the checkerboard coordinate system as a calibration reference can reduce the reference deviation of the robotic arm itself, ensure the accuracy of the establishment of the robotic arm coordinate system, and thus improve the accuracy of the establishment of the grasping coordinate system.

[0110] In embodiments of this disclosure, based on Figure 1 The embodiment shown, Figure 2 This is a flowchart illustrating a control method for a robotic arm provided in this embodiment. This embodiment provides an example of acquiring scene space data of a stacked object group collected by a first sensor of the robotic arm. The first sensor is used to collect scene point clouds, which are then segmented and their poses estimated to obtain the 6D poses of m candidate bolts (i.e., the scene space data of the stacked object group). This embodiment provides an example of determining the object to be moved in the stacked object group based on the scene space data. Based on the scene point cloud, the robotic arm's 3D mesh, and the poses of the candidate bolts, the target bolt is determined. This embodiment provides an example of controlling the robotic arm to move the object to be moved to a first target position. The graspable pose of the target bolt is determined, and the robotic arm is controlled to grasp the target bolt based on the grasping pose coordinate system calibration results. After successful grasping, the grasped bolt (hereinafter referred to as the loading bolt) is placed in a preset position A.

[0111] In embodiments of this disclosure, examples are provided for controlling a robotic arm to grasp a target object in response to a grasping task. A first sensor is used to acquire scene images and point clouds, and the 6D pose of the bolt to be loaded is obtained. The robotic arm is then controlled to pinch the tail end of the target bolt with two fingers and lift it in place. Embodiments of this disclosure also provide examples for determining the tilt angle parameter in the posture parameters and obtaining the corresponding reference tilt angle parameter. A second sensor is used to acquire a target visual image including the bolt to be loaded, and the axial tilt angle of the bolt to be loaded in the target visual image is determined. A tactile sensor is used to acquire the current target tactile image, and the indentation tilt angle of the bolt to be loaded due to contact and compression with the tactile sensor in the target tactile image is determined. Furthermore, embodiments of this disclosure provide examples for determining the tilt angle error of a target object based on the tilt angle parameter and the reference tilt angle parameter. The positional error between the bolt to be loaded and the desired alignment position is determined based on the bolt's axial tilt angle and indentation tilt angle, as well as the reference axial tilt angle and the reference indentation tilt angle.

[0112] In embodiments of this disclosure, an example is provided whereby a robotic arm is controlled to adjust the posture of a target object until the tilt angle error is less than a preset error threshold. This involves acquiring a first pose of the second target position relative to a first sensor of the robotic arm, a second pose of the first sensor relative to the base coordinate system of the robotic arm, and a third pose of the target object relative to a second sensor of the robotic arm. Based on the first, second, and third poses, the amount of motion of the robotic arm moving the target object is determined. When the position error is greater than or equal to a preset threshold, the control amount (i.e., the amount of motion) of the robotic arm is determined based on the position error and a preset mapping relationship. This disclosure also provides an example of controlling the robotic arm to move the adjusted target object to a second target position based on the amount of motion. The robotic arm is controlled to move, driving the second sensor and tactile sensor to acquire current visual and tactile images. It is determined whether the bolt to be loaded is held in the robotic arm's hand. When the position error is less than the preset threshold, the robotic arm, holding the bolt, moves to the support platform and places the bolt onto the support column.

[0113] In the embodiments of this disclosure, based on bolt size parameters and attitude estimation results, this disclosure proposes a grasping feasibility judgment method based on spatial data intersection criteria. By judging the spatial overlap relationship between the closed mesh of the manipulator and the scene spatial data, the graspability of each candidate pose is accurately determined. Based on the bolt size information and the poses of m bolts returned by the pose estimation network, a series of grasping poses are sampled. Then, based on the scene spatial data and the 3D mesh of the manipulator, the m candidate bolts are analyzed one by one. Using the spatial data intersection principle, the final graspable target bolt and its corresponding graspable pose are determined, thereby controlling the manipulator to grasp the target bolt. By judging whether the clipped scene spatial data is located inside the closed manipulator model, grasping feasibility judgment based on geometric intersection is achieved. Compared with traditional methods based on planar normals or depth projections, this method can accurately handle the graspability of non-planar and irregular bolts.

[0114] In embodiments of this disclosure, based on Figure 2 The embodiment shown, Figure 3 This is a flowchart of a control method for a robotic arm provided in an embodiment of this disclosure. This embodiment provides an example of obtaining the simulated grasping pose of the robotic arm when simulating the grasping of each candidate object, and obtaining the number of collision points between the robotic arm model in the simulated grasping pose and the scene space data. The method involves reading the scene point cloud and the 3D mesh of the robotic arm; for the first candidate bolt, accurately cropping the point cloud to be detected in the object coordinate system based on the bolt pose and the minimum bounding box of the bolt; sampling n grasping poses based on the bolt pose and the minimum bounding box of the bolt; sequentially transforming the robotic arm model to the first coordinate system according to the n grasping poses to obtain n robotic arm models in the first coordinate system; sequentially determining whether any points in the point cloud to be detected fall inside the robotic arm model in the first coordinate system, and recording the points falling inside the robotic arm mesh as collision points; returning n sets of collision point counts for the n grasping poses; repeating the above operations for the 2nd to mth candidate bolts to obtain m... n candidate grasp poses, and m The number of collision points is n. This disclosure provides an example of determining the object to be moved based on the simulated grasping pose and the number of collision points, based on m... n candidate grasping poses and m Given n sets of collision points, a unique target bolt is determined. This embodiment provides an example of controlling a robotic arm to grasp a target object, determining the grasping pose of the unique target bolt; based on the graspable pose, controlling the robotic arm to move with a predefined grasping gesture to the target bolt; controlling the robotic arm to perform a closing action to complete the grasping of the target bolt. The following input data is obtained: scene space data, where each point is represented in a first coordinate system; the minimum and maximum coordinates of the minimum bounding box of the target bolt in its own coordinate system; and a closed 3D mesh model of the robotic arm. The scene space data is transformed from the first coordinate system to the bolt coordinate system. The minimum bounding box of the bolt is then used for the transformation. Calculate the first magnified coordinates of the minimum bounding box of the enlarged bolt. Second magnified coordinates For each point in the scene spatial data Determine whether it satisfies: If the conditions are met, the point is considered a candidate point; otherwise, it is discarded. The final set of clipped points is obtained. Unlike spatial data filtering based on 2D projection or depth thresholds, this embodiment directly determines whether a point is within the minimum bounding box of the expanding bolt in 3D space, avoiding false filtering caused by pose changes and improving clipping accuracy.

[0115] In the embodiments of this disclosure, the sampling grasping pose is generated by combining sampling around the bolt axis with equidistant sampling along the axis, forming a uniformly distributed candidate set to ensure coverage of grasping possibilities under different orientations. Specifically, the origin of the bolt pose coordinate system returned by the pose estimation network is located at the geometric center of the bolt, and the Y-axis points along the bolt axis towards the bolt head. However, since the bolt is a rotationally symmetric body, the X and Z axes of the bolt coordinate system can be any direction that rotates around the bolt axis and is perpendicular to the axis. To ensure the uniqueness of the pose, this invention limits the X-axis of the bolt coordinate system to be parallel to the world coordinate system, and the Z-axis is obtained by the cross product of the X and Y axes. Figure 3 The embodiment shown, Figure 4 This diagram illustrates a grasping coordinate system provided in an embodiment of this disclosure. The grasping gesture is defined as a three-finger grasp, with the thumb positioned between the index and middle fingers, and the fingertips of all three fingers coplanar. The grasping coordinate system is defined as follows: the X-axis points from the tip of the thumb to the midpoint between the index and middle fingers; the Y-axis is parallel to the vector pointing from the tip of the middle finger to the tip of the index finger, and the thumb, middle, and index fingers are equidistant from the Y-axis; the Z-axis is obtained by the cross product of the X-axis and Y-axis. Figure 4 The red arrow in the image can represent the X-axis. Figure 4 The green arrow in the image represents the Y-axis. Figure 4 The blue arrow in the image represents the Z-axis.

[0116] In embodiments of this disclosure, the bolt posture is determined as follows: In the base coordinate system of the robot arm, based on the angle between the bolt's Y-axis unit vector and the XY plane, it is determined whether the bolt is lying down or standing upright. Figure 4 The embodiment shown, Figure 5 This diagram illustrates one gripping method provided in this disclosure. If the bolt is upright, the finger descends along the bolt's axis (Y-axis) and grips it. The gripping point is set at the end of the bolt's upward-facing side (e.g., gripping the tail end if the tail is upward, gripping the head if the head is upward). n candidate gripping poses are sampled around the bolt's axis, with the origin located on the bolt's axis at a certain distance from the end. Based on... Figure 4 The embodiment shown, Figure 6 This diagram illustrates a gripping method provided in this disclosure. If the object is lying down, the robotic arm grips horizontally, meaning the Y-axis of the gripping coordinate system coincides with the Y-axis of the bolt, and the X-axis of the gripping coordinate system coincides with the X-axis of the bolt. The origin of the gripping pose coordinate system is located on the bolt's axis. By sampling n1 gripping points at equal intervals along the axis, and at each gripping point, sampling n2 gripping angles around the X-axis, n=n1 is determined. n2 candidate capture poses.

[0117] In the bolt coordinate system, the X and Z components of the origin coordinates of the n grabbing pose coordinate systems are 0, and the set of values ​​for the Y component is as follows:

[0118] in, This indicates rounding down to zero. The length of the candidate object, n is the width of the finger, and n1 is the number of gripping points in the gripping pose.

[0119] This disclosure proposes a third coordinate system calibration method based on a chessboard grid. According to the geometric constraint relationship, the following equation can be obtained:

[0120] The calibration results are as follows:

[0121] in, Let be the pose matrix of the second coordinate system relative to the base coordinate system. Let be the pose matrix of the chessboard coordinate system relative to the first coordinate system. Let be the pose matrix of the third coordinate system relative to the chessboard coordinate system. Let be the pose matrix of the robot's end effector coordinate system relative to the checkerboard coordinate system. is the pose matrix of the third coordinate system relative to the coordinate system of the robot's end effector.

[0122] With the calibration results, the control quantity can be obtained according to the following formula. The robotic arm is controlled to reach the target bolt using a predefined grasping gesture.

[0123]

[0124] in, Let be the pose matrix of the third coordinate system relative to the grasping coordinate system. To capture the pose matrix of the coordinate system relative to the first coordinate system.

[0125] Third-coordinate system calibration is effective not only in the disordered grasping stage but also in the subsequent bolt tail-end grasping stage for attitude adjustment, and is therefore crucial. Attitude adjustment requires grasping the tail end, demanding higher grasping accuracy compared to the disordered grasping stage. This calibration method can be extended to various robot arm structures, possessing universality and reconfigurability. The third-coordinate system calibration method improves grasping accuracy and success rate. This calibration method is universal and applicable to different robot arm structures and multi-finger grasping modes, providing accurate fingertip pose references for subsequent attitude adjustment. The robot arm's joints are controlled to rotate to a predefined angle, executing a closing motion to complete the grasping of the target bolt. After grasping the bolt to be loaded, it is placed at the preset position A.

[0126] In embodiments of this disclosure, based on Figure 2 The embodiment shown, Figure 7 This is a flowchart illustrating a control method for a robotic arm provided in an embodiment of this disclosure. The disclosure provides an example of acquiring a simulated grasping pose of the robotic arm in simulating the grasping of a target object. A first sensor is used to capture an image to be processed and a point cloud to be processed, including a bolt to be loaded. Based on a visual detection model, a sub-region of the bolt to be loaded in the image to be processed is determined. The sub-region point cloud is then processed to obtain the 6D pose of the bolt to be loaded. This disclosure also provides an example of controlling a robotic arm to grasp a target object based on its simulated grasping pose. Based on the 6D pose and bolt size information of the bolt, the robotic arm is controlled to grasp the tail end of the bolt using a two-finger pinch gesture. The robotic arm is then controlled to raise the grasped bolt in its original position.

[0127] In the embodiments of this disclosure, a posture detection method based on vision-tactile fusion is used, where the visual channel is responsible for detecting the bolt axis angle and the tactile channel is used for extracting the indentation angle, achieving multimodal complementary perception. A second sensor is used to capture a target visual image including the bolt to be loaded, and the axis tilt angle of the bolt in the target visual image is determined. Specifically, a rotation target detection network is used to return a rotation bounding box of the bolt, and the tilt angle of this bounding box is calculated as the bolt axis tilt angle, with a detection accuracy of ±2 degrees. Alternatively, the bolt axis can be extracted using traditional image processing algorithms to determine the bolt tilt angle in the image. Simultaneously, a tactile sensor is used to acquire the current target tactile image, and the indentation angle of the bolt to be loaded, caused by contact and compression with the tactile sensor, is determined.

[0128] In the embodiments of this disclosure, a tactile image acquired using a tactile sensor shows the thread indentation. Using an image processing algorithm, the inclination angle of the thread in the image can be obtained. Other tactile sensors, such as piezoresistive tactile sensors, can also acquire thread images and calculate the indentation angle. The positional error between the bolt to be loaded and the desired alignment position is determined based on the bolt's axial inclination angle and indentation angle, as well as reference axial inclination angle and reference indentation angle. The reference axial inclination angle and reference indentation angle are the angles of the axial inclination angle in the visual image and the indentation angle in the tactile image when the bolt to be loaded is held vertically downwards by two fingers of a robotic arm.

[0129] When the position error is greater than or equal to a preset threshold, the closing angle of the mechanical finger is adjusted based on the multi-tilt error using a closed-loop control algorithm (such as Proportional-Integral-Derivative (PID) control). :

[0130] in, For multiple tilt angle errors, This is the proportionality coefficient. The integral coefficient is... is the differential coefficient.

[0131] The robotic arm is controlled to adjust its posture until the vertical direction of the bolt's main axis aligns with the direction of gravity (position error less than a preset threshold) or the bolt falling condition is triggered (visual and tactile detection results are empty). Through repeated multimodal closed-loop adjustment control, the multi-angle errors gradually converge to within the preset error threshold, ensuring the bolt's verticality and stability. When the multi-angle errors converge to the preset error threshold, the straightening is considered complete. This embodiment can achieve automatic bolt straightening without external fixture assistance. The second sensor and tactile sensor are driven to acquire current visual and tactile images to determine whether the bolt to be loaded is held in the robotic arm's hand. If both visual and tactile detections show an empty bolt, a re-grabbing logic is triggered, enabling the system to have self-recovery capabilities and ensuring continuous and stable operation during the posture adjustment phase.

[0132] based on Figure 2 The embodiment shown, Figure 8 This is a flowchart of a control method for a robotic arm provided in an embodiment of this disclosure. The disclosure provides methods for obtaining a first pose of the second target position relative to a first sensor of the robotic arm, a second pose of the first sensor relative to the base coordinate system of the robotic arm, and a third pose of the target object relative to the second sensor of the robotic arm. Based on the first, second, and third poses, the motion amount of the robotic arm moving the target object is determined. Based on the motion amount, the robotic arm is controlled to move the adjusted target object to the second target position. For example, the robotic arm is controlled to move to a vehicle body support platform. An image of the vehicle body support platform is acquired using the first sensor, and the relative pose 1 of the vehicle body support platform relative to the first sensor is determined based on the size parameters of the checkerboard pattern. An image and point cloud containing the bolt to be loaded are acquired using the second sensor. Visual processing is used to obtain the relative pose 2 of the bolt to be loaded relative to the second sensor. Based on the relative pose 1, relative pose 2, the relative pose of the first sensor relative to the base coordinate system, and the extrinsic parameters of the first sensor, the motion amount of the robotic arm is determined. A force-position hybrid control algorithm is used to control the robotic arm to place the bolt to be loaded onto the bolt support column.

[0133] In the embodiments of this disclosure, a scenario example of robotic arm control is provided. The robotic arm picks up a bolt on a bolt platform, straightens the bolt at a preset position and picks it up again, moves the bolt picked up by the robotic arm to the vehicle body support platform, and installs the bolt onto the support column, thereby realizing automated bolt loading.

[0134] Corresponding to the control method for the robotic arm described above, this invention also proposes a control device for the robotic arm. Since the device embodiments of this invention correspond to the method embodiments described above, details not disclosed in the device embodiments can be referred to in the method embodiments described above, and will not be repeated here.

[0135] Figure 9This is a schematic diagram of the structure of a control device for a robotic arm provided in an embodiment of the present disclosure. The control device for the robotic arm includes: The acquisition unit 21 is used to acquire the posture parameters of the target object in response to the grasping task of the robotic arm; the target object is located at the first target position, and the target object is obtained by analyzing a scene image including multiple objects; The first control unit 22 is used to control the robot arm to adjust the posture of the target object until the posture parameters meet the preset posture conditions. The second control unit 23 is used to control the robotic arm to move the adjusted target object to the second target position.

[0136] According to the control device for the robotic arm disclosed herein, the device includes: acquiring the posture parameters of a target object in response to the robotic arm's grasping task; the target object being located at a first target position, the target object being obtained based on scene image analysis including multiple objects; controlling the robotic arm to adjust the posture of the target object until the posture parameters meet preset posture conditions; and controlling the robotic arm to move the adjusted target object to a second target position. This control strategy of adjusting the target object's posture before moving it ensures that the target object is transferred in a suitable posture, improving the stability and reliability of the robotic arm's movement of the target object.

[0137] In one possible implementation of this disclosure, the first control unit 22 is further configured to: Determine the tilt angle parameter in the attitude parameters; Obtain the reference tilt angle parameter corresponding to the tilt angle parameter; Determine the tilt angle error of the target object based on the tilt angle parameters and the reference tilt angle parameters; The robot arm is controlled to adjust the posture of the target object until the tilt angle error is less than the preset error threshold.

[0138] By determining the tilt angle error based on the tilt angle parameters and reference tilt angle parameters, the difference between the current attitude and the target attitude of the target object is quantified into a numerical value, thereby improving the accuracy of target object attitude monitoring.

[0139] In one possible implementation of this disclosure, the first control unit 22 is further configured to: Determine the visual tilt angle and tactile tilt angle in the tilt angle parameters; the visual tilt angle is obtained from the visual image analysis of the target object, and the tactile tilt angle is obtained from the tactile image analysis of the target object; Determine the reference visual tilt angle and reference tactile tilt angle in the reference tilt angle parameters; Based on the visual tilt angle, tactile tilt angle, reference visual tilt angle, and reference tactile tilt angle, the multi-tilt angle error of the target object is determined; the tilt angle error is the multi-tilt angle error, which is obtained by analyzing the error between the visual tilt angle and the reference visual tilt angle, and the error between the tactile tilt angle and the reference tactile tilt angle.

[0140] By simultaneously utilizing both visual and tactile tilt angles for posture analysis, more posture information can be obtained compared to posture detection methods that rely on a single posture parameter, thereby improving the accuracy of posture detection.

[0141] In one possible implementation of this disclosure, the first control unit 22 is further configured to: Determine the first weight corresponding to the visual tilt angle and the second weight corresponding to the tactile tilt angle; The multi-tilt error is determined based on the first difference between the visual tilt angle and the reference visual tilt angle, the second difference between the tactile tilt angle and the reference tactile tilt angle, the first weight, and the second weight.

[0142] By setting different first and second weights, the influence of the two detection dimensions on multi-tilt error can be adjusted according to the accuracy differences or timing requirements of visual tilt detection and tactile tilt detection, making the multi-tilt error adaptable to the actual application scenario and improving the accuracy of multi-tilt error calculation.

[0143] In one possible implementation of this disclosure, the first control unit 22 is further configured to: Obtain the first variance estimate of the visual tilt angle and the second variance estimate of the tactile tilt angle; the first variance estimate is obtained by analyzing the detection results of the historical visual tilt angle, and the second variance estimate is obtained by analyzing the detection results of the historical tactile tilt angle. The first weight and the second weight are determined based on the first variance estimate and the second variance estimate.

[0144] The variance estimate reflects the dispersion of the detection results. The greater the dispersion, the lower the reliability of the sensor. By allocating weights based on the variance estimate, the weight allocation becomes more reasonable, thereby improving the accuracy of multi-tilt error calculation.

[0145] In one possible implementation of this disclosure, the second control unit 23 is further configured to: The system acquires the first pose of the second target relative to the first sensor of the robot arm, the second pose of the first sensor relative to the base coordinate system of the robot arm, and the third pose of the target object relative to the second sensor of the robot arm; the base coordinate system is a coordinate system based on the robot arm, and the installation position of the first sensor is higher than the installation position of the second sensor. Based on the first pose, second pose, and third pose, determine the amount of motion of the robot arm to move the target object; Based on the amount of motion, the robotic arm is controlled to move the adjusted target object to the second target position.

[0146] The first pose reflects the positional relationship between the second target position and the first sensor, the second pose reflects the positional relationship between the first sensor and the base coordinate system, and the third pose reflects the positional relationship between the target object and the second sensor. Based on the first pose, the second pose, and the third pose, the spatial association between the second target position, the first sensor, the second sensor, and the target object can be realized, thereby improving the accuracy of motion determination.

[0147] In one possible implementation of this disclosure, the second control unit 23 is further configured to: Obtain the real-time position of the target object during its movement; The control force of the robotic arm is determined based on its real-time position and motion. Based on the control force and the amount of motion, the control robot moves the adjusted target object to the second target position.

[0148] The control force is dynamically adjusted according to the real-time position and motion of the target object, so that the control force adapts to the actual movement process, making the robot arm move the target object more smoothly.

[0149] In one possible implementation of the embodiments of this disclosure, such as Figure 10 As shown, the device also includes: The acquisition unit 21 is also used to acquire the first position of the first target in the scene image; Determining unit 24 is used to determine the second position of each of the multiple objects in the scene image; The determining unit 24 is also used to determine the target object from multiple objects based on the similarity between the first orientation and the second orientation.

[0150] By calculating the similarity between the orientation of each of the multiple objects and the orientation of the first target location, the quantitative selection of the target object is achieved. The similarity converts the spatial proximity into a comparable value, so that the selection result reflects the spatial relationship between the object and the first target location.

[0151] In some embodiments, the apparatus further includes: The acquisition unit 21 is also used to acquire scene space data of the stacked object group collected by the first sensor of the robot arm; the stacked object group is a group of objects stacked together before the target object moves to the first target position; The determining unit 24 is also used to determine the object to be moved in the stacked object group based on the scene space data; The second control unit 23 is also used to control the robotic arm to move the object to be moved to the first target position; the target object is the object to be moved in the first target position. The second control unit 23 is also used to control the robotic arm to grasp the target object in response to the robotic arm's grasping task.

[0152] Scene spatial data reflects the three-dimensional geometric information of stacked object groups. Compared with two-dimensional images, scene spatial data contains the spatial depth relationship between objects, can distinguish the layers of stacked objects, and improves the recognition rate of stacked objects.

[0153] In one possible implementation of this disclosure, the determining unit 24 is further configured to: From the stacked groups of objects in the scene spatial data that contain identifiable spatial data, identify the spatial data set of each object; Based on the spatial data set of each object, multiple candidate objects are identified in the stacked object group; The object to be moved is determined from multiple candidate objects.

[0154] By identifying the spatial data set of each object in the scene spatial data that contains identifiable spatial data, the separation and filtering of each object in the stacked object group is realized. The spatial data reflects the geometric information and spatial position of the object. Identifying objects based on the spatial data can improve the object recognition accuracy.

[0155] In one possible implementation of this disclosure, the determining unit 24 is further configured to: Based on the number of spatial data in the spatial dataset, the spatial dataset for each object is filtered to obtain the filtered spatial dataset. Obtain the target depth value of the filtered spatial data set; Based on the target depth value, multiple candidate objects are determined from the objects corresponding to the filtered spatial data set.

[0156] By filtering through both the quantity and depth of spatial data, objects with incomplete spatial data or insufficient depth can be eliminated, thus improving the efficiency of identifying multiple candidate objects.

[0157] In one possible implementation of this disclosure, the determining unit 24 is further configured to: The initial depth value of the filtered spatial data set collected by the first sensor is obtained, and the second pose of the first sensor relative to the base coordinate system of the robot is obtained; the base coordinate system is a coordinate system with the robot as the reference. Based on the second pose, the initial depth value is adjusted to obtain the target depth value.

[0158] Based on the second pose of the first sensor relative to the base coordinate system, the initial depth value can be adjusted to the target depth value in the vertical direction, so that the target depth value better reflects the true vertical relationship of each object in the stacked object group.

[0159] In one possible implementation of this disclosure, the determining unit 24 is further configured to: Obtain the simulated grasping pose of the robotic arm when it simulates grasping each candidate object, and obtain the number of collision points between the robotic arm model in the simulated grasping pose and the scene space data. The object to be moved is determined based on the simulated grasping pose and the number of collision points.

[0160] By introducing simulated grasping postures for virtual grasping verification, no physical movement of the robotic arm is required, reducing the risk of collision or grasping failure due to improper grasping posture and lowering the risk of damage to the robotic arm.

[0161] In one possible implementation of this disclosure, the determining unit 24 is further configured to: Determine the target placement state for each candidate object; the target placement state can be either vertical or horizontal. In response to the target being placed in a vertical position, a first preset number of simulated grasping poses are randomly selected around the axis of the candidate object in the vertical position.

[0162] Distinguishing between the placement states of objects can improve the matching degree between simulated grasping poses and actual geometric conditions. By differentiating between vertical and horizontal placement states, differentiated grasping strategies can be implemented for different object postures.

[0163] In one possible implementation of this disclosure, the determining unit 24 is further configured to: In response to the target being placed in a horizontal position, the length of the candidate object in the horizontal position, the width of the robotic arm's fingers in the grasping state, and a second preset number of grasping angles of the candidate object in the horizontal position are obtained. Based on the length and finger width, determine the number of gripping points along the axis of the candidate object in a horizontally placed state. Based on the gripping point and gripping angle, determine the target number of candidate objects in a horizontally placed state and simulate gripping pose; the target number is obtained by analyzing the number of gripping points and the second preset number.

[0164] By calculating the number of gripping points based on the object's length and the finger's width, an adaptive distribution of gripping positions is achieved. The adaptive mechanism matches the gripping point layout with the object's size, improving gripping stability and flexibility.

[0165] In one possible implementation of this disclosure, the determining unit 24 is further configured to: Get the minimum and maximum coordinates of the minimum bounding box of the candidate object in the object coordinate system, and get the magnification factor of the minimum bounding box; Based on the magnification factor, determine the first magnified coordinate of the minimum coordinate and the second magnified coordinate of the maximum coordinate; Based on the first magnification coordinate and the second magnification coordinate, the scene space data is filtered to obtain the filtered scene space data; The number of collision points is determined based on the robotic arm model and the selected scene space data.

[0166] By introducing a minimum bounding box and a magnification factor, the collision detection area is adaptively adjusted. The screening area generated by combining the minimum bounding box and the magnification factor includes both the object itself and the safety space required for the robot arm operation.

[0167] In one possible implementation of this disclosure, the determining unit 24 is further configured to: Obtain the fourth pose of the first sensor relative to the candidate object; Based on the fourth pose, the scene space data is transformed from the first coordinate system of the first sensor to the object coordinate system of the candidate object; Based on the first and second magnified coordinates, the transformed scene space data is filtered to obtain the filtered scene space data.

[0168] By introducing a fourth pose for coordinate system transformation, the unification of scene spatial data and object local space is achieved.

[0169] In one possible implementation of this disclosure, the determining unit 24 is further configured to: The fifth pose of the simulated grasping pose is obtained relative to the second coordinate system of the second sensor of the robot arm; the mounting position of the first sensor is higher than the mounting position of the second sensor. Based on the fifth pose, transform the robotic arm model into the second coordinate system; The number of collision points is determined based on the transformed robotic arm model and the filtered scene space data.

[0170] The fifth pose is obtained and the coordinates of the robot model are transformed to make the coordinate reference of the robot model consistent with the filtered scene space data, thereby improving the accuracy of collision detection.

[0171] In one possible implementation of this disclosure, the determining unit 24 is further configured to: In response to a minimum number of collision points of one, the candidate object corresponding to the simulated grasping pose of the minimum value is determined as the object to be moved; or, Since there are multiple minimum values ​​for the number of collision points, the candidate object with the largest component in any of the simulated grasping poses corresponding to the multiple minimum values ​​is determined as the object to be moved; any component is the pose of any component axis in the grasping coordinate system of the simulated grasping pose.

[0172] By selecting the pose with the fewest collision points, the robotic arm minimizes contact with environmental objects, reducing the risk of collision.

[0173] Since the apparatus provided in this embodiment corresponds to the methods provided in the above embodiments, the implementation of the methods is also applicable to the apparatus provided in this embodiment, and will not be described in detail in this embodiment.

[0174] The methods and apparatus provided in the embodiments of this application have been described above. To implement the functions of the methods provided in the embodiments of this application, the robotic arm may include hardware structures and software modules, and may implement the above functions in the form of hardware structures, software modules, or a combination of hardware structures and software modules. One of the above functions may be executed in the form of hardware structures, software modules, or a combination of hardware structures and software modules.

[0175] Figure 11 This is a schematic diagram of the structure of a robotic arm 1200 for implementing the control method of the robotic arm described above, according to an exemplary embodiment.

[0176] Reference Figure 11 The robotic arm 1200 may include one or more of the following components: a processing component 1202, a memory 1204, a power supply component 1206, a multimedia component 1208, an audio component 1210, an input / output (I / O) interface 1212, a sensor component 1214, and a communication component 1216.

[0177] Processing component 1202 typically controls the overall operation of robotic arm 1200, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 1202 may include one or more processors 1220 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 1202 may include one or more modules to facilitate interaction between processing component 1202 and other components. For example, processing component 1202 may include a multimedia module to facilitate interaction between multimedia component 1208 and processing component 1202.

[0178] Memory 1204 is configured to store various types of data to support the operation of robot 1200. Examples of this data include instructions for any application or method used to operate on robot 1200, contact data, phone book data, messages, pictures, videos, etc. Memory 1204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0179] The power supply assembly 1206 provides power to the various components of the robot arm 1200. The power supply assembly 1206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the robot arm 1200.

[0180] The multimedia component 1208 includes a screen that provides an output interface between the robotic arm 1200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 1208 includes a front-facing camera and / or a rear-facing camera. When the robotic arm 1200 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0181] Audio component 1210 is configured to output and / or input audio signals. For example, audio component 1210 includes a microphone (MIC) configured to receive external audio signals when the robot arm 1200 is in an operating mode, such as a call mode, a recording mode, or a voice recognition mode. The received audio signals may be further stored in memory 1204 or transmitted via communication component 1216. In some embodiments, audio component 1210 also includes a speaker for outputting audio signals.

[0182] I / O interface 1212 provides an interface between processing component 1202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0183] Sensor assembly 1214 includes one or more sensors for providing status assessments of various aspects of the robotic arm 1200. For example, sensor assembly 1214 can detect the open / closed state of the robotic arm 1200, the relative positioning of components such as the display and keypad of the robotic arm 1200, changes in position of the robotic arm 1200 or one of its components, the presence or absence of user contact with the robotic arm 1200, the orientation or acceleration / deceleration of the robotic arm 1200, and temperature changes of the robotic arm 1200. Sensor assembly 1214 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1214 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 1214 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0184] Communication component 1216 is configured to facilitate wired or wireless communication between robot 1200 and other devices. Robot 1200 can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G LTE, 12G NR (NewRadio), or combinations thereof. In one exemplary embodiment, communication component 1216 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 1216 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0185] In an exemplary embodiment, the robotic arm 1200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0186] According to the robotic arm disclosed herein, the robotic arm includes: acquiring the posture parameters of a target object in response to a grasping task; the target object being located at a first target position, the target object being obtained based on scene image analysis including multiple objects; controlling the robotic arm to adjust the posture of the target object until the posture parameters meet preset posture conditions; and controlling the robotic arm to move the adjusted target object to a second target position. By first adjusting the posture of the target object and then moving it, the target object is transferred in a suitable posture, improving the stability and reliability of the robotic arm in moving the target object.

[0187] Embodiments of this disclosure also provide a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods described in the above embodiments of this disclosure.

[0188] According to the non-transitory computer-readable storage medium disclosed herein, the non-transitory computer-readable storage medium includes, in response to a grasping task of a robotic arm, acquiring the posture parameters of a target object; the target object being located at a first target position, the target object being obtained based on scene image analysis including multiple objects; controlling the robotic arm to adjust the posture of the target object until the posture parameters meet preset posture conditions; and controlling the robotic arm to move the adjusted target object to a second target position. By first adjusting the posture of the target object and then moving it, the target object is transferred in a suitable posture, improving the stability and reliability of the robotic arm in moving the target object.

[0189] Embodiments of this disclosure also provide a computer program product comprising a computer program executable by a programmable device, the computer program having, when executed by the programmable device, the method described in the above embodiments of this disclosure.

[0190] According to the computer program product disclosed herein, the computer program product includes, in response to a grasping task by a robotic arm, acquiring the posture parameters of a target object; the target object being located at a first target position, the target object being obtained based on scene image analysis including multiple objects; controlling the robotic arm to adjust the posture of the target object until the posture parameters meet preset posture conditions; and controlling the robotic arm to move the adjusted target object to a second target position. By first adjusting the posture of the target object and then moving it, the target object is transferred in a suitable posture, improving the stability and reliability of the robotic arm in moving the target object.

[0191] For cases where the robotic arm may include a chip or chip system, see [link to relevant documentation]. Figure 12 The diagram shows the structure of the chip. Figure 12The chip shown includes a processor 1301 and an interface 1302. There can be one or more processors 1301, and multiple interfaces 1302.

[0192] Optionally, the chip also includes a memory 1303, which is used to store necessary computer programs and data.

[0193] According to the chip disclosed herein, the chip includes: responding to a robotic arm's grasping task by acquiring the posture parameters of a target object; the target object being located at a first target position, the target object being obtained based on scene image analysis including multiple objects; controlling the robotic arm to adjust the posture of the target object until the posture parameters meet preset posture conditions; and controlling the robotic arm to move the adjusted target object to a second target position. By employing a control strategy of first adjusting the posture of the target object and then moving it, the target object is transferred in a suitable posture, improving the stability and reliability of the robotic arm's movement of the target object.

[0194] Those skilled in the art will also understand that the various illustrative logical blocks and steps listed in the embodiments of this application can be implemented by electronic hardware, computer software, or a combination of both. Whether such functionality is implemented through hardware or software depends on the specific application and the overall system design requirements. Those skilled in the art can implement the functionality using various methods for each specific application, but such implementation should not be construed as exceeding the scope of protection of the embodiments of this application.

[0195] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0196] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0197] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0198] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processing module, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (control method), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic device, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0199] It should be understood that various parts of the embodiments of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0200] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium. When executed, the program includes one or a combination of the steps of the method embodiments.

[0201] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc.

[0202] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A control method for a robotic arm, characterized in that, The method includes: In response to the grasping task of the robotic arm, the posture parameters of the target object are acquired; the target object is located at a first target position, and the target object is obtained by analyzing a scene image including multiple objects; The robotic arm is controlled to adjust the posture of the target object until the posture parameters meet the preset posture conditions. The robotic arm is controlled to move the adjusted target object to the second target position.

2. The method according to claim 1, characterized in that, The step of controlling the robotic arm to adjust the posture of the target object until the posture parameters meet preset posture conditions includes: Determine the tilt angle parameter in the attitude parameters, the tilt angle parameter being used to represent the tilt angle of the target object relative to the reference surface; Obtain the reference tilt angle parameter corresponding to the tilt angle parameter; Based on the tilt angle parameters and the reference tilt angle parameters, the tilt angle error of the target object is determined; The robotic arm is controlled to adjust the posture of the target object until the tilt angle error is less than a preset error threshold.

3. The method according to claim 2, characterized in that, The step of determining the tilt angle error of the target object based on the tilt angle parameter and the reference tilt angle parameter includes: The visual tilt angle and tactile tilt angle in the tilt angle parameters are determined; the visual tilt angle is obtained based on the visual image analysis of the target object, and the tactile tilt angle is obtained based on the tactile image analysis of the target object. Determine the reference visual tilt angle and the reference tactile tilt angle in the reference tilt angle parameters; Based on the visual tilt angle, the tactile tilt angle, the reference visual tilt angle, and the reference tactile tilt angle, the multi-tilt angle error of the target object is determined; the tilt angle error is the multi-tilt angle error, which is obtained by analyzing the error between the visual tilt angle and the reference visual tilt angle and the error between the tactile tilt angle and the reference tactile tilt angle.

4. The method according to claim 3, characterized in that, The determination of the multi-tilt error of the target object based on the visual tilt angle, the tactile tilt angle, the reference visual tilt angle, and the reference tactile tilt angle includes: Determine the first weight corresponding to the visual tilt angle and the second weight corresponding to the tactile tilt angle; The multi-tilt error is determined based on the first difference between the visual tilt angle and the reference visual tilt angle, the second difference between the tactile tilt angle and the reference tactile tilt angle, the first weight, and the second weight.

5. The method according to claim 4, characterized in that, Determining the first weight corresponding to the visual tilt angle and the second weight corresponding to the tactile tilt angle includes: Obtain a first variance estimate of the visual tilt angle and a second variance estimate of the tactile tilt angle; the first variance estimate is obtained by analyzing the detection results of historical visual tilt angles, and the second variance estimate is obtained by analyzing the detection results of historical tactile tilt angles. The first weight and the second weight are determined based on the first variance estimate and the second variance estimate.

6. The method according to claim 1, characterized in that, The step of controlling the robotic arm to move the adjusted target object to the second target position includes: The system acquires the first pose of the second target position relative to the first sensor of the robot, the second pose of the first sensor relative to the base coordinate system of the robot, and the third pose of the target object relative to the second sensor of the robot; the base coordinate system is a coordinate system based on the robot, and the installation position of the first sensor is higher than the installation position of the second sensor. Based on the first pose, the second pose, and the third pose, the amount of motion of the robotic arm in moving the target object is determined; Based on the amount of motion, the robotic arm is controlled to move the adjusted target object to the second target position.

7. The method according to claim 6, characterized in that, The step of controlling the robotic arm to move the adjusted target object to the second target position based on the amount of motion includes: Obtain the real-time position of the target object during its movement; The control force of the robotic arm is determined based on the real-time position and the amount of motion. Based on the control force and the amount of motion, the robotic arm is controlled to move the adjusted target object to the second target position.

8. The method according to claim 1, characterized in that, The target object is obtained from scene image analysis that includes multiple objects, including: Obtain the first orientation of the first target position in the scene image; Determine the second position of each of the plurality of objects in the scene image; The target object is determined from the plurality of objects based on the similarity between the first orientation and the second orientation.

9. The method according to claim 1, characterized in that, The method further includes: The scene space data of the stacked object group collected by the first sensor of the robotic arm is obtained; the stacked object group is a group of objects stacked together before the target object moves to the first target position; Based on the scene space data, determine the objects to be moved in the stacked object group; The robotic arm is controlled to move the object to be moved to the first target position; the target object is the object to be moved at the first target position. In response to the grasping task of the robotic arm, the robotic arm is controlled to grasp the target object.

10. The method according to claim 9, characterized in that, The step of determining the object to be moved in the stacked object group based on the scene space data includes: From the stacked object group containing identifiable spatial data in the scene spatial data, identify the spatial data set of each object; Based on the spatial data set of each object, multiple candidate objects are determined in the stacked object group; The object to be moved is determined from the plurality of candidate objects.

11. The method according to claim 10, characterized in that, The step of determining multiple candidate objects in the stacked object group based on the spatial data set of each object includes: Based on the number of spatial data in the spatial data set, the spatial data set of each object is filtered to obtain the filtered spatial data set. Obtain the target depth value of the filtered spatial data set; Based on the target depth value, the plurality of candidate objects are determined from the objects corresponding to the filtered spatial data set.

12. The method according to claim 11, characterized in that, The step of obtaining the target depth value of the filtered spatial data set includes: The initial depth value of the filtered spatial data set collected by the first sensor is obtained, and the second pose of the first sensor relative to the base coordinate system of the manipulator is obtained; the base coordinate system is a coordinate system with the manipulator as the reference. Based on the second pose, the initial depth value is adjusted to obtain the target depth value.

13. The method according to claim 10, characterized in that, The step of determining the object to be moved from the plurality of candidate objects includes: Obtain the simulated grasping pose of the robotic arm when it simulates grasping each candidate object, and obtain the number of collision points between the robotic arm model in the simulated grasping pose and the scene space data. The object to be moved is determined based on the simulated grasping pose and the number of collision points.

14. The method according to claim 13, characterized in that, The process of obtaining the simulated grasping pose of the robotic arm for each candidate object includes: Determine the target placement state for each candidate object; the target placement state is either a vertical placement state or a horizontal placement state. In response to the target placement state being the vertical placement state, a first preset number of simulated grasping poses are randomly selected around the axis of the candidate object in the vertical placement state.

15. The method according to claim 14, characterized in that, The process of obtaining the simulated grasping pose of the robotic arm for each candidate object includes: In response to the target being placed in a horizontal position, the length of the candidate object in the horizontal position, the width of the fingers of the robotic arm in the grasping state, and a second preset number of grasping angles of the candidate object in the horizontal position are obtained. Based on the length and the finger width, determine the number of gripping points along the axis of the candidate object in the horizontally placed state. Based on the grasping points and the grasping angle, a target number of simulated grasping poses for the candidate objects in the horizontally placed state are determined; the target number is obtained by analyzing the number of grasping points and the second preset number.

16. The method according to claim 13, characterized in that, The number of collision points between the robotic arm model that acquires the simulated grasping pose and the scene space data includes: Obtain the minimum and maximum coordinates of the minimum bounding box of the candidate object in the object coordinate system of the candidate object, and obtain the magnification factor of the minimum bounding box; Based on the magnification factor, determine the first magnified coordinate of the minimum coordinate and the second magnified coordinate of the maximum coordinate; The scene space data is filtered based on the first magnified coordinate and the second magnified coordinate to obtain filtered scene space data; The number of collision points is determined based on the robotic arm model and the filtered scene space data.

17. The method according to claim 16, characterized in that, The step of filtering the scene spatial data based on the first magnified coordinates and the second magnified coordinates to obtain the filtered scene spatial data includes: Obtain the fourth pose of the first sensor relative to the candidate object; Based on the fourth pose, the scene space data is converted from the first coordinate system of the first sensor to the object coordinate system of the candidate object; Based on the first magnified coordinates and the second magnified coordinates, the converted scene space data is filtered to obtain the filtered scene space data.

18. The method according to claim 16, characterized in that, Determining the number of collision points based on the robotic arm model and the filtered scene space data includes: Obtain the fifth pose of the grasping coordinate system of the simulated grasping pose relative to the second coordinate system of the second sensor of the robot; the installation position of the first sensor is higher than the installation position of the second sensor; Based on the fifth pose, the robotic arm model is transformed into the second coordinate system; The number of collision points is determined based on the transformed robotic arm model and the filtered scene space data.

19. The method according to claim 13, characterized in that, The step of determining the object to be moved based on the simulated grasping pose and the number of collision points includes: In response to the minimum number of collision points being one, the candidate object corresponding to the simulated grasping pose of the minimum value is determined as the object to be moved; or, Since there are multiple minimum values ​​for the number of collision points, the candidate object with the largest component among the simulated grasping poses corresponding to the multiple minimum values ​​is determined as the object to be moved; the "any component" is the pose of the simulated grasping pose on any component axis in the grasping coordinate system of the simulated grasping pose.

20. A control device for a robotic arm, characterized in that, The device includes: The acquisition unit is used to acquire the posture parameters of the target object in response to the grasping task of the robotic arm; the target object is located at a first target position, and the target object is obtained by analyzing a scene image including multiple objects; The first control unit is used to control the robot arm to adjust the posture of the target object until the posture parameters meet the preset posture conditions. The second control unit is used to control the robotic arm to move the adjusted target object to the second target position.

21. The apparatus according to claim 20, characterized in that, The first control unit is also used for: Determine the tilt angle parameter among the attitude parameters; Obtain the reference tilt angle parameter corresponding to the tilt angle parameter; Based on the tilt angle parameters and the reference tilt angle parameters, the tilt angle error of the target object is determined; The robotic arm is controlled to adjust the posture of the target object until the tilt angle error is less than a preset error threshold.

22. The apparatus according to claim 20, characterized in that, The second control unit is also used for: The system acquires the first pose of the second target position relative to the first sensor of the robot, the second pose of the first sensor relative to the base coordinate system of the robot, and the third pose of the target object relative to the second sensor of the robot; the base coordinate system is a coordinate system based on the robot, and the installation position of the first sensor is higher than the installation position of the second sensor. Based on the first pose, the second pose, and the third pose, the amount of motion of the robotic arm in moving the target object is determined; Based on the amount of motion, the robotic arm is controlled to move the adjusted target object to the second target position.

23. A robotic arm, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-19.

24. A robot, characterized in that, Used to perform the method of any one of claims 1-19, or includes the robotic arm of claim 23.

25. A computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-19.

26. A program product, characterized in that, Includes computer instructions for causing a computer to perform the method of any one of claims 1-19.

Images