Robot adaptive grasping control method and system suitable for complex curved surface workpiece

Abstract

The application discloses a robot adaptive grabbing control method and system suitable for complex curved surface workpieces, and belongs to the technical field of robot control, and the technical scheme points of the application include the following: acquiring workpiece data in a working range, the workpiece data including three-dimensional point cloud data and two-dimensional image data; identifying each workpiece and the posture corresponding to each workpiece according to the three-dimensional point cloud data; obtaining a motion trajectory probability atlas according to the speed of a conveying belt, the motion trajectory probability atlas including predicted trajectory points corresponding to each workpiece and confidence; obtaining a grabbing priority sequence according to the three-dimensional point cloud data and the motion trajectory probability atlas; determining the grabbing points corresponding to each workpiece according to the three-dimensional point cloud data, the two-dimensional image data and the posture corresponding to each workpiece, and completing workpiece grabbing according to the grabbing points, and the application separates stacked workpieces through three-dimensional point cloud data and clustering analysis, and generates grabbing priorities through the communication mechanism between the motion trajectory of the workpieces and the robot, so that the grabbing efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and more specifically to a robot adaptive grasping control method and system applicable to complex curved surface workpieces. Background Technology

[0002] With the development of high-end manufacturing fields such as automobiles and aerospace, the demand for complex curved surface workpieces has increased significantly. These workpieces often need to be continuously transported by conveyor belts and have extremely high requirements for gripping accuracy, efficiency and system stability. However, complex curved surface workpieces have problems such as irregular geometric shape and variable stacking posture. Traditional gripping solutions mostly use a single industrial camera to acquire two-dimensional images, which cannot restore the three-dimensional contour of complex curved surfaces. This results in large errors in curvature recognition and recognition of stacked areas, making it easy to select gripping points in weak areas and increasing the workpiece slippage rate. Therefore, existing technologies have shortcomings. Summary of the Invention

[0003] To address the shortcomings of existing technologies, the present invention aims to provide a robot adaptive grasping control method and system suitable for complex curved workpieces. By using 3D point cloud data and cluster analysis, stacked workpieces are effectively identified and segmented based on features such as the curvature of the workpieces. A motion trajectory probability map is obtained by predicting the motion trajectory of the workpieces, and a grasping priority sequence is generated by combining the motion trajectory probability map. Ultimately, the grasping efficiency is improved while accurately grasping the workpieces.

[0004] To achieve the above objectives, the present invention provides the following technical solution:

[0005] This invention provides a robot adaptive gripping control method applicable to complex curved surface workpieces, executed by each robot. The robot adaptive gripping control method includes:

[0006] Acquire workpiece data within the working range, the workpiece data including three-dimensional point cloud data and two-dimensional image data;

[0007] Identify each workpiece and its corresponding posture based on the three-dimensional point cloud data;

[0008] The motion trajectory probability map is obtained based on the speed of the conveyor belt. The motion trajectory probability map includes the predicted trajectory points and confidence levels corresponding to each workpiece.

[0009] A grasping priority sequence is obtained based on the three-dimensional point cloud data and the motion trajectory probability map;

[0010] Based on the 3D point cloud data, the 2D image data, and the posture corresponding to each workpiece, the gripping point corresponding to each workpiece is determined, and the workpiece gripping is completed based on the gripping point.

[0011] As a further improvement of the present invention, identifying each workpiece and the corresponding posture of each workpiece based on the three-dimensional point cloud data includes:

[0012] Cluster analysis is performed on the three-dimensional point cloud data to obtain multiple first categories, and each first category corresponds to a workpiece.

[0013] Based on the angle between the normal vectors corresponding to each first category, multiple point cloud clusters are obtained, and each workpiece is identified based on the point cloud clusters;

[0014] Obtain the point cloud template corresponding to each workpiece, and obtain the posture corresponding to each workpiece based on the point cloud template.

[0015] As a further improvement of the present invention, multiple point cloud clusters are obtained based on the included angle between the normal vectors corresponding to each first category, including:

[0016] For each first category, the covariance matrix is ​​calculated based on each data point corresponding to the first category;

[0017] The first category is updated based on the eigenvalues ​​in the covariance matrix and the curvature corresponding to each data point to obtain the second category;

[0018] The covariance matrix is ​​calculated based on each data point corresponding to the second category.

[0019] Based on the covariance matrix, eigenvalue decomposition is performed to obtain multiple normal vectors;

[0020] Multiple point cloud clusters are obtained based on the angle between the normal vectors and a preset threshold.

[0021] As a further improvement of the present invention, the posture corresponding to each workpiece is obtained according to the point cloud template, including:

[0022] For each second category, perform an iterative operation, which includes: for each data point in the current second category, determine the corresponding data point in the corresponding point cloud template, obtain the transformation matrix corresponding to the current second category according to the preset objective function, determine whether the preset termination condition has been met, if not, update the current second category, until the preset termination condition is met, and output the transformation matrix corresponding to the current second category.

[0023] The posture corresponding to each workpiece is obtained based on the transformation matrix.

[0024] As a further improvement of the present invention, the step of obtaining the motion trajectory probability map based on the speed of the conveyor belt includes:

[0025] Based on the speed of the conveyor belt and the historical trajectory of each workpiece, the feature vector corresponding to each workpiece is obtained;

[0026] Based on the feature vector and the preset model, the predicted trajectory points corresponding to each workpiece and the probability distribution corresponding to each predicted trajectory point are obtained;

[0027] The confidence level is obtained based on the probability distribution, and the motion trajectory probability map is obtained based on the predicted trajectory points and the confidence level.

[0028] As a further improvement of the present invention, a grasping priority sequence is obtained based on the three-dimensional point cloud data and the motion trajectory probability map, including:

[0029] The priority value corresponding to each workpiece is obtained based on the three-dimensional point cloud data and the preset priority decision function, thus obtaining the first priority sequence;

[0030] The first priority sequence is broadcast to the neighboring robots, and a second priority sequence is received, wherein the second priority sequence is the first priority sequence corresponding to the neighboring robot;

[0031] The grabbing priority sequence is obtained based on the motion trajectory probability map, the first priority sequence, and the second priority sequence.

[0032] As a further improvement of the present invention, a grasping priority sequence is obtained based on the motion trajectory probability map, the first priority sequence, and the second priority sequence, including:

[0033] Identify conflicting artifacts in the first priority sequence and the second priority sequence;

[0034] The confidence level corresponding to the conflicting workpiece is obtained based on the motion trajectory probability map.

[0035] The first grasping time window corresponding to the conflicting workpiece is obtained based on the confidence level;

[0036] The first priority sequence is updated according to the first crawling time window to obtain the crawling priority sequence.

[0037] As a further improvement of the present invention, the first priority sequence is updated according to the first crawling time window to obtain the crawling priority sequence, including:

[0038] Send the first grasping time window to the adjacent robot and receive the second grasping time window;

[0039] If the first grasping time window is smaller than the second grasping time window, the conflicting workpiece is removed from the first priority sequence to obtain the grasping priority sequence;

[0040] If the first grasping time window is equal to the second grasping time window, the first priority sequence is updated according to the robot's current capability value to obtain the grasping priority sequence.

[0041] As a further improvement of the present invention, based on the three-dimensional point cloud data, the two-dimensional image data, and the posture corresponding to each workpiece, a gripping point corresponding to each workpiece is determined, and workpiece gripping is completed based on the gripping point, including:

[0042] The contact area is determined based on the curvature and position of each data point in the three-dimensional point cloud data;

[0043] The grasping point is determined based on the distance between each data point in the grasping area and the contact area, and the angle between the data point and the direction of the robot's end effector movement.

[0044] The surface state of the gripping point is obtained from the two-dimensional image, and the gripping force is obtained based on the surface state and the preset prediction model.

[0045] The workpiece is grasped based on the grasping force and the posture corresponding to each workpiece.

[0046] This invention provides a robot adaptive gripping control system suitable for complex curved surface workpieces, comprising:

[0047] The acquisition module is used to acquire workpiece data within the working range, including three-dimensional point cloud data and two-dimensional image data.

[0048] The recognition module is used to identify each workpiece and the posture corresponding to each workpiece based on the three-dimensional point cloud data.

[0049] The prediction module is used to obtain a motion trajectory probability map based on the speed of the conveyor belt. The motion trajectory probability map includes the predicted trajectory points and confidence levels corresponding to each workpiece.

[0050] The communication module is used to obtain a grasping priority sequence based on the three-dimensional point cloud data and the motion trajectory probability map;

[0051] The grasping module is used to determine the grasping point corresponding to each workpiece based on the three-dimensional point cloud data, the two-dimensional image data and the posture corresponding to each workpiece, and to complete the workpiece grasping based on the grasping point.

[0052] This invention effectively identifies and segments stacked workpieces based on features such as workpiece curvature using 3D point cloud data and cluster analysis. It also obtains a motion trajectory probability map by predicting the motion trajectory of the workpieces. Simultaneously, it obtains the first priority sequence corresponding to the adjacent robots based on the communication mechanism between adjacent robots. When a conflict occurs with the first priority sequence corresponding to the adjacent robot, it generates a grasping priority sequence by combining the motion trajectory probability map. Ultimately, it improves grasping efficiency while accurately grasping the workpiece. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the method steps of the present invention;

[0054] Figure 2 This is a schematic diagram illustrating the steps of an iterative operation.

[0055] Figure 3 A step-by-step diagram illustrating the process of capturing the window in real time;

[0056] Figure 4 A schematic diagram illustrating the steps to obtain the capture priority sequence. Detailed Implementation

[0057] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof.

[0058] Identical parts are indicated by the same reference numerals. It should be noted that the terms "front," "rear," "left," "right," "up," and "down" used in the following description refer to directions in the accompanying drawings, while the terms "bottom surface," "top surface," "inner," and "outer" refer to directions toward or away from the geometric center of a specific part, respectively.

[0059] The term "and / or" in the following text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0060] like Figure 1 As shown, this application provides a robot adaptive gripping control method suitable for complex curved surface workpieces, including:

[0061] Acquire workpiece data within the working area, including 3D point cloud data and 2D image data;

[0062] Identify each workpiece and its corresponding posture based on 3D point cloud data;

[0063] The motion trajectory probability map is obtained based on the speed of the conveyor belt. The motion trajectory probability map includes the predicted trajectory points and confidence levels for each workpiece.

[0064] A grasping priority sequence is obtained based on 3D point cloud data and motion trajectory probability map;

[0065] Based on the 3D point cloud data, 2D image data, and the posture of each workpiece, the gripping point corresponding to each workpiece is determined, and the workpiece gripping is completed according to the gripping point.

[0066] In this embodiment, the curved workpiece is transported via a conveyor belt, with multiple robots distributed on both sides. The adaptive grasping control method is executed by each robot. Three-dimensional point cloud data and two-dimensional image data are collected by sensors mounted on the robots, such as a laser contour sensor for three-dimensional point cloud data and an industrial area scan camera for two-dimensional image data. The angles collected by both types of sensors should completely cover the grasping area of ​​the curved workpiece on the conveyor belt. This can be achieved through physical installation positioning and algorithm calibration. The size of the grasping area for each robot corresponds to its working range, and the specific value can be determined based on the viewing angle of the new sensor and the maximum horizontal working radius of the robot's end effector.

[0067] The posture of each workpiece specifically refers to its rotation relative to the point cloud template. The motion trajectory probability map includes the predicted trajectory point and confidence level for each workpiece, where the confidence level indicates the reliability of the predicted trajectory point. The grasping priority sequence refers to the grasping order of each workpiece within the current working range. The method provided in this embodiment can be executed after grasping each workpiece, or it can be executed after grasping each workpiece in the current grasping priority sequence. This embodiment does not impose any restrictions on this.

[0068] This embodiment uses 3D point cloud data and cluster analysis to effectively identify and segment stacked workpieces based on features such as workpiece curvature. It also obtains a motion trajectory probability map by predicting the motion trajectory of the workpieces. Simultaneously, it obtains the first priority sequence corresponding to the adjacent robots based on the communication mechanism between adjacent robots. When a conflict occurs with the first priority sequence corresponding to the adjacent robot, it generates a grasping priority sequence by combining the motion trajectory probability map. Ultimately, it improves grasping efficiency while accurately grasping the workpiece.

[0069] Furthermore, this embodiment provides a step for identifying each workpiece and its corresponding pose based on 3D point cloud data, including:

[0070] Cluster analysis was performed on the 3D point cloud data to obtain multiple first categories, and each first category corresponds to a workpiece.

[0071] Based on the angle between the normal vectors corresponding to each first category, multiple point cloud clusters are obtained;

[0072] For each point cloud cluster, obtain the corresponding point cloud template, and obtain the pose of each workpiece based on the point cloud template.

[0073] Specifically, this embodiment first measures the spatial proximity of the three-dimensional point cloud based on Euclidean distance, and classifies points with a distance less than a threshold into the same first category, thus initially distinguishing workpieces in different stacking layers, i.e., one first category corresponds to one workpiece.

[0074] For example, firstly, a data point is randomly selected from the 3D point cloud data to create the first category, and this data point is added to this category. Then, all other data points are traversed, and the distance between each data point and every other data point is calculated. If the distance is less than a threshold, the data point is added to the first category; otherwise, it is not assigned, and the calculation of the distance to the next data point continues. When there are multiple data points in the first category, calculating the distance to the next data point requires calculating the distance between each data point in the first category and the next data point. If the distance between any data point in the first category and the next data point is less than the threshold, the next data point is added to the first category, until no data point can be added to the first category. The above steps are then repeated for the remaining data points to obtain multiple first categories. This embodiment does not limit the specific value of the threshold.

[0075] After obtaining multiple first categories, multiple point cloud clusters are obtained based on the angle between the normal vectors corresponding to each first category. Each point cloud cluster corresponds to a region in the workpiece. Finally, based on the similarity between the region features and the point cloud template, the specific workpiece type and the workpiece posture are identified.

[0076] This embodiment takes into account that although there are contact areas in the stacked complex curved surfaces, in the overall structure, there are still spatial gaps between the main parts of the workpieces in different layers in the vertical direction (such as the height direction of the conveyor belt) or the horizontal direction (such as the misalignment of the workpiece edges). Although there is local contact, the distance between non-contact areas should be greater than or equal to the distance threshold. Since the point cloud data of the same workpiece surface is a continuous entity, the spatial distance between data points is much smaller than the distance threshold. Therefore, in this embodiment, when stacking complex curved surface workpieces, based on the characteristic that there are obvious distance differences between different layers of workpieces in space, different workpieces are divided by cluster analysis. Then, based on the constraints of the normal vector and the regional features, the workpiece type and posture are further identified to ensure grasping efficiency. This embodiment does not limit the specific value of the threshold.

[0077] Furthermore, this embodiment provides a step for obtaining multiple point cloud clusters based on the angle between the normal vectors corresponding to each first category, including:

[0078] For each first category, the covariance matrix is ​​calculated based on each data point corresponding to the first category;

[0079] Based on the eigenvalues ​​in the covariance matrix and the curvature corresponding to each data point, the first category is updated to obtain the second category;

[0080] The covariance matrix is ​​calculated for each data point corresponding to the second category.

[0081] Eigenvalue decomposition is performed on the covariance matrix to obtain multiple normal vectors;

[0082] Multiple point cloud clusters are obtained based on the angle between the normal vectors and a preset threshold.

[0083] For each first category, each data point it contains corresponds to a covariance matrix. For example, for a data point in one of the first categories... First, multiple data points within a preset neighborhood are obtained. In this embodiment, the size of the preset neighborhood is not limited; for example, it can be set as a region centered on the data point with a preset distance as its radius. The preset neighborhood should be smaller than the aforementioned distance threshold. Next, its corresponding covariance matrix is ​​obtained as follows:

[0084]

[0085] in, This indicates the number of data points included in the preset neighborhood. To determine the centroid of this preset neighborhood, the average coordinates of the data points within the neighborhood can be used as the centroid. Represents the first term of the preset neighborhood. There are 10 data points. Then the covariance matrix will be... Eigenvalue decomposition yields:

[0086]

[0087] in, It is an eigenvalue diagonal matrix. The eigenvector matrix is ​​used to obtain data points from its eigenvalues. The corresponding curvature, for example, assumes that the three eigenvalues ​​obtained from the eigenvalue diagonal matrix are arranged in ascending order. , and data points Corresponding curvature .

[0088] Next, based on historical workpiece data, the curvature distribution under normal conditions is statistically analyzed. Normal conditions refer to the absence of noise points, resulting in a curvature range. Finally, the data points are... The corresponding curvature is compared with the range of curvature. If If the curvature exceeds this range, the data point is considered... If a data point is considered noise, it is removed from the first category; otherwise, it is retained.

[0089] Repeat the above steps for each first category. The first category after removing noise points is recorded as the second category. Then, calculate the covariance matrix corresponding to each data point in each second category and perform eigenvalue decomposition to obtain the eigenvalue diagonal matrix and eigenvector matrix corresponding to each data point in each second category. The calculation method is the same as above, and will not be repeated here in this embodiment. Finally, the eigenvector corresponding to the smallest eigenvalue is recorded as the normal vector corresponding to the data point, and the normal vector is normalized.

[0090] Next, for each second category, construct its corresponding angle matrix. Each element in the angle matrix corresponds to a normal vector angle. For example, assume there are a total of [missing information - likely related to the second category]. If there are 10 data points, then the included angle matrix is: The matrix, where the first... Line number The element of the column is the first element in the second category. The and the first The normal vectors of each data point are angled together. Then, cluster analysis is performed based on the normal vector angles between each data point. First, a data point is randomly selected, and then the normal vector angle between each of the other data points and that data point is iterated. If the angle is less than a preset threshold, the two data points are grouped into the same point cloud cluster, resulting in multiple point cloud clusters. The specific principle is the same as the principle for obtaining the first category, and will not be elaborated upon in this embodiment. In this embodiment, the preset threshold is not limited; those skilled in the art can set it experimentally, for example, setting the preset threshold to 10°. Each point cloud cluster corresponds to a region in the workpiece. For example, if the direction of the normal vector of each data point within a point cloud cluster is consistent and the curvature is small, it indicates that the point cloud cluster corresponds to a planar region. If the direction of the normal vector of each data point within a point cloud cluster diverges outward, it indicates that the point cloud cluster corresponds to a convex curved surface region. If the direction of the normal vector of each data point within a point cloud cluster converges inward, it indicates that the point cloud cluster corresponds to the inner region of a groove.

[0091] This embodiment takes into account that complex curved workpieces usually contain multiple geometric structures, such as protrusions, depressions, and planes. The normal vector directions of these regions have significant differences. Therefore, this embodiment further divides the point cloud clusters based on the classification of normal vectors, accurately distinguishing regions with different geometric features. This facilitates subsequent identification of workpiece type and posture based on region features, helping the robot to more accurately judge the specific situation of each workpiece when multiple workpieces are stacked, avoiding identification errors caused by inaccurate workpiece region division, and improving workpiece grasping efficiency. At the same time, this embodiment takes into account the large difference between the curvature of noise points and the curvature of the workpiece itself, and removes noise points according to the curvature range to ensure the accuracy of the point cloud clusters. The noise points include impurities such as debris on the conveyor belt and error points caused by electromagnetic interference to the sensors.

[0092] Furthermore, such as Figure 2 As shown, this embodiment provides a step for obtaining the pose corresponding to each workpiece based on a point cloud template, including:

[0093] For each second category, perform an iterative operation. The iterative operation includes: for each data point in the current second category, determine the corresponding data point in the corresponding point cloud template, obtain the transformation matrix corresponding to the current second category according to the preset objective function, determine whether the preset termination condition has been met, if not, update the current second category, until the preset termination condition is met, and output the transformation matrix corresponding to the current second category.

[0094] The posture of each workpiece is obtained based on the transformation matrix.

[0095] After obtaining the point cloud clusters, the first step is to identify the workpiece type. Specifically, for each second category, each corresponding point cloud cluster is obtained, and the corresponding regional features are extracted. These regional features include basic regional features, curvature features, and inter-regional relationship features. Basic regional features include regional area, regional volume, and regional centroid coordinates. The regional area can be obtained by the number of data points within the point cloud cluster and the resolution of a single point cloud. The regional volume can be obtained using the convex hull algorithm. The regional centroid coordinates are the mean of all data points within the point cloud cluster. Curvature features include the average curvature and standard deviation of the curvature of the data points within the point cloud cluster. Inter-regional relationship features include the relative position vector and the angle between the normal vectors. The relative position vector is obtained by concatenating the distances between the centroid coordinates of this point cloud cluster and the centroid coordinates of other point cloud clusters. The angle between the normal vectors is the mean of the angles between the normal vectors of all data points within the point cloud cluster.

[0096] After normalizing the features of each region, the feature vector corresponding to the point cloud cluster is obtained. The feature vectors corresponding to each point cloud cluster included in each second category are concatenated to obtain the global feature vector. Finally, the similarity between the global feature vector and the point cloud template of each workpiece is calculated. The workpiece type corresponding to the point cloud template with the highest similarity is taken as the workpiece type corresponding to the second category. The dimension of the vector corresponding to the point cloud template should be the same as the dimension of the global feature vector.

[0097] After obtaining the workpiece type, a point cloud template corresponding to each second category is determined based on the workpiece type, and an iterative operation is performed on each second category. For example, for one second category, each data point in that second category is determined. The nearest data point in the corresponding point cloud template And determine the transformation matrix according to the preset objective function. Translation vector The objective function is:

[0098]

[0099] in, This indicates the number of data points included in the current second category. The index is used to iterate through each data point in the current second category. Then, it checks if a preset termination condition is met. The preset termination condition is that the difference between the objective functions of two iterations is less than a preset difference. Since this is the first iteration, the preset termination condition is not met, and the second category needs to be updated. Each data point in the updated second category is represented as... Then, based on the updated second category, the transformation matrix, translation vector, and objective function are recalculated until the preset termination condition is met. The transformation matrix corresponding to the current second category is then output. The above steps are repeated for each second category to obtain the transformation matrix corresponding to each second category, which is the transformation matrix corresponding to each workpiece. The transformation matrix describes the degree of deviation of the workpiece's posture on the current conveyor belt from the standard posture in the point cloud template.

[0100] This embodiment first determines the workpiece type using a point cloud template, and then matches the point cloud data corresponding to the identified workpiece type with the point cloud template according to the iterative steps to calculate the corresponding transformation matrix. This yields the rotation angle and other postures of the current workpiece relative to the point cloud template, ensuring that the robot can contact the workpiece at a suitable angle and position, thereby improving the accuracy and stability of the grasping.

[0101] Furthermore, this embodiment provides a step for obtaining a motion trajectory probability map based on the speed of the conveyor belt, including:

[0102] Based on the speed of the conveyor belt and the historical trajectory of each workpiece, the feature vector corresponding to each workpiece is obtained;

[0103] Based on the feature vector and the preset model, the predicted trajectory points corresponding to each workpiece and the probability distribution corresponding to each predicted trajectory point are obtained;

[0104] The confidence level is obtained from the probability distribution, and the probability map of the motion trajectory is obtained from the predicted trajectory points and the confidence level.

[0105] The conveyor belt speed can be acquired via an encoder. This data is then combined with workpiece data collected by the robot itself and other robots at historical moments to obtain the historical trajectory of each workpiece within the current working range. A position sequence for each workpiece is generated based on this historical trajectory. This embodiment does not limit the length of the sequence. For example, if the obtained historical trajectory of the workpiece is within the past 10 seconds, with a sampling interval of 1 second, the resulting position sequence includes 10 positions. A relative displacement sequence is then obtained based on the position sequence, where each element represents the offset between two adjacent positions. Therefore, the relative displacement sequence includes 9 elements. Next, the average curvature and material density corresponding to each workpiece are obtained. Finally, for each workpiece, the relative displacement sequence, the conveyor belt speed and acceleration, the average curvature, and the material density are concatenated and normalized to obtain the feature vector corresponding to each workpiece.

[0106] The feature vector is then input into a preset model. For example, the preset model can be a deep learning model based on an attention mechanism, specifically including a self-attention layer, a cross-attention layer, and a Transformer decoder. For each workpiece, when its corresponding feature vector is input into the preset model, different layers will receive different features. First, the self-attention layer receives the elements corresponding to the relative displacement sequence and outputs a feature representation based on time relationship to the cross-attention layer. The cross-attention layer uses this feature representation as a query and receives the elements corresponding to the speed and acceleration of the conveyor belt, the average curvature, and the material density, using them as keys and values. Finally, it outputs an enhanced feature that integrates motion time sequence and external factors. The Transformer decoder receives this enhanced feature and obtains the probability distribution corresponding to each future moment based on the multilayer perceptron. In this embodiment, there are no restrictions on the number of future moments or the form of the probability distribution. For example, the predicted trajectory points for the next 10 moments can be predicted, and the probability distribution is a two-dimensional Gaussian distribution. Then, the position with the highest probability density in the probability distribution corresponding to each moment is taken as the predicted trajectory point corresponding to that moment. For each probability distribution, its corresponding entropy can be calculated as follows:

[0107]

[0108] in, It is a constant. In a two-dimensional Gaussian distribution Standard deviation in direction In a two-dimensional Gaussian distribution Standard deviation in direction Representing covariance, its corresponding confidence level can be obtained from the entropy value. The higher the confidence level, the more accurate the predicted trajectory point at that moment. Finally, a motion trajectory map is obtained based on each predicted trajectory point and its confidence level. Each row in the map represents a workpiece, each column represents a moment, and each element corresponds to a predicted trajectory point. The value of the element is the confidence level, and the color is determined according to the value of the confidence level. For example, it is set to red when the confidence level is greater than 90%. This embodiment does not impose any restrictions on this.

[0109] Furthermore, this embodiment provides a step for obtaining a grasping priority sequence based on 3D point cloud data and motion trajectory probability map, including:

[0110] The priority value corresponding to each workpiece is obtained based on the 3D point cloud data and the preset priority decision function, thus obtaining the first priority sequence;

[0111] The first priority sequence is broadcast to neighboring robots, and the second priority sequence is received. The second priority sequence is the first priority sequence corresponding to the neighboring robot.

[0112] The grabbing priority sequence is obtained based on the motion trajectory probability map, the first priority sequence, and the second priority sequence.

[0113] When multiple workpieces exist within the working range, the priority value of each workpiece needs to be determined according to a preset priority function. The preset priority function is a weighted function of grasping success rate and grasping cost. The grasping success rate is determined based on the surface complexity and stacking separation difficulty of the workpiece. Specifically, the surface complexity is determined based on curvature. According to the above analysis, each second category corresponds to one workpiece. Therefore, the point cloud data corresponding to each workpiece can be obtained, and then the curvature corresponding to each data point in the point cloud data can be obtained. The proportion of data points with curvature less than the preset curvature corresponding to each workpiece is counted to obtain the surface complexity corresponding to each workpiece. The stacking separation difficulty is the number of stacking layers. Since the above steps to obtain the first category are the steps to separate stacked workpieces, the number of stacking layers corresponding to each workpiece can be obtained based on the first category, and then the stacking separation difficulty corresponding to each workpiece can be obtained. Finally, the surface complexity and stacking separation difficulty are normalized and weighted to obtain the grasping success rate corresponding to each workpiece. The grasping cost is determined based on the distance between the centroid of each workpiece within the current working range and the robot's end effector. The distance is then mapped to the 0-1 range. The grasping success rate and grasping cost are weighted to obtain the priority value for each workpiece. Each workpiece is then arranged in descending order of priority value to obtain the first priority sequence.

[0114] Since multiple robots are distributed on both sides of the conveyor belt, and the working range of each robot overlaps, the workpieces included in the first priority sequence of each robot overlap. In order to avoid repeated grasping actions, each robot needs to transmit its own first priority sequence to the adjacent robot and receive the first priority sequence (second priority sequence) of the adjacent robot. The final grasping priority sequence is obtained by comparing the first priority sequence and the second priority sequence.

[0115] Furthermore, this embodiment provides a step for obtaining a grasping priority sequence based on a motion trajectory probability map, a first priority sequence, and a second priority sequence, including:

[0116] Identify conflicting artifacts in the first priority sequence and the second priority sequence;

[0117] The confidence level corresponding to the conflicting workpiece is obtained based on the probability map of the motion trajectory.

[0118] The first grasping time window corresponding to the conflicting workpiece is obtained based on the confidence level;

[0119] The first priority sequence is updated based on the first crawling time window to obtain the crawling priority sequence.

[0120] Specifically, such as Figure 3As shown, upon receiving the second priority, the robot first needs to identify the workpieces that appear repeatedly in the first and second priority sequences and determine their positions within those sequences. Then, it identifies the position with the highest priority and determines whether this position appears in only one sequence. For example, if it appears only in the first priority sequence, the current robot performs the grasping of the workpiece and sends information to neighboring robots to remove it from their corresponding first priority sequences. If the position appears in multiple sequences, the workpiece is marked as a conflicting workpiece. The current robot then searches for the confidence level of each conflicting workpiece in its corresponding motion trajectory probability map. Since a confidence level greater than 90% is indicated by red, the length of the red interval corresponding to each conflicting workpiece can be obtained, along with the time period corresponding to that interval, which serves as the first grasping time window. Similarly, neighboring robots can also search for the confidence level of each conflicting workpiece in their corresponding motion trajectory probability map to obtain the first grasping time window for each conflicting workpiece for each neighboring robot.

[0121] Furthermore, such as Figure 4 As shown, this embodiment provides a step for updating a first priority sequence according to a first fetching time window to obtain a fetching priority sequence, including:

[0122] Send the first grasping time window to the adjacent robot and receive the second grasping time window;

[0123] If the first grasping time window is smaller than the second grasping time window, the conflicting workpieces are removed from the first priority sequence to obtain the grasping priority sequence;

[0124] If the first grasping time window is equal to the second grasping time window, the first priority sequence is updated according to the robot's current capability value to obtain the grasping priority sequence.

[0125] Specifically, after the current robot obtains the first grasping time window corresponding to each conflicting workpiece, it sends the first grasping time window corresponding to each conflicting workpiece to neighboring robots and receives the first grasping time window (second grasping time window) corresponding to each neighboring robot. For each conflicting workpiece, if the first grasping time window is shorter than the second grasping time window, the conflicting workpiece is removed from the first priority sequence, resulting in a grasping priority sequence. If the first grasping time window of each conflicting workpiece is longer than the second grasping time window, the first priority sequence is still used as the grasping priority sequence. If the first grasping time window is equal to the second grasping time window, the robot calculates its own capability value. For example, normalization and weighted calculation can be performed based on the temperature and remaining battery power of each joint of the robot, and the weighted result is used as the capability value. Then, the robot sends its own capability value to neighboring robots and obtains the capability values ​​of the neighboring robots. Based on the comparison of capability values, the robot with the higher capability value performs the grasping.

[0126] This embodiment takes into account that workpieces with different curvatures will be affected to varying degrees by factors such as conveyor belt vibration, start-stop, and acceleration when moving on the conveyor belt. Therefore, this embodiment first generates feature vectors based on conveyor belt speed and curvature, and then accurately predicts the motion trajectory probability map. This solves the problem of difficulty in predicting the motion trend of complex curved workpieces when they move dynamically on the conveyor belt and have variable stacking postures. Furthermore, based on 3D point cloud data, the priority is calculated by weighting the grasping success rate and grasping cost, allowing the robot to prioritize grasping workpieces that are easy to grasp and close to the target. For example, workpieces with a high proportion of low curvature areas and fewer stacking layers will be grasped first, which reduces the grasping failure rate and avoids resource waste. At the same time, based on the robot's communication mechanism, combined with the confidence level of the motion trajectory probability map and the grasping time window, and further assisted by the robot's capability value, the robot with a longer grasping time window (the more stable the workpiece's motion state) will perform the grasping, avoiding the problem of collisions caused by multiple robots moving simultaneously.

[0127] Furthermore, this embodiment provides a step of determining the gripping point corresponding to each workpiece based on 3D point cloud data, 2D image data, and the pose corresponding to each workpiece, and completing the workpiece gripping based on the gripping point, including:

[0128] The contact area is determined based on the curvature and position of each data point in the 3D point cloud data;

[0129] The grasping point is determined based on the distance between each data point in the grasping area and the contact area, and the angle between the data point and the direction of the robot's end effector movement.

[0130] The surface state of the gripping point is obtained from the two-dimensional image, and the gripping force is obtained based on the surface state and the preset prediction model.

[0131] The workpiece is grasped based on the gripping force and the corresponding posture of each workpiece.

[0132] Specifically, when gripping workpieces according to the gripping priority sequence, for each workpiece, firstly, each data point in its corresponding second category is acquired. For any second category, the distance between each data point it includes and each data point in other second categories is calculated. The area where the distance is less than a threshold is recorded as the contact area. Based on this, the contact area between any two workpieces can be obtained. Then, the gripping point for each workpiece is selected according to the contact area. The gripping point must meet the following conditions: the distance from the contact area is less than a preset value, the angle between the normal vector and the robot's end effector motion direction is less than a preset angle, and the robot's gripping structure is satisfied. Next, based on the curvature, surface condition, and workpiece mass corresponding to the gripping point, a preset prediction model is invoked to obtain the gripping force. The surface condition includes clean surfaces, oily surfaces, etc. Finally, the robot's actuator end effector is controlled to complete the gripping based on the gripping force and the posture corresponding to each workpiece. The preset prediction model can be trained based on historical data. For example, for each type of workpiece, successful gripping samples of that type are obtained based on historical data, and the surface condition, curvature, mass, and gripping force corresponding to each sample are obtained. Finally, a preset prediction model corresponding to each workpiece type is trained based on the successful gripping samples. This embodiment does not limit the type of preset model. For example, a multilayer perceptron model and a random forest model can be used.

[0133] This embodiment determines the contact area based on the curvature and position of the 3D point cloud, defining a reliable spatial benchmark for selecting gripping points and accurately distinguishing the contact interface between the workpiece and surrounding objects. Simultaneously, this embodiment considers the presence of friction, pressure, or deformation interference in the contact area. If the gripping points are too close, it can easily lead to unstable gripping. If the angle between the robot's end effector movement direction and the gripping point's normal vector is too large, it can cause the workpiece to slide or the end effector to slip against the workpiece. Therefore, this embodiment sets selection conditions for gripping points to make the gripping more stable. Then, it uses 2D images to perceive the surface state of the gripping points and derives the gripping force through a preset prediction model, making the calculation of the gripping force more consistent with actual friction and load-bearing characteristics, thereby improving gripping efficiency.

[0134] Furthermore, embodiments of this application provide a robot adaptive gripping control system suitable for complex curved surface workpieces, including:

[0135] The acquisition module is used to acquire workpiece data within the working range. The workpiece data includes 3D point cloud data and 2D image data.

[0136] The recognition module is used to identify each workpiece and its corresponding posture based on 3D point cloud data.

[0137] The prediction module is used to obtain a motion trajectory probability map based on the speed of the conveyor belt. The motion trajectory probability map includes the predicted trajectory points and confidence levels for each workpiece.

[0138] The communication module is used to obtain the grasping priority sequence based on 3D point cloud data and motion trajectory probability map;

[0139] The grasping module is used to determine the grasping point for each workpiece based on the 3D point cloud data, 2D image data and the posture of each workpiece, and to complete the workpiece grasping based on the grasping point.

[0140] Specifically, the system is located on the robot, meaning that the robot in this embodiment can execute the above-mentioned grasping method on its own without the need for control by a central controller or server.

[0141] This application provides a robot adaptive grasping control method and system applicable to complex curved surface workpieces. By using 3D point cloud data and cluster analysis, it effectively identifies and segments stacked workpieces based on features such as the curvature of the workpieces. It also obtains a motion trajectory probability map by predicting the motion trajectory of the workpieces. Simultaneously, based on the communication mechanism between adjacent robots, it obtains the first priority sequence corresponding to the adjacent robots. When a conflict occurs with the first priority sequence corresponding to the adjacent robots, it generates a grasping priority sequence by combining the motion trajectory probability map. Ultimately, it improves grasping efficiency while accurately grasping the workpieces.

[0142] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0143] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0144] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0145] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A robot adaptive grasping control method suitable for complex curved surface workpieces, characterized in that, The robot adaptive grasping control method is executed by each robot, and the robot adaptive grasping control method includes: Acquire workpiece data within the working range, the workpiece data including three-dimensional point cloud data and two-dimensional image data; Identify each workpiece and its corresponding posture based on the three-dimensional point cloud data; The motion trajectory probability map is obtained based on the speed of the conveyor belt. The motion trajectory probability map includes the predicted trajectory points and confidence levels corresponding to each workpiece. The priority value corresponding to each workpiece is obtained based on the 3D point cloud data and a preset priority decision function, resulting in a first priority sequence. The first priority sequence is broadcast to neighboring robots, and a second priority sequence is received, where the second priority sequence is the first priority sequence corresponding to the neighboring robot. Conflicting workpieces are identified in the first and second priority sequences. The confidence level corresponding to the conflicting workpiece is obtained based on the motion trajectory probability map. A first grasping time window corresponding to the conflicting workpiece is obtained based on the confidence level. The first priority sequence is updated based on the first grasping time window to obtain a grasping priority sequence. Based on the 3D point cloud data, the 2D image data, and the posture corresponding to each workpiece, the gripping point corresponding to each workpiece is determined, and the workpiece gripping is completed based on the gripping point.

2. The robot adaptive gripping control method for complex curved surface workpieces according to claim 1, characterized in that, Identifying each workpiece and its corresponding pose based on the 3D point cloud data includes: Cluster analysis is performed on the three-dimensional point cloud data to obtain multiple first categories, and each first category corresponds to a workpiece. Based on the angle between the normal vectors corresponding to each first category, multiple point cloud clusters are obtained, and each workpiece is identified based on the point cloud clusters; Obtain the point cloud template corresponding to each workpiece, and obtain the posture corresponding to each workpiece based on the point cloud template.

3. The robot adaptive gripping control method for complex curved surface workpieces according to claim 2, characterized in that, Based on the angle between the normal vectors corresponding to each first category, multiple point cloud clusters are obtained, including: For each first category, the covariance matrix is ​​calculated based on each data point corresponding to the first category; The first category is updated based on the eigenvalues ​​in the covariance matrix and the curvature corresponding to each data point to obtain the second category. The covariance matrix is ​​calculated based on each data point corresponding to the second category. Based on the covariance matrix, eigenvalue decomposition is performed to obtain multiple normal vectors; Multiple point cloud clusters are obtained based on the angle between the normal vectors and a preset threshold.

4. The robot adaptive gripping control method for complex curved surface workpieces according to claim 3, characterized in that, The pose of each workpiece is obtained based on the point cloud template, including: For each second category, an iterative operation is performed. The iterative operation includes: for each data point in the current second category, determining the corresponding data point in the corresponding point cloud template, obtaining the transformation matrix corresponding to the current second category according to the preset objective function, determining whether the preset termination condition has been met, and if not, updating the current second category until the preset termination condition is met, and outputting the transformation matrix corresponding to the current second category. The transformation matrix represents the degree of deviation of the posture of the workpiece on the current conveyor belt from the standard posture in the point cloud template. The posture corresponding to each workpiece is obtained based on the transformation matrix.

5. The robot adaptive gripping control method for complex curved surface workpieces according to claim 1, characterized in that, The process of obtaining the motion trajectory probability map based on the speed of the conveyor belt includes: Based on the speed of the conveyor belt and the historical trajectory of each workpiece, the feature vector corresponding to each workpiece is obtained; Based on the feature vector and the preset model, the predicted trajectory points corresponding to each workpiece and the probability distribution corresponding to each predicted trajectory point are obtained; The confidence level is obtained based on the probability distribution, and the motion trajectory probability map is obtained based on the predicted trajectory points and the confidence level.

6. The robot adaptive gripping control method for complex curved surface workpieces according to claim 1, characterized in that, The first priority sequence is updated according to the first crawling time window to obtain the crawling priority sequence, including: Send the first grasping time window to the adjacent robot and receive the second grasping time window; If the first grasping time window is smaller than the second grasping time window, the conflicting workpiece is removed from the first priority sequence to obtain the grasping priority sequence; If the first grasping time window is equal to the second grasping time window, the first priority sequence is updated according to the robot's current capability value to obtain the grasping priority sequence.

7. The robot adaptive gripping control method for complex curved surface workpieces according to claim 1, characterized in that, Based on the 3D point cloud data, the 2D image data, and the pose corresponding to each workpiece, determine the gripping point corresponding to each workpiece, and complete the workpiece gripping based on the gripping point, including: The contact area is determined based on the curvature and position of each data point in the three-dimensional point cloud data; The grasping point is determined based on the distance between each data point in the grasping area and the contact area, and the angle between the data point and the direction of the robot's end effector movement. The surface state of the gripping point is obtained from the two-dimensional image, and the gripping force is obtained based on the surface state and the preset prediction model. The workpiece is grasped based on the grasping force and the posture corresponding to each workpiece.

8. A robot adaptive gripping control system suitable for complex curved surface workpieces, characterized in that, include: The acquisition module is used to acquire workpiece data within the working range, including three-dimensional point cloud data and two-dimensional image data. The recognition module is used to identify each workpiece and the posture corresponding to each workpiece based on the three-dimensional point cloud data. The prediction module is used to obtain a motion trajectory probability map based on the speed of the conveyor belt. The motion trajectory probability map includes the predicted trajectory points and confidence levels corresponding to each workpiece. The communication module is used to obtain the priority value corresponding to each workpiece based on the three-dimensional point cloud data and a preset priority decision function, thereby obtaining a first priority sequence; broadcast the first priority sequence to adjacent robots, and receive a second priority sequence, wherein the second priority sequence is the first priority sequence corresponding to the adjacent robot; and determine the conflicting workpieces in the first priority sequence and the second priority sequence. The confidence level corresponding to the conflicting workpiece is obtained based on the motion trajectory probability map; the first grasping time window corresponding to the conflicting workpiece is obtained based on the confidence level; the first priority sequence is updated based on the first grasping time window to obtain the grasping priority sequence. The grasping module is used to determine the grasping point corresponding to each workpiece based on the three-dimensional point cloud data, the two-dimensional image data and the posture corresponding to each workpiece, and to complete the workpiece grasping based on the grasping point.

Images