Interactive predictive learning and training method, device and computer equipment

Abstract

The present application relates to an interactive prediction learning and training method, device and computer equipment. The method comprises: obtaining multiple original feature points of an object to be clamped by the robot, and obtaining three-dimensional point cloud data of each of the multiple original feature points; based on the three-dimensional point cloud data of each of the multiple original feature points, selecting geometric feature points representing the geometric shape of the object to be clamped from the multiple original feature points; determining the associated features of each geometric feature point, and based on the associated features of each geometric feature point, predicting the robot's clamping probability for each geometric feature point; based on the clamping probability, selecting a target clamping point from each geometric feature point, and the target clamping point is used by the robot to clamp the object to be clamped. The use of this method can improve the probability of successful interaction between the robot and the object to be clamped.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an interactive predictive learning and training method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] In the field of industrial automation and robotics, the ability to interact with the environment is the key to achieving intelligence, and using a gripper to accurately grasp objects is one of its core requirements.

[0003] However, traditional robotic grasping methods rely on manually set rules and limited sensor data. They are not smart and efficient enough when faced with complex and changing environments, require a lot of human intervention, and have significant limitations in the data preparation stage. Summary of the invention

[0004] Based on this, it is necessary to provide an interaction prediction learning and training method, device, computer equipment, computer-readable storage medium and computer program product that can improve the probability of successful interaction between the robot and the object to be clamped in order to solve the above technical problems.

[0005] In a first aspect, the present application provides an interactive prediction learning and training method, comprising:

[0006] Acquire multiple original feature points of the object to be clamped by the robot, and acquire three-dimensional point cloud data of each of the multiple original feature points;

[0007] Based on the three-dimensional point cloud data of each of the plurality of original feature points, geometric feature points representing the geometric shape of the object to be clamped are selected from the plurality of original feature points;

[0008] Determine the associated features of each of the geometric feature points, and predict the gripping probability of the robot for each of the geometric feature points based on the associated features of each of the geometric feature points;

[0009] Based on the clamping probability, a target clamping point is screened out from each of the geometric feature points, and the target clamping point is used by the robot to clamp the object to be clamped.

[0010] In one embodiment, based on the three-dimensional point cloud data of each of the plurality of original feature points, selecting geometric feature points representing the geometric shape of the object to be clamped from the plurality of original feature points includes:

[0011] Based on the three-dimensional point cloud data of each of the multiple original feature points, the associated features of each original feature point are extracted; based on the associated features of each original feature point, geometric feature points representing the geometric shape of the object to be clamped are screened out from the multiple original feature points.

[0012] In one embodiment, obtaining a plurality of original feature points of an object to be gripped by a robot and obtaining three-dimensional point cloud data of each of the plurality of original feature points includes:

[0013] A depth image of an object to be gripped by a robot under multiple viewing angles captured by a camera is obtained, wherein the depth image includes depth information and position information corresponding to each pixel point; based on the depth information corresponding to each pixel point in the depth image and a preset filtering condition, a plurality of original feature points are screened out for each pixel point in the depth image; the shooting parameters of the camera and the posture matrix of the camera relative to a preset coordinate system are obtained, and based on the shooting parameters and the posture matrix, the depth information and position information corresponding to the original feature points are converted into three-dimensional point cloud data.

[0014] In one embodiment, based on the associated features of each original feature point, selecting geometric feature points representing the geometric shape of the object to be clamped from the plurality of original feature points includes:

[0015] For the three-dimensional point cloud data corresponding to each original feature point, feature extraction at multiple different scales is performed on the three-dimensional point cloud data corresponding to the original feature point to obtain the associated features corresponding to the original feature point; the associated features corresponding to each original feature point are fused to obtain the global features of the object to be clamped; based on the associated features and the global features, geometric feature points representing the geometric shape of the object to be clamped are screened out from the multiple original feature points.

[0016] In one of the embodiments, the target posture of the robot gripping the object to be gripped is determined based on the associated features of each target gripping point.

[0017] In one embodiment, the target position and posture includes a target posture; and determining the target position and posture of the robot gripping the object to be gripped according to the associated features of each target gripping point includes:

[0018] Select a reference clamping point from multiple target clamping points, and determine the position information of target clamping points other than the reference clamping point among the multiple target clamping points relative to the reference clamping point; determine the rotation information and translation information of the robot relative to the object to be clamped based on each piece of position information; and determine the target posture of the robot clamping the object to be clamped based on the rotation information and translation information of the robot relative to the object to be clamped.

[0019] In one embodiment, the target posture includes a target direction; and determining the target posture of the robot gripping the object to be gripped according to the associated features of each target gripping point includes:

[0020] A reference clamping point is selected from a plurality of target clamping points, and position information of target clamping points other than the reference clamping point among the plurality of target clamping points relative to the reference clamping point is determined; based on each piece of position information, direction information of the robot relative to the object to be clamped is determined; based on the direction information of the robot relative to the object to be clamped, a target direction of the robot clamping the object to be clamped is determined.

[0021] In one of the embodiments, a moving route of the robot for clamping the object to be clamped within a preset time period is obtained; based on the moving route, a moving mode of the robot within the preset time period is determined; when the moving mode is translation, the target posture and target clamping point of the robot for clamping the object to be clamped in the next time period of the preset time period are maintained; when the moving mode is rotation, the center of rotation and the rotation axis are determined based on the moving route; based on the rotation center and the rotation axis, the target posture and target clamping point of the object to be clamped are adjusted, and the adjusted target posture and target clamping point are used as the posture and clamping point of the robot for clamping the object to be clamped in the next time period of the preset time period.

[0022] In a second aspect, the present application also provides an interactive prediction learning and training device, comprising:

[0023] An acquisition module, used to acquire a plurality of original feature points of an object to be clamped by the robot, and to acquire three-dimensional point cloud data of each of the plurality of original feature points;

[0024] A first screening module, configured to screen out geometric feature points representing the geometric shape of the object to be clamped from the plurality of original feature points based on the three-dimensional point cloud data of each of the plurality of original feature points;

[0025] A prediction module, used to determine the associated features of each of the geometric feature points, and based on the associated features of each of the geometric feature points, predict the gripping probability of the robot for each of the geometric feature points;

[0026] The second screening module is used to screen out a target clamping point from each of the geometric feature points based on the clamping probability, and the target clamping point is used for the robot to clamp the object to be clamped.

[0027] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0028] Acquire multiple original feature points of the object to be clamped by the robot, and acquire three-dimensional point cloud data of each of the multiple original feature points;

[0029] Based on the three-dimensional point cloud data of each of the plurality of original feature points, geometric feature points representing the geometric shape of the object to be clamped are selected from the plurality of original feature points;

[0030] Determine the associated features of each of the geometric feature points, and predict the gripping probability of the robot for each of the geometric feature points based on the associated features of each of the geometric feature points;

[0031] Based on the clamping probability, a target clamping point is screened out from each of the geometric feature points, and the target clamping point is used by the robot to clamp the object to be clamped.

[0032] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0033] Acquire multiple original feature points of the object to be clamped by the robot, and acquire three-dimensional point cloud data of each of the multiple original feature points;

[0034] Based on the three-dimensional point cloud data of each of the plurality of original feature points, geometric feature points representing the geometric shape of the object to be clamped are selected from the plurality of original feature points;

[0035] Determine the associated features of each of the geometric feature points, and predict the gripping probability of the robot for each of the geometric feature points based on the associated features of each of the geometric feature points;

[0036] Based on the clamping probability, a target clamping point is screened out from each of the geometric feature points, and the target clamping point is used by the robot to clamp the object to be clamped.

[0037] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0038] Acquire multiple original feature points of the object to be clamped by the robot, and acquire three-dimensional point cloud data of each of the multiple original feature points;

[0039] Based on the three-dimensional point cloud data of each of the plurality of original feature points, geometric feature points representing the geometric shape of the object to be clamped are selected from the plurality of original feature points;

[0040] Determine the associated features of each of the geometric feature points, and predict the gripping probability of the robot for each of the geometric feature points based on the associated features of each of the geometric feature points;

[0041] Based on the clamping probability, a target clamping point is screened out from each of the geometric feature points, and the target clamping point is used by the robot to clamp the object to be clamped.

[0042] The interactive prediction learning and training method, device, computer equipment, computer readable storage medium and computer program product described above automatically extract feature points from three-dimensional space without relying on pre-set rules or limited sensor information, thereby reducing the need for human intervention by obtaining multiple original feature points of the object to be gripped by the robot and obtaining three-dimensional point cloud data of each of the multiple original feature points.

[0043] Based on the three-dimensional point cloud data of each of the multiple original feature points, the geometric feature points representing the geometric shape of the object to be clamped are selected from the multiple original feature points to accurately identify the key geometric features of the object, better deal with the sparsity and disorder of the point cloud data, and ensure that the system can operate stably under different conditions.

[0044] Determine the associated features of each geometric feature point, and based on the associated features of each geometric feature point, predict the gripping probability of the robot for each geometric feature point. Evaluate the interaction potential of each geometric feature point through interaction probability, and provide a scientific and reasonable decision-making basis for subsequent robot operations.

[0045] Based on the clamping probability, a target clamping point is selected from each of the geometric feature points, and the target clamping point is used by the robot to clamp the object to be clamped. By selecting the geometric feature point with the highest clamping probability as the final target clamping point, the accuracy of the clamping action is ensured. In addition, the number of unnecessary attempts is reduced, the success rate and efficiency of the robot's task execution are improved, and the time cost and resource consumption are reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A diagram of an application environment of an interactive prediction learning and training method in one embodiment;

[0048] Figure 2 A schematic diagram of a flow chart of an interactive prediction learning and training method in one embodiment;

[0049] Figure 3A schematic diagram of a framework of an interaction probability prediction network in one embodiment;

[0050] Figure 4 A schematic diagram of a framework of an interaction direction prediction network in one embodiment;

[0051] Figure 5 A schematic diagram of a framework of an interactive posture prediction network in one embodiment;

[0052] Figure 6 A schematic diagram of a flow chart of an interactive prediction learning and training method in another embodiment;

[0053] Figure 7 It is a structural block diagram of an interactive prediction learning and training device in one embodiment;

[0054] Figure 8 It is a structural block diagram of an interactive prediction learning and training device in another embodiment;

[0055] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] The robot clamping method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 is used to generate an interactive prediction learning and training request, and send the interactive prediction learning and training request to the server 104, so that the server 104 can filter out the target clamping point from each geometric feature point based on the clamping probability. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0058] In an exemplary embodiment, Figure 2 As shown, an interactive prediction learning and training method is provided, which is applied to Figure 1 The server 104 in the example is used as an example to illustrate, including the following steps 202 to 208. Among them:

[0059] Step 202: Acquire multiple original feature points of the object to be gripped by the robot, and acquire three-dimensional point cloud data of each of the multiple original feature points.

[0060] Herein, the robot in this application refers to an automated device or a robotic arm capable of performing gripping tasks. Optionally, the robot is equipped with sensors (such as a color depth camera) and actuators (such as a gripper) for sensing the environment and performing interactive actions such as grasping.

[0061] Objects to be gripped are target objects that the robot needs to grasp or hold. These objects can be physical items of various shapes and sizes, such as door handles, kitchen utensils, furniture parts, etc.

[0062] The original feature points refer to a series of initial point cloud data points obtained from the surface of the object to be clamped. Optionally, each original feature point represents a specific position on the surface of the object and carries color, depth and other possible spatial information about the position.

[0063] 3D point cloud data refers to a set of discrete points on the surface of the object to be clamped captured by a scanning device. Each point contains its coordinates (x, y, z) in 3D space and possible additional color information (R, G, B). These points together describe the geometric shape of the object.

[0064] Specifically, first ensure that the color depth camera (RGB-D camera) equipped with the robot works properly and is ready to capture data of the object to be gripped from different perspectives. And determine the target object to be grasped, that is, the object to be gripped, which can be any three-dimensional shape with joint parts or other interactive features.

[0065] Then use a color depth camera to scan the target object from multiple angles. Multi-view coverage is achieved by adjusting the angle of the camera or the mobile robot itself to obtain more comprehensive object information. Optionally, at each viewpoint, boundary clipping technology is applied to remove background interference, retain the effective area of ​​the target object, and ensure the accuracy of subsequent processing. The color image and depth map information obtained from each viewpoint are fused to construct a complete three-dimensional point cloud model containing color information. Optionally, if only single-view point cloud is used at the current stage, in order to support future multi-view tasks, multi-view point cloud data is directly used to better provide location information and adapt to complex geometric structures.

[0066] Finally, the sampling distribution is randomly initialized from the synthesized point cloud data, and several coordinate points (for example, 24) are sampled based on the distribution. These points will serve as preliminary candidates for original feature points. Using the multimodal large language model, the best few (for example, 3) sampling points are queried. Based on the selected best points, the new sampling distribution is fitted using maximum likelihood estimation. Continue the above sampling and optimization process multiple times (for example, five times) until the standard deviation of the sampling distribution converges and its mean stabilizes within a small range, thereby accurately locating the two-dimensional interaction position. Combined with the color depth map and the two-dimensional coordinates, the corresponding three-dimensional coordinates are inferred, and finally a set of accurate original feature points and their three-dimensional point cloud data are obtained.

[0067] Step 204 , based on the three-dimensional point cloud data of each of the multiple original feature points, select geometric feature points representing the geometric shape of the object to be clamped from the multiple original feature points.

[0068] Among them, geometry refers to the basis for characterizing the overall shape of the object to be clamped and its interactive parts. By identifying the geometric shape of the object, it is possible to determine which areas are suitable for clamping operations, such as cabinet drawer handles, door edges, etc.

[0069] Geometric feature points refer to specific locations that are selected from the original feature points and can significantly reflect the geometric shape characteristics of an object. These points are usually located at key locations on the surface of an object, such as corners, concave and convex changes, or other places that are representative of the object's shape.

[0070] Specifically, a neural network architecture suitable for processing disordered point cloud data is first selected as the feature extraction network for further analysis and screening of feature points.

[0071] Then, for each original feature point, the feature extraction network is used to calculate the local geometric properties around it, such as curvature, normal direction, neighborhood density, etc. These properties help to understand the surface morphology of the area where the point is located. Optionally, by setting a threshold or using a clustering algorithm, redundant points that obviously do not contain important geometric information are removed to reduce the amount of data for subsequent processing.

[0072] Next, the hierarchical sampling and grouping strategy preset by the feature extraction network is used to extract multi-scale local and global geometric features. This step aims to discover important structural elements on the surface of objects, such as edges, corners, planes, etc.

[0073] Finally, based on the above calculated local geometric characteristics and key structure identification results, a set of criteria is formulated to define which points can be regarded as geometric feature points. Optionally, these points should have the following characteristics: the first characteristic is that the geometric feature points can accurately describe the geometric shape of the object, and the second characteristic is that the geometric feature points change significantly on the surface of the object, which is suitable as the clamping candidate points, i.e., geometric feature points.

[0074] Step 206 , determining the associated features of each geometric feature point, and based on the associated features of each geometric feature point, predicting the gripping probability of the robot for each geometric feature point.

[0075] Among them, associated features refer to local or global attributes closely related to geometric feature points, which can help describe the spatial structure around the point and its position characteristics on the surface of the object. They provide necessary contextual information for subsequent interactive predictions. Optionally, associated features include local geometric characteristics, spatial relationships, texture information, and structural information. Local geometric characteristics can be curvature, normal direction, neighborhood density, etc., which reflect the surface morphology around the point. Spatial relationships represent the relative position between a geometric feature point and other nearby points, such as distance, angle, etc. Texture information can be color information, which is used to help distinguish different materials or surface types. Structural information such as planes, edges, corners, etc., is used to enhance the understanding of geometric shapes.

[0076] The gripping probability is an evaluation based on geometric feature points and their associated features to measure the likelihood of the robot successfully gripping at that point. It is a value between 0 and 1, indicating the effectiveness of the point as a gripping point.

[0077] Optionally, first obtain the weight values ​​and bias values ​​corresponding to each of the multiple clamping categories; for each geometric feature point, multiply the associated features of the geometric feature point and the weight values ​​corresponding to each of the multiple clamping categories to obtain the weighted values ​​corresponding to each of the multiple clamping categories; add the weighted values ​​and bias values ​​corresponding to each of the multiple clamping categories to obtain the activation values ​​corresponding to each of the multiple clamping categories. Then, convert the activation values ​​corresponding to each of the multiple clamping categories into the probability that the geometric feature point is classified into the corresponding multiple clamping categories; based on the probability that the geometric feature point is classified into the corresponding multiple clamping categories, determine the clamping probability of the robot for the geometric feature point.

[0078] Specifically, first identify different types of clamping operations (such as pulling, pushing, etc.) and assign a unique identifier to each clamping category. And obtain the weight value and bias value corresponding to each clamping category from a pre-trained model or other means. For each geometric feature point, extract its associated feature vector, which contains various attributes describing the point and its surrounding environment, such as curvature, normal direction, neighborhood density, etc. Multiply the associated feature vector of the geometric feature point with the weight value corresponding to each clamping category to obtain the weighted value of each clamping category. It can be understood that this step aims to quantify the correlation of geometric feature points under different clamping types.

[0079] Then, the corresponding bias value is added to the weighted value of each clamping category to adjust the final score. The role of the bias value is to ensure that the model can give reasonable prediction results even when the input feature is close to zero. The result of the above operation is the activation value of each clamping category, which represents the original prediction score without normalization. Use an appropriate activation function (such as the Softmax function) to convert the activation values ​​of all clamping categories into a probability distribution. The Softmax function can map multiple values ​​to the interval [0,1] and ensure that their sum is equal to 1. The purpose of this is to represent the relative possibility that the geometric feature point belongs to each clamping category.

[0080] Finally, based on the converted probability distribution, the gripping category with the highest probability is selected as the best prediction result. The probability value of this category is the gripping probability of the robot for this geometric feature point. Finally, the probability of each geometric feature point being classified into each gripping category is output, especially the highest probability value, as a measure of whether the system is suitable as a gripping point.

[0081] In one example, reference Figure 3 , input point cloud data into the interaction probability prediction network to represent the 3D point cloud of the object to be grasped. Extract the features of the point cloud through the encoder module. Generate action suggestions through the decoder module. Finally, output the interaction probability prediction result.

[0082] Step 208 , based on the clamping probability, select a target clamping point from each geometric feature point, and the target clamping point is used by the robot to clamp the object to be clamped.

[0083] Among them, the target gripping point refers to the geometric feature point that is finally determined after a series of analysis and evaluation and is most suitable for the robot to perform the gripping operation. Specifically, the target gripping point is the point or points with the highest gripping probability selected from all geometric feature points. These points are considered to be the best positions for the robot to successfully grasp the object to be gripped. The main purpose of selecting the target gripping point is to improve the success rate of the robot in performing the gripping task. By predicting the gripping probability of each geometric feature point and selecting the best candidate point from them, the reliability in actual operation can be significantly improved.

[0084] Specifically, first ensure that the gripping probability prediction for each geometric feature point has been completed. These probability values ​​reflect the probability of the robot successfully performing a gripping operation at that point. Define the criteria for selecting the target gripping point, such as setting a minimum gripping probability threshold (such as 0.5), or deciding to select the top N points with the highest probability as candidates.

[0085] Then, for all geometric feature points, check whether their clamping probability exceeds the preset threshold. Only those points that meet the conditions, that is, points with a higher success probability, are retained. For example, the method of selecting the top N points is adopted to sort all points from high to low according to the clamping probability, and select several points with the highest ranking as preliminary candidate clamping points.

[0086] Finally, specific gripping instructions are generated for the robot, instructing it how to move to the selected target gripping point and perform the gripping action. This may involve calculating precise spatial coordinates, rotation angles, and other necessary motion parameters. The relevant information of the target gripping point is passed to the robot's actuator (such as a robotic arm) so that it can accurately reach the specified position and implement the gripping task. The robot goes to the target gripping point according to the received instructions and attempts to perform the gripping operation.

[0087] Optionally, in order to further optimize the selection, a multimodal large language model can be used to assist in the evaluation of the initially selected candidate clamping points. Then, through multiple rounds of iterative visual optimization algorithms and other methods, the most likely successful clamping position is confirmed. Based on the preliminary screening results and possible multiple rounds of reasoning optimization, the various indicators of each candidate clamping point are comprehensively evaluated, including but not limited to clamping probability, geometric stability, ease of operation, etc. Finally, one or several positions that are most suitable as target clamping points are selected. These points not only have the highest clamping success rate, but also can provide stable and effective clamping effects in actual operations.

[0088] In one of the embodiments, based on the three-dimensional point cloud data of each of the multiple original feature points, the associated features of each original feature point are extracted; based on the associated features of each original feature point, geometric feature points representing the geometric shape of the object to be clamped are screened out from the multiple original feature points.

[0089] Among them, associated features refer to local or global attributes that are closely related to each original feature point, which can help describe the spatial structure around the point and its position characteristics on the surface of the object. Local geometric attributes can be curvature, normal direction, neighborhood density, etc., which reflect the surface morphology around the point. It can also be a spatial relationship, which is used to represent the relative position between a geometric feature point and other nearby points, such as distance, angle, etc. This helps to determine the importance of the point in the entire object structure. It can also be texture information, which is used to represent color information that helps distinguish different materials or surface types. It can also be structural information, such as planes, edges, corners, etc., to enhance the understanding of geometric shapes.

[0090] Specifically, a neural network architecture suitable for processing disordered point cloud data is first selected as a feature extraction network for further analysis and screening of feature points. For each original feature point, a two-dimensional or three-dimensional sampling distribution (such as a normal distribution) is initialized for subsequent iterative optimization. Optionally, the quality of point cloud data is enhanced through multi-view synthesis to provide richer spatial information.

[0091] The selected feature extraction network then processes the 3D point cloud data of each original feature point and extracts various local geometric features as associated features. The various local geometric features include but are not limited to:

[0092] Curvature (a measure of how curved a surface is), normal direction (a vector indicating the direction of a surface), neighborhood density (indicates how densely points are distributed around a point), texture information (including color information, which helps distinguish between different materials or surface types), and structural information (such as planes, edges, corners, etc.).

[0093] Next, based on the extracted associated features, a preliminary evaluation is made as to whether each original feature point can significantly characterize the geometric shape of the object to be clamped. Points located at the object boundary, where the concave and convex changes, or other structural mutations are usually more representative. Optionally, redundant points that obviously do not contain important geometric information can be removed by setting a threshold or using a clustering algorithm to reduce the amount of data to be processed later.

[0094] Next, randomly sample several points from the initialized sampling distribution and mark them in the image by labeling. Use a multimodal large language model to determine the best few (for example, 3) sampling points. Based on the selected best points, use maximum likelihood estimation to fit a new sampling distribution, and adjust its mean and variance to better focus on potential interaction areas. Continue the above sampling and optimization process multiple times (for example, five times) until the standard deviation of the sampling distribution converges and its mean stabilizes within a small range, thereby accurately locating the two-dimensional interaction position and converting it into three-dimensional coordinates.

[0095] Finally, combined with the results of iterative optimization, the geometric characteristics and clamping potential of each point are comprehensively evaluated, and the points that best represent the geometric shape of the object and have a higher clamping probability are selected as geometric feature points. Optionally, the screened and optimized geometric feature points are sorted into a list as input for the next step to determine the associated features and predict the clamping probability.

[0096] Screening out suitable geometric feature points in advance means that the robot can directly perform the clamping task on these points, avoiding the possibility of failure caused by attempting to clamp at an inappropriate position, thereby improving the success rate of the overall operation. At the same time, it also reduces the amount of data to be processed later, making the entire calculation process more efficient. Only the most valuable points are retained for further analysis, which speeds up the selection of feature points.

[0097] In one of the embodiments, a depth image of an object to be gripped by a robot captured by a camera at multiple viewing angles is obtained, the depth image including depth information and position information corresponding to each pixel; based on the depth information corresponding to each pixel in the depth image and preset filtering conditions, a plurality of original feature points are screened out for each pixel in the depth image; the shooting parameters of the camera and the posture matrix of the camera relative to a preset coordinate system are obtained, and based on the shooting parameters and the posture matrix, the depth information and position information corresponding to the original feature points are converted into three-dimensional point cloud data.

[0098] Among them, the camera refers to a device used in the robot system to obtain image data of the object to be clamped, which can be a color depth camera. This camera can not only capture the color information (RGB) of the object, but also record the depth information of each pixel at the same time. A depth image refers to a two-dimensional image captured by a camera, in which each pixel contains not only color information (if any), but also the depth value corresponding to the pixel. The depth information is the distance from each pixel to the camera, expressed as a numerical value, usually in millimeters or meters. It reflects the near and far distribution of the surface of the object in three-dimensional space. The position information is the two-dimensional coordinates (x, y) of each pixel in the image, and its exact position in three-dimensional space can be determined in combination with the depth information. The preset filtering conditions refer to some rules or standards set to filter out meaningful original feature points from the depth image. For example, minimum / maximum depth thresholds, edge detection results, curvature limits, etc. These conditions are used to remove background noise, non-target areas, and points that do not meet the requirements of geometric features, ensuring that the final selected points can accurately represent the geometric shape of the object to be clamped. Shooting parameters refer to a set of parameters that describe camera characteristics and imaging settings, such as focal length, aperture size, sensor resolution, etc. The purpose of the shooting parameters is to help convert the 2D coordinates in the image into 3D coordinates in the real world. The pose matrix is ​​a 4×4 matrix that represents the spatial pose of the camera relative to a preset coordinate system (such as the world coordinate system), which contains rotation and translation information. It is used to transform points in the local coordinate system (i.e. the camera coordinate system) to the global coordinate system for fusion with other sensor data or for combination with robot motion planning. Optionally, the pose matrix is ​​divided into a rotation part (3×3 submatrix) and a translation part (a 3D vector), which together describe the change in the pose of the camera.

[0099] Specifically, first ensure that the color depth camera (RGB-D camera) equipped with the robot is working properly. And select multiple viewing angles at different angles to ensure that all surface details of the object to be clamped can be fully captured. Use the camera to shoot the object to be clamped from multiple selected viewing angles to obtain a series of depth images. Each depth image not only contains color information (if any), but also records the depth information corresponding to each pixel (the distance from each pixel to the camera, expressed as a numerical value, reflecting the far and near distribution of the object surface in three-dimensional space) and position information (the two-dimensional coordinates (x, y) of each pixel in the image, combined with the depth information, can determine its exact position in three-dimensional space).

[0100] Then, based on the depth information of each pixel in the depth image, combined with preset filtering conditions (such as minimum / maximum depth thresholds, edge detection results, curvature limits, etc.), background noise, non-target areas, and points that do not meet the geometric feature requirements are removed. Through the above filtering process, multiple original feature points are screened out in the depth image, which are the basis for subsequent processing.

[0101] Next, obtain the camera's shooting parameters, including focal length, aperture size, sensor resolution, etc. These parameters are very important for correcting lens distortion and calculating the actual size of the object. And obtain the camera's attitude matrix relative to the preset coordinate system (such as the world coordinate system). This matrix describes the camera's spatial attitude and consists of two parts: rotation and translation. It is used to transform points in the local coordinate system to the global coordinate system.

[0102] Finally, the original feature points selected are converted from the image coordinate system to the three-dimensional space coordinate system using the camera's shooting parameters and attitude matrix. Specifically, the depth information and two-dimensional position information of each point are converted into three-dimensional coordinates (x, y, z) to form a complete three-dimensional point cloud data. Optionally, the three-dimensional coordinate information of each original feature point is finally output to form a complete three-dimensional point cloud model, which provides a basis for subsequent geometric feature point extraction, associated feature calculation and clamping probability prediction.

[0103] Since the depth information and position information corresponding to each pixel point are obtained through multi-view depth images, the surface details of the object are fully captured. This enables the robot to understand the geometric shape of the object more accurately and select the best gripping point. And by removing background noise, non-target areas and points that do not meet the geometric feature requirements through preset filtering conditions, the quality of subsequent processing data is guaranteed, making the system more robust. And by using the camera's shooting parameters and posture matrix, the original feature points are converted into three-dimensional point cloud data, providing high-precision three-dimensional reconstruction results, which is helpful for further analysis of the geometric characteristics of the object.

[0104] In one of the embodiments, for the three-dimensional point cloud data corresponding to each original feature point, feature extraction is performed on the three-dimensional point cloud data corresponding to the original feature point at multiple different scales to obtain associated features corresponding to the original feature point; the associated features corresponding to each original feature point are fused to obtain the global features of the object to be clamped; based on the associated features and the global features, geometric feature points representing the geometric shape of the object to be clamped are screened out from the multiple original feature points.

[0105] Among them, different scales refer to the extraction of geometric features from different levels from local to global when processing 3D point cloud data. These scales can be changes in spatial resolution or changes in the size of the analysis window. Global features refer to a set of parameters that describe the overall geometric characteristics of the entire object to be clamped, which comprehensively reflects the main morphological and structural information of the object. It can be understood that combined with local correlation features, global features help to screen out those positions that are most suitable as geometric feature points, thereby guiding the robot to select the best clamping points.

[0106] Specifically, first ensure that multiple original feature points of the robot's object to be gripped and their corresponding 3D point cloud data have been obtained. These data contain the spatial coordinates (x, y, z) of each point and possible color information. And select a neural network architecture suitable for processing unordered point cloud data as the feature extraction network for further analysis and screening of feature points.

[0107] Then, multiple different scale levels are determined, such as local scale, medium scale and global scale. Each scale level corresponds to a different analysis window size or resolution. For the three-dimensional point cloud data corresponding to each original feature point, feature extraction is performed at multiple different scales to obtain the associated features corresponding to the point, including extracting local geometric characteristics in a smaller neighborhood range for local scale, such as curvature, normal direction, neighborhood density, etc., and extracting macroscopic geometric characteristics in the entire object or larger area for global scale, such as overall shape, symmetry, etc.

[0108] Next, the associated features extracted from each original feature point at different scales are fused to form a comprehensive feature vector. This can be achieved by weighted averaging, splicing, etc. Based on the comprehensive feature vector of all original feature points, the global features of the entire object to be clamped are calculated and summarized. Statistical methods (such as mean value, variance) or advanced algorithms (such as PCA principal component analysis, shape matching) can be used to extract key global feature parameters.

[0109] Finally, for each original feature point, its own associated features and the global features of the entire object are combined to evaluate whether the point can significantly represent the geometric shape of the object to be clamped. According to the set criteria (such as a specific threshold or ranking mechanism), the points that best represent the geometric shape of the object are selected from all the original feature points as geometric feature points. These points are usually located where the surface of the object changes significantly, such as edges, corners or other structural mutations, and are suitable as ideal locations for clamping operations.

[0110] Since multi-scale feature extraction enhances the system's ability to understand geometric features at different scales, it can obtain relatively complete point cloud data even in the face of complex geometric structures or occlusions, improving the system's adaptability and robustness. And the local features of each original feature point are fused with the global features to form a more complete and accurate feature representation, providing a solid foundation for subsequent screening. The fused features are more resistant to the influence of local noise because they contain information from a larger range, improving the stability and reliability of the features.

[0111] In one of the embodiments, the target posture of the robot gripping the object to be gripped is determined based on the associated features of each target gripping point.

[0112] Among them, the target posture refers to the spatial position and posture that the robot's end effector (such as a manipulator or gripper) needs to reach in order to successfully clamp the object to be clamped.

[0113] Specifically, based on the associated features of each target gripping point (such as curvature, normal direction, neighborhood density, etc.), the spatial structure and its geometric properties around the point are evaluated. Understandably, these features help determine the most appropriate approach path and gripping method to ensure that the robot can stably and effectively grasp the object. Based on the position information and associated features of the target gripping point, the optimal path of the robot end effector from the current position to the target pose is calculated. Based on the above analysis results, one or a group of target poses are set for each target gripping point. These poses must not only ensure that the robot can accurately reach the gripping point, but also ensure the successful implementation of the gripping action.

[0114] In one example, when facing a cabinet, if the target gripping point is on the drawer handle, the target pose includes the specific position that the robot should reach (near the center of the handle) and its angle relative to the handle (perpendicular to the handle surface). This ensures that the robot can smoothly grasp the handle and open the drawer.

[0115] Since choosing the right target pose is crucial to ensure that the robot can complete the gripping task efficiently and accurately, it not only reduces the number of unnecessary attempts, but also improves the efficiency of the entire automation process.

[0116] In one of the embodiments, a reference clamping point is selected from a plurality of target clamping points, and position information of target clamping points other than the reference clamping point among the plurality of target clamping points relative to the reference clamping point is determined; based on each position information, rotation information and translation information of the robot relative to the object to be clamped is determined; based on the rotation information and translation information of the robot relative to the object to be clamped, the target posture of the robot clamping the object to be clamped is determined.

[0117] The reference gripping point refers to a specific point selected from multiple target gripping points as the reference for the positional relationship of other target gripping points. This point is usually located at a key part of the object, such as a handle or edge, and can provide a stable reference. It is used to provide a relative position reference for other target gripping points, simplifying the robot motion planning process. By fixing the reference point, it can ensure that all calculations are based on a consistent coordinate system to avoid error accumulation.

[0118] Rotation information refers to the rotation angle or direction of the robot end effector relative to the object to be gripped in three-dimensional space. It determines how the end effector is aligned with the surface of the object to ensure correct contact and grasping.

[0119] Translation information refers to the linear displacement information of the robot end effector relative to the object to be gripped in three-dimensional space. It specifies the specific distance and direction that the end effector needs to move. Optionally, it is represented by a three-dimensional vector (x, y, z), where each component corresponds to the displacement on a coordinate axis.

[0120] The target posture refers to the target posture that combines position and translation information, that is, the spatial position and posture that the robot end effector needs to achieve in order to successfully grip the object to be gripped. The spatial position refers to the specific coordinates (x, y, z) of the robot end effector in three-dimensional space. The posture refers to the direction or angle of the end effector, which can be expressed by a rotation matrix, quaternion or Euler angle.

[0121] Specifically, first evaluate the geometric characteristics and stability of each point from multiple target gripping points, and select a position that is most suitable as a reference gripping point. This point is usually located at a key part of the object (such as a handle, edge, etc.) and can provide a stable reference. The selected reference gripping point is determined as the basis for the positional relationship of all other target gripping points. This step establishes the basis of the relative coordinate system.

[0122] Then, for each target clamping point, calculate its position information relative to the reference clamping point. This includes two aspects: the first aspect is the linear distance, that is, calculating the linear distance of each point in three-dimensional space. The second aspect is the direction vector, that is, determining the direction vector of each point relative to the reference point. The relative position information obtained by the above calculation is recorded to form a relationship table describing the relative positions between all target clamping points. This helps to simplify subsequent calculations and ensure that all operations are based on a consistent coordinate system.

[0123] Next, based on the recorded positional relationships, analyze the specific rotations and translations that the robot end effector needs to make to ensure that it can properly approach and grasp the target gripping point. Optionally, use a rotation matrix, quaternion, or Euler angle to represent the rotation angle and direction that the robot end effector needs to make relative to the object to be gripped. This determines how the end effector is aligned with the target gripping point. Determine the specific distance and direction that the robot end effector needs to move relative to the object to be gripped, represented by a three-dimensional vector (x, y, z). This clarifies the exact position that the end effector should reach.

[0124] Finally, the calculated rotation information and translation information are combined to form a complete motion instruction to instruct the robot end effector on how to adjust its posture to complete the gripping task. Finally, the target posture of the robot gripping the object to be gripped is determined. This includes the specific position (x, y, z) that the end effector should reach and its angle relative to the object (such as pitch, roll, yaw). Ensure the successful implementation of the gripping action.

[0125] In one of the embodiments, a moving route of the robot for clamping an object to be clamped within a preset time period is obtained; based on the moving route, a moving mode of the robot within the preset time period is determined; when the moving mode is translation, the target posture and target clamping point of the robot for clamping the object to be clamped in the next time period of the preset time period are maintained; when the moving mode is rotation, the center of rotation and the rotation axis are determined based on the moving route; based on the center of rotation and the rotation axis, the target posture and target clamping point of the object to be clamped are adjusted, and the adjusted target posture and target clamping point are used as the posture and clamping point of the robot for clamping the object to be clamped in the next time period of the preset time period.

[0126] Specifically, the spatial structure and its geometric characteristics around the target gripping point are first evaluated based on the associated features of the target gripping point (such as curvature, normal direction, etc.). The robot's movement mode (translation or rotation) is determined by calculating the optimal path from the current position of the robot end effector to the target pose and combining the position information of the target gripping point.

[0127] Then, if the movement mode is translation, the robot maintains the current target posture and target gripping point unchanged in the next time period. If the movement mode is rotation, the rotation center and rotation axis are determined based on the movement route by analyzing the relative position relationship between the target gripping points. The target posture and target gripping point of the object to be gripped are adjusted according to the rotation center and axis to ensure that the robot can correctly align with the target gripping point. The adjusted target posture and target gripping point are used as the new position and posture of the robot to grip the object to be gripped in the next time period.

[0128] Point cloud data and geometric feature points are used to determine the movement route and corresponding movement mode of the robot when clamping the object to be clamped. In the case of translation, the original target posture and clamping point are directly maintained; in the case of rotation, the rotation center and axis need to be further calculated to adjust the target posture and clamping point, so as to ensure that the robot can accurately complete the clamping task in a changing environment.

[0129] In one example, reference Figure 4 , the input to the interactive posture prediction network is point cloud data, which represents the 3D point cloud of the object to be grasped. The features of the point cloud are extracted through the encoder module. The extracted point cloud features are intermediately processed through the transformation layer. The action suggestions are generated through the decoder module. Finally, the interactive posture prediction result is output.

[0130] Since a stable reference gripping point is selected as the benchmark, it provides a reliable position reference for all other points, ensuring the consistency and accuracy of the entire calculation process. Points located at key parts of the object (such as handles, edges, etc.) are usually selected as references. These points can provide better geometric stability and are conducive to subsequent precise operations. The final target posture combines rotation and translation information, provides a specific arrival position and alignment method for the target gripping point, and ensures the successful implementation of the gripping operation.

[0131] In one of the embodiments, a reference clamping point is selected from a plurality of target clamping points, and position information of target clamping points other than the reference clamping point among the plurality of target clamping points relative to the reference clamping point is determined; based on each position information, direction information of the robot relative to the object to be clamped is determined; based on the direction information of the robot relative to the object to be clamped, the target direction of the robot clamping the object to be clamped is determined.

[0132] The target direction refers to the direction or angle that the robot end effector needs to reach relative to the object in three-dimensional space in order to successfully grip the object. It determines how the end effector is aligned with the target gripping point to ensure that it can properly contact and grasp the object.

[0133] Direction information describes the rotation angle or direction of the robot end effector relative to the object to be gripped in three-dimensional space. It specifies the specific rotation that the end effector needs to make to align with the target gripping point.

[0134] Specifically, the geometric characteristics and stability of each point are evaluated from multiple target clamping points, and a position that is most suitable as a reference clamping point is selected. This point is usually located at a key part of the object (such as a handle, edge, etc.) and can provide a stable reference. For each target clamping point, its position information relative to the reference clamping point is calculated, including: linear distance, that is, calculating the linear distance of each point in three-dimensional space, and direction vector, that is, determining the direction vector of each point relative to the reference point.

[0135] Then, based on the relative position information of each target clamping point and their associated features (such as curvature, normal direction, etc.), the geometric relationship of each point is analyzed to determine the most appropriate approach path and clamping method. Use rotation matrices, quaternions, or Euler angles to represent the rotation angle and direction required for the robot end effector relative to the object to be clamped. Specifically: The rotation matrix is ​​a 3x3 matrix that directly represents the rotation transformation. Quaternions are vectors composed of four values ​​and are often used to represent rotations in three-dimensional space. Euler angles are three angle values ​​(pitch, roll, yaw), which describe rotations around three axes respectively.

[0136] Finally, based on the above analysis results, the direction information of the robot relative to the object to be clamped is determined, that is, the specific rotation that the robot end effector needs to perform to correctly align with the target clamping point. Based on the calculated direction information, the collaborative work between all target clamping points is comprehensively considered to finally determine the target direction of the robot clamping the object to be clamped. It can be understood that the final determination of the target direction of the robot clamping the object to be clamped includes the angle (such as pitch, roll, yaw) that the end effector should be relative to the object to ensure the successful implementation of the clamping action. Adjust the position and posture of the robot end effector according to the set target direction to prepare for the clamping operation.

[0137] In one example, reference Figure 5 , the input to the interaction direction prediction network is point cloud data, which represents the 3D point cloud of the object to be grasped. The features of the point cloud are extracted through the encoder module. The extracted point cloud features are processed intermediately through the transformation layer. The action suggestions are generated through the decoder module. Finally, the interaction direction prediction result is output.

[0138] By accurately setting the target direction, the robot end effector is ensured to contact and grasp the object at the best angle, which enables the robot to handle objects of different shapes and sizes and reduce the influence of external factors.

[0139] In an exemplary embodiment, Figure 6 As shown, it includes steps 602 to 610. Among them:

[0140] Step 602, obtaining a depth image of the object to be gripped by the robot at multiple viewing angles captured by a camera, wherein the depth image includes depth information and position information corresponding to each pixel; based on the depth information corresponding to each pixel in the depth image and a preset filtering condition, filtering out multiple original feature points for each pixel in the depth image; obtaining the shooting parameters of the camera and the posture matrix of the camera relative to a preset coordinate system, and based on the shooting parameters and the posture matrix, converting the depth information and position information corresponding to the original feature points into three-dimensional point cloud data;

[0141] Step 604: extracting the associated features of each original feature point based on the three-dimensional point cloud data of each of the multiple original feature points; performing feature extraction on multiple different scales on the three-dimensional point cloud data corresponding to each original feature point to obtain the associated features corresponding to the original feature point; fusing the associated features corresponding to each original feature point to obtain the global features of the object to be clamped; and selecting geometric feature points representing the geometric shape of the object to be clamped from the multiple original feature points based on the associated features and the global features;

[0142] Step 606, determining the associated features of each geometric feature point, and based on the associated features of each geometric feature point, predicting the gripping probability of the robot for each geometric feature point;

[0143] Step 608, based on the clamping probability, select a target clamping point from each geometric feature point, and the target clamping point is used by the robot to clamp the object to be clamped;

[0144] Step 610, select a reference clamping point from multiple target clamping points, determine the position information of target clamping points other than the reference clamping point among the multiple target clamping points, relative to the reference clamping point; determine the rotation information and translation information of the robot relative to the object to be clamped based on each position information; determine the target posture of the robot clamping the object to be clamped based on the rotation information and translation information of the robot relative to the object to be clamped; select a reference clamping point from multiple target clamping points, determine the position information of target clamping points other than the reference clamping point among the multiple target clamping points, relative to the reference clamping point; determine the direction information of the robot relative to the object to be clamped based on each position information; determine the target direction of the robot clamping the object to be clamped based on the direction information of the robot relative to the object to be clamped.

[0145] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0146] Based on the same inventive concept, the embodiment of the present application also provides an interactive prediction learning and training device for implementing the interactive prediction learning and training method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more interactive prediction learning and training device embodiments provided below can refer to the limitations of the interactive prediction learning and training method above, and will not be repeated here.

[0147] In an exemplary embodiment, Figure 7 As shown, an interactive prediction learning and training device 700 is provided, comprising: an acquisition module 702, a first screening module 704, a prediction module 706 and a second screening module 708, wherein:

[0148] An acquisition module 702 is used to acquire a plurality of original feature points of an object to be clamped by the robot, and to acquire three-dimensional point cloud data of each of the plurality of original feature points;

[0149] A first screening module 704 is used to screen out geometric feature points representing the geometric shape of the object to be clamped from the multiple original feature points based on the three-dimensional point cloud data of each of the multiple original feature points;

[0150] A prediction module 706 is used to determine the associated features of each geometric feature point, and based on the associated features of each geometric feature point, predict the gripping probability of the robot for each geometric feature point;

[0151] The second screening module 708 is used to screen out target clamping points from various geometric feature points based on the clamping probability, and the target clamping points are used by the robot to clamp the object to be clamped.

[0152] In one embodiment, the first screening module 704 is used to extract the associated features of each original feature point based on the three-dimensional point cloud data of each of the multiple original feature points; based on the associated features of each original feature point, the geometric feature points representing the geometric shape of the object to be clamped are screened out from the multiple original feature points.

[0153] In one embodiment, the acquisition module 702 is used to obtain a depth image of an object to be clamped by the robot captured by a camera at multiple perspectives, the depth image including depth information and position information corresponding to each pixel; based on the depth information corresponding to each pixel in the depth image and a preset filtering condition, a plurality of original feature points are screened out for each pixel in the depth image; the shooting parameters of the camera and the posture matrix of the camera relative to a preset coordinate system are obtained, and based on the shooting parameters and the posture matrix, the depth information and position information corresponding to the original feature points are converted into three-dimensional point cloud data.

[0154] In one embodiment, the first screening module 704 is used to perform feature extraction at multiple scales on the three-dimensional point cloud data corresponding to each original feature point to obtain associated features corresponding to the original feature point; fuse the associated features corresponding to each original feature point to obtain the global features of the object to be clamped; and screen out geometric feature points that represent the geometric shape of the object to be clamped from multiple original feature points based on the associated features and the global features.

[0155] In one of the embodiments, the robot gripping device further includes a determination module 710 for determining a target posture of the robot gripping the object to be gripped according to the associated features of each target gripping point.

[0156] In one embodiment, the determination module 710 is used to select a reference clamping point from multiple target clamping points, determine the position information of the target clamping points other than the reference clamping point among the multiple target clamping points relative to the reference clamping point; determine the rotation information and translation information of the robot relative to the object to be clamped based on each position information; and determine the target posture of the robot clamping the object to be clamped based on the rotation information and translation information of the robot relative to the object to be clamped.

[0157] In one embodiment, the determination module 710 is used to select a reference clamping point from multiple target clamping points, determine the position information of the target clamping points other than the reference clamping point among the multiple target clamping points relative to the reference clamping point; determine the direction information of the robot relative to the object to be clamped based on each position information; and determine the target direction of the robot clamping the object to be clamped based on the direction information of the robot relative to the object to be clamped.

[0158] In another embodiment, if Figure 8 As shown, Figure 8 7 is a structural block diagram of an interactive prediction learning and training device 700 in another embodiment, comprising: an acquisition module 702, a first screening module 704, a prediction module 706 and a second screening module 708. The interactive prediction learning and training device further comprises a determination module 710, which is used to select a reference clamping point from a plurality of target clamping points, determine the position information of the target clamping points other than the reference clamping point among the plurality of target clamping points relative to the reference clamping point; determine the rotation information and translation information of the robot relative to the object to be clamped according to each position information; determine the target posture of the robot clamping the object to be clamped based on the rotation information and translation information of the robot relative to the object to be clamped, and select a reference clamping point from a plurality of target clamping points, determine the position information of the target clamping points other than the reference clamping point among the plurality of target clamping points relative to the reference clamping point; determine the direction information of the robot relative to the object to be clamped according to each position information; and determine the target direction of the robot clamping the object to be clamped based on the direction information of the robot relative to the object to be clamped.

[0159] Each module in the above interactive prediction learning and training device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0160] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Fig. 9As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data related to interactive predictive learning and training. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an interactive predictive learning and training method is implemented.

[0161] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0162] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0163] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0164] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0165] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0166] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0167] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0168] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An interactive prediction learning and training method, characterized in that: The method comprises: Acquire multiple original feature points of the object to be clamped by the robot, and acquire three-dimensional point cloud data of each of the multiple original feature points; Based on the three-dimensional point cloud data of each of the multiple original feature points, extract the associated features of each original feature point; for the three-dimensional point cloud data corresponding to each original feature point, perform feature extraction on multiple different scales on the three-dimensional point cloud data corresponding to the original feature point to obtain the associated features corresponding to the original feature point; fuse the associated features corresponding to each original feature point to obtain the global features of the object to be clamped; based on the associated features and the global features, select geometric feature points that characterize the geometric shape of the object to be clamped from the multiple original feature points; Determine the associated features of each of the geometric feature points, and predict the gripping probability of the robot for each of the geometric feature points based on the associated features of each of the geometric feature points; Based on the clamping probability, a target clamping point is screened out from each of the geometric feature points, and the target clamping point is used by the robot to clamp the object to be clamped.

2. The method according to claim 1, characterized in that The step of obtaining a plurality of original feature points of the object to be clamped by the robot and obtaining three-dimensional point cloud data of each of the plurality of original feature points includes: Acquire a depth image of an object to be clamped by the robot at multiple viewing angles captured by a camera, wherein the depth image includes depth information and position information corresponding to each pixel point; Based on the depth information corresponding to each pixel in the depth image and the preset filtering condition, a plurality of original feature points are filtered out from each pixel in the depth image; The shooting parameters of the camera and the posture matrix of the camera relative to a preset coordinate system are obtained, and based on the shooting parameters and the posture matrix, the depth information and position information corresponding to the original feature points are converted into three-dimensional point cloud data.

3. The method according to claim 1, characterized in that The method further comprises: The target posture of the robot for clamping the object to be clamped is determined according to the associated features of each target clamping point.

4. The method according to claim 3, characterized in that The target position includes a target posture; Determining the target posture of the robot clamping the object to be clamped according to the associated features of each target clamping point includes: Selecting a reference clamping point from a plurality of target clamping points, and determining position information of target clamping points other than the reference clamping point among the plurality of target clamping points relative to the reference clamping point; Determining rotation information and translation information of the robot relative to the object to be clamped according to each of the position information; Based on the rotation information and translation information of the robot relative to the object to be clamped, a target posture of the robot clamping the object to be clamped is determined.

5. The method according to claim 3, characterized in that: The target posture includes a target direction; and determining the target posture of the robot clamping the object to be clamped according to the associated features of each target clamping point includes: Selecting a reference clamping point from a plurality of target clamping points, and determining position information of target clamping points other than the reference clamping point among the plurality of target clamping points relative to the reference clamping point; Determining direction information of the robot relative to the object to be clamped according to each of the position information; Based on the direction information of the robot relative to the object to be clamped, a target direction for the robot to clamp the object to be clamped is determined.

6. The method according to claim 3, characterized in that The method further comprises: Acquiring a moving route of the robot for clamping the object to be clamped within a preset time period; Based on the moving route, determining a moving mode of the robot within the preset time period; When the movement mode is translation, maintaining the target posture and target clamping point of the robot for clamping the object to be clamped in the next time period of the preset time period; In the case where the moving mode is rotation, determining the rotation center and the rotation axis based on the moving route; Based on the rotation center and the rotation axis, the target posture and target clamping point of the object to be clamped are adjusted, and the adjusted target posture and target clamping point are used as the posture and clamping point of the robot clamping the object to be clamped in the next time period of the preset time period.

7. An interactive prediction learning and training device, characterized in that: The device comprises: An acquisition module, used to acquire a plurality of original feature points of an object to be clamped by the robot, and to acquire three-dimensional point cloud data of each of the plurality of original feature points; The first screening module is used to extract the associated features of each original feature point based on the three-dimensional point cloud data of each of the multiple original feature points; for the three-dimensional point cloud data corresponding to each original feature point, perform feature extraction on multiple different scales on the three-dimensional point cloud data corresponding to the original feature point to obtain the associated features corresponding to the original feature point; fuse the associated features corresponding to each original feature point to obtain the global features of the object to be clamped; and based on the associated features and the global features, screen out geometric feature points that characterize the geometric shape of the object to be clamped from the multiple original feature points; A prediction module, used to determine the associated features of each of the geometric feature points, and based on the associated features of each of the geometric feature points, predict the gripping probability of the robot for each of the geometric feature points; The second screening module is used to screen out a target clamping point from each of the geometric feature points based on the clamping probability, and the target clamping point is used for the robot to clamp the object to be clamped.

8. The device according to claim 7, characterized in that The acquisition module is used to acquire a depth image of the object to be clamped by the robot at multiple viewing angles captured by a camera, wherein the depth image includes depth information and position information corresponding to each pixel point; Based on the depth information corresponding to each pixel in the depth image and the preset filtering condition, a plurality of original feature points are filtered out from each pixel in the depth image; The shooting parameters of the camera and the posture matrix of the camera relative to a preset coordinate system are obtained, and based on the shooting parameters and the posture matrix, the depth information and position information corresponding to the original feature points are converted into three-dimensional point cloud data.

9. The device according to claim 7, characterized in that The device also includes a determination module, which is used to determine the target posture of the robot clamping the object to be clamped according to the associated features of each target clamping point.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

Images