Intelligent robot loading method and system

Abstract

The application provides a kind of intelligent robot feeding method and system, the application relates to humanoid robot technical field, the method includes obtaining test rubber product image;Based on test rubber product image, using initial stress point model determines the multiple initial stress points of test rubber product;Based on the multiple initial stress points of test rubber product determines three finger stress points;Based on the multiple initial stress points of test rubber product, three finger stress points generate multiple sets of grabbing scheme, and each set of grabbing scheme includes five finger stress points;Obtain the grabbing video of each set of grabbing scheme;Based on the grabbing video of each set of grabbing scheme determines the remaining two finger target grabbing point;Based on three finger stress points, the remaining two finger target grabbing point controls five finger robot and feeds the remaining rubber product and grabs, the method can efficiently and accurately determine the adaptive grabbing point of rubber product to realize robot stable feeding and grabbing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of humanoid robots, in particular to an intelligent robot feeding method and system. BACKGROUND

[0002] Rubber products are widely used in the fields of automobiles, electronics, machinery, etc. due to their elasticity, wear resistance and sealing advantages. As a key link in the production and assembly of rubber products, the stability and precision of robot feeding directly affect the efficiency of the production line and the quality of the products. However, the physical properties of rubber products pose many difficulties for robot grasping. Rubber products often have irregular shapes, and some products also have surface irregularities, wrinkles and other detailed features. Traditional fixed-point grasping methods cannot adapt to different shapes of products and may cause grasping position deviation. In addition, rubber materials have strong elasticity and flexibility. If the force point is not selected properly during grasping, the product may deform, affecting the subsequent processing precision, or it may slip due to insufficient friction, causing material loss or production interruption. In the traditional production mode, robot feeding relies on fixed programs preset by humans. For different specifications and shapes of rubber products, technicians need to manually adjust the grasping parameters and points, which is not only time-consuming and labor-intensive, but also easily affected by the experience level and subjective factors of technicians, making it difficult to ensure that the grasping scheme is in the optimal state.

[0003] Therefore, how to efficiently and accurately determine the adaptive grasping point of rubber products to achieve stable robot feeding and grasping is a problem to be solved. SUMMARY

[0004] The technical problem solved by the present application is how to efficiently and accurately determine the adaptive grasping point of rubber products to achieve stable robot feeding and grasping.

[0005] According to a first aspect, the present application provides an intelligent robot feeding method, comprising: acquiring a test rubber product image; determining a plurality of initial force points of the test rubber product using an initial force point model based on the test rubber product image; determining a three-finger force point based on the plurality of initial force points of the test rubber product; generating a plurality of sets of grasping schemes based on the plurality of initial force points of the test rubber product and the three-finger force point, each set of grasping schemes including five-finger force points; controlling a five-finger robot to grasp the test rubber product multiple times based on each set of grasping schemes and acquiring grasping videos of each set of grasping schemes; determining the remaining two-finger target grasping points based on the grasping videos of each set of grasping schemes; and controlling the five-finger robot to feed and grasp the remaining rubber product based on the three-finger force point and the remaining two-finger target grasping points.

[0006] In a possible implementation, the determining the three-finger force point based on the plurality of initial force points of the test rubber product comprises: generating a plurality of initial force point sets based on the plurality of initial force points of the test rubber product, each initial force point set including three initial three-finger force points; controlling the five-finger robot to perform a plurality of grasping operations on the test rubber product using the three fingers respectively based on each initial force point set, and obtaining a grasping video of each initial force point set, the three fingers being the thumb, the index finger, and the middle finger respectively; and determining the three-finger force point based on the grasping video of each initial force point set.

[0007] In a possible implementation, the determining the target grasping points of the remaining two fingers based on the grasping video of each grasping scheme comprises: determining a plurality of key support points of the remaining two fingers based on the grasping video of each grasping scheme; constructing a support graph, the support graph including a plurality of nodes and a plurality of edges between the nodes, each node representing a key support point, a node feature of each node including position information of the key support point, the grasping video of each grasping scheme, and the three-finger force point, and each edge between the nodes representing a distance between the nodes; and processing the support graph based on a graph neural network to obtain the target grasping points of the remaining two fingers.

[0008] In a possible implementation, the generating the plurality of initial force point sets based on the plurality of initial force points of the test rubber product comprises: clustering the plurality of initial force points of the test rubber product based on a clustering algorithm to obtain K clusters, where K is 5; and obtaining the plurality of initial force point sets based on the K clusters.

[0009] According to a second aspect, the present application provides an intelligent robot feeding system, comprising: an acquisition module configured to acquire a test rubber product image; an initial force point determination module configured to determine a plurality of initial force points of the test rubber product based on the test rubber product image using an initial force point model; a three-finger force point determination module configured to determine a three-finger force point based on the plurality of initial force points of the test rubber product; a grasping scheme generation module configured to generate a plurality of grasping schemes based on the plurality of initial force points of the test rubber product and the three-finger force point, each grasping scheme including five-finger force points; a grasping test module configured to control the five-finger robot to perform a plurality of grasping operations on the test rubber product based on each grasping scheme and obtain a grasping video of each grasping scheme; a remaining grasping point determination module configured to determine target grasping points of the remaining two fingers based on the grasping video of each grasping scheme; and a feeding grasping module configured to control the five-finger robot to perform a feeding grasping operation on a remaining rubber product based on the three-finger force point and the target grasping points of the remaining two fingers.

[0010] In one possible implementation, the three-finger force point determination module is further configured to: generate multiple sets of initial force points based on multiple initial force points of the test rubber product, wherein each set of initial force points includes three initial three-finger force points; control a five-finger robot to use three fingers to grasp the test rubber product multiple times based on each set of initial force points, and acquire grasping videos of each set of initial force points, wherein the three fingers are the thumb, index finger, and middle finger; and determine the three-finger force points based on the grasping videos of each set of initial force points.

[0011] In one possible implementation, the remaining grasping point determination module is further configured to: determine multiple key support points for the remaining two fingers based on the grasping video of each grasping scheme; construct a support graph, which includes multiple nodes and multiple edges between nodes, each node representing a key support point, and the node features of each node including the location information of the key support point, the grasping video of each grasping scheme, the force points of the three fingers, and the edges between nodes representing the distance between nodes; and process the support graph based on a graph neural network to obtain the target grasping points for the remaining two fingers.

[0012] In one possible implementation, generating multiple sets of initial stress points based on multiple initial stress points of the test rubber product includes: clustering the multiple initial stress points of the test rubber product to obtain K clusters, where K is 5; and obtaining multiple sets of initial stress points based on the K clusters.

[0013] According to a third aspect, the present invention provides an electronic device, comprising: a processor; a memory; and a computer program; wherein the computer program is stored in the memory and configured to be executed by the processor to implement the method described above, the method comprising: acquiring an image of a test rubber product; determining a plurality of initial force points of the test rubber product using an initial force point model based on the image of the test rubber product; determining three-finger force points based on the plurality of initial force points of the test rubber product; generating a plurality of gripping schemes based on the plurality of initial force points of the test rubber product and the three-finger force points, each gripping scheme including five-finger force points; controlling a five-finger robot to grip the test rubber product multiple times based on each gripping scheme, and acquiring a gripping video of each gripping scheme; determining the remaining two-finger target gripping points based on the gripping video of each gripping scheme; and controlling the five-finger robot to load and grip the remaining rubber product based on the three-finger force points and the remaining two-finger target gripping points.

[0014] According to a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the aforementioned intelligent robot loading method, the method comprising: acquiring an image of a test rubber product; determining multiple initial force points of the test rubber product using an initial force point model based on the image of the test rubber product; determining three-finger force points based on the multiple initial force points of the test rubber product; generating multiple gripping schemes based on the multiple initial force points and the three-finger force points of the test rubber product, each gripping scheme including five-finger force points; controlling a five-finger robot to grip the test rubber product multiple times based on each gripping scheme, and acquiring a gripping video of each gripping scheme; determining the remaining two-finger target gripping points based on the gripping video of each gripping scheme; and controlling the five-finger robot to load and grip the remaining rubber product based on the three-finger force points and the remaining two-finger target gripping points.

[0015] This invention provides an intelligent robot feeding method and system. The method includes acquiring an image of a test rubber product; determining multiple initial force points of the test rubber product using an initial force point model based on the image; determining three-finger force points based on the multiple initial force points of the test rubber product; generating multiple gripping schemes based on the multiple initial force points and the three-finger force points, each gripping scheme including five-finger force points; controlling a five-finger robot to grip the test rubber product multiple times based on each gripping scheme, and acquiring a gripping video for each gripping scheme; determining the remaining two-finger target gripping points based on the gripping video for each gripping scheme; and controlling the five-finger robot to feed and grip the remaining rubber product based on the three-finger force points and the remaining two-finger target gripping points. This method can efficiently and accurately determine the appropriate gripping points for the rubber product to achieve stable robot feeding and gripping. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating an intelligent robot feeding method provided in an embodiment of the present invention.

[0017] Figure 2 This is a schematic diagram of a process for determining the force points of three fingers, provided as an embodiment of the present invention.

[0018] Figure 3 This is a schematic diagram of a process for generating multiple sets of initial force points, provided as an embodiment of the present invention.

[0019] Figure 4 This is a schematic diagram of a five-fingered robot provided in an embodiment of the present invention.

[0020] Figure 5 This is a schematic diagram of a process for determining the remaining two target grasping points according to an embodiment of the present invention.

[0021] Figure 6 This is a schematic diagram of an intelligent robot feeding system provided in an embodiment of the present invention. Detailed Implementation

[0022] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of the invention. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, certain operations related to the present invention are not shown or described in the specification. This is to avoid obscuring the core parts of the invention with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0023] In this embodiment of the invention, the following are provided: Figure 1 The method for feeding materials with an intelligent robot shown includes steps S1 to S7: Step S1, acquiring an image of a test rubber product.

[0024] The test rubber product images are three-dimensional images captured by high-definition camera acquisition equipment installed near the robot's working area, which can clearly show the complete outline, surface details and spatial shape of the test rubber product.

[0025] Images of tested rubber products can show the appearance and structural features of the tested rubber products.

[0026] Step S2: Based on the image of the test rubber product, determine multiple initial stress points of the test rubber product using the initial stress point model.

[0027] In some embodiments, the initial stress point model is a convolutional neural network model, the input of the initial stress point model is the image of the test rubber product, and the output of the initial stress point model is multiple initial stress points of the test rubber product.

[0028] Convolutional Neural Network (CNN) models are a type of deep learning model capable of processing images. A CNN can contain an input layer, convolutional layers, pooling layers, fully connected layers, and an output layer. CNNs can accurately extract latent features from images.

[0029] The multiple initial stress points of the test rubber product are determined by the initial stress point model, which identifies several points that may be suitable for the robot to grasp the test rubber product. Each initial stress point of the test rubber product includes its specific coordinates in the image of the test rubber product, the direction of its normal vector, the magnitude of its local curvature, and its surface texture gradient.

[0030] The pixel information in the image of the tested rubber product can completely record its geometric shape and surface details. The contours, bumps, wrinkles, and other areas of the rubber product can be represented by different brightness, color, and texture distributions in the image. The model can use these visual features to determine whether a point is suitable as a stress point. A convolutional neural network (CNN) can learn from this raw pixel data which combinations of visual features correspond to a stable, non-slip gripping position. The convolutional layers of the CNN can perform sliding convolution operations on the image of the tested rubber product using multiple convolutional kernels of different sizes to extract edge features, texture features, and local structural features, such as the corner positions and surface protrusions. After multiple rounds of convolution and pooling operations, the CNN can obtain high-level abstract features of the tested rubber product. These high-level abstract features can comprehensively reflect the structural characteristics and potential stress areas of the tested rubber product. The model then inputs high-level abstract features into a fully connected layer. This fully connected layer uses non-linear transformations of multiple neurons to convert the feature information into probabilities of whether each pixel on the test rubber product is a stress point, and filters out pixels with probabilities higher than a threshold as candidate stress points. Finally, the model deduplicates and optimizes these candidate stress points, removing those that are too close or overlap, ultimately determining multiple initial stress points on the test rubber product with different coordinates.

[0031] In some embodiments, determining multiple initial stress points of the test rubber product using an initial stress point model based on the test rubber product image includes steps S21 to S23: Step S21, determining the geometric morphology information and surface feature information of the test rubber product based on the test rubber product image.

[0032] In some embodiments, a convolutional neural network can be used to determine the geometric morphology information and surface feature information of the tested rubber product.

[0033] The geometric morphology information of the tested rubber product is a quantitative information reflecting the spatial structure and shape characteristics of the tested rubber product, extracted from images using a convolutional neural network. This geometric morphology information includes the complete outline data and local structural features of the tested rubber product.

[0034] The complete outline data of the tested rubber product includes the edge position, overall configuration and spatial morphology data of the tested rubber product.

[0035] Local structural features include the location and outline of areas such as corners, protrusions, depressions, and folds.

[0036] The surface feature information of the tested rubber products is extracted from the images of the tested rubber products using a convolutional neural network, reflecting the physical state of the surface. This surface feature information includes differences in surface brightness and color distribution, surface texture details, and texture gradient variations.

[0037] Surface texture details include texture direction and density.

[0038] Surface texture details can reflect the frictional characteristics of the contact surface and affect the gripping and anti-slip effect.

[0039] Convolutional neural networks (CNNs) can extract information layer by layer from pixel data of test rubber product images through their multi-layered feature extraction capabilities. The bottom convolutional layers capture basic visual features such as edges and textures of the test rubber products. Middle layers combine these basic features to form local structural features such as convexities and depressions. Higher layers integrate these local features and construct a complete outline and surface state. This hierarchical structure of CNNs is similar to the human visual system's process of perceiving objects and can effectively analyze the spatial structural information in test rubber product images, thereby extracting features related to the geometric shape of the test rubber products, such as outlines, dimensions, and local structures. Simultaneously, the weight-sharing mechanism of the convolutional layers makes the model more sensitive to surface features such as texture details and brightness variations in the image. The model can learn the feature representations of different texture patterns to distinguish attributes such as the roughness and texture direction of the test rubber product surface.

[0040] Step S22: Based on the geometric morphology information and surface feature information of the test rubber product, determine multiple spatial structural partitions of the test rubber product, the anti-slip performance level of each partition, the deformation risk coefficient, and the spatial positional relationship of each partition.

[0041] In some embodiments, deep neural networks can be used to determine multiple spatial structural zones of the tested rubber product, the anti-slip performance level of each zone, the deformation risk coefficient, and the spatial positional relationship of each zone.

[0042] Deep neural networks (DNNs) are computational models composed of multiple processing layers, each containing a large number of neurons. Data enters from the input layer, undergoes nonlinear transformations through multiple hidden layers, and finally reaches the output layer. The output of each neuron in a layer is a weighted sum of the outputs of the neurons in the previous layer, processed by a nonlinear activation function. By training on large amounts of data, deep neural networks can learn extremely complex and abstract patterns and rules within the data, and can use this learned knowledge to predict or classify new data. Deep neural networks can handle high-dimensional, complex relational mapping problems.

[0043] Multiple spatial structural partitions refer to the structured division of the surface of a test rubber product using a deep neural network based on its geometric morphology and surface feature information. Each spatial structural partition is a continuous region with similar geometric and surface properties, and there are clearly distinguishable boundaries between each spatial structural partition.

[0044] Geometric properties include curvature and flatness.

[0045] Surface properties include texture and roughness.

[0046] The anti-slip performance level is a grading index output by a deep neural network to represent the ability of the surface of each spatial structure partition to resist relative sliding. The anti-slip performance level is divided into high, medium and low levels. The higher the anti-slip performance level, the less likely the surface of that partition will slip when in contact.

[0047] The deformation risk coefficient is a numerical index output by a deep neural network used to quantify the likelihood of morphological changes in each spatial structural partition when subjected to external forces. The higher the deformation risk coefficient, the more likely the partition is to undergo significant deformation after being subjected to force.

[0048] The spatial positional relationship of each partition is the relative positional characteristics of multiple spatial structure partitions in three-dimensional space output by a deep neural network. The spatial positional relationship of each partition includes the straight-line distance between the centroids of the partitions, the azimuth angle between the partitions, and the coordinate distribution law of the partitions in the overall coordinate system of the tested rubber product.

[0049] Distribution patterns include symmetrical distribution, gradient distribution, concentrated distribution, and uniform distribution.

[0050] The spatial relationship between the partitions can be used to describe the spatial distribution of the partitions.

[0051] Deep neural networks, through their multi-layered, hierarchical nonlinear transformations and complex feature fusion capabilities, can deeply process input geometric and surface feature information. The model's multiple hidden layers can capture high-order correlations between the geometric and surface feature information of the tested rubber products. For example, the model can correlate curvature and flatness in geometric attributes with texture and roughness in surface attributes, thereby achieving precise spatial structural partitioning. By learning the mapping relationship between partitions and anti-slip performance and deformation risk in samples, the model can transform extracted features into quantified anti-slip performance levels and deformation risk coefficients. Furthermore, by analyzing the positional data of each candidate partition, deep neural networks can uncover spatial relationships such as centroid distance, azimuth angle, and overall distribution patterns between partitions.

[0052] Step S23: Determine multiple initial stress points of the test rubber product based on the multiple spatial structure partitions, the anti-slip performance level of each partition, the deformation risk coefficient, and the spatial positional relationship of each partition.

[0053] In some embodiments, a deep neural network can be used to determine multiple initial stress points on the test rubber article.

[0054] Deep neural networks, with their powerful multi-feature fusion and complex decision-making capabilities, can comprehensively analyze the spatial structure partitions, anti-slip performance levels, deformation risk coefficients, and spatial positional relationships of the input. The deep network structure of deep neural networks can capture the implicit correlations between various parameters. For example, it can combine partitions with high anti-slip levels and low deformation risk with reasonable spatial distribution patterns. Furthermore, by learning the selection logic of high-quality stress points in the samples, it can associate the attributes of spatial structure partitions, such as anti-slip levels and deformation risk coefficients, with the spatial distribution characteristics of the partitions to form a correspondence rule from these specific parameters to the initial stress point positions, thereby determining multiple initial stress points of the tested rubber products.

[0055] Step S3: Determine the three-finger force points based on the multiple initial force points of the tested rubber product.

[0056] In some embodiments, Figure 2 The present invention provides a flowchart for determining the force points of three fingers. The determination of the force points of three fingers includes steps S31 to S33: Step S31, generating multiple sets of initial force points based on multiple initial force points of the test rubber product, wherein one set of initial force points includes three initial three-finger force points.

[0057] In some embodiments, Figure 3The present invention provides a schematic diagram of a process for generating multiple sets of initial stress points, wherein generating multiple sets of initial stress points includes steps S311 to S312: Step S311, based on the multiple initial stress points of the test rubber product, K clusters are obtained, where K is 5.

[0058] The clustering algorithm described is the K-means clustering algorithm. K-means clustering is an iterative unsupervised clustering algorithm. The principle of K-means clustering is to first randomly select K objects as initial cluster centers, then calculate the distance between each object and each cluster center, and assign each object to the nearest cluster center. After all objects have been assigned, the cluster centers of each cluster are recalculated. This assignment and update process is repeated until the positions of the cluster centers no longer change significantly, or until a preset number of iterations is reached. Ultimately, K-means clustering can divide the dataset into K compact and independent clusters.

[0059] The K clusters are five independent sets of points formed by dividing multiple initial stress points of the test rubber product in space and feature dimensions using the K-means clustering algorithm. Each cluster represents a specific region on the test rubber product, where all initial stress points are geographically adjacent and exhibit high similarity in accompanying geometric features such as normal vectors and curvature.

[0060] The process of clustering multiple initial stress points of a test rubber product using the K-means clustering algorithm is as follows: First, five initial stress points are randomly selected as the initial centers of five clusters. In each iteration, the algorithm traverses all other initial stress points and calculates the multidimensional distance from each point to the centers of these five clusters. This distance is a weighted distance that combines spatial location differences and geometric feature information differences. After calculation, each point is assigned to the cluster containing the center of the cluster with the shortest distance. After all points are assigned, the algorithm recalculates the new center of each cluster, which is the average value of all points within that cluster in the multidimensional feature space. This iterative process continues until the center point of each cluster no longer changes, indicating that the points within the cluster have reached a high degree of cohesion, and the points between clusters have significant differences. Finally, the K-means clustering algorithm can divide all initial stress points into K stable clusters, each cluster representing a region on the product with similar gripping attributes.

[0061] Step S312: Obtain multiple sets of initial force points based on K clusters.

[0062] In some embodiments, multiple sets of initial force points can be obtained based on K clusters using a force point combination model. The force point combination model is a deep neural network model. The input to the force point combination model is K clusters, and the output of the force point combination model is multiple sets of initial force points.

[0063] Multiple initial force points are generated by analyzing and combining K candidate force point clusters that meet the basic screening conditions through a force point combination model. Each initial force point set contains three initial three-finger force points.

[0064] K clusters divide the scattered initial force points into K representative regions. Points within each region have similar physical properties, which greatly simplifies the model's analysis task. The model no longer needs to search through a massive number of individual points; instead, it can combine and filter based on the macroscopic characteristics of these five regions. For example, the model can evaluate the relative positions and distances between different clusters, as well as the average geometric characteristics of points within each cluster, to determine which three clusters are most likely to form a stable three-finger grip in terms of geometric configuration.

[0065] Deep neural networks can encode the overall features of each cluster, such as calculating the centroid location of each cluster, the distribution range of points within the cluster, and the mean and variance of the geometric features of all points within the cluster. These encoded cluster features constitute the input of the neural network. Deep neural networks can learn the stability rules of different cluster combinations through hidden layers. For example, a model can learn through training that a stable three-finger grip requires two clusters that are symmetrical in position, and a cluster in between with different geometric properties. Deep neural networks can simulate various combinations of three clusters selected from five clusters and evaluate each combination. The evaluation is based on whether the relative geometric relationship of these clusters can form an envelope-like, torsional-resistant gripping posture. Ultimately, the deep neural network can output multiple sets of combinations that it considers to have the highest stability score. Each set of combinations contains a representative point selected from each of the three different clusters, and these three points together constitute a set of initial force points.

[0066] Step S32: Based on each set of initial force points, control the five-finger robot to use three fingers to grasp the test rubber product multiple times, and obtain the grasping video of each set of initial force points. The three fingers are the thumb, index finger, and middle finger.

[0067] Each set of initial force point grasping videos is a video sequence synchronously recorded by a camera as the five-finger robot repeatedly grasps the real test rubber product according to the planned positions and postures of each initial force point. Figure 4 This is a schematic diagram of a five-fingered robot provided in an embodiment of the present invention.

[0068] The grasping video can record in detail the entire dynamic process from the robot's three fingers contacting the product, applying pressure, to attempting to lift it, and includes key physical interaction information such as whether the grasp was successful, whether the product was deformed, slipped, or fell.

[0069] Step S33: Determine the three-finger force points based on the grasping video of each set of initial force points.

[0070] In some embodiments, a grasping analysis model can be used to determine the three-finger pressure points. The grasping analysis model is a Transformer model. The input to the grasping analysis model is the grasping video of each set of initial pressure points, and the output of the grasping analysis model is the three-finger pressure points.

[0071] The Transformer model is a deep learning model based on the self-attention mechanism. The Transformer model can process sequential data and capture long-distance dependencies between elements within the sequence. It can also process the entire sequence in parallel. The Transformer model can use the self-attention mechanism to calculate the importance weight of each element in the sequence to all other elements, thereby dynamically focusing on the most relevant information during the encoding and decoding process.

[0072] The three-finger force point is a set of three optimal and most stable force points selected and determined from multiple sets of initial force points by comprehensively analyzing the force point grabbing videos of each set of initial force points using a grabbing analysis model.

[0073] The three-finger gripping point was verified to be the most reliable way to grasp and lift the tested rubber product, which formed the basis for the subsequent determination of the five-finger gripping scheme.

[0074] Each set of initial force-application videos provides the grasping analysis model with dynamic data to evaluate the actual effectiveness of each grasping scheme. Each frame of the video contains visual information such as the precise location of the contact point between the robot's fingers and the rubber product, the degree of deformation of the product, and whether any slight slippage occurs. By analyzing the video sequence, the model can observe the complete process of the test rubber product's shape changing over time under the grasping force. Successful grasping videos show a smooth image of the test rubber product being stably gripped and smoothly lifted, while failed grasping videos record the process of the test rubber product slipping from the fingertips or undergoing drastic deformation due to uneven force. This dynamic visual feedback is the basis for the model to judge the quality of a set of initial force-application methods.

[0075] The Transformer model can process time-series data from each set of initial force-application videos as input. Through its self-attention mechanism, the Transformer model can simultaneously focus on multiple key time points in the video sequence. For example, the model can focus on analyzing video frames at the moments when the robot's fingers first contact the object, when the grasping force reaches its peak, and when the object begins to be lifted. The Transformer model can extract the centroid displacement, the contour change rate of the contact area, and the flow field changes of the surface texture of the test rubber object during the grasping process in each grasping video to determine whether slippage occurs. The Transformer model can learn the intrinsic relationship between the deformation process and slippage trend of the test rubber object in the grasping video and the final grasping success rate. The model can analyze and score each set of initial force-application videos; high-scoring solutions indicate minimal object deformation, no slippage, and a smooth lifting process. Then, the model can comprehensively compare the scores of all solutions and select the three-finger force-application video corresponding to the initial force-application video that consistently performs the most stably and has the highest success rate in multiple repeated grasping experiments.

[0076] Step S4: Based on the multiple initial force points of the test rubber product and the three-finger force points, generate multiple gripping schemes, each gripping scheme including five-finger force points.

[0077] In some embodiments, a grasping scheme generation model can be used to generate multiple grasping schemes. The grasping scheme generation model is a deep neural network model. The input to the grasping scheme generation model is multiple initial force points of the tested rubber product and the three-finger force points; the output of the grasping scheme generation model is multiple grasping schemes.

[0078] Multiple gripping schemes are sets of complete five-finger gripping schemes output by the gripping scheme generation model to guide the five-finger robot in performing a complete gripping operation on the test rubber product. Each gripping scheme includes the force points of the five fingers.

[0079] Multiple initial force points on the tested rubber product can provide the model with candidate positions for the remaining two fingers out of the five-finger force points, while the three-finger force points provide the basis for the model's grasping. Because the positions of the three-finger force points are stable and reliable, the model can select the optimal auxiliary support points for the ring and little fingers of the robot when grasping the tested rubber product from the candidate positions based on the three-finger force points, thereby optimizing the overall grasping posture.

[0080] Deep neural networks possess the ability to learn and reason about complex spatial relationships. They can use proven, stable, and reliable three-finger force points as the fixed geometric foundation for constructing a five-finger gripping pattern. Furthermore, multiple initial force points from tested rubber products can form a broad set of candidate points for the ring and little fingers. Through training on a large amount of gripping data, deep neural networks can understand the nonlinear mapping relationship between the spatial distribution of the five force points and the gripping success rate. The model can systematically evaluate the five-point spatial configuration formed by combining the three-finger force points with any two points from the candidate point pool. Based on the learned knowledge, it can predict the comprehensive performance of this five-point spatial configuration in resisting gravity, inertial forces, and preventing product slippage and rotation. Finally, the model can output multiple sets of five-finger force point combinations with high predictive stability and unique characteristics in gripping posture. Each set of five-finger force point combinations constitutes a complete gripping scheme.

[0081] Step S5: Based on each grasping scheme, control the five-finger robot to grasp the test rubber product multiple times and obtain the grasping video of each grasping scheme.

[0082] The gripping video for each gripping scheme is a sequence of videos recorded synchronously by a camera as the five-fingered robot repeatedly grips the test rubber product according to the five force points defined in each gripping scheme.

[0083] Each grasping scheme's grasping video fully records the robot's dynamic interaction process during five-finger coordinated grasping, with a focus on the actual role and effect of the ring finger and little finger as auxiliary support points in stabilizing the entire grasping process.

[0084] Step S6: Determine the remaining two target capture points based on the capture video of each capture scheme.

[0085] In some embodiments, Figure 5 This is a flowchart illustrating the process of determining the remaining two finger target grasping points according to an embodiment of the present invention. The process of determining the remaining two finger target grasping points includes steps S61 to S63: Step S61, determining multiple key support points of the remaining two fingers based on the grasping video of each grasping scheme.

[0086] In some embodiments, a second grasping analysis model can be used to determine multiple key support points for the remaining two fingers. The second grasping analysis model is a Transformer model. The input to the second grasping analysis model is the grasping video of each grasping scheme, and the output of the second grasping analysis model is the multiple key support points for the remaining two fingers.

[0087] The remaining two fingers are the ring and little fingers of the five-fingered robot. During the grasping process, these two fingers work in coordination with the thumb, index finger, and middle finger to jointly perform the grasping operation on the tested rubber product.

[0088] The remaining two fingers' multiple key support points are a set of multiple points that play an important supporting role in improving the stability of grasping in actual grasping, identified and extracted from all tested candidate points of the ring finger and little finger after the second grasping analysis model analyzes the grasping video of each grasping scheme.

[0089] The remaining two fingers have multiple key support points that can effectively prevent the product from rotating, shaking or coming off while the three fingers apply the main gripping force, and can further enhance the stability of the five-fingered robot in gripping the test rubber product.

[0090] The gripping videos for each gripping scheme provide the second gripping analysis model with dynamic data to evaluate the actual function of the robot's remaining two fingers in different five-finger gripping schemes. Compared to the gripping video at the initial force point, the gripping videos for each gripping scheme focus on observing the overall stability and minute dynamics of the rubber product under the coordinated action of the five fingers. The model can accurately track the deformation of the rubber product near the contact points of the ring and little fingers from the gripping videos of each gripping scheme, and whether these two points effectively suppress the vibration and displacement of the product during lifting and movement.

[0091] The Transformer model can perform temporal analysis on the grasping videos for each input grasping scheme. Utilizing a self-attention mechanism, the Transformer model can capture subtle dynamic changes in the video related to grasping stability, particularly visual feature changes around the contact points of the ring and little fingers. The model can analyze how these support points distribute pressure, increase friction, and restrict the object's degrees of freedom when the robot applies force. For example, the model can identify a support point on the ring finger in a grasping scheme where, although the force is small, its location precisely suppresses the object's rotational tendency at the moment of lifting; the model will then classify this point as a critical support point. By analyzing the videos of all schemes, the model can filter out the ring and little finger points that consistently contribute to successful grasping in multiple experiments and aggregate these points to form multiple sets of critical support points for the remaining two fingers.

[0092] Step S62: Construct a support graph. The support graph includes multiple nodes and multiple edges between the nodes. Each node represents a key support point. The node features of each node include the location information of the key support point, the capture video of each capture scheme, and the three-finger force point. The edges between nodes represent the distance between the nodes.

[0093] A support graph is a structured data representation used to describe the relationships between multiple key support points of the remaining two fingers, and their connection to the overall grasping task. In the support graph, each key support point is abstracted as a node. The node features of each node include its location information, the grasping video for each grasping scheme, and the three finger support points serving as the grasping reference. The edges of the support graph represent the physical spatial distance between nodes, i.e., between key support points, connecting these discrete key support points into a network.

[0094] The support map can preserve the independent attributes of each key support point, and can also establish the topological relationship between multiple key support points.

[0095] Step S63: Process the support map based on the graph neural network to obtain the remaining two target grasping points.

[0096] Graph Neural Networks (GNNs) are deep learning models used to process graph data. GNNs effectively learn the features of nodes and edges in a graph through aggregation and update operations defined on nodes. The input to the GNN is the supporting graph, and the output is the remaining two target grasping points.

[0097] The remaining two target grasping points are the two best grasping points ultimately selected from multiple key support points of the remaining two fingers after comprehensive analysis and reasoning of the support map by graph neural network.

[0098] The remaining two target gripping points are used for the ring and little fingers of the five-fingered robot. These two target gripping points, combined with the already determined force points of the three fingers, can form the optimal five-fingered gripping pattern, thereby achieving the most efficient and stable grip on rubber products.

[0099] Support graphs provide graph neural networks with analytical objects rich in contextual information and structured relationships. Through nodes and edges, support graphs can place previously isolated key support points into an interconnected network, allowing the model to consider not only the individual properties of each point but also its spatial relationships with all other candidate points, rather than evaluating each point in isolation. The model can learn about the location and actual grasping performance of nodes through their features. For example, the model can analyze that two key support points, although performing well individually, may not be as effective when combined as a combination forming a better mechanical support at another location. Support graphs enable the model to transcend local optima and find the remaining two target grasping points with the best overall synergistic effect.

[0100] Graph neural networks (Graph Neural Networks) enable information propagation and feature aggregation on a support graph, where each node can collect feature information from all its neighbors. After multiple layers of processing by the Graph Neural Network, the final feature representation of each node in the support graph not only contains its own original information but also encodes its structural and relational information within the entire support graph. The Graph Neural Network can then apply a decision layer on top of these node embeddings containing global information. This decision layer evaluates the overall stability of any two node combinations working in conjunction with a fixed three-finger gripping point. By scoring all possible combinations, the model ultimately determines the pair of nodes with the highest score, which represents the remaining two finger gripping points.

[0101] Step S7: Based on the three-finger force points and the remaining two-finger target grasping points, control the five-finger robot to feed and grasp the remaining rubber products.

[0102] Once the three-finger force-bearing points and the remaining two-finger target gripping points are determined, the five-finger robot is controlled to perform batch feeding operations on the remaining rubber products of the same type based on the complete five-finger gripping scheme formed by the three-finger force-bearing points and the remaining two-finger target gripping points.

[0103] Based on the same inventive concept Figure 6 This is a schematic diagram of an intelligent robot loading system provided in an embodiment of the present invention. The intelligent robot loading system includes: an acquisition module 81, used to acquire images of test rubber products.

[0104] The initial stress point determination module 82 is used to determine multiple initial stress points of the test rubber product based on the image of the test rubber product and using the initial stress point model.

[0105] The three-finger force point determination module 83 is used to determine the three-finger force point based on multiple initial force points of the test rubber product.

[0106] The gripping scheme generation module 84 is used to generate multiple gripping schemes based on multiple initial force points of the test rubber product and the three-finger force points. Each gripping scheme includes five-finger force points.

[0107] The grasping test module 85 is used to control the five-finger robot to grasp the test rubber product multiple times based on each grasping scheme and to acquire the grasping video of each grasping scheme.

[0108] The remaining capture point determination module 86 is used to determine the remaining two target capture points based on the capture video of each capture scheme.

[0109] The feeding and gripping module 87 is used to control the five-finger robot to feed and grip the remaining rubber products based on the three-finger force points and the remaining two-finger target gripping points.

[0110] It should be noted that, in order to simplify the descriptions disclosed herein and thus aid in the understanding of one or more embodiments of the invention, the foregoing description of embodiments of this specification may sometimes combine multiple features into a single embodiment, drawing, or description thereof. However, this method of disclosure does not imply that the subject matter of this specification requires more features than those mentioned in the claims. In fact, the embodiments contain fewer features than all the features of a single embodiment disclosed above.

[0111] Finally, it should be understood that the embodiments described in this specification are merely illustrative of the principles of the embodiments described herein. Other variations may also fall within the scope of this specification. Therefore, alternative configurations of the embodiments described herein are intended to be illustrative rather than limiting, and should be considered consistent with the teachings of this specification. Accordingly, the embodiments described herein are not limited to those explicitly introduced and described herein.

Claims

1. A method for feeding materials using an intelligent robot, characterized in that, include: Acquire images of the test rubber products; Based on the image of the tested rubber product, multiple initial stress points of the tested rubber product are determined using an initial stress point model; The three-finger force points are determined based on multiple initial force points of the tested rubber product; Based on the multiple initial force points of the tested rubber product and the three-finger force points, multiple gripping schemes are generated, each gripping scheme including five-finger force points; Based on each grasping scheme, the five-finger robot is controlled to grasp the test rubber product multiple times, and the grasping video of each grasping scheme is obtained. The remaining two target capture points are determined based on the capture video of each capture scheme. Based on the three force points and the remaining two target grasping points, the five-finger robot is controlled to feed and grasp the remaining rubber products.

2. The intelligent robot feeding method as described in claim 1, characterized in that, The determination of the three-finger force point based on multiple initial force points of the tested rubber product includes: Multiple sets of initial force points are generated based on the multiple initial force points of the test rubber product. Each set of initial force points includes three initial three-finger force points. Based on each initial force point, the five-finger robot uses three fingers to grasp the test rubber product multiple times and acquires the grasping video of each initial force point. The three fingers are the thumb, index finger, and middle finger. The three-finger force points are determined based on the grasping video of each set of initial force points.

3. The intelligent robot feeding method as described in claim 1, characterized in that, The determination of the remaining two target capture points based on the capture video of each capture scheme includes: Based on the capture video of each capture scheme, determine multiple key support points for the remaining two fingers. Construct a support graph, which includes multiple nodes and multiple edges between nodes. Each node represents a key support point. The node features of each node include the location information of the key support point, the capture video of each capture scheme, and the three-finger force point. The edges between nodes represent the distance between nodes. The remaining two target grasping points are obtained by processing the support map using a graph neural network.

4. The intelligent robot feeding method as described in claim 2, characterized in that, The generation of multiple sets of initial stress points based on multiple initial stress points of the tested rubber product includes: Based on the multiple initial stress points of the tested rubber product, K clusters were obtained, where K is 5; Multiple initial force points are obtained based on K clusters.

5. An intelligent robot feeding system, characterized in that, include: The acquisition module is used to acquire images of the test rubber products; The initial stress point determination module is used to determine multiple initial stress points of the test rubber product based on the image of the test rubber product and using the initial stress point model. The three-finger force point determination module is used to determine the three-finger force point based on multiple initial force points of the tested rubber product. The gripping scheme generation module is used to generate multiple gripping schemes based on multiple initial force points of the test rubber product and the three-finger force points. Each gripping scheme includes five-finger force points. The grasping test module is used to control the five-finger robot to grasp the test rubber product multiple times based on each grasping scheme and to acquire the grasping video of each grasping scheme. The remaining capture point determination module is used to determine the remaining two target capture points based on the capture video of each capture scheme. The feeding and gripping module is used to control the five-finger robot to feed and grip the remaining rubber products based on the three-finger force points and the remaining two-finger target gripping points.

6. The intelligent robot feeding system as described in claim 5, characterized in that, The three-finger force point determination module is also used for: Multiple sets of initial force points are generated based on the multiple initial force points of the test rubber product. Each set of initial force points includes three initial three-finger force points. Based on each initial force point, the five-finger robot uses three fingers to grasp the test rubber product multiple times and acquires the grasping video of each initial force point. The three fingers are the thumb, index finger, and middle finger. The three-finger force points are determined based on the grasping video of each set of initial force points.

7. The intelligent robot feeding system as described in claim 5, characterized in that, The remaining capture point determination module is also used for: Based on the capture video of each capture scheme, determine multiple key support points for the remaining two fingers. Construct a support graph, which includes multiple nodes and multiple edges between nodes. Each node represents a key support point. The node features of each node include the location information of the key support point, the capture video of each capture scheme, and the three-finger force point. The edges between nodes represent the distance between nodes. The remaining two target grasping points are obtained by processing the support map using a graph neural network.

8. The intelligent robot feeding system as described in claim 6, characterized in that, The generation of multiple sets of initial stress points based on multiple initial stress points of the tested rubber product includes: Based on the multiple initial stress points of the tested rubber product, K clusters were obtained, where K is 5; Multiple initial force points are obtained based on K clusters.

Images