Grasping posture positioning method, computer storage medium and terminal device

Abstract

The present invention relates to a grasping posture positioning method, a computer storage medium and a terminal device. First, based on a neural network method, a category judgment model is constructed to determine whether the current object to be grasped is a regular part or an irregular part; then, according to the determined part type, a rotation search method or a probability search method is selectively adopted. Advantages: On the one hand, the two search methods are shared, and the optimal grasping posture can be positioned to a great extent. Compared with the method of using only the rotation search method, the grasping success rate is greatly improved; on the other hand, the trained category judgment model has a lower time overhead when performing forward reasoning. Compared with the method of using only the probability search method and the method of first using the rotation search method and then the probability search method, the average time consumption is less, especially in industrial scenarios with a large number of irregular parts, the technical advantages are more obvious. Overall, it is a positioning method that comprehensively considers positioning accuracy and time overhead, which greatly improves posture accuracy and response timeliness.

Description

Technical Field

[0001] The present invention relates to the field of control, and in particular to a grasping posture positioning method. Background Art

[0002] In the process of upgrading intelligent manufacturing of engineering machinery, it is common to use mechanized grasping equipment to grasp parts and wrap the objects waiting to be grasped. For example, the suction cup grasping parts is used as an example. Figure 1-2 The images shown are scaled images of the suction cup's underside and the part's surface. The steps for determining the optimal grasping pose can essentially be summarized as maximizing the overlap area of ​​target regions in different images under certain constraints. Current research has attempted to replace traditional manual grasping with intelligent grasping algorithms, with some success.

[0003] Specifically, existing techniques typically employ a center-of-gravity rotation search method. This involves horizontally aligning the suction cup image and the part image at their respective centers of gravity, then rotating them clockwise at regular intervals to determine the position of maximum gripping force and locate the grasping posture. However, the customized parts used in construction machinery often have complex shapes and a wide range of sizes. This results in the grasping posture determined by this search method being applicable only to certain parts, such as regular parts. For irregular parts, especially large ones, it may not meet actual requirements, leading to grasping failures and low success rates, limiting its application scenarios.

[0004] Based on this, existing technologies have proposed some population-based probabilistic search methods. These methods can, to a certain extent, handle complex and variable customized parts, enabling more accurate grasping of their positions. However, compared to the center of gravity rotation search method, which can calculate in hundreds of seconds, the search process is more time-consuming, typically requiring over 300 seconds. If this probabilistic search method is used for all parts, it would be a waste of time for regular parts.

[0005] Therefore, existing technologies typically use a rotation search method followed by a probabilistic search method to determine the grasping position. Specifically, the center of gravity rotation search method is first used, which takes approximately 100 seconds. If the grasp is successful, the entire positioning process only takes about 100 seconds. If the grasp is not successful, the initial grasp will fail. The probabilistic search method is then used, and the entire positioning process takes approximately 100 seconds plus 300 seconds, for a total of over 400 seconds. This time-consuming search process hinders further improvement in the production line's production cycle time.

[0006] Therefore, for industrial machinery scenarios, there is still much room for improvement in the accuracy and real-time performance of the process of positioning and grasping posture. How to comprehensively consider the posture positioning process from the perspective of accuracy and time cost is a technical problem that needs to be solved urgently in this field. Summary of the Invention

[0007] To solve one of the above technical problems, the present invention provides a grasping posture positioning method based on a neural network, comprising:

[0008] T1: Based on the neural network method, a category judgment model is constructed to determine whether the current object to be grasped is a regular part or an irregular part;

[0009] T2: If it is a regular part, the rotation search method is used to locate the grasping posture of the current object to be grasped;

[0010] T3: If it is an irregular part, a population-based probabilistic search method is used to locate the grasping posture of the current object to be grasped.

[0011] Furthermore, the category judgment model includes:

[0012] An input layer, used to input at least an image of an object to be grasped;

[0013] The feature extraction layer is used to extract the features of the input content of the input layer;

[0014] The fully connected layer is used to fuse the features of each layer in the feature extraction layer to obtain the classification result;

[0015] The output layer is used to output the classification results as regular or irregular items.

[0016] Furthermore, the input layer includes:

[0017] The first channel is used to input the grasped part image;

[0018] The second channel is used to input the image of the object to be grasped;

[0019] The third channel is used to input parameters related to the problem of maximizing the overlapping area between the grasped object image and the image of the object to be grasped;

[0020] The synthesis channel is used to synthesize the grasped object image and the image of the object to be grasped.

[0021] Furthermore, the parameters related to the overlap area maximization problem include: the positive and negative extreme values ​​of the maximum rotation angle of the grasping member; the horizontal and vertical coordinates of the center of gravity of the object to be grasped; the positive and negative extreme values ​​of the width of the object to be grasped; and at least one or more groups of positive and negative extreme values ​​of the height of the object to be grasped.

[0022] Furthermore, in the third channel, the data in each column is set to a repeated cycle of parameters related to the overlap area maximization problem, and the data in each row is set to be the same.

[0023] Furthermore, step T1 includes:

[0024] S1: Based on a neural network, a classification judgment model is constructed, which takes at least an image of the object to be grasped as input and outputs whether the object to be grasped is a regular object or an irregular object;

[0025] S2: Collect at least a number of samples of images of the object to be grasped, and label the images of the object to be grasped to obtain a sample set;

[0026] S3: Input the samples of the sample set into the category judgment model in sequence to obtain the trained category judgment model;

[0027] S4: At least collect an image of the object to be grasped, input the image into the trained category judgment model, and determine whether the object to be grasped is a regular object or an irregular object.

[0028] Furthermore, in step T3, any one of genetic algorithm, particle swarm algorithm, grey wolf optimization algorithm, and firefly algorithm is adopted.

[0029] Furthermore, specifically, the grey wolf optimization algorithm is adopted, and sampling initialization based on the center of gravity Gaussian distribution or sampling initialization based on the uniform distribution.

[0030] On the other hand, the present invention also provides a computer storage medium storing an executable program code; the executable program code is used to execute any of the above-mentioned grasping posture positioning methods.

[0031] On the other hand, the present invention also provides a terminal device, including a memory and a processor; the memory stores program code that can be executed by the processor; the program code is used to execute any of the above-mentioned grasping posture positioning methods.

[0032] The grasping posture positioning method provided by the present invention is based on a neural network method to construct a category judgment model, based on which it first determines whether the current object to be grasped is a regular part or an irregular part, and then selectively adopts a rotation search method or a probability search method according to the determined part type. Advantages: On the one hand, since the two search methods are shared, the optimal grasping posture can be located to a great extent for both regular and irregular parts. Compared with the method of using only the rotation search method, the grasping success rate is greatly improved. Experiments have shown that the grasping success rate is as high as 99.7%. On the other hand, since the trained category judgment model has a low time overhead when performing forward reasoning, experiments have shown that it only takes about 0.3 seconds. Therefore, compared with the method of using only the probability search method (both regular and irregular parts take about 300 seconds) and the method of first using the rotation search method and then the probability search method (regular parts take more than 100 seconds, irregular parts take more than 400 seconds), the average time consumption is less (regular parts take 0.3 seconds + more than 100 seconds; irregular parts take 0.3 seconds + more than 300 seconds). The technical advantages are especially obvious in industrial scenarios with a large number of irregular parts, which can further improve the accuracy and real-time performance of the entire industrial scenario and improve the production rhythm of the entire production line. Overall, it is a posture positioning method that comprehensively considers positioning accuracy and time cost, which greatly improves posture accuracy and response timeliness. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 An image of the bottom surface of a suction cup according to an example of the present invention;

[0034] Figure 2 The surface image of the part according to the example of the present invention;

[0035] Figure 3 This is a flow chart of an embodiment of the grasping posture positioning method of the present invention;

[0036] Figure 4 Flowchart of step T1 of the grasping posture positioning method according to an embodiment of the present invention;

[0037] Figure 5 This is a flow chart of another embodiment of step T1 of the grasping posture positioning method of the present invention;

[0038] Figure 6 This is an example diagram of a binarized image of a gripped part image according to the present invention;

[0039] Figure 7 This is an example diagram of a binary image of an object to be grasped according to the present invention;

[0040] Figure 8 A schematic structural diagram of an embodiment of the type determination model of the present invention;

[0041] Figure 9 This is an example diagram of a composite image of a binary image of a grasping member and a binary image of an object to be grasped according to the present invention;

[0042] Figure 10 This is an example diagram of the input form of the third channel of the input layer of the category judgment model of the present invention;

[0043] Figure 11 The statistical graph of the average accuracy of different neural network models;

[0044] Figure 12 is the confusion matrix of DenseNet-121 on the validation set;

[0045] Figure 13 The average ratio score and time cost statistics of different wolf pack sizes on the validation set in the gray wolf optimization algorithm;

[0046] Figure 14 This is a statistical graph of the average ratio score and time cost of different iteration numbers on the validation set in the Gray Wolf optimization algorithm;

[0047] Figure 15 Statistics of the average ratio scores and time costs of different probabilistic search methods on the validation set. DETAILED DESCRIPTION

[0048] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0049] It should be noted that if the embodiments of the present invention involve directional indications, such as up, down, left, right, front, back, etc., then the directional indications are only used to explain the relative position relationship, movement status, etc. between the components in a certain specific posture. If the specific posture changes, the directional indications will also change accordingly. In addition, if the embodiments of the present invention involve descriptions of "first, second", "S1, S2", "step one, step two", etc., then such descriptions are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features or indicating the execution order of the method, etc. Those skilled in the art can understand that anything that does not violate the main points of the invention under the technical concept of the invention should be included in the scope of protection of the present invention.

[0050] like Figure 3 As shown, the present invention provides a grasping posture positioning method based on a neural network, comprising:

[0051] T1: Based on the neural network method, a category judgment model is constructed to determine whether the current object to be grasped is a regular part or an irregular part. Specifically, the category judgment model can be optionally but not limited to being constructed based on various classic neural network models, such as VGGNet, ResNet, DenseNet, EfficientNet, Vit, etc., and at least takes the image of the object to be grasped as input and whether the object to be grasped is a regular part or an irregular part as output; the trained category judgment model is obtained through sample set verification, and the input image of the current object to be grasped is classified into image categories to determine whether the current object to be grasped is a regular part or an irregular part.

[0052] Preferably, the category judgment model may optionally include but is not limited to:

[0053] An input layer, used to input at least an image of an object to be grasped;

[0054] The feature extraction layer is used to extract the features of the input content of the input layer;

[0055] The fully connected layer is used to fuse the features of each layer in the feature extraction layer to obtain the classification result;

[0056] The output layer is used to output the classification results as regular or irregular items.

[0057] T2: If it is a regular part, the rotation search method is used to locate the grasping posture of the current object to be grasped; specifically, the rotation search method is a common method in the prior art and will not be described in detail here, that is: the image of the grasped part and the image of the object to be grasped, such as the image of the suction cup and the image of the part, are horizontally overlapped at the center of gravity of each other, and then rotated clockwise at a certain angle interval to search and determine the maximum grasping force position to locate the grasping posture.

[0058] T3: If the object is irregular, a population-based probabilistic search method is used to locate the grasping position of the current object to be grasped. Specifically, this population-based probabilistic search method is a common method in the prior art and will not be described in detail here. Optional methods include, but are not limited to, genetic algorithms, particle swarm algorithms, grey wolf optimization (GWO) algorithms, and firefly algorithms.

[0059] In this embodiment, a grasping posture positioning method of the present invention is provided. Based on the rapidly developing deep learning and neural networks, its core invention lies in constructing a category judgment model based on the neural network method, thereby first judging whether the current object to be grasped is a regular part or an irregular part, and then selectively adopting a rotation search method or a probability search method according to the judged part type. Advantages: On the one hand, since the two search methods are shared, the optimal grasping posture can be located to a great extent for both regular and irregular parts. Compared with the method of using only the rotation search method, the grasping success rate is greatly improved. Experiments have shown that the grasping success rate is as high as 99.7%. On the other hand, since the trained category judgment model has a low time overhead when performing forward reasoning, experiments have shown that it only takes about 0.3 seconds. Therefore, compared with the method of using only the probability search method (both regular and irregular parts take about 300 seconds) and the method of first using the rotation search method and then the probability search method (regular parts take more than 100 seconds, irregular parts take more than 400 seconds), the average time consumption is less (regular parts take 0.3 seconds + more than 100 seconds; irregular parts take 0.3 seconds + more than 300 seconds). The technical advantages are especially obvious in industrial scenarios with a large number of irregular parts, which can further improve the accuracy and real-time performance of the entire industrial scenario and improve the production rhythm of the entire production line. Overall, it is a posture positioning method that comprehensively considers positioning accuracy and time cost, which greatly improves posture accuracy and response timeliness.

[0060] 1. About step T1

[0061] 1-1:

[0062] Specifically, such as Figure 4 As shown, a neural network-based category judgment method is also provided, which can be optionally but not limited to step T1. Specifically, step T1 can optionally but not limited to include:

[0063] S1: Based on a neural network, a classification judgment model is constructed, which takes at least an image of the object to be grasped as input and outputs whether the object to be grasped is a regular object or an irregular object;

[0064] S2: Collect at least a number of samples of images of the objects to be grasped, and label the images of the objects to be grasped to obtain a sample set. Specifically, it is optional but not limited to first collecting a batch of proportional image data of industrial parts, and then manually labeling them to construct a sample set.

[0065] S3: Inputting samples of the sample set into the category judgment model in sequence to obtain a trained category judgment model; specifically, the sample set may be randomly sampled and divided into a training set and a validation set, and various parameters may be iteratively updated and optimized to train and validate the category judgment model;

[0066] S4: At least collect an image of the object to be grasped, input the image into the trained category judgment model, and determine whether the object to be grasped is a regular object or an irregular object.

[0067] In this embodiment, a preferred embodiment of step T1 of the present invention is given, which can be optionally but not limited to any model based on a neural network method, to construct a category judgment model with the image of the object to be grasped as input and the output of whether the object to be grasped is a regular part or an irregular part, then collect samples for annotation, and verify the category judgment model through sample set training to obtain a trained neural network model, and then collect the current image of the object to be grasped, input it into the trained category judgment model, and output the result of whether it is a regular part or an irregular part. It is worth noting that the specific form of the category judgment model and the training process can be arbitrarily set by those skilled in the art. The key to the present invention is to think of a method based on a neural network to construct a category judgment model to first determine whether the object to be grasped is a regular part or an irregular part, and then selectively adopt a rotation search method or a probability search method, rather than adopting a single search method or tentatively adopting two search methods in succession.

[0068] 1-2:

[0069] Preferably, in order to improve the accuracy of the category judgment model of the present invention, the present invention is for step T1, the core and technical points of the present invention, such as Figure 5 It is preferred to provide a category determination method based on a neural network, comprising:

[0070] S1': Based on a neural network, a category judgment model is constructed, which takes the image of the grasped object, the image of the object to be grasped, and the parameters related to the problem of maximizing the overlapping area of ​​the two images as input, and outputs whether the object to be grasped is a regular object or an irregular object; compared with step S1, its input has two more contents, combining the image of the grasped object and the parameters related to the problem of maximizing the overlapping area of ​​the two images. Although it is only an increase in input content, it is the result of repeated verification and experimentation. Its model construction is more comprehensive, making the subsequent judgment results of regular and irregular objects more accurate.

[0071] Specifically, such as Figure 6 As shown, the grasping part image may be, but is not limited to, a binary image of a grasping device such as a suction cup; Figure 7 As shown in FIG, the image of the object to be grasped may be, but is not limited to, a binary image of a part or package to be grasped; as shown in Table 1, the parameters related to the problem of maximizing the overlapping area of ​​the two images may be, but are not limited to, including: the positive and negative extreme values ​​of the maximum rotation angle of the grasping part; the horizontal and vertical coordinates of the center of gravity of the object to be grasped; the positive and negative extreme values ​​of the width of the object to be grasped; and at least one or more groups of positive and negative extreme values ​​of the height of the object to be grasped.

[0072] Table 1 Parameters related to the overlap area maximization problem

[0073]

[0074] More specifically, the type judgment model can be constructed based on, but not limited to, any neural network; for example, Figure 8 As shown, optional but not limited to include:

[0075] A. Input layer, which is used to input the image of the grasped object, the image of the object to be grasped, and the parameters related to the problem of maximizing the overlapping area of ​​the two images;

[0076] B. Feature extraction layer, used to extract features of parameters related to the grasped object image, the image of the object to be grasped, and the problem of maximizing the overlapping area of ​​the two images; for example, it is optional but not limited to taking the classic convolutional neural network - DenseNet as an example, which includes two modules: DenseBlock and Transition; among them, 1. DenseBlock module, including multiple layers (DenseBlock1, Dense Block2...), each layer has the same feature map size, and dense connections are used between layers; 2. Transition module, including convolution layer (Convolution) and pooling layer (Pooling), used to connect two adjacent DenseBlocks, and reduce the feature map size through the average pooling layer. More preferably, considering the balance between accuracy and time overhead, it is optional but not limited to prefer DenseNet-121 as the classifier, that is, a network structure containing 121 layers,

[0077] C. Fully connected layer, which fuses features from each layer to obtain the classification result. For example, through the above-mentioned DenseBlock and Transition, the stacking of the two modules is input into the fully connected layer (Linear) to form the final deep network classifier.

[0078] D. Output layer, which outputs the classification results; for example, whether the object to be grasped is a regular part or an irregular part.

[0079] More preferably, A, the input layer, can also optionally but not limited to use transfer learning and input image synthesis method; specifically, including: the first channel, input grasping part binary image, for example, the suction cup binary image, such as Figure 2 As shown, it is input as the first channel; the second channel inputs the binary image of the object to be grasped, for example, the binary image of the part to be grasped, such as Figure 3 As shown, it is used as the second channel input; the synthesis channel is used to synthesize the binary image of the grasped object and the binary image of the object to be grasped to obtain a synthetic image, such as Figure 9 As shown; in the third channel, the data of each column is set to the repeated cycle of the parameters related to the overlap area maximization problem, and the data of each row is set to be the same; as shown Figure 10 As shown, the parameters related to the overlap area maximization problem include: the positive and negative extremes of the maximum rotation angle of the gripper (-120, 120), the horizontal and vertical coordinates of the center of gravity of the object to be grasped (788, 792), the positive and negative extremes of the width of the object to be grasped (-750, 750), and the positive and negative extremes of the height of the object to be grasped (-750, 750). For a total of eight parameters, each column is set to a repeated cycle of these eight parameters, and each row is set to the same data. Of course, the order of these eight parameters can be uniformly swapped. In addition, the relevant parameters can be selected, but not limited to, all or any one or more of these eight parameters. Other relevant parameters can also be selected, such as the positive and negative extremes of the horizontal and vertical coordinates of the gripper, the positive and negative extremes of the width, and the positive and negative extremes of the height.

[0080] S2': Collect several samples of images of the grasped object, images of the object to be grasped, and parameters related to the problem of maximizing the overlap area between the two images, and annotate the images of the object to be grasped to obtain a sample set. Specifically, but not limited to, collecting binary images of the object to be grasped, images of the object to be grasped, and corresponding parameters related to the problem of maximizing the overlap area, and annotating the images of the object to be grasped as either regular or irregular objects, to form several samples and obtain a sample set.

[0081] More specifically, to improve the accuracy of annotation, in the above step S2 or S2', the step of annotating the image of the object to be grasped may optionally include but is not limited to:

[0082] S21': Using the center of gravity rotation search method, the gripper searches for the maximum gripping force required to grasp the object. Specifically, using the magnetic chuck grasping a part as an example, the center of gravity rotation search method determines the optimal position of the chuck grasping the part. This position is represented by the chuck's planar position coordinates and its rotation angle during grasping, denoted as the grasping pose (pose = [x, y, angle]. Based on this, the chuck grasps the part in this optimal position, achieving maximum suction and gripping force, ensuring successful grasping.

[0083] More specifically, in engineering machinery scenarios, parts are relatively stationary. Taking the above-mentioned suction cup and part grasping process as an example, the magnetic force is generated by the contact area between the suction cup and the part. The opening and closing of the magnetic block is determined by the ratio of the contact area to the magnetic block area. When the ratio is greater than or equal to the set threshold α, it indicates that the magnetic block is open and takes 1. When the ratio is less than the set threshold α, it indicates that the magnetic block is closed and takes 0. In actual engineering experience, this ratio can be selected but not limited to being set to above 0.9. The above optimization problem can be mathematically defined as formula (1):

[0084]

[0085] Where i represents the i-th magnetic block in the suction cup, i ranges from 1 to N, N is the total number of magnetic blocks in the suction cup, S r represents the contact area between the i-th magnetic block and the part, S i represents the magnetic block area of ​​the i-th magnetic block, α represents the ratio threshold of the bonding area to the magnetic block area, I i represents the indicator function of the i-th magnetic block; F i represents the rated suction force of the i-th magnetic block, and G represents the gravity of the object to be grasped.

[0086] Under the above mathematical definition, the optimization goal of the optimization problem is to search for the best gripping position of the suction cup so that when the suction cup grips the part in the best position, the ratio of the suction force generated to the weight of the part is maximized. The larger the ratio, the better the gripping position, and the actual gripping will be more stable and safe.

[0087] S22': Specifically, based on the mathematical definition of formula (1), the judgment rule may be selected but not limited to the mathematical definition of formula (2);

[0088] S23': If yes, mark the object to be grasped as a regular part; if no, mark the object to be grasped as an irregular part.

[0089]

[0090] That is to say, if the maximum ratio of the suction force to the weight of the part is greater than or equal to the second set threshold β, which can be selected but not limited to 1.2-1.8, and is preferably 1.3, it indicates that the best grasping posture found under the rotation search method can successfully grasp the object to be grasped, and the object to be grasped can be defined as a regular part, otherwise the object to be grasped is defined as an irregular part.

[0091] S3': Input the samples of the sample set into the category judgment model in sequence to obtain the trained category judgment model; specifically, input the samples into the category judgment model in sequence, iteratively update various parameters, and obtain the trained category judgment model.

[0092] S4': Collect the current grasped object image, the current to-be-grasped object image and current data of related parameters, input them into the trained category judgment model, and determine whether the current to-be-grasped object is a regular object or an irregular object.

[0093] In this embodiment, another preferred embodiment of step T1 of the present invention is provided. This embodiment can optionally employ, but is not limited to, any model based on a neural network approach. A classification judgment model is constructed, taking as input an image of the grasped object, an image of the object to be grasped, and parameters related to maximizing the overlap area between the two images, and outputting whether the object to be grasped is a regular or irregular object. Samples are then collected and labeled, and the classification judgment model is trained and verified using the sample set to obtain a trained neural network model. Furthermore, current data, including the current grasped object image, the current image of the object to be grasped, and related parameters, are collected and input into the trained classification judgment model, which outputs a result indicating whether the object is a regular or irregular object. Based on this, a preferred model structure for the classification judgment model and a preferred labeling method for sample labeling are provided. Experimental comparisons show that the method of steps S1'-S4', compared to the method of steps S1-S4, has three input channels, resulting in more comprehensive feature extraction. Furthermore, it uses the ratio of maximum gripping force to gravity as the dividing line, providing a clear and precise labeling method for determining whether the object is a regular or irregular object. This makes the classification judgment model's results more accurate, with experiments demonstrating an accuracy of 99.3% in determining whether the object is a regular or irregular object. It is worth noting that steps S1-S4 and S1'-S4' are preferred embodiments of step T1 of the present invention, and in particular, steps S1'-S4' are the construction, training and optimization methods of the category judgment model proposed by the inventors of the present invention after many attempts, but are not limited to this. For example, the input of the category judgment model can also be optional but not limited to the image of the object to be grasped, combined with the image of the grasped object, any one or two of the problems of maximizing the overlapping area of ​​the two images, and even other content related to the grasping posture, such as the axially symmetric image of the object to be grasped, etc. The corresponding input layer, feature extraction layer, fully connected layer and output layer of the type judgment model will be adjusted accordingly. On this basis, the corresponding steps S2 / S2' and S4 / S4' will collect the current content corresponding to the input content to determine whether the current object to be grasped is a regular part or an irregular part. It is worth noting that the more input layer channels the type judgment model has and the more relevant input content, the higher the accuracy of its type judgment will be. However, the corresponding feature extraction layer and fully connected layer structures will inevitably be more complex, resulting in a longer time consumption in the type judgment process. Those skilled in the art can selectively determine the input content of the input layer according to the accuracy requirements and time requirements, such as the type judgment model for protection in claims 2-6.

[0094] 2. About step T3

[0095] 2-1:

[0096] Regarding the probabilistic search method for the population, the present invention preferably adopts the Grey Wolf Optimization (GWO) algorithm based on experimental comparison. More preferably, in the Grey Wolf Optimization algorithm, it is optional but not limited to adopting the initialization of sampling based on the center of gravity of Gaussian distribution or sampling based on uniform distribution.

[0097] Specifically, the Gray Wolf Optimization Algorithm, as a type of swarm intelligence algorithm, is an iterative optimization algorithm with poor real-time performance. The initialization of the group generally has an important impact on the search process and results. An ideal initialized group helps the process converge quickly and obtain better results. Poor initialization may cause the process to converge slowly or fall into a local optimum. Based on the Gray Wolf Optimization Algorithm optimized by the probability search method, the present invention further improves its initialization process, preferably uses the center-of-gravity-based Gaussian distribution sampling initialization or the uniform distribution sampling initialization technology to accelerate the search process of the Gray Wolf Optimization Algorithm, and optimizes the search results at the same time, which can further improve the search speed and accuracy.

[0098] 2-2:

[0099] Specifically, the sampling initialization based on the center of gravity Gaussian distribution can be optionally but not limited to recording the center of gravity of the object to be grasped as (center_x, center_y) as part of the position parameter. The first component sampling of the grasping posture is used as an example to illustrate. The sampling principle of other components is the same, and only the relevant parameters are changed. Assume that the random variable Y obeys a Gaussian distribution with a position parameter of center_x and a scale parameter of σ (set σ in the experiment) 2 =100, which is an empirical parameter), as shown in formula (3), then its probability density function is shown in formula (4), and the first component of the pose, that is, initialization, will be sampled from this distribution.

[0100] Y~N(center x ,σ 2 ) (3)

[0101]

[0102] 2-3:

[0103] More specifically, we will use the sampling of the first component of the grasped pose as an example for initialization based on uniform distribution sampling. Assuming that the random variable U follows a uniform distribution as shown in Equation (5), and width is the width of the part image, its probability density function is shown in Equation (6). The first component of the pose, i.e., initialization, will be sampled from this distribution.

[0104] U~(0,width) (5)

[0105]

[0106] In summary, the grasping posture positioning step of the grasping part grasping the object to be grasped is optional but not limited to the grasping scenario of engineering machinery parts. The present invention proposes a two-stage grasping posture positioning method based on the neural network method. The first stage is based on the neural network method to construct a category judgment model, and preferably adopts a three-channel Densenet to improve the category judgment accuracy, up to 99.3%; the second stage is based on the above category judgment, and selectively adopts a rotation search method or a probability search method to improve the accuracy and speed of posture positioning from the dual angles of accuracy and time consumption. On this basis, for the probability search method, the gray wolf optimization algorithm (GWO) is selected based on experimental comparison, and its initialization step is further improved. Gaussian distribution sampling initialization based on the center of gravity and uniform distribution sampling initialization are proposed, which further improves the convergence speed and accuracy of its probability search method to further improve the accuracy and speed of posture positioning.

[0107] 3. Experimental comparison

[0108] 3-1:

[0109] Specifically, the classification model of the present invention can be implemented using, but is not limited to, an 11th Gen Intel(R) Core(TM) i5-11300H processor at 3.10GHz to 3.11GHz, 16GB of RAM, no GPU acceleration, Windows 10 operating system, and Python 3.8 programming language. The classification model is implemented using the PyTorch deep learning framework. Taking suction cup grasping of parts as an example, the training set of the part image classification dataset contains 2201 images, including 1470 images of regular parts and 731 images of irregular parts; the validation set contains 549 images, including 367 images of regular parts and 182 images of irregular parts.

[0110] In this experiment, nine representative neural network classification models were selected for comparative analysis, including DenseNet, VGGNet, ResNet, MobileNet_V2, ShuffleNet_V2, DenseNet, EfficientNet_V1, RegNet, Vit, and GoogleNet. It can be seen that the above-mentioned category judgment model can be constructed based on any neural network model. The present invention has constructed many forms of category judgment models, and the above-mentioned three-channel, DenseNet-based neural network model is preferred. More specifically, in the grasping pose search, we conducted a series of test experiments to determine the relevant hyperparameters of the GWO algorithm: the number of wolf packs and the total number of iterations; at the same time, we compared the search results and time overhead of the three algorithms proposed in this invention: GWO, PSO, and center of gravity rotation search.

[0111] 3-2:

[0112] Specifically, for step T1, verification of part category judgment:

[0113] 3-2-1: Classification Accuracy Verification

[0114] To evaluate the performance of the type judgment model, this experiment used average accuracy as the model evaluation metric and compared the time overhead of DenseNet-121 model inference and center of gravity rotation search. First, 9 neural network models were trained on the original part image training set and tested on the validation set. Each model was preferably trained with or without transfer learning for comparison. The experimental results are shown in Figure 2. Figure 11 As shown in the figure. It can be seen that the classification models using transfer learning outperformed the classification models without transfer learning in terms of average accuracy, with DenseNet-121 and EfficientNet_V1 achieving the highest average accuracy of 95.6%. It is worth noting that Vit, which performed best on the ImageNet dataset, did not achieve the expected results. More notably, when not using transfer learning, DenseNet-121 had a higher average accuracy than EfficientNet_V1. Therefore, when constructing the category judgment model described above, DenseNet, especially DenseNet-121, is preferably used as the basis for the final category judgment model.

[0115] In order to further improve the classification accuracy of the model, the present invention proposes steps S1'-S4' based on this. The input includes not only the image of the object to be grasped, but also the image of the grasped object and the parameters related to the problem of maximizing the overlap area of ​​the two images. DenseNet-121 is used for training on the synthetic image dataset. The experimental results are as follows: Figure 12 The confusion matrix of the trained DenseNet-121 on the validation set shows an average accuracy of (182+363) / (182+4+363+0)≈99.3%, a 3.7% improvement over the highest performance of 95.6% without synthetic data. This suggests that incorporating information such as the splicing cup, part dimensions, and the actual constraints of the problem—namely, the image of the gripped part and the parameters related to maximizing the overlap area—into the original part image can help the neural network model further improve classification accuracy.

[0116] 3-2-2: Classification Time Overhead Verification

[0117] Table 2 shows a comparison of the time costs of DenseNet-121 forward reasoning and center of gravity rotation search. Each data point is averaged after running 10 times, and the data in parentheses is the variance of the 10 data points. As shown in Table 2, the time cost of DenseNet-121 forward reasoning is only 0.296s, which is 487 times lower than the 144.45s of center of gravity rotation search. This shows that the use of neural network methods to construct a category judgment model first determines whether the object to be grasped is regular or irregular, and then selectively uses a rotation search method or a probabilistic search method. Compared with the existing method of first using a rotation search method and then using a probabilistic search method if the grasping is unsuccessful, the calculation speed is significantly faster, which can greatly improve the grasping position positioning speed and improve the performance of industrial production lines.

[0118] Table 2 Time cost of DenseNet-121 and center of gravity rotation search

[0119]

[0120] In summary, experiments have proved that using the neural network-based grasping posture positioning method proposed in the present invention to pre-classify the synthesized images of objects to be grasped can effectively improve the classification accuracy of the model and solve the problem of time-consuming center of gravity rotation search.

[0121] 3-3:

[0122] Verification of step T3 grasp pose search

[0123] 3-3-1: Quantitative Analysis

[0124] Before running the Grey Wolf Optimization (GWO) algorithm, it is necessary to determine the key hyperparameters in the GWO algorithm: the number of wolves and the total number of iterations. The following will quantitatively analyze the GWO algorithm. Figure 12 The figure shows the average ratio score (suction / gravity) and time cost of the GWO algorithm on the irregular parts dataset with a fixed total number of iterations and different numbers of wolf packs.

[0125] Depend on Figure 13As can be seen from the middle curve, as the number of wolf packs increases, the average ratio score corresponding to GWO gradually increases, from the lowest 1.033 to 1.208, an increase of 16.94%. In addition, it can be observed that in the comparison of the two preferred initialization methods of the present invention, the initialization based on the Gaussian distribution sampling of the center of gravity achieved a higher score in the population of no more than 30, while the initialization based on the uniform distribution sampling achieved a higher score in the population of more than 30. That is to say, in the grasping posture positioning algorithm, when there are more candidate solutions, evenly distributing these candidate solutions in the solution space is conducive to finding a better grasping posture. Finally, based on the balance between performance and time overhead, the present invention preferably uses 30 or 35 for the hyperparameter of the number of wolf packs.

[0126] exist Figure 14 The figure shows the average ratio score and time cost of the GWO algorithm on an irregular parts dataset, with a fixed wolf pack size (35) and different total iteration counts. The curves in the figure show that as the total number of iterations increases, the average ratio score of the GWO algorithm gradually increases, from a low of 1.160 to 1.228, a 5.86% increase. However, based on the same balance between performance and time cost, the present invention prefers a total number of iterations hyperparameter of 30.

[0127] 3-3-2: Qualitative Analysis

[0128] The following will compare the advantages and disadvantages of the preferred GWO pose search algorithm of the present invention with other methods. Other methods can be selected but not limited to PSO-Levy and center of gravity rotation search method. Figure 15 As shown in Figure 2, the experimental data of the comparison is shown, including the average ratio score and time cost. Figure 15 It can be seen that the average ratio score of the center of gravity rotation search method is the lowest, at 0.624; PSO-Levy is 1.110; GWO based on the center of gravity Gaussian distribution sampling initialization is 1.193 (the second highest accuracy), and GWO based on the uniform distribution sampling initialization is 1.196 (the highest accuracy). In addition, in terms of time cost, the GWO algorithm is superior to PSO-Levy. Therefore, when positioning the grasping posture of irregular parts, the present invention prefers the GWO algorithm, especially the center of gravity Gaussian distribution sampling initialization and the uniform distribution sampling initialization, which achieves a certain balance between performance and time cost, greatly improving the positioning speed and accuracy of the grasping posture.

[0129] 4. Conclusion

[0130] This paper proposes a two-stage grasping pose localization method. Taking part grasping pose search as an example, it first uses a model such as DenseNet-121 to quickly select irregular parts. A three-channel input, namely an image data synthesis method, achieves an average accuracy of 99.3% for the DenseNet-121 model, approaching 100%, laying a solid foundation for the second stage of grasping pose search. Next, a GWO (Grounded Word Oriented Wolf Pack) is preferably used to search for grasping poses. To further improve GWO performance, the paper proposes initializing the wolf pack using either a center-of-gravity Gaussian or uniform distribution sampling. Experiments on a part image dataset demonstrate that the proposed algorithm significantly improves the grasping average ratio score compared to other algorithms. The proposed grasping pose localization method can effectively and quickly match appropriate poses from existing data, significantly improving grasping accuracy and real-time performance, meeting the real-time requirements of industrial scenarios. While the grasping pose localization method of the present invention uses the aforementioned suction cup grasping of parts as an example, this is not limiting. Examples also include, but are not limited to, gripping packages with a gripper.

[0131] On the other hand, the present invention also provides a computer storage medium storing an executable program code; the executable program code is used to execute any of the above-mentioned grasping posture positioning methods.

[0132] On the other hand, the present invention also provides a terminal device, including a memory and a processor; the memory stores program code that can be executed by the processor; the program code is used to execute any of the above-mentioned grasping posture positioning methods.

[0133] Exemplarily, the program code may be divided into one or more modules / units, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution process of the program code in the terminal device.

[0134] The terminal device may be a computing device such as a desktop computer, laptop, PDA, or cloud server. The terminal device may include, but is not limited to, a processor and memory. Those skilled in the art will appreciate that the terminal device may also include input / output devices, network access devices, buses, and the like.

[0135] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0136] The memory may be an internal storage unit of the terminal device, such as a hard disk or memory. The memory may also be an external storage device of the terminal device, such as a plug-in hard disk equipped on the terminal device, a SmartMedia Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Furthermore, the memory may include both an internal storage unit of the terminal device and an external storage device. The memory is used to store the program code and other programs and data required by the terminal device. The memory may also be used to temporarily store data that has been output or is about to be output.

[0137] The above-mentioned computer storage medium and terminal device are created based on the above-mentioned grasping posture positioning method. Their technical functions and beneficial effects are no longer repeated here. The various technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the various technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0138] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be determined by the appended claims.

Claims

1. A grasping posture positioning method based on neural network, characterized in that: include: T1: Based on a neural network approach, a classification judgment model is constructed and trained that takes as input an image of the grasped object, an image of the object to be grasped, and parameters related to the problem of maximizing the overlap area between the two images, and outputs whether the object to be grasped is a regular object or an irregular object. The relevant parameters include: the positive and negative extreme values ​​of the maximum rotation angle of the grasped object, the positive and negative extreme values ​​of the horizontal and vertical coordinates, the positive and negative extreme values ​​of the width, and the positive and negative extreme values ​​of the height; at least one or more sets of horizontal and vertical coordinates of the center of gravity of the object to be grasped, the positive and negative extreme values ​​of the width, and the positive and negative extreme values ​​of the height. When inputting the relevant parameters, the data in each column is set to a repeated cycle of the parameters related to the problem of maximizing the overlap area, and the data in each row is set to be the same. The image of the current grasped object, the image of the object to be grasped, and the parameters related to the problem of maximizing the overlapping area of ​​the two images are collected and input into the trained category judgment model to determine whether the current object to be grasped is a regular object or an irregular object; T2: If it is a regular part, the rotation search method is used to locate the grasping posture of the current object to be grasped; T3: If it is an irregular part, a population-based probabilistic search method is used to locate the grasping posture of the current object to be grasped.

2. The grasping posture positioning method according to claim 1, characterized in that: Category judgment model, including: The input layer is used to input the image of the grasped object, the image of the object to be grasped, and the parameters related to the problem of maximizing the overlapping area of ​​the two images; The feature extraction layer is used to extract the features of the input content of the input layer; The fully connected layer is used to fuse the features of each layer in the feature extraction layer to obtain the classification result; The output layer is used to output the classification results as regular or irregular items.

3. The grasping posture positioning method according to claim 2, characterized in that: Input layer, including: The first channel is used to input the grasped part image; The second channel is used to input the image of the object to be grasped; The third channel is used to input parameters related to the problem of maximizing the overlap area between the grasped part image and the image of the object to be grasped; the data in each column is set to a repeated cycle of the parameters related to the problem of maximizing the overlap area, and the data in each row is set to be the same; The synthesis channel is used to synthesize the grasped object image and the image of the object to be grasped.

4. The grasping posture positioning method according to claim 2, characterized in that: The feature extraction layer consists of two modules: DenseBlock and Transition. The DenseBlock module consists of multiple layers, each with the same feature map size, and dense connections between layers. The Transition module includes a convolutional layer and a pooling layer, which is used to connect two adjacent DenseBlocks and reduce the feature map size through the average pooling layer.

5. The grasping posture positioning method according to claim 1, characterized in that: Training of the category judgment model includes: Collect a number of samples of grasped object images, images of objects to be grasped, and parameters related to the problem of maximizing the overlapping area of ​​the two images, and annotate the images of objects to be grasped as regular objects or irregular objects to form a number of samples and obtain a sample set; The samples in the sample set are input into the category judgment model in sequence, and the parameters are updated iteratively to obtain the trained category judgment model.

6. The grasping posture positioning method according to claim 5, characterized in that: The steps of labeling include: The center of gravity rotation search method is used to search for the maximum gripping force of the gripper to grasp the object to be grasped; Determine whether the maximum ratio of the maximum gripping force to the weight of the object to be grasped is greater than or equal to a set threshold. If so, mark the object to be grasped as a regular part; otherwise, mark the object to be grasped as an irregular part.

7. The grasping posture positioning method according to any one of claims 1 to 6, characterized in that: In step T3, any one of genetic algorithm, particle swarm algorithm, grey wolf optimization algorithm, and firefly algorithm is used.

8. The grasping posture positioning method according to claim 7, characterized in that: Step T3 specifically adopts the Grey Wolf Optimization Algorithm and initializes the sampling based on the Gaussian distribution of the center of gravity or the uniform distribution.

9. A computer storage medium, characterized in that An executable program code is stored; the executable program code is used to execute the grasping posture positioning method described in any one of claims 1-8.

10. A terminal device, characterized in that: It comprises a memory and a processor; the memory stores a program code that can be executed by the processor; the program code is used to execute the grasping posture positioning method described in any one of claims 1-8.

Images