Category determination and grasping pose positioning method, storage medium and terminal device

Abstract

The application relates to a category judgment and grasping pose positioning method, a storage medium and a terminal device, and comprises the following steps: constructing a category judgment model based on a neural network, taking a grasping piece image, a to-be-grasped object image and parameters related to a maximum overlap area problem of the two images as input, and taking a regular piece or an irregular piece as output; collecting samples of the grasping piece image, the to-be-grasped object image and the parameters related to the maximum overlap area problem of the two images, and labeling the to-be-grasped object image to obtain a sample set; inputting the samples of the sample set into the category judgment model in sequence to obtain a trained category judgment model; collecting current data of the current grasping piece image, the current to-be-grasped object image and the related parameters, inputting the current data into the trained category judgment model, and determining whether the current to-be-grasped object is a regular piece or an irregular piece.

Description

Technical Field

[0001] This invention relates to the field of control, and in particular to a method for classifying objects. Background Technology

[0002] In the process of upgrading engineering machinery to intelligent manufacturing, customized parts are characterized by complex shapes and wide size ranges, resulting in many irregularly shaped parts. There are significant differences between regular and irregular parts, whether in image recognition or grasping and positioning methods. Therefore, in industrial scenarios, distinguishing between regular and irregular parts and using different methods for identification and positioning based on their different types is of great significance.

[0003] However, in the above scenarios, regularity and irregularity mainly refer to the differences in the shape of parts. Currently, there is no clear geometric definition to distinguish between regular and irregular parts. As a result, in actual industrial scenarios, methods that are generally applicable to regular parts, such as recognition methods and grasping pose positioning methods, are often applied to irregular parts, leading to recognition errors, grasping failures, etc., which seriously affect the normal operation and efficiency of the entire production line.

[0004] Therefore, how to identify regular and irregular parts is an indispensable research direction in industrial production lines, and how to provide a highly accurate method for determining part categories is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] To address one of the aforementioned technical problems, this invention provides a category determination method based on a neural network, comprising:

[0006] Based on neural networks, a category judgment model is constructed, which takes the image of the object to be grasped, the image of the object to be grasped, and the relevant parameters of the problem of maximizing the overlapping area of ​​the two images as input, and the object to be grasped as regular or irregular as output.

[0007] Collect several images of the object to be grabbed, images of the object to be grabbed, and samples of parameters related to the problem of maximizing the overlapping area of ​​the two images, and label the images of the object to be grabbed to obtain a sample set;

[0008] The samples in the sample set are input into the category judgment model in sequence to obtain the trained category judgment model;

[0009] Collect the current image of the object to be grabbed, the current image of the object to be grabbed, and the current data of relevant parameters, input them into the trained category judgment model, and determine whether the current object to be grabbed is a regular object or an irregular object.

[0010] Furthermore, the image of the grabbed item is a binarized image of the grabbed item; the image of the object to be grabbed is a binarized image of the object to be grabbed.

[0011] Furthermore, the relevant parameters for the problem of maximizing the overlapping area are at least one or more sets of positive and negative extreme values ​​of the maximum rotation angle of the grasping object; the horizontal and vertical coordinates of the center of gravity of the object to be grasped; the positive and negative extreme values ​​of the width of the object to be grasped; and the positive and negative extreme values ​​of the height of the object to be grasped.

[0012] Furthermore, the steps for annotating the images to be captured include:

[0013] The center of gravity rotation search method is used to search for the maximum gripping force of the gripper when gripping the object to be gripped;

[0014] Determine if the maximum gripping force exceeds a set threshold;

[0015] If yes, then mark the object to be grabbed as a regular object; otherwise, mark the object to be grabbed as an irregular object.

[0016] Furthermore, the steps for annotating the image to be captured can be represented as follows:

[0017]

[0018] Where i represents the i-th magnetic block in the suction cup, i ranges from 1 to N, N is the total number of magnetic blocks in the suction cup, and S r S represents the contact area between the i-th magnet and the part. i I represents the area of ​​the i-th magnetic block. i F represents the indicator function for the i-th magnetic block; i β represents the rated attraction force of the i-th magnetic block, G represents the weight of the object to be grasped, and β represents the set threshold value of the maximum ratio of attraction force to the weight of the part.

[0019] Furthermore, the category judgment model includes:

[0020] The input layer is used to input the image of the object to be grabbed, the image of the object to be grabbed, and the parameters related to the problem of maximizing the overlapping area of ​​the two images;

[0021] The feature extraction layer is used to extract features from the input content of the input layer.

[0022] Fully connected layers are used to fuse features from each layer in the feature extraction layer to obtain classification results;

[0023] The output layer is used to output the classification results as regular or irregular components.

[0024] Furthermore, the input layer includes:

[0025] The first channel is used to input the image of the captured object;

[0026] The second channel is used to input the image of the object to be captured.

[0027] The third channel is used to input parameters related to maximizing the overlap area between the image of the object to be grabbed and the image of the object to be grabbed.

[0028] The compositing channel is used to composite the image of the object to be grabbed and the image of the object to be grabbed.

[0029] On the other hand, the present invention also provides a grasping pose localization method, comprising:

[0030] T1: Based on any of the above category determination methods, determine whether the current object to be grabbed is a regular object or an irregular object;

[0031] T2: If it is a regular object, the rotation search method is used to locate the current grasping pose of the object to be grasped;

[0032] T3: If it is an irregular object, a population-based probabilistic search method is used to locate the grasping pose of the object to be grasped.

[0033] On the other hand, the present invention also provides a computer storage medium storing executable program code; the executable program code is used to execute any of the above-mentioned category determination methods or any of the above-mentioned grasping pose localization methods.

[0034] On the other hand, the present invention also provides a terminal device, including a memory and a processor; the memory stores program code that can be executed by the processor; the program code is used to execute any of the above-mentioned category determination methods or any of the above-mentioned grasping pose localization methods.

[0035] The present invention provides a method, storage medium, and terminal device for category determination and grasping pose localization. Based on an arbitrary model using a neural network method, it constructs a category determination model that takes as input the image of the grasped part, the image of the object to be grasped, and parameters related to maximizing the overlapping area of ​​the two images, and outputs whether the object to be grasped is a regular or irregular part. Samples are collected and labeled, and the category determination model is trained and validated using the sample set to obtain a trained neural network model. Then, current data of the current grasped part image, the current image of the object to be grasped, and related parameters are collected and input into the trained category determination model, outputting whether the object is a regular or irregular part. Crucially, its input includes the grasped part image, the image of the object to be grasped, and parameters related to maximizing the overlapping area of ​​the two images, resulting in more comprehensive feature extraction and significantly improving the accuracy of part category determination. Attached Figure Description

[0036] Figure 1 This is an image of the bottom surface of the suction cup, as exemplified by the present invention.

[0037] Figure 2 This is an example of a part surface image from the present invention;

[0038] Figure 3This is a flowchart of an embodiment of the pose capture and localization method of the present invention;

[0039] Figure 4 This is a flowchart of an embodiment of step T1 of the pose capture and localization method of the present invention;

[0040] Figure 5 This is a flowchart of another embodiment of step T1 of the pose capture and positioning method of the present invention;

[0041] Figure 6 This is an example image of the binarized image of the object being grasped in this invention;

[0042] Figure 7 This is an example of a binarized image of the object to be grasped in this invention;

[0043] Figure 8 This is a schematic diagram of the structure of an embodiment of the type determination model of the present invention;

[0044] Figure 9 This is an example of a composite image of the binarized image of the grasping component and the binarized image of the object to be grasped, according to the present invention.

[0045] Figure 10 This is an example diagram showing the input format of the third channel of the input layer of the category judgment model of the present invention;

[0046] Figure 11 A statistical graph showing the average accuracy of different neural network models;

[0047] Figure 12 This is the confusion matrix of DenseNet-121 on the validation set;

[0048] Figure 13 A statistical chart showing the average ratio score and time cost of different wolf pack sizes on the validation set in the gray wolf optimization algorithm.

[0049] Figure 14 A statistical chart showing the average ratio score and time cost of different iterations on the validation set in the Grey Wolf optimization algorithm.

[0050] Figure 15 This is a statistical chart showing the average ratio score and time cost of different probability search methods on the validation set. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0052] It should be noted that if the embodiments of the present invention involve directional indications, such as up, down, left, right, front, back, etc., these directional indications are only used to explain the relative positional relationships and movement of the components in a specific posture. If the specific posture changes, the directional indications will also change accordingly. Furthermore, if the embodiments of the present invention involve descriptions such as "first," "second," "S1," "S2," "step one," "step two," etc., these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance, or implicitly indicating the number of technical features indicated or the order of method execution. Those skilled in the art will understand that anything that does not violate the inventive concept and is within the scope of the present invention should be included in the protection scope of the present invention.

[0053] like Figure 3 As shown, this invention provides a neural network-based grasping pose localization method, comprising:

[0054] T1: Construct a category judgment model based on neural network methods to determine whether the current object to be grasped is a regular or irregular object. Specifically, this category judgment model can be, but is not limited to, based on various classic neural network models such as VGGNet, ResNet, DenseNet, EfficientNet, Vit, etc., and at least takes the image of the object to be grasped as input and the output as whether the object to be grasped is a regular or irregular object. The trained category judgment model is validated through a sample set, and the input image of the current object to be grasped is classified into image categories to determine whether the current object to be grasped is a regular or irregular object.

[0055] Preferred category judgment models may include, but are not limited to:

[0056] The input layer is used to input at least an image of the object to be captured.

[0057] The feature extraction layer is used to extract features from the input content of the input layer.

[0058] Fully connected layers are used to fuse features from each layer in the feature extraction layer to obtain classification results;

[0059] The output layer is used to output the classification results as regular or irregular components.

[0060] T2: If it is a regular object, the rotation search method is used to locate the current grasping pose of the object to be grasped. Specifically, the rotation search method is a common method in the existing technology, which will not be described in detail here. That is, the image of the object to be grasped and the image of the object to be grasped, such as the image of the suction cup and the image of the part, are horizontally overlapped at their center points and then rotated clockwise at certain angular intervals to search and determine the position of maximum gripping force to locate the grasping pose.

[0061] T3: If the object is irregular, a population-based probabilistic search method is used to locate the grasping pose of the object to be grasped. Specifically, this population-based probabilistic search method is a commonly used method in existing technologies and will not be elaborated here. Other methods that can be used, but are not limited to, include genetic algorithms, particle swarm optimization, gray wolf optimization (GWO) algorithms, and firefly algorithms.

[0062] In this embodiment, a grasping pose localization method of the present invention is presented. Based on the rapidly developing deep learning and neural network, its core invention lies in constructing a category judgment model based on the neural network method. Based on this model, it first determines whether the object to be grasped is a regular part or an irregular part. Then, according to the determined part type, it selectively adopts a rotation search method or a probability search method. Advantages: Firstly, since both search methods are used, the optimal grasping pose can be located to a great extent for both regular and irregular parts, significantly improving the grasping success rate compared to using only the rotation search method. Experiments show that the grasping success rate is as high as 99.7%. Secondly, because the trained category judgment model has low time overhead during forward inference, experiments show that it only takes about 0.3 seconds. Therefore, compared to using only the probability search method (which takes about 300 seconds for both regular and irregular parts) and using the rotation search method followed by the probability search method (which takes more than 100 seconds for regular parts and more than 400 seconds for irregular parts), its average time consumption is less (0.3 seconds + 100+ seconds for regular parts; 0.3 seconds + 300+ seconds for irregular parts). The technical advantages are particularly obvious in industrial scenarios with many irregular parts, which can further improve the accuracy and real-time performance of the entire industrial scenario and increase the production cycle of the entire production line. Overall, it is a pose localization method that takes into account both positioning accuracy and time cost, which greatly improves pose accuracy and response timeliness.

[0063] 1. Regarding step T1

[0064] 1-1:

[0065] Specifically, such as Figure 4 As shown, a category determination method based on a neural network is also provided, which can be used, but is not limited to, step T1. Specifically, step T1 can include, but is not limited to, the following:

[0066] S1: Based on neural networks, construct a category judgment model that takes at least the image of the object to be grasped as input and the object to be grasped as a regular or irregular object as output;

[0067] S2: Collect at least a number of sample images of the objects to be grabbed and label the images of the objects to be grabbed to obtain a sample set; specifically, it is optional but not limited to first collecting a batch of proportional image data of industrial parts, and then manually labeling them to construct a sample set;

[0068] S3: Input the samples from the sample set into the category judgment model in sequence to obtain the trained category judgment model; specifically, optional but not limited to randomly sampling the above sample set to divide it into a training set and a validation set, iteratively updating and optimizing various parameters to train and validate the category judgment model;

[0069] S4: At least collect the image of the object to be grasped, input it into the trained category judgment model, and determine whether the object to be grasped is a regular object or an irregular object.

[0070] In this embodiment, a preferred embodiment of step T1 of the present invention is given. It can be, but is not limited to, any model based on a neural network method. A category judgment model is constructed, taking the image of the object to be grasped as input and outputting whether the object is a regular or irregular object. Then, samples are collected and labeled. The category judgment model is trained and verified using the sample set to obtain a trained neural network model. Then, the current image of the object to be grasped is collected, input into the trained category judgment model, and the result of whether it is a regular or irregular object is output. It is worth noting that the specific form of this category judgment model and the training process can be arbitrarily set by those skilled in the art. The key to the present invention is the idea of ​​constructing a category judgment model based on a neural network method, first determining whether the object to be grasped is a regular or irregular object, and then selectively using a rotation search method or a probabilistic search method, rather than using a single search method or experimentally using two search methods sequentially.

[0071] 1-2:

[0072] Preferably, to improve the accuracy of the category judgment model of the present invention, for step T1, the core and key technical points of the present invention are as follows: Figure 5 The method shown preferably provides a category determination method based on a neural network, which may be used in step T1, but is not limited to, including:

[0073] S1': Based on a neural network, a category judgment model is constructed with the image of the object to be grasped, the image of the object to be grasped, and the parameters related to maximizing the overlapping area of ​​the two images as input, and the output being whether the object to be grasped is a regular object or an irregular object. Compared with step S1, its input has two additional contents, combining the image of the object to be grasped and the parameters related to maximizing the overlapping area of ​​the two images. Although it is only an increase in the input contents, it is the result of repeated verification and attempts. Its model construction is more comprehensive, making the subsequent judgment results of regular and irregular objects more accurate.

[0074] Specifically, such as Figure 6 As shown, the image of the grasped object can be, but is not limited to, a binarized image of a grasping device such as a suction cup; for example... Figure 7As shown, the image of the object to be grasped can be, but is not limited to, a binarized image of a part or package to be grasped; as shown in Table 1, the parameters related to the problem of maximizing the overlapping area of ​​the two images can be, but are not limited to, including: the positive and negative extreme values ​​of the maximum rotation angle of the grasping part; the horizontal and vertical coordinates of the centroid of the object to be grasped; the positive and negative extreme values ​​of the width of the object to be grasped; and at least one or more sets of the positive and negative extreme values ​​of the height of the object to be grasped.

[0075] Table 1. Parameters related to the problem of maximizing the overlapping area.

[0076]

[0077] More specifically, this type of judgment model can be, but is not limited to, built based on any kind of neural network; for example, such as Figure 8 As shown, optional, but not limited to, inclusion of:

[0078] A. Input layer, used to input the image of the object to be grabbed, the image of the object to be grabbed, and the parameters related to the problem of maximizing the overlapping area of ​​the two images;

[0079] B. Feature Extraction Layer: This layer extracts features related to the problem of maximizing the overlap area between the image to be grasped, the image to be grasped, and other parameters. For example, but not limited to, the classic convolutional neural network DenseNet, it includes two modules: DenseBlock and Transition. Specifically: 1. The DenseBlock module contains multiple layers (DenseBlock1, DenseBlock2, etc.), each with the same feature map size, and dense connections between layers; 2. The Transition module includes convolutional layers and pooling layers, used to connect two adjacent DenseBlocks, and uses average pooling to reduce the feature map size. More preferably, considering a balance between accuracy and time complexity, DenseNet-121 (a network structure with 121 layers) can be used as the classifier.

[0080] C. Fully connected layer: fuses features from each layer to obtain the classification result; for example, the stacked DenseBlock and Transition modules are input to the fully connected layer (Linear) to form the final deep network classifier.

[0081] D. Output layer, outputs the classification results; for example, whether the object to be crawled is a regular or irregular object.

[0082] More preferably, A, the input layer, may optionally, but is not limited to, using transfer learning and input image synthesis methods; specifically, it includes: a first channel, the input grasp binarized image, for example, the binarized image of the suction cup, such as... Figure 2 As shown, the first channel is the input; the second channel is the input binarized image of the object to be grasped. For example, the binarized image of the part to be grasped is shown below. Figure 3 As shown, the second input channel is used; the synthesis channel is used to synthesize the binarized image of the grasped object and the binarized image of the object to be grasped, resulting in a synthesized image, as shown. Figure 9 As shown; the third channel sets the data in each column to repeat the parameters related to the problem of maximizing the overlapping area, and sets the data in each row to be the same; as shown. Figure 10 As shown, the parameters related to the problem of maximizing the overlapping area include: the positive and negative extreme values ​​of the maximum rotation angle of the grasped object (-120, 120), the horizontal and vertical coordinates of the center of gravity of the object to be grasped (788, 792), the positive and negative extreme values ​​of the width of the object to be grasped (-750, 750), and the positive and negative extreme values ​​of the height of the object to be grasped (-750, 750), for a total of 8 parameters. Each column is set to repeat the 8 parameters, and each row is set to the same data. Of course, the order of these 8 parameters can be uniformly changed. In addition, the relevant parameters can be selected, but are not limited to, all or any one or more of the 8 parameters. Other relevant parameters can also be selected, such as the positive and negative extreme values ​​of the horizontal and vertical coordinates of the grasped object, the positive and negative extreme values ​​of the width, the positive and negative extreme values ​​of the height, etc.

[0083] S2': Collect several images of the object to be grasped, images of the object to be grasped, and samples of parameters related to maximizing the overlapping area of ​​the two images. Label the images of the object to be grasped to obtain a sample set. Specifically, this may include, but is not limited to, collecting a binarized image of the object to be grasped, a binarized image of the object to be grasped, and corresponding parameters related to maximizing the overlapping area. Label the images of the object to be grasped, indicating whether they are regular or irregular objects, forming several samples to obtain a sample set.

[0084] More specifically, to improve annotation accuracy, the step of annotating the image of the object to be captured in step S2 or S2' above may include, but is not limited to, the following:

[0085] S21': Employs a center-of-gravity rotation search method to find the maximum gripping force required for the gripper to grasp the object. Specifically, taking a magnetic chuck grasping a part as an example, the optimal posture for the chuck to grasp the part is determined using the center-of-gravity rotation search method. An example is the planar position coordinates of the chuck during grasping and its own rotation angle, denoted as the grasping posture pose = [x, y, angle]. Based on this, the chuck can grasp the part at this optimal posture, maximizing its suction force and ensuring a successful grasp.

[0086] More specifically, in engineering machinery scenarios, parts are relatively stationary. Taking the above-mentioned suction cup gripping process as an example, magnetic force is generated through the contact area between the suction cup and the part. The opening and closing of the magnetic block is determined by the ratio of the contact area to the area of ​​the magnetic block. When the ratio is greater than or equal to the set threshold α, it means the magnetic block is open (1); when the ratio is less than the set threshold α, it means the magnetic block is closed (0). In actual engineering experience, this ratio can be selected, but is not limited to, being above 0.9. The above optimization problem can be mathematically defined as formula (1):

[0087]

[0088] Where i represents the i-th magnetic block in the suction cup, i ranges from 1 to N, N is the total number of magnetic blocks in the suction cup, and S r S represents the contact area between the i-th magnet and the part. i Let I represent the area of ​​the i-th magnetic block, α represent the threshold ratio of the bonding area to the magnetic block area, and I i F represents the indicator function for the i-th magnetic block; i Let G represent the rated attraction force of the i-th magnetic block, and let G represent the weight of the object to be grasped.

[0089] Under the above mathematical definition, the optimization objective of the optimization problem is to search for the optimal gripping pose of the suction cup, so that when the suction cup grips the part in the optimal pose, the ratio of the suction force generated to the weight of the part is maximized. This is because the larger the ratio, the better the gripping pose, and the more stable and safer the actual gripping will be.

[0090] S22': Specifically, based on the above mathematical definition formula (1), this judgment rule can be selected, but is not limited to, the mathematical definition formula (2);

[0091] S23': If yes, mark the object to be grabbed as a regular object; otherwise, mark the object to be grabbed as an irregular object.

[0092]

[0093] In other words, if the maximum ratio of suction force to the weight of the part is greater than or equal to the second set threshold β, which can be selected but not limited to 1.2-1.8, and preferably 1.3, it indicates that the optimal grasping pose found under the rotation search method can successfully grasp the object to be grasped, and the object to be grasped can be defined as a regular part; otherwise, the object to be grasped is defined as an irregular part.

[0094] S3': Input the samples from the sample set into the category judgment model in sequence to obtain the trained category judgment model; specifically, input the samples into the category judgment model in sequence, iteratively update the parameters, and obtain the trained category judgment model.

[0095] S4': Collect the current image of the object to be grabbed, the current image of the object to be grabbed, and the current data of related parameters, input them into the trained category judgment model, and determine whether the current object to be grabbed is a regular object or an irregular object.

[0096] In this embodiment, another preferred embodiment of step T1 of the present invention is given. Any model based on a neural network method can be selected, but is not limited to, one. A category judgment model is constructed with the image of the object to be grasped, the image of the object to be grasped, and parameters related to maximizing the overlapping area of ​​the two images as input, and the output being whether the object to be grasped is a regular or irregular object. Samples are then collected and labeled. The category judgment model is trained and verified using the sample set to obtain the trained neural network model. Then, the current data of the current image of the object to be grasped, the current image of the object to be grasped, and the relevant parameters are collected and input into the trained category judgment model, outputting the result as either a regular or irregular object. Based on this, a preferred model structure for the category judgment model and a preferred labeling method for the samples are given. Through experimental comparison, the method in steps S1'-S4', compared to the method in steps S1-S4, has two advantages: firstly, its input layer includes three channels, resulting in more comprehensive feature extraction; secondly, it uses the ratio of maximum gripping force to gravity as the dividing line, providing a clear and accurate labeling method for regular or irregular objects; this makes the judgment result of the category judgment model more accurate. Experiments show that its accuracy in judging regular and irregular objects reaches 99.3%. It is worth noting that steps S1-S4 and S1'-S4' are preferred embodiments of step T1 of the present invention. In particular, steps S1'-S4' are the construction, training, and optimization methods for the category judgment model proposed by the inventors after numerous attempts, but are not limited to this. For example, the input of this category judgment model can also be selected, but is not limited to, the image of the object to be grasped, any one or two of the image of the object to be grasped, the problem of maximizing the overlapping area of ​​two images, and even other content related to the grasping pose, such as the axisymmetric image of the object to be grasped. The input layer, feature extraction layer, fully connected layer, and output layer of the corresponding category judgment model will be adjusted accordingly. Based on this, the corresponding steps S2 / S2' and S4 / S4' collect the current content corresponding to the input content and determine whether the current object to be grasped is a regular object or an irregular object. It is worth noting that the more input layer channels a type judgment model has and the more relevant input content it contains, the higher its type judgment accuracy will be. However, the corresponding feature extraction layer and fully connected layer structure will inevitably be more complex, resulting in a greater time consumption for the type judgment process. Those skilled in the art can selectively determine the input content of the input layer according to the accuracy and time requirements, as exemplified by the type judgment model claimed in claims 2-6.

[0097] 2. Regarding step T3

[0098] 2-1:

[0099] Regarding probabilistic search methods for populations, this invention, based on experimental comparisons, preferably employs the Grey Wolf Optimization (GWO) algorithm. More preferably, in the Grey Wolf Optimization algorithm, initialization can be selected, but is not limited to, using centroid-based Gaussian distribution sampling initialization or uniform distribution sampling initialization.

[0100] Specifically, the Grey Wolf Optimization Algorithm, as a type of swarm intelligence algorithm, is an iterative optimization algorithm with poor real-time performance. The initialization of the swarm generally has a significant impact on the search process and results. Ideal swarm initialization helps the process converge quickly and achieve better results, while poor initialization may lead to slow convergence or getting trapped in local optima. This invention, based on the probabilistic search method for optimizing the Grey Wolf Optimization Algorithm, further improves its initialization process. It preferably uses centroid-based Gaussian distribution sampling initialization or uniform distribution sampling initialization techniques to accelerate the search process of the Grey Wolf Optimization Algorithm, while simultaneously optimizing the search results, thereby further improving search speed and accuracy.

[0101] 2-2:

[0102] Specifically, the initialization is based on Gaussian distribution sampling with the center of gravity. Optionally, but not limited to, the center of gravity of the object to be grasped can be denoted as (center_x, center_y) as part of the position parameters. The sampling of the first component of the grasping pose is used as an example; the sampling principle for other components is the same, only the relevant parameters change. Assume the random variable Y follows a Gaussian distribution with position parameter center_x and scale parameter σ (σ is set in the experiment). 2 =100, which is an empirical parameter), as shown in formula (3), then its probability density function is shown in formula (4). The first component of the pose, i.e. the initialization, will be sampled from this distribution.

[0103] Y~(center x , σ 2 (3)

[0104]

[0105] 2-3:

[0106] More specifically, based on uniform distribution sampling initialization, we will take the sampling of the first component of the pose as an example. Assuming that the random variable U follows a uniform distribution, as shown in formula (5), and width is the width of the part image, its probability density function is shown in formula (6). The first component of the pose, i.e. the initialization, will be sampled from this distribution.

[0107] U~(0, width) (5)

[0108]

[0109] In summary, for the grasping pose localization steps of the grasping device, which can be selected but not limited to the grasping scenario of engineering machinery parts, this invention proposes a two-stage grasping pose localization method based on neural network methods. The first stage is based on neural network methods to construct a category judgment model, preferably using a three-channel DenseNet to improve the accuracy of category judgment, up to 99.3%. The second stage, based on the above category judgment, selectively uses rotation search method or probabilistic search method to improve the accuracy and speed of pose localization from both accuracy and time consumption perspectives. On this basis, for the probabilistic search method, the Grey Wolf Optimization Algorithm (GWO) is selected based on experimental comparison, and its initialization steps are further improved by proposing Gaussian distribution sampling initialization based on centroid and uniform distribution sampling initialization, which further improves the convergence speed and accuracy of the probabilistic search method, thereby further improving the accuracy and speed of pose localization.

[0110] 3. Experimental Comparison

[0111] 3-1:

[0112] Specifically, the following configuration is optional, but not limited to: 11th Gen Intel(R) Core(TM) i5-11300H @ 3.10GHz 3.11GHz hardware platform, 16GB RAM, no GPU acceleration, Windows 10 operating system, Python 3.8 programming language, and the PyTorch deep learning framework used to implement the category judgment model of this invention. Taking suction cup gripping of parts as an example, in the parts image classification dataset, the training set contains 2201 images, including 1470 regular parts images and 731 irregular parts images; the validation set contains 549 images, including 367 regular parts images and 182 irregular parts images.

[0113] In this experiment, nine representative neural network classification models were selected for comparative analysis, including DenseNet, VGGNet, ResNet, MobileNet_V2, ShuffleNet_V2, DenseNet, EfficientNet_V1, RegNet, Vit, and GoogleNet. This demonstrates that the aforementioned category judgment models can be constructed based on any neural network model. This invention constructs numerous forms of category judgment models, with the three-channel, DenseNet-based neural network model being the preferred choice. More specifically, in the pose search for grasping, we conducted a series of tests to determine the relevant hyperparameters of the GWO algorithm: the number of wolves and the total number of iterations. Simultaneously, we compared the search results and time costs of the three algorithms proposed in this invention: GWO, PSO, and centroid rotation search.

[0114] 3-2:

[0115] Specifically, for step T1, the verification of the part category determination:

[0116] 3-2-1: Classification Accuracy Validation

[0117] To evaluate the performance of the type judgment model, this experiment used average accuracy as the evaluation metric and compared the time cost of inference and centroid rotation search using the DenseNet-121 model. First, nine neural network models were trained on the original part image training set and tested on the validation set. For comparison, each selected model was trained with or without transfer learning during training. The experimental results are as follows: Figure 11 As shown, the classification models using transfer learning consistently outperform those without in terms of average accuracy. DenseNet-121 and EfficientNet_V1 achieved the highest average accuracy of 95.6%. It's noteworthy that Vit, which performed best on the ImageNet dataset, did not achieve the expected results. More importantly, without transfer learning, DenseNet-121 has a higher average accuracy than EfficientNet_V1. Therefore, in constructing the category classification model described above, DenseNet, especially DenseNet-121, is preferred as the foundation for the final category classification model.

[0118] To further improve the classification accuracy of the model, this invention proposes steps S1'-S4'. The input includes not only the image of the object to be grasped, but also the image of the object to be grasped and parameters related to maximizing the overlap area of ​​the two images. DenseNet-121 is used for training on a synthetic image dataset. Experimental results are as follows... Figure 12 As shown in the figure, the confusion matrix of the trained DenseNet-121 on the validation set shows an average accuracy of (182+363) / (182+4+363+0)≈99.3%, which is 3.7% higher than the highest performance of 95.6% without synthetic data. Therefore, it can be inferred that adding information such as the splicing suction cup, part size, and actual problem constraints—that is, parameters related to the grasping image and maximizing the overlapping area problem—to the original part images helps the neural network model further improve its classification accuracy.

[0119] 3-2-2: Validation of Classification Time Cost

[0120] Table 2 shows a comparison of the time overhead between DenseNet-121 forward inference and centroid rotation search. Each data point is the average after 10 runs, and the data in parentheses represents the variance of the 10 runs. As shown in Table 2, DenseNet-121 forward inference has a time overhead of only 0.296s, compared to 144.45s for centroid rotation search, a reduction of 487 times. This demonstrates that using a neural network method to construct a category judgment model, first determining whether the object to be grasped is a regular or irregular object, and then selectively applying either rotation search or probabilistic search, significantly speeds up computation compared to existing techniques that first use rotation search and then switch to probabilistic search if grasping fails. This greatly improves the speed of grasping pose localization and enhances the performance of industrial production lines.

[0121] Table 2 shows the time cost of DenseNet-121 and centroid rotation search.

[0122]

[0123] In summary, the experiments demonstrate that using the neural network-based grasping pose localization method proposed in this invention to pre-classify the synthesized image of the object to be grasped can effectively improve the classification accuracy of the model and solve the problem of excessive time consumption in the centroid rotation search.

[0124] 3-3:

[0125] Verification of step T3: Capturing Pose Search

[0126] 3-3-1: Quantitative Analysis

[0127] Before running the Gray Wolf Optimization Algorithm (GWO), it is necessary to determine the key hyperparameters of the GWO algorithm: the number of wolves and the total number of iterations. The following section will provide a quantitative analysis of the GWO algorithm. Figure 12 The paper shows the average ratio score (attraction / gravity) and time cost of the GWO algorithm on an irregular parts dataset with a fixed total number of iterations and different wolf pack sizes.

[0128] Depend on Figure 13As shown in the curve, with the increase in the number of wolves, the average ratio score corresponding to GWO gradually increases, from a minimum of 1.033 to 1.208, an increase of 16.94%. Furthermore, it can be observed that, in the comparison of the two preferred initialization methods of this invention, the centroid-based Gaussian distribution sampling initialization achieved a higher score in populations of no more than 30 wolves, while the uniform distribution sampling initialization achieved a higher score in populations of more than 30 wolves. That is to say, in the grasping pose localization algorithm, when there are more candidate solutions, uniformly distributing these candidate solutions in the solution space is beneficial for discovering better grasping poses. Finally, based on the balance between performance and time cost, this invention preferably uses 30 or 35 for the hyperparameter of wolf population size.

[0129] exist Figure 14 The figure shows the average ratio score and time cost of the GWO algorithm on an irregular parts dataset with a fixed wolf pack size (35) and different total number of iterations. As the curves in the figure show, the average ratio score of GWO gradually increases with the increase in the total number of iterations, from a minimum of 1.160 to 1.228, an increase of 5.86%. However, based on the balance between performance and time cost, this invention preferably uses 30 for the total number of iterations hyperparameter.

[0130] 3-3-2: Qualitative Analysis

[0131] The following will compare the advantages and disadvantages of the preferred GWO pose search algorithm of this invention with other methods. Other methods may be selected, but are not limited to, PSO-Levy and centroid rotation search methods. Figure 15 As shown, the experimental data for comparison are presented, including mean odds scores and time costs. (By...) Figure 15 It can be seen that the average ratio score of the centroid rotation search method is the lowest, at 0.624; PSO-Levy is 1.110; GWO based on centroid-based Gaussian distribution sampling initialization is 1.193 (second highest accuracy), and GWO based on uniform distribution sampling initialization is 1.196 (highest accuracy). Furthermore, in terms of time overhead, the GWO algorithm is superior to PSO-Levy. Therefore, this invention prefers the GWO algorithm, especially the centroid-based Gaussian distribution sampling initialization and the uniform distribution sampling initialization, when locating the grasping pose of irregular parts. This achieves a certain balance between performance and time overhead, greatly improving the positioning speed and accuracy of the grasping pose.

[0132] 4. Conclusion:

[0133] This invention proposes a two-stage grasping pose localization method. Taking part grasping pose search as an example, the first step uses a model such as DenseNet-121 to quickly select irregular parts. A three-channel input method, i.e., an image data synthesis method, is proposed, resulting in an average accuracy of 99.3% for the DenseNet-121 model, approaching 100%, laying a solid foundation for the second-stage grasping pose search. Then, GWO (Gross-Wolf) search is preferred for grasping pose. To further improve GWO performance, this invention proposes initializing the wolf pack using Gaussian or uniform distribution sampling based on the centroid. Experiments on a part image dataset show that the proposed algorithm significantly improves the average grasping ratio score compared to other algorithms. The proposed grasping pose localization method can effectively and quickly match suitable poses from existing data, making significant progress in grasping accuracy and real-time performance, meeting the real-time requirements of industrial scenarios. Of course, this invention's grasping pose localization method, using the suction cup grasping of parts as an example, is not limited to this; other examples include, but are not limited to, gripper grasping of packages.

[0134] On the other hand, the present invention also provides a computer storage medium storing executable program code; the executable program code is used to execute any of the above-mentioned grasping pose localization methods.

[0135] On the other hand, the present invention also provides a terminal device, including a memory and a processor; the memory stores program code that can be executed by the processor; the program code is used to execute any of the above-mentioned grasping pose localization methods.

[0136] For example, the program code can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the program code in the terminal device.

[0137] The terminal device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the terminal device may also include input / output devices, network access devices, buses, etc.

[0138] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0139] The memory can be an internal storage unit of the terminal device, such as a hard drive or RAM. The memory can also be an external storage device of the terminal device, such as a plug-in hard drive, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory can include both internal and external storage units of the terminal device. The memory is used to store the program code and other programs and data required by the terminal device. The memory can also be used to temporarily store data that has been output or will be output.

[0140] The computer storage medium and terminal device described above are created based on the above-described grasping pose positioning method. Their technical functions and beneficial effects will not be elaborated here. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0141] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A category determination method based on neural networks, characterized in that, include: Based on neural networks, a category judgment model is constructed, which takes the image of the object to be grasped, the image of the object to be grasped, and the relevant parameters of the problem of maximizing the overlapping area of ​​the two images as input, and the object to be grasped as regular or irregular as output. Collect several images of the object to be grabbed, images of the object to be grabbed, and samples of parameters related to the problem of maximizing the overlapping area of ​​the two images, and label the images of the object to be grabbed to obtain a sample set; The samples in the sample set are input into the category judgment model in sequence to obtain the trained category judgment model; Collect the current image of the currently grasped object, the current image of the object to be grasped, and the current data of relevant parameters, input them into the trained category judgment model, and determine whether the current object to be grasped is a regular object or an irregular object; The parameters related to the problem of maximizing the overlapping area are the positive and negative extreme values ​​of the maximum rotation angle of the grasping object; the horizontal and vertical coordinates of the center of gravity of the object to be grasped; the positive and negative extreme values ​​of the width of the object to be grasped; and at least one or more sets of the positive and negative extreme values ​​of the height of the object to be grasped. The steps for annotating the captured image include: The center of gravity rotation search method is used to search for the maximum gripping force of the gripper when gripping the object to be gripped; Determine if the maximum gripping force exceeds a set threshold; If yes, then mark the object to be grabbed as a regular object; otherwise, mark the object to be grabbed as an irregular object.

2. The category determination method according to claim 1, characterized in that, The image of the grabbed item is a binarized image of the grabbed item; the image of the object to be grabbed is a binarized image of the object to be grabbed.

3. The category determination method according to claim 1, characterized in that, The steps for annotating the captured image are represented as follows: ； Where i represents the i-th magnetic block in the suction cup, i ranges from 1 to N, N is the total number of magnetic blocks in the suction cup, and S r S represents the contact area between the i-th magnet and the part. i I represents the area of ​​the i-th magnetic block. i F represents the indicator function for the i-th magnetic block; i β represents the rated attraction force of the i-th magnetic block, G represents the weight of the object to be grasped, and β represents the set threshold value of the maximum ratio of attraction force to the weight of the part.

4. The category determination method according to any one of claims 1-3, characterized in that, Category judgment models include: The input layer is used to input the image of the object to be grabbed, the image of the object to be grabbed, and the parameters related to the problem of maximizing the overlapping area of ​​the two images; The feature extraction layer is used to extract features from the input content of the input layer. Fully connected layers are used to fuse features from each layer in the feature extraction layer to obtain classification results; The output layer is used to output the classification results as regular or irregular components.

5. The category determination method according to claim 4, characterized in that, The input layer includes: The first channel is used to input the image of the captured object; The second channel is used to input the image of the object to be captured. The third channel is used to input parameters related to maximizing the overlap area between the image of the object to be grabbed and the image of the object to be grabbed. The compositing channel is used to composite the image of the object to be grabbed and the image of the object to be grabbed.

6. A method for grasping pose localization, characterized in that, include: T1: According to any one of the category determination methods in claims 1-5, determine whether the current object to be grabbed is a regular part or an irregular part; T2: If it is a regular object, the rotation search method is used to locate the current grasping pose of the object to be grasped; T3: If it is an irregular object, a population-based probabilistic search method is used to locate the grasping pose of the object to be grasped.

7. A computer storage medium, characterized in that, It stores executable program code; the executable program code is used to execute the category determination method according to any one of claims 1-5 or the grasping pose localization method according to claim 6.

8. A terminal device, characterized in that, It includes a memory and a processor; the memory stores program code that can be executed by the processor; the program code is used to execute the category determination method according to any one of claims 1-5 or the grasping pose localization method according to claim 6.

Images