Workpiece grasping capability continuous learning method based on mixed image data

Abstract

The application relates to the technical field of image learning, in particular to a workpiece grasping capability continuous learning method based on mixed image data. The method comprises the following steps: acquiring original image data sets of target workpieces, generating virtual image data sets, combining the original image data sets and the virtual image data sets to form mixed image data sets, calling a saved grasping learning model, performing general learning network training on the grasping learning model after implanting a plasticity factor in the grasping learning model, generating a continuous learning model, and analyzing grasping parameters of the target workpieces under actual working conditions based on the continuous learning model. The application implants the plasticity factor in the grasping learning model, retains the grasping features in the process of obtaining the optimal grasping posture by previously executing the workpiece grasping capability continuous learning method, and applies the grasping features to the process of subsequently executing the workpiece grasping capability continuous learning method, so that forgetting of the grasping features is effectively avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image learning, in particular to a workpiece grasping capability continuous learning method based on mixed image data. BACKGROUND

[0002] Robotic grasping has a wide range of applications in industrial manufacturing. By enabling robots to intelligently perceive, plan, and execute precise tasks, robotic grasping models endow robots with the ability to handle diverse workpieces. Ensuring the success rate of grasping is the basis of robotic grasping applications, and deep learning-based robotic grasping models perform well in most cases for unknown objects. These models analyze RGB images to extract visual features, and then predict the optimal grasping pose by evaluating spatial relationships and stability indicators.

[0003] However, existing robotic general grasping detection techniques are difficult to improve the grasping ability for specific target workpieces due to catastrophic forgetting, and the root cause lies in the contradiction between data distribution changes and parameter update mechanisms during model training. When the model is trained on an image dataset, its parameters will converge to a global optimal solution, but subsequent incremental training for specific workpieces will introduce new data distribution, causing the current model to prioritize optimizing the loss function of the current task during backpropagation, thereby overriding the original parameter weights. This mechanism directly weakens the model's ability to remember previously learned grasping features, resulting in catastrophic forgetting. SUMMARY

[0004] To solve or partially solve the problems in the related art, the present application provides a workpiece grasping capability continuous learning method based on mixed image data, which can record the key parameters of the workpiece grasping capability continuous learning method each time it is executed, and avoid catastrophic forgetting.

[0005] The first aspect of the present application provides a workpiece grasping capability continuous learning method based on mixed image data, the workpiece grasping capability continuous learning method comprising:

[0006] obtaining an original image of a target workpiece, generating an original image dataset based on the original image, and generating a virtual image dataset based on the original image dataset;

[0007] combining the original image dataset and the virtual image dataset to form a mixed image dataset;

[0008] calling a grasping learning model and implanting a plasticity factor in the grasping learning model;

[0009] inputting the mixed image dataset into the grasping learning model for general learning network training to generate a continuous learning model;

[0010] The original image of the target workpiece under the actual working condition is input into the continuous learning model, and the grabbing parameter of the target workpiece is analyzed, and the target workpiece is grabbed according to the grabbing parameter;

[0011] The plasticity factor is the contribution utility minimum unit position of the stored continuous learning model after the workpiece grabbing ability continuous learning method is performed on a target workpiece.

[0012] The general learning network is composed of historical data of the execution process of the workpiece grabbing ability continuous learning method.

[0013] The general learning network training is based on the plasticity factor saved after the previous execution of the workpiece grabbing ability continuous learning method, and the training of the grabbing learning model through the mixed image data set in the process of the current execution of the workpiece grabbing ability continuous learning method.

[0014] The grabbing learning model is constructed in the process of the first execution of the workpiece grabbing ability continuous learning method, and the generated continuous learning model is saved after the execution of the workpiece grabbing ability continuous learning method, and is called as the grabbing learning model in the next execution of the workpiece grabbing ability continuous learning method.

[0015] As an embodiment of the first aspect of the application, the step of implanting the plasticity factor into the grabbing learning model comprises:

[0016] The stored continuous learning model and contribution utility minimum unit position of the previous target workpiece after the execution of the workpiece grabbing ability continuous learning method are acquired.

[0017] Based on the continuous learning model and the contribution utility minimum unit position, the same unit in the grabbing learning model in the process of the current execution of the workpiece grabbing ability continuous learning method on the target workpiece is located.

[0018] The same unit in the workpiece grabbing ability continuous learning method used for the target workpiece this time is re-updated and initialized, and after initialization, the plasticity factor of the grabbing learning model is implanted.

[0019] As an embodiment of the first aspect of the application, the contribution utility minimum unit position is determined by the following method:

[0020] The contribution utility is calculated by formula (1) to take the exponential decay average value of the instantaneous weight of each unit.

[0021] (1);

[0022] Wherein, lThe number of times the described workpiece grasping capability continuous learning method is executed in the general learning network in this instance. i The unit order in the grasping learning model trained by the continuous learning method for workpiece grasping capability described in this execution. t For the required time, u l,t [ i [The first character in the general learning network is] the [number]th [character]. l The first execution of the workpiece gripping capability continuous learning method during the second execution process i The contribution utility of each unit at time t. The preset attenuation rate;

[0023] h l,i,t The output value of the calculated unit. w l,i,k,t The connection weight of the calculated unit ，n l+1 For the first l+ The number of units that execute the workpiece gripping capability continuous learning method in one run. k The connection weight between the calculated unit and the two units before and after it;

[0024] Compare the exponentially decaying average of the instantaneous weights of all units, and determine the unit with the smallest exponentially decaying average of instantaneous weights as the unit with the smallest contribution utility;

[0025] The position of the unit with the smallest contribution utility in the grasping learning model is preserved.

[0026] As one embodiment of the first aspect of this application, the training of the general learning network involves multiple mini-batch training cycles;

[0027] After each mini-batch training cycle, the unit with the smallest contribution utility is selected for re-updating and initialization based on the contribution utility, the preset reset ratio, and the maximum number of resetable units.

[0028] Each unit in the grasping learning model has a maturity standard, and the unit with the smallest contribution utility that has reached the maturity threshold is included in the re-update scope;

[0029] In this process, after each mini-batch training cycle, the maturity of each unit in the grasping learning model is increased.

[0030] As one embodiment of the first aspect of this application, the step of generating a virtual image dataset based on the original image dataset includes:

[0031] inputting the original image dataset as a training set into a self-attention generative adversarial network model, and training the self-attention generative adversarial network model;

[0032] generating a target workpiece virtual image using the trained self-attention generative adversarial network model;

[0033] filtering the target workpiece virtual image to retain a high-quality workpiece virtual image;

[0034] performing data labeling on the high-quality workpiece virtual image to generate the virtual image dataset.

[0035] As an embodiment of the first aspect of the present application, the step of training the self-attention generative adversarial network model comprises:

[0036] After inputting the original image dataset into the self-attention generative adversarial network model, the self-attention generative adversarial network model determines a self-attention value of each pixel block in each original image in the original image dataset based on a self-attention mechanism;

[0037] The self-attention generative adversarial network model captures long-distance dependencies and detailed features in each original image based on the self-attention value;

[0038] The target workpiece virtual image is generated based on the long-distance dependencies and detailed features;

[0039] The difference between the target workpiece virtual image and the original image corresponding to the target workpiece virtual image is verified, and when the difference between the target workpiece virtual image and the original image corresponding to the target workpiece virtual image meets a set threshold, it is determined that the target workpiece virtual image passes the verification;

[0040] When the pass rate of the target workpiece virtual image meets a pre-set pass rate threshold, it is determined that the training of the self-attention generative adversarial network model is completed.

[0041] As an embodiment of the first aspect of the present application, the trained self-attention generative adversarial network model generates the target workpiece virtual image with different appearances, postures and details by adjusting a random noise vector and a target workpiece parameter.

[0042] As an embodiment of the first aspect of the present application, the step of filtering the target workpiece virtual image comprises:

[0043] The quality evaluation index of each target workpiece virtual image is calculated by formula (2);

[0044] (2);

[0045] In, y q For the quality evaluation index, x For the target workpiece virtual image, M For the image pixel scale, N () is a normalization operation, l () is transparency, c () is contrast, p () is the Gaussian function of image structure, si For the original image data set, i For the starting serial number of the filtered target workpiece virtual image, j For the starting position of the image pixel;

[0046] The similarity index of each target workpiece virtual image and the original data set is calculated by formula (3);

[0047] (3);

[0048] Wherein, y s For the similarity index of the original data set, f () is a Resnet transformation;

[0049] According to the quality evaluation index and the similarity index, the first quality evaluation value of each target workpiece virtual image is calculated;

[0050] The first quality evaluation value is calculated by formula (4);

[0051] (4);

[0052] y 1 For the first quality evaluation value;

[0053] According to the first quality evaluation value, the target workpiece virtual image is filtered to obtain a preliminary filtered target workpiece virtual image;

[0054] The dissimilarity index between the preliminary filtered target workpiece virtual image is calculated;

[0055] The dissimilarity index is calculated by formula (5);

[0056] (5);

[0057] Wherein, y d For the dissimilarity index, t ia virtual image data set of the target workpiece;

[0058] calculating a second quality evaluation value of each of the preliminary filtered target workpiece virtual images according to the similarity index and the dissimilarity index;

[0059] calculating the second quality evaluation value through formula (6);

[0060] (6);

[0061] wherein, y 2 the second quality evaluation value;

[0062] further filtering the preliminary filtered target workpiece virtual images according to the second quality evaluation value, and determining the remaining images as the high-quality target workpiece virtual images after the filtering is completed.

[0063] As an embodiment of the first aspect of the present application, the original image data set is composed of the original images;

[0064] The original images are preprocessed and data-labeled after being collected;

[0065] The preprocessing is a clipping, scaling and normalization processing of the original images;

[0066] The data labeling is to label the position of the target workpiece grabbed by the robot, the angle of the robot relative to the target workpiece and the required opening width of the robot in the original images.

[0067] As an embodiment of the first aspect of the present application, the step of data-labeling the high-quality target workpiece virtual images comprises:

[0068] constructing a data labeling model, inputting the original image data set into the data labeling model, and training the data labeling ability of the data labeling model;

[0069] After the training is completed, inputting the high-quality target workpiece virtual images into the data labeling model, and automatically completing the data labeling of the high-quality target workpiece virtual images by the data labeling model.

[0070] The technical scheme provided in the application can have the following beneficial effects: the application provides a workpiece grasping capability continuous learning method based on mixed image data, generates a virtual image data set from an original image data set of a target workpiece, combines to form a mixed image data set, calls a saved grasping learning model, implants a plasticity factor, and then performs general learning network training to generate a continuous learning model, and analyzes grasping parameters of the target workpiece under actual working conditions based on the continuous learning model. Based on the plasticity factor implantation mechanism, the continuous learning model obtains grasping features obtained by executing the workpiece grasping capability continuous learning method previously, reduces forgetting of historical grasping features, enables the continuous learning model to continuously learn different target workpiece grasping poses, continuously optimizes analysis capability of the continuous learning model based on this, enhances the ability of the continuous learning model to adapt to new workpieces, can timely correct training errors, avoids performance degradation of the continuous learning model, ensures long-term reliable operation, improves the target workpiece grasping capability, and maintains the robustness of the overall model.

[0071] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the application. BRIEF DESCRIPTION OF DRAWINGS

[0072] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which like reference characters refer to like parts throughout the several views, and in which:

[0073] Figure 1 is an execution flow schematic diagram of the workpiece grasping capability continuous learning method shown in the embodiments of the application;

[0074] Figure 2 is a step schematic diagram of implanting a plasticity factor into a grasping learning model shown in the embodiments of the application;

[0075] Figure 3 is a step schematic diagram of generating a virtual image data set based on an original image data set shown in the embodiments of the application;

[0076] Figure 4 is a step schematic diagram of training a self-attention generative adversarial network model shown in the embodiments of the application;

[0077] Figure 5 is a step schematic diagram of filtering a target workpiece virtual image shown in the embodiments of the application;

[0078] Figure 6 is a step schematic diagram of data labeling a high-quality workpiece virtual image shown in the embodiments of the application. DETAILED DESCRIPTION

[0079] Embodiments of the present application will be described in more detail with reference to the drawings. Although the embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0080] The terminology used in the present application is merely for the purpose of describing particular embodiments and is not intended to limit the present application. As used in the present application and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0081] It should be understood that although the terms "first", "second", "third", etc. can be used in the present application to describe various information, these information should not be limited by these terms. These terms are only used to distinguish the same type of information from each other. For example, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information without departing from the scope of the present application. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0082] Robotic hand grasping has a wide range of applications in industrial manufacturing, and in the prior art, the parameters of the robot hand converge to the global optimal solution during the training of the optimal grasping pose, but subsequent incremental training for the target workpiece introduces new training parameters, causing the model to preferentially optimize the loss function of the current task, thereby covering the original parameter weights and weakening the model's memory ability for previously learned grasping features.

[0083] To solve the above problems, the embodiment of the present application provides a workpiece grasping capability continuous learning method based on mixed image data, and the execution process of the workpiece grasping capability continuous learning method is as shown in Figure 1 As shown in the figure, the workpiece grasping capability continuous learning method comprises:

[0084] S101, obtaining an original image of a target workpiece, generating an original image dataset based on the original image, and generating a virtual image dataset based on the original image dataset.

[0085] Wherein, the number of original images of the target workpiece is 100, which can be increased or decreased according to actual use requirements.

[0086] S102, combine the original image dataset and the virtual image dataset to form a mixed image dataset

[0087] The image data in the original image dataset and the virtual image dataset are data-labeled, the image data of the original image dataset and the virtual image dataset are put into a data folder, the data-labeled image data of the original image dataset and the virtual image dataset are put into a label folder, and the image and the data-labeled image are associated with the same number, and the data and the label folder are used as the mixed image dataset.

[0088] The image data in the original image dataset and the virtual image dataset and the data-labeled image data are arranged in the folder according to the collection order.

[0089] S103, call the grabbing learning model, and implant a plasticity factor in the grabbing learning model.

[0090] S104, input the mixed image dataset into the grabbing learning model to perform general learning network training, and generate a continuous learning model.

[0091] The general learning network training trains the analysis ability of the grabbing learning model to the optimal grabbing posture of the target workpiece, and generates the continuous learning model after the training is completed.

[0092] S105, input the original image of the target workpiece under the actual working condition into the continuous learning model, analyze the grabbing parameters of the target workpiece, and grab the target workpiece according to the grabbing parameters.

[0093] Further, the analyzed grabbing parameters include the position of the manipulator grabbing the workpiece, the angle of the manipulator relative to the workpiece, and the width of the manipulator.

[0094] The plasticity factor is the position of the contribution utility minimum unit stored after the workpiece grabbing capability continuous learning method is executed on a target workpiece.

[0095] After each execution of the workpiece grabbing capability continuous learning method, the generated continuous learning model and the position of the contribution utility minimum unit determined in the general learning network training are stored.

[0096] The general learning network is composed of historical data of the execution process of the workpiece grabbing capability continuous learning method.

[0097] The plasticity factor is stored in the historical data of the execution process of the corresponding workpiece grabbing capability continuous learning method.

[0098] Further, the execution process of the workpiece grabbing capability continuous learning method is arranged in the general learning network according to the execution order.

[0099] The general learning network training is based on the plasticity factor saved by the previous execution of the workpiece grasping capability continuous learning method, and the training of the grasping learning model is performed through the mixed image dataset.

[0100] The grasping learning model is constructed during the first execution of the workpiece grasping capability continuous learning method, and the generated continuous learning model is saved after the execution of the workpiece grasping capability continuous learning method, which is called as the grasping learning model during the next execution of the workpiece grasping capability continuous learning method.

[0101] The grasping learning model is preferably a convolutional neural network model, which can be replaced by other network models such as GR-ConyNet, SKGNet, etc. according to actual use requirements.

[0102] Further, during the first execution of the workpiece grasping capability continuous learning method, the plasticity factor is not implanted, and only the current continuous learning model obtained by the execution of the workpiece grasping capability continuous learning method and the contribution utility minimum unit position are recorded.

[0103] Specifically, during the first execution of the workpiece grasping capability continuous learning method, the mixed image dataset is obtained through steps S101-S102, and the grasping learning model is constructed.

[0104] After the construction is completed, the mixed image dataset is input into the grasping learning model for general learning network training, and a continuous learning model is generated.

[0105] The generated continuous learning model and the contribution utility minimum unit position determined in the general learning network training are stored.

[0106] The original image of the target workpiece under actual working conditions is transmitted into the generated continuous learning model, and the grasping parameters of the target workpiece are analyzed, and the target workpiece is grasped according to the grasping parameters.

[0107] During the second execution of the workpiece grasping capability continuous learning method, after the mixed image dataset is obtained through steps S101-S102, the stored continuous learning model is called as the grasping learning model, and the plasticity factor is implanted based on the stored contribution utility minimum unit position.

[0108] The grasping learning model is trained through the new mixed image dataset of the target workpiece, and a new continuous learning model is generated.

[0109] The generated new continuous learning model and the new contribution utility minimum unit position determined in the general learning network training are stored as the plasticity factor, which is called during the next execution of the workpiece grasping capability continuous learning method.

[0110] The original image of the new target workpiece under the actual working condition is input into the generated continuous learning model, and the grabbing parameters of the new target workpiece are analyzed, and the new target workpiece is grabbed according to the grabbing parameters.

[0111] In addition to the first execution of the workpiece grabbing capability continuous learning method, each subsequent execution of the workpiece grabbing capability continuous learning method calls the continuous learning model stored by the last execution of the workpiece grabbing capability continuous learning method and the contribution utility minimum unit position as a plasticity factor, and performs plasticity factor implantation.

[0112] In this embodiment, a virtual image data set is generated through the original image data set of the target workpiece, and a hybrid image data set is formed by combining, and a general learning network training is performed after calling the saved grabbing learning model implanting the plasticity factor, to generate a continuous learning model, and the grabbing parameters of the target workpiece under the actual working condition are analyzed based on the continuous learning model. Based on the plasticity factor implantation mechanism, the continuous learning model obtains the grabbing features obtained by the previous execution of the workpiece grabbing capability continuous learning method, reduces the forgetting of historical grabbing features, enables the continuous learning model to continuously learn different target workpiece grabbing poses, and continuously optimizes the analysis capability of the continuous learning model based on this, enhances the ability of the continuous learning model to adapt to new workpieces, and can timely correct training errors, avoid performance degradation of the continuous learning model, and ensure long-term reliable operation, while improving the target workpiece grabbing capability and maintaining the robustness of the overall model.

[0113] In the embodiments of the present application, the step of implanting the plasticity factor to the grabbing learning model is as shown in Figure 2 , including:

[0114] S201, obtaining the stored continuous learning model and the contribution utility minimum unit position of the previous target workpiece after executing the workpiece grabbing capability continuous learning method.

[0115] The stored contribution utility minimum unit position is the distribution position of the unit with the minimum contribution utility in the model in the continuous learning model.

[0116] S202, based on the continuous learning model and the contribution utility minimum unit position, positioning the same unit in the grabbing learning model in the process of executing the workpiece grabbing capability continuous learning method on the target workpiece this time.

[0117] S203, re-updating and initializing the same unit in the workpiece grabbing capability continuous learning method used for the target workpiece this time, and completing the plasticity factor implantation to the grabbing learning model after initialization.

[0118] Wherein, in the process of training the general learning network of the grasping learning model, the change range of the unit in the grasping learning model will decrease with the training, at this time the learning ability of the grasping learning model decreases, since the change range of the contribution utility minimum unit is the smallest, by positioning the contribution utility minimum unit position, the contribution utility minimum unit is re-updated and initialized, the change range of the contribution utility minimum unit is reset to the initial range, and the learning ability of the unit is enhanced.

[0119] In the embodiment, by acquiring the plasticity factor of the previous target workpiece after the workpiece grasping ability continuous learning method is executed, and positioning the unit that needs to be re-updated based on the plasticity factor, the learning ability of the workpiece grasping ability continuous learning method is enhanced, the limitation of the historical target workpiece on the training of the current grasping learning model is reduced, and the grasping ability of the grasping learning model for new target workpieces is improved.

[0120] In the embodiment of the present application, the position of the contribution utility minimum unit is determined by the following method:

[0121] The contribution utility is calculated by formula (1) to take the exponentially decaying average value of the instantaneous weight of each unit;

[0122] (1);

[0123] Wherein, l is the number of times of executing the workpiece grasping ability continuous learning method in the general learning network this time, i is the order of the unit in the grasping learning model trained by the workpiece grasping ability continuous learning method this time, t is the required time, u l,t is the contribution utility of the unit in the general learning network in the process of executing the workpiece grasping ability continuous learning method the i th time at time t, l is the contribution utility of the unit in the general learning network in the process of executing the workpiece grasping ability continuous learning method the i th time at time t, is a preset decay rate, generally set to 0.99.

[0124] In formula (1), the h l,i,t is the output value of the calculated unit, w l,i,k,t is the connection weight of the calculated unit, n l+1 is the number of units of the workpiece grasping ability continuous learning method executed the l+ 1th time, k is the connection weight of the calculated unit and the two units before and after it.

[0125] Specifically, the output value of the calculation unit is a linear weighted sum of the node value and the weight value of the same unit when the workpiece grasping capability continuous learning method is last executed, the node value of the unit is randomly assigned when the workpiece grasping capability continuous learning method is first executed, and the node value of the unit is calculated in the general learning network training when the workpiece grasping capability continuous learning method is subsequently executed. After each calculation is completed, it is stored.

[0126] The exponential decay average value of the instant weight of all units is compared, and the unit with the minimum exponential decay average value of the instant weight is determined as the contribution utility minimum unit.

[0127] The position of the contribution utility minimum unit in the grasping learning model is saved.

[0128] In the embodiment, by determining and saving the position of the contribution utility minimum unit, the workpiece grasping capability continuous learning method can quickly locate the position of the contribution utility minimum unit and perform re-updating in the training of the grasping learning model, enhance the utilization rate of the unit, and improve the training accuracy of the grasping learning model.

[0129] In the embodiment of the present application, in the general learning network training, there are multiple small batch training cycles.

[0130] Wherein, each small batch training cycle selects part of images in the mixed image dataset for training, and the number of images selected each time is divided into the mixed image dataset according to the number of small batch training cycles.

[0131] After each small batch training cycle ends, the unit with the minimum contribution utility is selected for re-updating and initialization according to the contribution utility, the preset reset ratio and the maximum resettable unit quantity.

[0132] Wherein, the number of units re-updated each time is calculated by formula (7).

[0133] (7);

[0134] C l,t The number of units required for re-updating after the current small batch training cycle ends, The preset reset ratio, p l The maximum resettable unit quantity, C l,t-1 The number of units required for re-updating after the last small batch training cycle ends.

[0135] The preset reset ratio and the maximum resettable unit quantity are set according to actual use requirements. The preset reset ratio is generally set to 0.1%, and the maximum resettable unit quantity is generally set to re-set 20 units at most each time.

[0136] Each unit in the grasping learning model has a maturity standard, and the unit with the minimum contribution utility that reaches the maturity threshold is included in the range of re-updating.

[0137] Wherein, after each mini-batch training cycle, the maturity of each unit in the grasping learning model is increased.

[0138] In this embodiment, by dividing the training cycle into multiple mini-batch training cycles, the consumption of hardware resources during model training is significantly reduced, so that the model can be trained by a large-scale data set, and only part of the images are used for training in each training, improving the training efficiency. In addition, the training process of the mini-batch training cycle is relatively stable, which is convenient for real-time monitoring of model performance and dynamic adjustment. In addition, the model can be gradually deepened and trained with the addition of new images, and the grasping ability of the model for the target workpiece is enhanced.

[0139] In the embodiments of the present application, the step of generating a virtual image data set based on the original image data set is as shown in Figure 3 As shown in the figure, it includes:

[0140] S301, input the original image data set as a training set into the self-attention generative adversarial network model, and train the self-attention generative adversarial network model.

[0141] S302, using the trained self-attention generative adversarial network model to generate the target workpiece virtual image.

[0142] Wherein, the trained self-attention generative adversarial network model generates 150,000 target workpiece virtual images based on the original image, and the number of generated target workpiece virtual images can be increased or decreased according to actual needs.

[0143] S303, filtering the target workpiece virtual image and retaining the high-quality workpiece virtual image.

[0144] S304, data labeling of the high-quality workpiece virtual image to generate a virtual image data set.

[0145] In this embodiment, the self-attention generative adversarial network model is trained by the original image data set, which can generate a large number of target workpiece virtual images and form a virtual image data set, thereby providing a large number of samples covering different scenarios for the training of the grasping learning model, reducing the image acquisition cost and alleviating the problem of insufficient image data. At the same time, due to the large amount of image data obtained as the training set, the grasping learning model can be trained to access more diverse image data, thereby reducing the risk of overfitting and enhancing the generalization ability of the grasping learning model.

[0146] In the embodiments of the present application, the step of training the self-attention generative adversarial network model comprises the following steps as shown in Figure 4

[0147] S401, after inputting the original image dataset into the self-attention generative adversarial network model, the self-attention generative adversarial network model determines the self-attention value of each pixel block in each original image in the original image dataset based on the self-attention mechanism.

[0148] The self-attention mechanism can analyze the importance of each pixel block, and assign a corresponding self-attention value to each pixel block based on the importance. The higher the self-attention value of a pixel block, the more important the feature represented by the pixel block in the original image.

[0149] S402, the self-attention generative adversarial network model captures long-range dependencies and detailed features in each original image based on the self-attention value.

[0150] S403, generating a target workpiece virtual image based on the long-range dependencies and the detailed features.

[0151] The long-range dependencies can determine the dependencies between each pixel block in the original image, and the detailed features can highlight the important features of the current pixel block.

[0152] S404, verifying the difference between the target workpiece virtual image and the original image corresponding to the target workpiece virtual image, and determining that the target workpiece virtual image passes the verification when the difference between the target workpiece virtual image and the original image corresponding to the target workpiece virtual image meets the set threshold.

[0153] Further, the self-attention generative adversarial network model has a generator unit and a discriminator unit. The generator unit generates a target workpiece virtual image according to the long-range dependencies and the detailed features of the original image, and the discriminator unit analyzes the features in the original image, summarizes a verification threshold, and verifies the authenticity of the target workpiece virtual image based on the verification threshold.

[0154] S405, when the pass rate of the target workpiece virtual image meets the pre-set pass rate threshold, it is determined that the training of the self-attention generative adversarial network model is completed.

[0155] During the detection of the target workpiece virtual image generated by the generator unit in the initial stage, the pass rate is low. The generator unit will continuously update the image generation method based on the long-range dependencies and the detailed features of the original image until the probability of passing the verification of the generated target workpiece virtual image reaches the pass rate threshold, and the training of the self-attention generative adversarial network model is stopped.

[0156] ​In the embodiment, the original image dataset is input into the self-attention generative adversarial network model for training, and the self-attention generative adversarial network model is enabled to capture long-distance dependencies and detailed features in the image based on the self-attention mechanism, so as to generate a target workpiece virtual image with rich details. In addition, the training process of the self-attention generative adversarial network model can gradually optimize the generator and the discriminator, further improve the quality of the generated image, enhance the authenticity of the image, and make the generated image better meet the requirements, thereby providing a large amount of image data support for the training of the subsequent grasping learning model.

[0157] In the embodiment of the present application, the trained self-attention generative adversarial network model generates target workpiece virtual images with different appearances, postures and details by adjusting random noise vectors and target workpiece parameters.

[0158] In the embodiment, the self-attention generative adversarial network model is used to generate target workpiece virtual images with different appearances, postures and details, which improves the image generation speed and greatly enriches the diversity of the target workpiece virtual images.

[0159] In the embodiment of the present application, the step of filtering the target workpiece virtual image includes Figure 5 as shown in the figure.

[0160] S501, calculate the quality evaluation index of each target workpiece virtual image by formula (2).

[0161] (2);

[0162] wherein, y q is the quality evaluation index, x is the target workpiece virtual image, M is the image pixel size, N is the normalization operation, l is the transparency, c is the contrast, p is the Gaussian function of the image structure, si is the original image dataset, i is the starting serial number of the filtered target workpiece virtual image, j is the starting position of the image pixel.

[0163] S502, calculate the similarity index of each target workpiece virtual image and the original dataset by formula (3).

[0164] (3);

[0165] wherein, y sa similarity index of the original data set, f () is a Resnet transformation.

[0166] S503, calculate the first quality evaluation value of each target workpiece virtual image according to the quality evaluation index and the similarity index.

[0167] The first quality evaluation value is calculated by formula (4).

[0168] (4);

[0169] y 1 The first quality evaluation value is y 1 The first quality evaluation value is y 1 The first quality evaluation value is y 1 The first quality evaluation value is y 1 The first quality evaluation value is

[0170] y 1 The first quality evaluation value is

[0171] S504, filtering the target workpiece virtual image according to the first quality evaluation value to obtain the preliminary filtered target workpiece virtual image.

[0172] Wherein, after filtering according to the first quality evaluation value, the number of preliminary filtered target workpiece virtual images obtained is 8000, and the number of image filtering can be increased or decreased according to actual needs. The first quality evaluation value is the similarity evaluation standard of the target workpiece virtual image and the original image, the smaller the first quality evaluation value, the lower the similarity, and the target workpiece virtual image with low similarity is filtered.

[0173] S505, calculate the dissimilarity index between the preliminary filtered target workpiece virtual images.

[0174] The dissimilarity index is calculated by formula (5);

[0175] (5);

[0176] Wherein, y d The dissimilarity index is t i The virtual image data set is

[0177] S506, calculate the second quality evaluation value of each preliminary filtered target workpiece virtual image according to the similarity index and the dissimilarity index.

[0178] The second quality evaluation value is calculated by formula (6).

[0179] (6);

[0180] wherein, y 2 The second quality evaluation value.

[0181] S507, further filtering the preliminary filtered target workpiece virtual image according to the second quality evaluation value, and determining the remaining image as a high-quality workpiece virtual image after the filtering is completed.

[0182] Wherein, the filtering according to the second quality evaluation value is to select 2000 high-quality workpiece virtual images from 8000 preliminary filtered target workpiece virtual images as a virtual data set, the high-quality workpiece virtual images are not repeated, and the number of images in the virtual data set can be increased or reduced according to actual needs.

[0183] In this embodiment, through twice filtering of the target workpiece virtual image, the non-repeated high-quality workpiece virtual image is obtained, the quality of the virtual image data set is optimized, the diversity of the images in the training set is greatly increased, and the generalization ability of the grasping learning model is enhanced based on the rich image data set.

[0184] In the embodiment of the present application, the original image data set is composed of original images.

[0185] The original image is preprocessed and data labeled after being collected.

[0186] The preprocessing is to crop, scale and normalize the original image.

[0187] The data labeling is to label the position of the manipulator grasping target workpiece in the original image, the angle of the manipulator relative to the target workpiece, and the required opening width of the manipulator.

[0188] In this embodiment, by preprocessing the original image, the quality of the original image is enhanced, and the adaptability and learning efficiency of the grasping learning model to input data are improved. Moreover, through data labeling, the grasping learning model can understand the image content, thereby improving the accuracy of the grasping learning model in understanding the image. In addition, preprocessing and labeling can also reduce the sensitivity of the grasping learning model to noise and abnormal data, and ensure that the training process is more stable.

[0189] In the embodiment of the present application, the step of data labeling the high-quality workpiece virtual image is as shown in Figure 6 , comprising:

[0190] S601, constructing a data labeling model, inputting the original image data set into the data labeling model, and training the data labeling ability of the data labeling model.

[0191] The data labeling model is a convolutional neural network model, and the model type is consistent with the crawling learning model, but is not the same model. The model can be replaced with the crawling learning model according to actual needs, and the model types replaced by the two are consistent.

[0192] S602, after the training is completed, input the high-quality workpiece virtual image to the data labeling model, and automatically complete the data labeling of the high-quality workpiece virtual image by the data labeling model.

[0193] In the embodiment, the data labeling ability of the data labeling model is trained by the original image data set. When facing a large number of image data sets, the labeling efficiency is significantly improved, and the time and cost required for manual labeling are reduced. Secondly, the data labeling model can provide consistent and high-quality labeling results, avoid subjective differences and errors that may occur in manual labeling, and thus improve the accuracy and reliability of the image data. In addition, the labeling model can learn the detailed features in the image, and the labeling accuracy will be continuously improved as the training deepens, providing better training data for the training of the crawling learning model.

[0194] The above has described the embodiments of the present application, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes are obvious to those skilled in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles, practical application or improvement of technology in the market of the embodiments, or to enable other ordinary skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for continuous learning of workpiece grasping capability based on hybrid image data, characterized in that, The continuous learning method for workpiece gripping capability includes: Acquire the original image of the target workpiece, generate an original image dataset based on the original image, and generate a virtual image dataset based on the original image dataset; The original image dataset is combined with the virtual image dataset to form a hybrid image dataset; Invoke the crawling learning model and implant a plasticity factor into the crawling learning model; The hybrid image dataset is input into the grasping learning model to train a general learning network, thereby generating a continuous learning model. The original image of the target workpiece under actual working conditions is input into the continuous learning model to analyze the grasping parameters of the target workpiece, and the target workpiece is grasped according to the grasping parameters. Wherein, the plasticity factor is the stored continuous learning model and the position of the unit with the smallest contribution utility after the continuous learning method for grasping the workpiece is applied to a target workpiece. The general learning network is composed of historical data from the execution process of the continuous learning method for workpiece grasping capability; The general learning network is trained by training the grasping learning model on the mixed image dataset during the current execution of the workpiece grasping capability continuous learning method, based on the plasticity factor saved from the previous execution of the workpiece grasping capability continuous learning method. The grasping learning model is constructed during the first execution of the continuous learning method for workpiece grasping capability. After the continuous learning method for workpiece grasping capability is completed, the generated continuous learning model is saved and reserved for use as the grasping learning model when the continuous learning method for workpiece grasping capability is executed next time.

2. The continuous learning method for workpiece gripping capability according to claim 1, characterized in that, The step of implanting a plasticity factor into the grasping learning model includes: After the previous target workpiece was executed using the workpiece grasping capability continuous learning method, the stored continuous learning model and the position of the unit with the smallest contribution utility were obtained. Based on the continuous learning model and the position of the unit with the smallest contribution utility, locate the same unit in the grasping learning model during the current execution of the continuous learning method for grasping the target workpiece; The same units in the continuous learning method for workpiece grasping capability used on the target workpiece are updated and initialized again. After initialization, the plasticity factor is implanted into the grasping learning model.

3. The continuous learning method for workpiece gripping capability according to claim 2, characterized in that, The location of the unit with the smallest contributing utility is determined by the following method: The contribution utility is calculated by formula (1), which takes the exponentially decaying average value of the instantaneous weight of each unit. （1）； in, l The number of times the described workpiece grasping capability continuous learning method is executed in the general learning network in this instance. i The unit order in the grasping learning model trained by the continuous learning method for workpiece grasping capability described in this execution. t For the required time, u l,t [ i [The first character in the general learning network is] the [number]th [character]. l The first execution of the workpiece gripping capability continuous learning method during the second execution process i The contribution utility of each unit at time t. The preset attenuation rate; h l,i, t The output value of the calculated unit. w l,i,k,t The connection weights of the calculated units, n l+1 For the first l+ The number of units that execute the workpiece gripping capability continuous learning method in one run. k The connection weight between the calculated unit and the two units before and after it; Compare the exponentially decaying average of the instantaneous weights of all units, and determine the unit with the smallest exponentially decaying average of instantaneous weights as the unit with the smallest contribution utility; The position of the unit with the smallest contribution utility in the grasping learning model is preserved.

4. The continuous learning method for workpiece gripping capability according to claim 3, characterized in that, The training of the general learning network involves multiple mini-batch training cycles. After each mini-batch training cycle, the unit with the smallest contribution utility is selected for re-updating and initialization based on the contribution utility, the preset reset ratio, and the maximum number of resetable units. Each unit in the grasping learning model has a maturity standard, and the unit with the smallest contribution utility that has reached the maturity threshold is included in the re-update scope; In this process, after each mini-batch training cycle, the maturity of each unit in the grasping learning model is increased.

5. The continuous learning method for workpiece gripping capability according to claim 1, characterized in that, The steps for generating a virtual image dataset based on the original image dataset include: The original image dataset is used as the training set and input into the self-attention generative adversarial network model to train the self-attention generative adversarial network model. The trained self-attention generative adversarial network model is used to generate a virtual image of the target workpiece; the virtual image of the target workpiece is filtered to retain high-quality virtual images; Data annotation is performed on the high-quality virtual image of the workpiece to generate the virtual image dataset.

6. The continuous learning method for workpiece gripping capability according to claim 5, characterized in that, The steps for training the self-attention generative adversarial network model include: After the original image dataset is input into the self-attention generative adversarial network model, the self-attention generative adversarial network model determines the self-attention value of each pixel block in each original image in the original image dataset based on the self-attention mechanism. The self-attention generative adversarial network model captures long-range dependencies and detailed features in each original image based on self-attention values; The virtual image of the target workpiece is generated based on the long-distance dependency and detailed features; The difference between the virtual image of the target workpiece and the original image corresponding to the virtual image of the target workpiece is checked. When the difference between the virtual image of the target workpiece and the original image corresponding to the virtual image of the target workpiece meets a set threshold, the virtual image of the target workpiece is determined to pass the check. When the pass rate of the virtual image of the target workpiece meets the preset pass rate threshold, the training of the self-attention generative adversarial network model is determined to be complete.

7. The continuous learning method for workpiece gripping capability according to claim 6, characterized in that, The trained self-attention generative adversarial network model generates virtual images of the target workpiece with different appearances, poses, and details by adjusting the random noise vector and the target workpiece parameters.

8. The continuous learning method for workpiece gripping capability according to claim 7, characterized in that, The steps for filtering the virtual image of the target workpiece include: The quality evaluation index of each target workpiece virtual image is calculated using formula (2); （2）； in, y q The quality evaluation index is referred to above. x The virtual image of the target workpiece. M Image pixel scale, N () represents the normalization operation. l () represents transparency. c () represents contrast. p () is the Gaussian function for the image structure. si For the original image dataset, i The starting number of the filtered target workpiece virtual image. j This represents the starting position of the image pixels; The similarity index between each virtual image of the target workpiece and the original image dataset is calculated using formula (3); （3）； in, y s The similarity index of the original image dataset. f () represents the ResNet transform; A first quality evaluation value is calculated for each of the target workpiece virtual images based on the quality evaluation index and the similarity index; The first quality evaluation value is calculated using formula (4); （4）； y 1 This is the first quality evaluation value; The target workpiece virtual image is filtered based on the first quality evaluation value to obtain a preliminary filtered target workpiece virtual image; Calculate the dissimilarity index between the virtual images of the preliminary filtered target workpieces; The dissimilarity index is calculated using formula (5); （5）； in, y d The dissimilarity index is... t i The virtual image dataset; A second quality evaluation value is calculated for each of the preliminary filtered target workpiece virtual images based on the similarity index and the dissimilarity index; The second quality evaluation value is calculated using formula (6); （6）； in, y 2 This is the second quality evaluation value; The virtual image of the target workpiece is further filtered based on the second quality evaluation value. After the filtering is completed, the remaining image is determined as the high-quality virtual image of the workpiece.

9. The continuous learning method for workpiece gripping capability according to claim 8, characterized in that, The original image dataset consists of the original images; The original images are preprocessed and labeled after acquisition; The preprocessing involves cropping, scaling, and normalizing the original image. The data annotations are used to mark the position of the robot arm grasping the target workpiece in the original image, the angle of the robot arm relative to the target workpiece, and the required opening width of the robot arm.

10. The continuous learning method for workpiece gripping capability according to claim 9, characterized in that, The steps for data annotation of the high-quality workpiece virtual image include: Construct a data annotation model by inputting the original image dataset into the data annotation model and training the data annotation capability of the data annotation model; After training is completed, the high-quality virtual image of the workpiece is input into the data annotation model, and the data annotation model automatically completes the data annotation of the high-quality virtual image of the workpiece.

Images