Method, device and electronic equipment for grasping deep frame workpiece

Abstract

The present invention provides a method, device and electronic device for realizing deep frame workpiece grasping, constructing multiple workpiece three-dimensional models based on existing workpiece information, stacking multiple workpiece three-dimensional models to a deep frame model, and obtaining multiple workpiece simulation images under different simulation conditions. Based on the constructed structural similarity index, multiple workpiece simulation images are screened out from multiple workpiece simulation images, and the target information included in each screened workpiece simulation image is processed using a wavelet transform processing method to reduce the high reflectivity of the workpiece simulation image. The constructed network model is trained using the processed workpiece simulation image to obtain an instance segmentation model required for deep frame workpiece grasping. In this solution, diversified and highly authentic simulation data is generated to ensure the generalization ability and robustness of the trained model.

Description

Technical Field

[0001] The present invention relates to the field of control technology, and in particular to a method, a device and an electronic device for grasping a deep frame workpiece. Background Art

[0002] With the development of industrial automation and intelligence, the application of robot grasping technology in manufacturing, logistics, warehousing and other fields is becoming more and more widespread. However, in the face of disordered grasping of complex workpieces, traditional robot grasping systems still face many challenges. The complexity of disordered grasping tasks is reflected in the wide variety and shape of workpieces. The workpieces are not fixed in posture when stacked and placed, and there may be mutual occlusion and interference. This puts higher requirements on the robot's perception, recognition and grasping capabilities.

[0003] With the rise of deep learning technology, instance segmentation networks have made breakthrough progress in image recognition, object detection and other fields. By learning a large amount of labeled data, deep learning models can automatically extract and utilize high-dimensional features such as the shape and texture of workpieces, thereby significantly improving the accuracy and robustness of instance segmentation. However, the success of deep learning models depends on a large amount of high-quality training data and the design of network models. Therefore, existing instance segmentation methods have at least the following limitations for industrial scenarios:

[0004] 1. Difficulty in data set production: Complex workpieces vary in shape, structure, and surface characteristics. It is very difficult to obtain high-quality training data sets covering a wide range of workpiece types. In industrial scenarios, there are many different workpieces with complex specifications, and the collection conditions in the production environment are often varied. It is time-consuming and laborious to collect a sufficient amount of real data. At the same time, due to the high reflectivity of metal workpieces under different lighting conditions, their actual colors may be distorted in the image, making the shape and details of the metal objects unrecognizable. In addition, even if a sufficient amount of real workpiece data is collected, it is extremely challenging to accurately annotate these images. Manual annotation is often subjective, resulting in unstable performance of the detection model and prone to errors when faced with workpieces with complex geometric structures or textures.

[0005] 2. Insufficient model robustness in complex scenarios: In disordered stacking scenarios, occlusion between workpieces will cause the detection model to be unable to obtain complete workpiece features, affecting recognition accuracy and causing missed detections. At the same time, the existing model has limited processing capabilities for workpieces with complex shapes and large size variations, especially for workpieces with irregular shapes or multiple postures. The model is prone to false detection or missed detection. In addition, in actual industrial environments, lighting conditions are usually complex and changeable, and factors such as reflections and shadows will interfere with existing models, leading to an increase in false detections.

[0006] It can be seen that the model network based on deep learning relies on a large number of high-quality annotated data sets, while the real data collection and annotation of complex workpieces are costly and difficult. At the same time, the disordered placement of workpieces and various complex scene factors such as lighting, occlusion, and reflection increase the difficulty of model detection. The existing network performs poorly when facing workpieces with complex shapes and diverse textures, and is prone to missed detection or false detection. Summary of the invention

[0007] The purpose of the embodiments of the present invention is to provide a method, device and electronic device for deep frame workpiece grasping, so as to ensure the generalization ability and robustness of the trained model.

[0008] In a first aspect, the present invention provides a method for grasping a deep frame workpiece, the method comprising:

[0009] Construct multiple workpiece 3D models based on existing workpiece information;

[0010] stacking the multiple workpiece three-dimensional models into a deep frame model, and obtaining multiple workpiece simulation images under different simulation conditions;

[0011] Based on the constructed structural similarity index, screening out a plurality of workpiece simulation images from the plurality of workpiece simulation images;

[0012] Using a wavelet transform processing method to process the target information included in each of the screened workpiece simulation images, so as to reduce the high reflectivity of the workpiece simulation image;

[0013] The constructed network model is trained using the processed workpiece simulation images to obtain an instance segmentation model required for deep frame workpiece grasping.

[0014] In an optional embodiment, the step of selecting a plurality of workpiece simulation images from the plurality of workpiece simulation images based on the constructed structural similarity index comprises:

[0015] Calculating a structural similarity index between each of the workpiece simulation images and an existing real workpiece image based on the constructed structural similarity index;

[0016] The workpiece simulation images whose structural similarity index is greater than or equal to a preset threshold are screened out.

[0017] In an optional embodiment, the step of calculating the structural similarity index between each of the simulated workpiece images and the existing real workpiece images based on the constructed structural similarity index comprises:

[0018] Dividing each of the workpiece simulation images and the existing real workpiece image into a plurality of image windows respectively;

[0019] Calculating the image mean and image variance of each image window of each of the workpiece simulation images and the real workpiece image;

[0020] For each image window in each of the workpiece simulation images, the covariance is calculated based on the image mean and image variance of the image window and the image mean and image variance of the image window corresponding to the image window in the real workpiece image;

[0021] Calculating a structural similarity index based on the image mean, image variance and covariance of the image window;

[0022] A final structural similarity index is calculated based on multiple structural similarity indexes of multiple image windows of the workpiece simulation image.

[0023] In an optional embodiment, the step of processing the target information included in each of the filtered workpiece simulation images using a wavelet transform processing method includes:

[0024] For each of the screened workpiece simulation images, extract low-frequency information included in the workpiece simulation image as target information;

[0025] The target information is subjected to histogram averaging processing.

[0026] In an optional embodiment, the step of extracting the workpiece simulation image including low-frequency information comprises:

[0027] Processing the workpiece simulation image using a low-pass filter and a high-pass filter;

[0028] The processed workpiece simulation image is down-sampled to decompose the workpiece simulation image into low-frequency information and high-frequency information.

[0029] In an optional embodiment, each of the workpiece simulation images has a real label, and the real label includes a real category and a real frame;

[0030] The step of training the constructed network model using the processed workpiece simulation images to obtain the instance segmentation model required for deep frame workpiece capture includes:

[0031] Importing the processed workpiece simulation images into the constructed network model for processing, and outputting the predicted category and predicted frame of the workpiece simulation image;

[0032] Under the guidance of the constructed loss function, the model parameters of the network model are adjusted and the training is continued until the instance segmentation model required for deep frame workpiece grasping is obtained when the preset requirements are met;

[0033] Among them, the loss function includes a category loss function, a prediction box regression loss function and a segmentation loss function. The category loss function is constructed based on the real category and the predicted category of the workpiece simulation image, and the prediction box regression loss function is constructed based on the real box and the predicted box of the workpiece simulation image.

[0034] In an optional embodiment, the network model includes a feature extraction module, a feature fusion module and a post-processing module;

[0035] The feature extraction module includes a convolution layer, a conversion layer and a pooling layer, and a multi-scale attention module is connected to the back end of the conversion layer;

[0036] The step of training the constructed network model using the processed workpiece simulation images comprises:

[0037] Importing the processed workpiece simulation images into the feature extraction module included in the constructed network model, and processing the workpiece simulation images through the convolution layer, conversion layer, pooling layer and multi-scale attention module in the feature extraction module to obtain feature extraction results;

[0038] The feature extraction results are fused by the feature fusion module to obtain a fusion result;

[0039] The fusion processing result is post-processed by using the post-processing module to obtain a prediction result of the workpiece simulation image.

[0040] In an optional embodiment, the feature fusion module includes a conversion layer, a convolution layer, a splicing layer, an upsampling layer and an attention fusion module;

[0041] The attention fusion module is connected to the multi-scale attention module in the feature extraction module and the conversion layer in the feature fusion module, and is used to fuse the output results of the multi-scale attention module and the conversion layer, and output the fused results to the post-processing module.

[0042] In a second aspect, the present invention provides a device for grasping a deep frame workpiece, the device comprising:

[0043] A construction module, used for constructing multiple workpiece three-dimensional models based on existing workpiece information;

[0044] An acquisition module, used for stacking the multiple workpiece three-dimensional models into a deep frame model, and obtaining multiple workpiece simulation images under different simulation conditions;

[0045] A screening module, configured to screen out a plurality of workpiece simulation images from the plurality of workpiece simulation images based on the constructed structural similarity index;

[0046] A processing module, used for processing the target information included in each of the screened workpiece simulation images by using a wavelet transform processing method, so as to reduce the high reflectivity of the workpiece simulation image;

[0047] The training module is used to train the constructed network model using the processed workpiece simulation images to obtain the instance segmentation model required for deep frame workpiece grasping.

[0048] In a third aspect, the present invention provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, and when the processor executes the computer program, the steps of the method described in any one of the aforementioned implementation modes are implemented.

[0049] The present invention provides a method, device and electronic device for realizing deep frame workpiece grasping, constructing multiple workpiece three-dimensional models based on existing workpiece information, stacking multiple workpiece three-dimensional models to a deep frame model, and obtaining multiple workpiece simulation images under different simulation conditions. Based on the constructed structural similarity index, multiple workpiece simulation images are screened out from multiple workpiece simulation images, and the target information included in each screened workpiece simulation image is processed using a wavelet transform processing method to reduce the high reflectivity of the workpiece simulation image. The constructed network model is trained using the processed workpiece simulation image to obtain an instance segmentation model required for deep frame workpiece grasping. In this solution, diversified and highly authentic simulation data is generated to ensure the generalization ability and robustness of the trained model. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments of the present invention are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0051] Figure 1 A flow chart of a method for implementing deep frame workpiece grasping provided by an embodiment of the present invention;

[0052] Figure 2 for Figure 1 Flow chart of sub-steps included in step S13;

[0053] Figure 3 for Figure 2 Flow chart of sub-steps included in step S131;

[0054] Figure 4 for Figure 1Flow chart of sub-steps included in step S14;

[0055] Figure 5 for Figure 4 Flow chart of sub-steps included in step S141;

[0056] Figure 6 for Figure 1 Flow chart of the sub-steps included in step S15;

[0057] Figure 7 A schematic diagram of a model architecture of a network model provided in an embodiment of the present invention;

[0058] Figure 8 A functional module block diagram of a device for implementing deep frame workpiece grasping provided by an embodiment of the present invention;

[0059] Fig. 9 This is a structural block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0061] See also Figure 1 , which is a flow chart of a method for implementing deep frame workpiece grasping provided by an embodiment of the present invention, the method for implementing deep frame workpiece grasping can be executed by a device for implementing deep frame workpiece grasping, the device for implementing deep frame workpiece grasping can be implemented by software and / or hardware, and can be configured in an electronic device, the electronic device can be a computer device, a server, a programmable logic controller, etc. The detailed steps of the method for implementing deep frame workpiece grasping are introduced as follows.

[0062] S11, constructing multiple workpiece three-dimensional models based on the existing workpiece information.

[0063] S12, stacking the multiple workpiece three-dimensional models into a deep frame model, and obtaining multiple workpiece simulation images under different simulation conditions.

[0064] S13, selecting a plurality of workpiece simulation images from the plurality of workpiece simulation images based on the constructed structural similarity index.

[0065] S14, using a wavelet transform processing method to process the target information included in each of the screened workpiece simulation images, so as to reduce the high reflectivity of the workpiece simulation image.

[0066] S15, using the processed workpiece simulation images to train the constructed network model to obtain an instance segmentation model required for deep frame workpiece capture.

[0067] In this embodiment, the existing workpiece information may be an existing workpiece CAD model in the three-dimensional model library. Based on the existing workpiece CAD model, the workpiece CAD model may be subjected to high-precision 3D modeling to obtain a workpiece three-dimensional model. Simulation software such as Unity, Gazebo, NVIDIA Omniverse, and Blender may be used to perform real physical simulation. For example, Blender may be used to implement 3D modeling in this embodiment. In this process, the shape, structure, and surface characteristics of the workpiece may be retained, and material rendering may be performed to obtain a workpiece three-dimensional model.

[0068] In addition, a deep frame model can be loaded from the 3D model library, where the deep frame model corresponds to the frame used to place the workpiece in the actual scene. Multiple constructed workpiece 3D models are stacked into the deep frame model. On this basis, different simulation conditions can be set, including setting the workpiece 3D model in different posture states, changing different lighting conditions, changing different camera shooting positions and angles, etc.

[0069] A plurality of workpiece simulation images are obtained under different simulation conditions, so that a variety of workpiece simulation images can be generated.

[0070] In order to ensure that the constructed workpiece simulation image is consistent with the stacking situation of the real workpieces and improve the authenticity of the workpiece simulation image, in this embodiment, the constructed workpiece simulation image can also be screened based on the constructed structural similarity index.

[0071] In addition, considering that the workpiece is generally made of metal, due to the high reflectivity of the workpiece surface, the workpiece simulation image obtained by shooting under lighting conditions will be affected, thereby affecting the subsequent recognition of the workpiece in the workpiece simulation image. Therefore, in this embodiment, the target information included in the screened workpiece simulation image is processed using a wavelet transform processing method, and the purpose of the processing is to reduce the high reflectivity of the workpiece simulation image, thereby alleviating the impact of the high reflectivity of the workpiece on the workpiece simulation image.

[0072] In this embodiment, the above method can be used to construct a diverse workpiece simulation image that is consistent with the real workpiece image and suppresses the influence of the workpiece's high reflectivity to light. On this basis, the constructed network model is trained using the obtained workpiece simulation image to obtain an instance segmentation model, which can be used to realize deep frame workpiece grasping.

[0073] In this embodiment, by constructing workpiece simulation images under different simulation conditions, including different posture states, different lighting conditions, different shooting positions and angles, etc., and screening out workpiece simulation images that match the actual workpiece stacking scene, and suppressing the influence of the high reflectivity of the workpiece on the image. The high-quality workpiece simulation images finally obtained are used to train the instance segmentation model, so that the final instance segmentation model has stronger generalization and stronger robustness.

[0074] See also Figure 2 In this embodiment, the step of selecting multiple workpiece simulation images from multiple workpiece simulation images based on the constructed structural similarity index can be implemented in the following manner:

[0075] S131, calculating a structural similarity index between each of the workpiece simulation images and an existing real workpiece image based on the constructed structural similarity index.

[0076] S132, screening out workpiece simulation images whose structural similarity index is greater than or equal to a preset threshold.

[0077] In this embodiment, an existing real workpiece image can be obtained, and the real workpiece image is an image of a stacked workpiece in an actual scene. Each workpiece simulation image and the real workpiece image can be first converted into a grayscale image, and then the structural similarity index between each workpiece simulation image and the real workpiece image in the form of a grayscale image is calculated. The similarity between each workpiece simulation image and the real workpiece image can be evaluated in terms of image brightness information, contrast information, image texture information, and other dimensions.

[0078] Among them, the higher the obtained structural similarity index is, the higher the similarity between the two images is. Therefore, the structural similarity index corresponding to each workpiece simulation image can be compared with the preset threshold. If the structural similarity index is greater than or equal to the preset threshold, the workpiece simulation image is retained, otherwise the corresponding workpiece simulation image is deleted.

[0079] Through the above method, from the perspective of image structure similarity, workpiece simulation images that are relatively similar to the workpiece stacking images in the real scene can be screened out to ensure the authenticity of the retained workpiece simulation images.

[0080] See also Figure 3 In this embodiment, the step of calculating the structural similarity index between each workpiece simulation image and the existing real workpiece image based on the constructed structural similarity index can be specifically implemented in the following manner:

[0081] S1311, dividing each of the workpiece simulation images and the existing real workpiece images into a plurality of image windows respectively.

[0082] S1312, calculating the image mean and image variance of each image window of each of the workpiece simulation images and the real workpiece image.

[0083] S1313, for each image window in each of the workpiece simulation images, calculate the covariance based on the image mean and image variance of the image window and the image mean and image variance of the image window corresponding to the image window in the real workpiece image.

[0084] S1314: Calculate a structural similarity index based on the image mean, image variance and covariance of the image window.

[0085] S1315 , calculating a final structural similarity index based on multiple structural similarity indexes of multiple image windows of the workpiece simulation image.

[0086] In this embodiment, each workpiece simulation image and real workpiece image may be divided into a plurality of image windows, for example, may be divided into 11×11 image windows.

[0087] Assume that the images in each image window of the workpiece simulation image are represented as I ( x ), the image in each image window of the real workpiece image is represented as J ( x ). First, the image mean and image variance of each image window of the workpiece simulation image, as well as the image mean and image variance of each image window of the real workpiece image, can be calculated. The calculation formula is as follows:

[0088]

[0089]

[0090]

[0091]

[0092] in, Respectively represent image windows I ( x ) and the Image Window J ( x ), Respectively represent image windows I ( x ) and the Image Window J ( x ), N Indicates the number of pixels in the image window.

[0093] For each workpiece simulation image, the covariance between a certain image window of the workpiece simulation image and the corresponding image window in the real workpiece image can be calculated. , the calculation formula is as follows:

[0094]

[0095] On this basis, based on the image mean, image variance and covariance of the image window, the structural similarity index can be calculated according to the following formula: :

[0096]

[0097] in, C 1 and C 2 is a constant to avoid the initial zero, and usually takes a small value. Repeat the above calculation in a sliding window manner on the entire image of the workpiece simulation image to obtain the structural similarity index of each image window. Take the average of the structural similarity indexes of all image windows to obtain the final structural similarity index. Usually, the final structural similarity index is between [-1, 1], and the closer it is to 1, the more similar the two images are. Assuming the preset threshold is 0.5, the workpiece simulation images with a structural similarity index greater than or equal to 0.5 can be screened out, and the remaining workpiece simulation images can be deleted.

[0098] Considering that the metal surface of the workpiece is highly reflective to light, the presence of high reflectivity will affect the presentation of the image, specifically the low-frequency information in the image. Among them, low-frequency information mainly refers to the information of the main structure and slowly changing areas in the image, such as large flat areas or slowly changing colors, that is, low-frequency information is usually manifested as smooth and slowly changing areas in the image.

[0099] Based on this, see Figure 4 In this embodiment, the step of processing the target information included in each of the filtered workpiece simulation images using the wavelet transform processing method can be implemented in the following manner:

[0100] S141 , for each of the screened workpiece simulation images, extract low-frequency information included in the workpiece simulation image as target information.

[0101] S142, performing histogram averaging processing on the target information.

[0102] In this embodiment, low-frequency information in the workpiece simulation image can be separated by wavelet transform processing. Figure 5 In this embodiment, this step can be implemented in the following way:

[0103] S1411, processing the workpiece simulation image using a low-pass filter and a high-pass filter.

[0104] S1412, performing down-sampling processing on the processed workpiece simulation image to decompose the workpiece simulation image into low-frequency information and high-frequency information.

[0105] In this embodiment, the simulated image of the workpiece after screening is processed using Harr wavelet transform. The basic principle of Harr wavelet transform is to shift the basic wavelet function and make an inner product with the signal to be tested under different scale factors. The signal to be tested here is the simulated image of the workpiece after screening. The principle of the transform process is represented by the following formula:

[0106]

[0107] in, is the scale factor, is the signal to be measured, represents displacement, is the basic wavelet.

[0108] The two-dimensional pyramid decomposition algorithm is a way to implement the Harr wavelet transform. The two-dimensional pyramid decomposition algorithm can be used to decompose the workpiece simulation image into high-frequency parts and low-frequency parts. The decomposition formula is as follows:

[0109]

[0110] in, x , y Represents the horizontal and vertical coordinates of the pixel point, k , l is an integer, g , h Stands for high pass filter and low pass filter. and They are the vertical high-frequency part, horizontal high-frequency part, diagonal high-frequency part and low-frequency part of the image. Since the illumination change affects the brightness and contrast of the image, the low-frequency part is greatly affected by the illumination, so the low-frequency part of the image is histogram-averaged and then restored using two-dimensional pyramid reconstruction to reduce the impact of high reflectivity of metal.

[0111] The method of performing histogram averaging on the separated low-frequency information is mainly to perform pixel grayscale value statistics on each pixel of the low-frequency information part of the image, and calculate the average value of all pixel grayscale values, and then use the cumulative distribution function of the histogram to obtain the proportion of pixels at each grayscale level and below in the total number of pixels, and map the grayscale value of the image to a new grayscale through the cumulative distribution function, so that the average value of the new grayscale distribution is suppressed by the average value of the pixel grayscale value of the original image mentioned above. Then the calculated new grayscale value is assigned to each pixel in the image to obtain the processed image.

[0112] Through the above method, the grayscale distribution of the image in the low-frequency information part can be made more uniform, thereby enhancing the visual quality of the image. In the case of uneven lighting or low contrast, and the case of high reflectivity of the low-frequency information part to light, the adverse effects of high reflectivity on image quality can be improved.

[0113] After the workpiece simulation image is obtained through the above processing, the workpiece simulation image is used as a training sample to train the constructed network model. In this embodiment, the constructed network model includes a feature extraction module, a feature fusion module and a post-processing module. Among them, the feature extraction module includes a convolution layer, a conversion layer and a pooling layer, and a multi-scale attention module is also connected to the back end of the conversion layer.

[0114] See also Figure 6 In this embodiment, the step of training the network model using the workpiece simulation image can be implemented in the following way:

[0115] S151, importing the processed workpiece simulation images into the feature extraction module included in the constructed network model, and processing the workpiece simulation images through the convolution layer, conversion layer, pooling layer and multi-scale attention module in the feature extraction module to obtain feature extraction results.

[0116] S152, fusing the feature extraction results through the feature fusion module to obtain a fusion processing result.

[0117] S153, using the post-processing module to post-process the fusion processing result to obtain a prediction result of the workpiece simulation image.

[0118] In this embodiment, the training samples can be divided into a training set and a test set in a certain ratio, for example, a ratio of 7:3. The constructed network model is trained using the artifact simulation images in the training set, and the trained model is tested using the artifact simulation images in the test set until the model meets the requirements.

[0119] Please refer to Figure 7, where Backbone, Neck, and Head represent feature extraction modules, feature fusion modules, and post-processing modules, respectively. The feature extraction module includes multiple convolutional layers (Conv, where k, s, and p represent the size of the convolution kernel, the step size of the convolution kernel sliding on the input data, and the number of layers adding zeros around the boundary of the input data, respectively), multiple conversion layers (C2f), and pooling layers (SPPF). The back end of the conversion layer is also connected to a multi-scale attention module (EMA). For example, the back ends of the second, third, and fourth conversion layers C2f in the feature extraction module are respectively connected to multi-scale attention modules EMA. The multi-scale attention module refines the feature representation of the image by aggregating information across different spatial dimensions, and can calculate the similarity of global and local features to capture long-distance and short-distance dependencies in the image, thereby improving the recognition and segmentation capabilities of the network model for complex scenes.

[0120] The workpiece simulation image imported into the feature extraction module is processed by the convolution layer, conversion layer, pooling layer and multi-scale attention module to obtain the feature extraction result. Figure 7 It can be seen that the feature extraction results obtained by the convolution layer, conversion layer, pooling layer, and multi-scale attention module are output to the feature fusion module respectively. The feature fusion module combines the feature extraction results output by the convolution layer, conversion layer, pooling layer, and multi-scale attention module in the feature extraction module to obtain the fusion processing result.

[0121] The feature fusion module includes a convolution layer (Conv), a conversion layer (C2f), a concatenation layer (Concat), an upsampling layer (Upsample) and an attention fusion module (UAFM). The attention fusion module is connected to the multi-scale attention module in the feature extraction module and the conversion layer in the feature fusion module, and is used to fuse the output results of the multi-scale attention module and the conversion layer, and output the fused results to the post-processing module. For example, the input end of the attention fusion module is connected to the first multi-scale attention module in the feature extraction module and the first conversion layer in the feature fusion module, and the output end of the attention fusion module is connected to the post-processing module.

[0122] By adding an attention fusion module to the feature fusion module, low-level detail features and high-level semantic features can be fused to strengthen the feature representation of the network and improve the segmentation accuracy of the network.

[0123] As can be seen from the figure, the post-processing module includes a small sample network (ProtoNet) and multiple heads (Segment), where the small sample network is connected to the attention fusion module in the feature fusion module, and each head is connected to the conversion layer in the feature fusion module. The small sample network processes the feature fusion results output by the attention fusion module, and each head processes the received feature fusion results of the conversion layer. Finally, the processing results of the small sample network and each head are fused to obtain the final output result.

[0124] In this embodiment, an EMA module is inserted after each conversion layer in the feature extraction module, and the global and local information of the features are effectively fused through feature grouping processing and cross-dimensional interaction, so that the model can capture the global layout and local feature texture information of the workpiece in the deep basket scene, improve the problems of illumination transformation and occlusion in the image, and improve the accuracy of workpiece detection. In the feature fusion module, the multi-scale fusion part is inserted into the UAFM module, which not only retains the low-level detail features of the network in the feature extraction module and the high-level abstract semantic features after feature fusion, but also improves the accuracy of the network segmentation for workpieces.

[0125] Based on the above network model, the network model is trained using various workpiece simulation images, wherein each workpiece simulation image has a true label, and the true label includes a true category and a true box, wherein the true category refers to the category of each workpiece in the image, and the true box refers to the box that frames each workpiece.

[0126] When training the network model, each processed workpiece simulation image is imported into the constructed network model for processing, and the predicted category and prediction box of the workpiece simulation image are output, wherein the predicted category refers to the category of each workpiece in the workpiece simulation image obtained by the network model, and the prediction box is the framing box of each workpiece in the workpiece simulation image output by the network model.

[0127] Under the guidance of the constructed loss function, the model parameters of the network model are adjusted and training is continued until the preset requirements are met to obtain the instance segmentation model required for deep frame workpiece grasping.

[0128] The preset requirements may be that the training time reaches a preset time, or the number of training iterations reaches a preset number, or the loss function reaches convergence and no longer changes.

[0129] Among them, the loss function includes the category loss function, the prediction box regression loss function and the segmentation loss function. The category loss function is constructed based on the real category and predicted category of the workpiece simulation image, and the prediction box regression function is constructed based on the real box and predicted box of the workpiece simulation image. The final loss function can be obtained by superimposing the category loss function, the prediction box regression loss function and the segmentation loss function according to different weights.

[0130] In this embodiment, the category loss function Using binary cross entropy loss, the calculation formula is as follows:

[0131]

[0132] in, w Indicates the weight of each category. x n Represents the predicted category of the network model for the workpiece simulation image, y n Represents the true category of the artifact simulation image.

[0133] The prediction box regression loss function is composed of the optimized border regression loss function and the positioning accuracy loss function. It is expressed by the following formula:

[0134]

[0135]

[0136]

[0137] In the formula, is the distance between the center of the real box and the center of the predicted box; c is the diagonal length of the minimum enveloping box of the two; and is the width and height of the real frame; w and h is the width and height of the prediction box; v It is a penalty term introduced, which represents the difference in aspect ratio between the real box and the predicted box. The larger its value, the greater the difference between the two. IoU represents the ratio of the intersection area and the union area of ​​the real box and the predicted box.

[0138] In addition, the positioning accuracy loss function The calculation formula is as follows:

[0139]

[0140] The prediction result of the target frame coordinates of the existing network model corresponds to a real target frame coordinate and calculates its loss value. However, this calculation method is not suitable for the case where the target boundary is not clear. Assume that the label range , using a generalized distribution P(·) to represent the estimated values ​​corresponding to a series of discrete true values ,but = , P(·) is obtained through the Softmax operation, denoted as ; The global optimal solution of DFL is , , so that is close to the true value y, and represents the values ​​from the left and right near the continuous label y, satisfying .

[0141] The segmentation loss function uses the pixel-level cross entropy loss function. The calculation formula is as follows:

[0142]

[0143] M is the mask image predicted by the network module, is the real mask image, BCE(∙) represents the pixel-level binary cross entropy operation on the predicted mask and the real mask, The area of ​​the smallest bounding box that represents the true mask.

[0144] The final loss function is composed of the above loss functions, which are superimposed according to different weights, as shown below:

[0145]

[0146] in, is the weight coefficient of each loss function, for example, they can be set to 0.05, 0.15, 0.75, and 0.75 respectively.

[0147] In order to enhance the generalization performance of the model, data enhancement processing can be performed during the training process, including rotating, scaling, flipping and color adjustment of the workpiece simulation image to simulate various complex scenes, thereby ensuring that the trained model can adapt to the actual application environment.

[0148] The method for deep frame workpiece grasping provided in this embodiment generates a diversified virtual data set based on virtual simulation technology, structural similarity index and wavelet transform metal high reflectivity suppression method, covering a variety of lighting, background and occlusion scenes, ensuring that the trained model has stronger generalization ability and robustness in industrial environments. In addition, by generating a large amount of high-quality training data, the dependence on real workpiece data collection and manual annotation is reduced, and the cost and time of data production are reduced.

[0149] In addition, by introducing the attention mechanism to construct the network model, the network model's ability to learn the features of complex shapes and textured artifacts is enhanced, and it can adaptively focus on artifact features of different scales, greatly improving the accuracy of detection and segmentation of stacked, occluded, and complex posture artifacts.

[0150] Based on the same inventive concept, please refer to Figure 8 , an embodiment of the present invention further provides a functional module schematic diagram of a device for grasping a deep frame workpiece. This embodiment can divide the functional modules of the device for grasping a deep frame workpiece according to the above method embodiment. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present invention is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0151] For example, when each functional module is divided into corresponding functional modules, Figure 8 The device for implementing deep frame workpiece grasping shown is only a schematic diagram of the device. The device for implementing deep frame workpiece grasping may include a construction module, an acquisition module, a screening module, a processing module and a training module. The functions of each functional module of the device for implementing deep frame workpiece grasping are described in detail below.

[0152] A construction module, used for constructing multiple workpiece three-dimensional models based on existing workpiece information;

[0153] An acquisition module, used for stacking the multiple workpiece three-dimensional models into a deep frame model, and obtaining multiple workpiece simulation images under different simulation conditions;

[0154] A screening module, configured to screen out a plurality of workpiece simulation images from the plurality of workpiece simulation images based on the constructed structural similarity index;

[0155] A processing module, used for processing the target information included in each of the screened workpiece simulation images by using a wavelet transform processing method, so as to reduce the high reflectivity of the workpiece simulation image;

[0156] The training module is used to train the constructed network model using the processed workpiece simulation images to obtain the instance segmentation model required for deep frame workpiece grasping.

[0157] It can be understood that the above-mentioned building module, acquisition module, screening module, processing module and training module can be used to execute the above-mentioned S11 to S15. The detailed implementation method of the building module, acquisition module, screening module, processing module and training module can refer to the relevant contents of the above-mentioned S11 to S15.

[0158] In a possible implementation, the screening module may be used to:

[0159] Calculating a structural similarity index between each of the workpiece simulation images and an existing real workpiece image based on the constructed structural similarity index;

[0160] The workpiece simulation images whose structural similarity index is greater than or equal to a preset threshold are screened out.

[0161] In a possible implementation, the above screening module may be specifically used for:

[0162] Dividing each of the workpiece simulation images and the existing real workpiece image into a plurality of image windows respectively;

[0163] Calculating the image mean and image variance of each image window of each of the workpiece simulation images and the real workpiece image;

[0164] For each image window in each of the workpiece simulation images, the covariance is calculated based on the image mean and image variance of the image window and the image mean and image variance of the image window corresponding to the image window in the real workpiece image;

[0165] Calculating a structural similarity index based on the image mean, image variance and covariance of the image window;

[0166] A final structural similarity index is calculated based on multiple structural similarity indexes of multiple image windows of the workpiece simulation image.

[0167] In a possible implementation, the processing module may be used to:

[0168] For each of the screened workpiece simulation images, extract low-frequency information included in the workpiece simulation image as target information;

[0169] The target information is subjected to histogram averaging processing.

[0170] In a possible implementation, the processing module may be specifically used for:

[0171] Processing the workpiece simulation image using a low-pass filter and a high-pass filter;

[0172] The processed workpiece simulation image is down-sampled to decompose the workpiece simulation image into low-frequency information and high-frequency information.

[0173] In a possible implementation, each of the workpiece simulation images has a real label, and the real label includes a real category and a real frame; the above training module can be used for:

[0174] Importing the processed workpiece simulation images into the constructed network model for processing, and outputting the predicted category and predicted frame of the workpiece simulation image;

[0175] Under the guidance of the constructed loss function, the model parameters of the network model are adjusted and the training is continued until the instance segmentation model required for deep frame workpiece grasping is obtained when the preset requirements are met;

[0176] Among them, the loss function includes a category loss function, a prediction box regression loss function and a segmentation loss function. The category loss function is constructed based on the real category and the predicted category of the workpiece simulation image, and the prediction box regression loss function is constructed based on the real box and the predicted box of the workpiece simulation image.

[0177] In a possible implementation, the network model includes a feature extraction module, a feature fusion module and a post-processing module; the feature extraction module includes a convolution layer, a conversion layer and a pooling layer, and a multi-scale attention module is also connected to the back end of the conversion layer; the above training module can be used for:

[0178] Importing the processed workpiece simulation images into the feature extraction module included in the constructed network model, and processing the workpiece simulation images through the convolution layer, conversion layer, pooling layer and multi-scale attention module in the feature extraction module to obtain feature extraction results;

[0179] The feature extraction results are fused by the feature fusion module to obtain a fusion result;

[0180] The fusion processing result is post-processed by using the post-processing module to obtain a prediction result of the workpiece simulation image.

[0181] In a possible implementation, the feature fusion module includes a conversion layer, a convolution layer, a splicing layer, an upsampling layer and an attention fusion module;

[0182] The attention fusion module is connected to the multi-scale attention module in the feature extraction module and the conversion layer in the feature fusion module, and is used to fuse the output results of the multi-scale attention module and the conversion layer, and output the fused results to the post-processing module.

[0183] See also Fig. 9 , is a block diagram of an electronic device provided in an embodiment of the present invention, the electronic device may be a computer device, etc., and the electronic device includes a memory, a processor, and a communication module. The memory, the processor, and the communication module are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines.

[0184] The memory is used to store computer programs or data. The memory can be, but is not limited to, random access memory (RAM), read only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0185] The processor is used to read / write data or programs stored in the memory, and execute the method for implementing deep frame workpiece grasping provided by any embodiment of the present invention.

[0186] The communication module is used to establish a communication connection between the electronic device and other communication terminals through the network, and is used to send and receive data through the network.

[0187] It should be understood that Fig. 9 The structure shown is only a schematic diagram of the structure of the electronic device. The electronic device may also include Fig. 9 More or fewer components as shown, or with Fig. 9 Different configurations shown.

[0188] Furthermore, an embodiment of the present invention also provides a computer-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are executed, the method for implementing deep-frame workpiece grasping provided in the above embodiment is implemented.

[0189] Specifically, the computer-readable storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the computer-readable storage medium is executed, the above-mentioned method for implementing deep frame workpiece grasping can be executed. Regarding the process involved when the computer-readable storage medium and its executable instructions are executed, reference can be made to the relevant description in the above-mentioned method embodiment, which will not be described in detail here.

[0190] In summary, the method, device and electronic device for realizing deep frame workpiece grasping provided by the embodiments of the present invention construct multiple workpiece three-dimensional models based on the existing workpiece information, stack the multiple workpiece three-dimensional models into the deep frame model, and obtain multiple workpiece simulation images under different simulation conditions. Based on the constructed structural similarity index, multiple workpiece simulation images are screened out from multiple workpiece simulation images, and the target information included in each screened workpiece simulation image is processed using a wavelet transform processing method to reduce the high reflectivity of the workpiece simulation image. The constructed network model is trained using the processed workpiece simulation image to obtain the instance segmentation model required for deep frame workpiece grasping. In this solution, diversified and highly authentic simulation data is generated to ensure the generalization ability and robustness of the trained model.

[0191] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0192] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0193] Furthermore, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0194] It should be noted that if the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0195] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0196] The above description is only an embodiment of the present invention and is not intended to limit the protection scope of the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for grasping a deep frame workpiece, characterized in that: The method comprises: Construct multiple workpiece 3D models based on existing workpiece information; stacking the multiple workpiece three-dimensional models into a deep frame model, and obtaining multiple workpiece simulation images under different simulation conditions; Based on the constructed structural similarity index, screening out a plurality of workpiece simulation images from the plurality of workpiece simulation images; Using a wavelet transform processing method to process the target information included in each of the screened workpiece simulation images, so as to reduce the high reflectivity of the workpiece simulation image; The constructed network model is trained using the processed simulated images of the workpiece to obtain an instance segmentation model required for deep frame workpiece grasping; The step of selecting a plurality of workpiece simulation images from the plurality of workpiece simulation images based on the constructed structural similarity index comprises: Calculating the structural similarity index between each of the workpiece simulation images and the existing real workpiece image based on the constructed structural similarity index; screening out the workpiece simulation images whose structural similarity index is greater than or equal to a preset threshold; The step of processing the target information included in each of the filtered workpiece simulation images using the wavelet transform processing method includes: For each of the screened workpiece simulation images, low-frequency information included in the workpiece simulation image is extracted as target information; and histogram averaging processing is performed on the target information.

2. The method for grasping a deep frame workpiece according to claim 1, characterized in that: The step of calculating the structural similarity index between each of the simulated workpiece images and the existing real workpiece images based on the constructed structural similarity index comprises: Dividing each of the workpiece simulation images and the existing real workpiece image into a plurality of image windows respectively; Calculating the image mean and image variance of each image window of each of the workpiece simulation images and the real workpiece image; For each image window in each of the workpiece simulation images, the covariance is calculated based on the image mean and image variance of the image window and the image mean and image variance of the image window corresponding to the image window in the real workpiece image; Calculating a structural similarity index based on the image mean, image variance and covariance of the image window; A final structural similarity index is calculated based on multiple structural similarity indexes of multiple image windows of the workpiece simulation image.

3. The method for grasping a deep frame workpiece according to claim 1, characterized in that: The step of extracting the workpiece simulation image including low-frequency information comprises: Processing the workpiece simulation image using a low-pass filter and a high-pass filter; The processed workpiece simulation image is down-sampled to decompose the workpiece simulation image into low-frequency information and high-frequency information.

4. The method for grasping a deep frame workpiece according to claim 1, characterized in that: Each of the workpiece simulation images has a real label, and the real label includes a real category and a real frame; The step of training the constructed network model using the processed workpiece simulation images to obtain the instance segmentation model required for deep frame workpiece capture includes: Importing the processed workpiece simulation images into the constructed network model for processing, and outputting the predicted category and predicted frame of the workpiece simulation image; The model parameters of the network model are adjusted under the guidance of the constructed loss function, and the training is continued until the instance segmentation model required for deep frame workpiece grasping is obtained when the preset requirements are met; Among them, the loss function includes a category loss function, a prediction box regression loss function and a segmentation loss function. The category loss function is constructed based on the real category and the predicted category of the workpiece simulation image, and the prediction box regression loss function is constructed based on the real box and the predicted box of the workpiece simulation image.

5. The method for grasping a deep frame workpiece according to claim 1, characterized in that: The network model includes a feature extraction module, a feature fusion module and a post-processing module; The feature extraction module includes a convolution layer, a conversion layer and a pooling layer, and a multi-scale attention module is connected to the back end of the conversion layer; The step of training the constructed network model using the processed workpiece simulation images comprises: Importing the processed workpiece simulation images into the feature extraction module included in the constructed network model, and processing the workpiece simulation images through the convolution layer, conversion layer, pooling layer and multi-scale attention module in the feature extraction module to obtain feature extraction results; The feature extraction results are fused by the feature fusion module to obtain a fusion result; The fusion processing result is post-processed by using the post-processing module to obtain a prediction result of the workpiece simulation image.

6. The method for grasping a deep frame workpiece according to claim 5, characterized in that: The feature fusion module includes a conversion layer, a convolution layer, a splicing layer, an upsampling layer and an attention fusion module; The attention fusion module is connected to the multi-scale attention module in the feature extraction module and the conversion layer in the feature fusion module, and is used to fuse the output results of the multi-scale attention module and the conversion layer, and output the fused results to the post-processing module.

7. A device for grasping deep frame workpieces, characterized in that: The device is used to implement the method for grasping a deep frame workpiece according to any one of claims 1 to 6, comprising: A construction module, used for constructing multiple workpiece three-dimensional models based on existing workpiece information; An acquisition module, used for stacking the multiple workpiece three-dimensional models into a deep frame model, and obtaining multiple workpiece simulation images under different simulation conditions; A screening module, configured to screen out a plurality of workpiece simulation images from the plurality of workpiece simulation images based on the constructed structural similarity index; A processing module, used for processing the target information included in each of the screened workpiece simulation images by using a wavelet transform processing method, so as to reduce the high reflectivity of the workpiece simulation image; The training module is used to train the constructed network model using the processed workpiece simulation images to obtain the instance segmentation model required for deep frame workpiece grasping.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

Images