Method, apparatus, and electronic equipment for picking workpieces in deep containers.

Abstract

The present invention provides a method, apparatus, and electronic equipment for picking workpieces in a deep container. [Solution] The method involves constructing multiple three-dimensional work models based on existing work information, stacking the multiple three-dimensional work models in a deep container model, acquiring multiple work simulation images under different simulation conditions, selecting multiple work simulation images from the multiple work simulation images based on a structural similarity index, processing the target information contained in each selected work simulation image using wavelet transform processing, training a network model constructed using work simulation images with reduced high reflectivity of the work simulation images, and obtaining an instance segmentation model necessary for picking work in a deep container.

Description

[Technical Field]

[0001] The present invention relates to the field of control technology, and more specifically to a method, apparatus, and electronic equipment for picking workpieces in a deep container. [Background technology]

[0002] With the advancement of industrial automation and intelligence, robotic picking technology is seeing increased use in fields such as manufacturing, logistics, and warehouse management. However, conventional robotic picking systems still face many challenges when it comes to randomly retrieving complex workpieces. The complexity of random retrieval stems from the large variety of workpiece types and shapes, as well as the fact that the orientation of the workpieces is not fixed during stacking and placement, potentially leading to entanglement and interference between workpieces. This places a higher demand on the robot's perception, recognition, and grasping capabilities.

[0003] With the rise of deep learning technology, instance segmentation networks have made groundbreaking advancements in fields such as image recognition and object detection. Deep learning models can automatically extract and utilize high-dimensional features such as the shape and pattern of a workpiece by learning from large amounts of labeled data, significantly improving the accuracy and robustness of instance segmentation. However, the success of deep learning models depends on a large amount of high-quality training data and the design of the network model. Therefore, existing instance segmentation methods have at least the following drawbacks for industrial use.

[0004] 1. Difficulty in creating datasets. The shapes, structures, and surface properties of complex workpieces are diverse, making it extremely difficult to obtain high-quality training datasets that cover a wide range of workpiece types. In industrial settings, there are many different workpieces with complex specifications, and the collection conditions in the production environment often change, requiring time and effort to collect a sufficient amount of ground truth data. Furthermore, the high reflectivity of metal workpieces can distort the actual color in images under different lighting conditions, resulting in the inability to recognize the shape and details of metal objects. In addition, even if a sufficient amount of ground truth workpiece data is collected, accurately labeling these images is extremely difficult. Manual data labeling is often subjective, which not only destabilizes the performance of the detection model but also easily leads to misjudgments for workpieces with complex geometric structures and patterns.

[0005] 2. Lack of model robustness in complex scenarios In situations where objects are stacked randomly, entanglement between workpieces can prevent the detection model from acquiring complete workpiece features, affecting recognition accuracy and leading to detection failures. Furthermore, conventional models have limitations in processing workpieces with complex shapes and large dimensional variations, making them particularly prone to false detections and missed detections for workpieces with irregular shapes or diverse orientations. In addition, in industrial settings, lighting conditions are typically complex and changeable, and factors such as reflected light and shadows interfere with conventional models, leading to an increase in false detections.

[0006] From the above, it can be seen that deep learning-based model networks rely on large amounts of high-quality labeled datasets, but collecting and labeling ground truth data for complex workpieces is costly and difficult. Furthermore, complex environmental factors such as the random arrangement of workpieces, various lighting conditions, entanglement, and reflections increase the difficulty of detection by the model. Conventional networks perform poorly on workpieces with complex shapes and diverse patterns, and are prone to detection failures and false positives. [Overview of the Initiative]

[0007] The embodiment of the present invention aims to provide a method, apparatus, and electronic equipment for picking workpieces in a deep container to ensure the generalization performance and robustness of a model obtained through training.

[0008] In the first phase, the present invention provides a method for picking workpieces in a deep container, the method comprising: constructing a plurality of three-dimensional workpiece models based on existing workpiece information; stacking the plurality of three-dimensional workpiece models in a deep container model and obtaining a plurality of workpiece simulation images in different simulation states; selecting a plurality of workpiece simulation images from the plurality of workpiece simulation images based on a created structural similarity index; processing target information contained in each of the selected workpiece simulation images by wavelet transform processing to reduce the high reflectivity of the workpiece simulation images; and training a constructed network model using each of the processed workpiece simulation images to obtain an instance segmentation model necessary for picking workpieces in a deep container.

[0009] In an optional embodiment, the step of selecting a plurality of work simulation images from a plurality of work simulation images based on a created structural similarity index includes the steps of calculating a structural similarity index between each of the work simulation images and an existing ground truth work image based on the created structural similarity index, and selecting work simulation images whose structural similarity index is equal to or greater than a predetermined threshold.

[0010] In an optional embodiment, the step of calculating a structural similarity index between each of the work simulation images and an existing ground truth work image based on a created structural similarity index includes: partitioning each of the work simulation images and the existing ground truth work image into a plurality of image windows; calculating the mean image and image variance for each image window of each of the work simulation images and the ground truth work image; calculating a covariance for each image window in each of the work simulation images based on the mean image and image variance of the image window and the mean image and image variance of the image window in the ground truth work image; calculating a structural similarity index based on the mean image, image variance and covariance of the image window; and calculating a final structural similarity index based on a plurality of structural similarity indices of the plurality of image windows of the work simulation image.

[0011] In an optional embodiment, the step of processing target information contained in each of the selected work simulation images by wavelet transform processing includes, for each of the selected work simulation images, the step of extracting low-frequency information contained in the work simulation image as target information, and the step of performing histogram equalization processing on the target information.

[0012] In an optional embodiment, the step of extracting low-frequency information contained in the work simulation image includes processing the work simulation image using a low-pass filter and a high-pass filter, and decomposing the work simulation image into low-frequency information and high-frequency information by performing downsampling on the processed work simulation image.

[0013] In an optional embodiment, each of the work simulation images has a ground truth label including a ground truth class and a ground truth box, and the step of training a constructed network model using each of the processed work simulation images to obtain an instance segmentation model necessary for picking workpieces in a deep container includes the steps of introducing each of the processed work simulation images into the constructed network model and processing it to output a predicted class and a predicted box for the work simulation images, and adjusting the model parameters of the network model based on a constructed loss function and continuing training until predetermined requirements are met to obtain an instance segmentation model necessary for picking workpieces in a deep container, wherein the loss function includes a class loss function, a predicted box regression loss function, and a segmentation loss function, the class loss function is constructed based on the ground truth class and predicted class for the work simulation images, and the predicted box regression loss function is constructed based on the ground truth box and a predicted box for the work simulation images.

[0014] In an optional embodiment, the network model includes a feature extraction module, a feature fusion module, and a post-processing module, wherein the feature extraction module includes a convolutional layer, a transformation layer, and a pooling layer, with a multiscale attention module further connected behind the transformation layer. The step of training a network model constructed using each of the processed work simulation images includes: introducing each of the processed work simulation images into a feature extraction module included in the constructed network model; processing the work simulation images with the convolutional layer, transformation layer, pooling layer and multiscale attention module of the feature extraction module to obtain feature extraction results; performing a fusion process on the feature extraction results with the feature fusion module to obtain fusion processing results; and performing post-processing on the fusion processing results with the post-processing module to obtain prediction results for the work simulation images.

[0015] In an optional embodiment, the feature fusion module includes a conversion layer, a convolutional layer, a coupling layer, an upsampling layer, and an attention fusion module, the attention fusion module being connected to a multiscale attention module in the feature extraction module and to the conversion layer in the feature fusion module, performing fusion processing on the output results of the multiscale attention module and the conversion layer, and outputting the result after fusion processing to the post-processing module.

[0016] In the second phase, the present invention provides an apparatus for realizing workpiece picking in a deep container, the apparatus comprising: a construction module for constructing a plurality of three-dimensional workpiece models based on existing workpiece information; an acquisition module for stacking the plurality of three-dimensional workpiece models in a deep container model and acquiring a plurality of workpiece simulation images in different simulation states; a selection module for selecting a plurality of workpiece simulation images from the plurality of workpiece simulation images based on a created structural similarity index; a processing module for processing target information contained in each of the selected workpiece simulation images by wavelet transform processing to reduce the high reflectivity of the workpiece simulation images; and a training module for training a constructed network model using each of the processed workpiece simulation images to obtain an instance segmentation model necessary for picking workpieces in a deep container.

[0017] In a third aspect, the present invention provides an electronic device comprising a processor and a memory storing a computer program executable by the processor, wherein the processor, upon executing the computer program, realizes a step of the method described in any one of the embodiments described above.

[0018] The present invention provides a method, apparatus, and electronic device for picking workpieces in a deep container, constructs a plurality of workpiece three-dimensional models based on existing workpiece information, stacks the plurality of workpiece three-dimensional models in a deep container model, and obtains a plurality of workpiece simulation images in different simulation states. Based on the created structural similarity index, a plurality of workpiece simulation images are selected from the plurality of workpiece simulation images, and the target information included in each selected workpiece simulation image is processed by wavelet transform processing to reduce the high reflectivity of the workpiece simulation image. Using the processed workpiece simulation image, the constructed network model is trained to obtain an instance segmentation model required for picking workpieces in a deep container. In the present application, by generating diverse and high-accuracy simulation data, the generalization performance and robustness of the model obtained by training are ensured.

[0019] To more clearly illustrate the technical form of the embodiments of the present invention, the drawings necessary for the description of the embodiments of the present invention will be briefly described below. The drawings to be described only show some embodiments of the present invention and do not limit the scope. Those skilled in the art can obtain other related drawings based on these drawings without using inventive capabilities.

Brief Description of the Drawings

[0020] [Figure 1] It is a flowchart of a method for picking workpieces in a deep container according to an embodiment of the present invention. [Figure 2] It is a flowchart of the sub-steps included in step S13 in FIG. 1. [Figure 3] It is a flowchart of the sub-steps included in step S131 in FIG. 2. [Figure 4] It is a flowchart of the sub-steps included in step S14 in FIG. 1. [Figure 5] It is a flowchart of the sub-steps included in step S141 in FIG. 4. [Figure 6]Figure 1 shows the flow chat of the substeps included in step S15. [Figure 7] This is a schematic diagram of the model structure of a network model according to an embodiment of the present invention. [Figure 8] This is a functional module block diagram of an apparatus for realizing workpiece picking in a deep container according to an embodiment of the present invention. [Figure 9] This is a structural block diagram of an electronic device according to an embodiment of the present invention. [Modes for carrying out the invention]

[0021] Hereinafter, the technical embodiments of the embodiments of the present invention will be described with reference to the drawings of the embodiments of the present invention.

[0022] Figure 1 is a flowchart of a workpiece picking method in a deep container according to an embodiment of the present invention. The workpiece picking method in a deep container can be implemented by a workpiece picking implementation device. The workpiece picking implementation device in a deep container can be implemented by software and / or hardware and can be installed in electronic equipment. The electronic equipment may be a computer system, a server, a programmable logic controller, etc. The detailed steps of the workpiece picking method in a deep container are as follows.

[0023] Step S11: Construct multiple three-dimensional workpiece models based on existing workpiece information.

[0024] Step S12: The multiple three-dimensional workpiece models are stacked inside the deep container model, and multiple workpiece simulation images are obtained under different simulation conditions.

[0025] Step S13: Select multiple work simulation images from the multiple work simulation images based on the created structural similarity index.

[0026] Step S14: The target information contained in each of the selected work simulation images is processed by wavelet transform processing to reduce the high reflectivity of the work simulation images.

[0027] Step S15: Using each of the processed work simulation images, the constructed network model is trained to obtain an instance segmentation model necessary for picking workpieces in deep containers.

[0028] In this embodiment, existing workpiece information may be a workpiece CAD model based on a three-dimensional model. Based on the existing workpiece CAD model, a high-precision 3D model of the workpiece can be obtained by performing 3D modeling on it. Realistic physical simulations can be performed using simulation software such as Unity, Gazebo, NVIDIA Omniverse, and Blender. For example, in this embodiment, 3D modeling can be achieved using Blender. In this process, a workpiece 3D model can be obtained by rendering its material while maintaining the shape, structure, and surface properties of the workpiece.

[0029] Furthermore, it is possible to load a deep container model from a 3D model base, and the deep container model corresponds to a physical container for holding workpieces. Multiple constructed 3D workpiece models are stacked inside the deep container model. Then, by setting the 3D workpiece models to different orientations, changing lighting conditions, or changing the camera's shooting position and angle, various simulation states can be set.

[0030] By acquiring multiple workpiece simulation images under different simulation conditions, a variety of workpiece simulation images can be generated.

[0031] In this embodiment, to ensure that the generated work simulation images match the stacking state of the correct work and to improve the accuracy of the work simulation images, the generated work simulation images can also be selected based on the created structural similarity index.

[0032] Since workpieces are generally made of metal, the high reflectivity of the workpiece surface affects the workpiece simulation image captured under lighting conditions, and consequently, affects the recognition of the workpiece in subsequent workpiece simulation images. Therefore, in this embodiment, target information contained in the workpiece simulation image selected by wavelet transform processing is processed. The purpose of this processing is to reduce the high reflectivity of the workpiece simulation image and mitigate the effect of the workpiece's high reflectivity on the workpiece simulation image.

[0033] In this embodiment, the above method makes it possible to generate diverse work simulation images that match the image of the correct work and suppress the influence of the work's high reflectivity on light irradiation. Then, by training the constructed network model using the obtained work simulation images, an instance segmentation model is obtained, and this instance segmentation model can be used to realize work picking in deep containers.

[0034] In this embodiment, work simulation images are generated under different simulation conditions, such as different postures, different lighting conditions, and different shooting positions and angles. Work simulation images that match the stacking state of the correct work are then selected, and the influence of the work's high reflectivity on the image is suppressed. Finally, the instance segmentation model is trained using the obtained high-quality work simulation images to improve the generalization performance and robustness of the final instance segmentation model.

[0035] As shown in Figure 2, in this embodiment, the step of extracting multiple work simulation images from multiple work simulation images based on the structural similarity index created above can be implemented as follows.

[0036] Step S131: Calculate the structural similarity index between each work simulation image and an existing ground truth work image based on the created structural similarity index.

[0037] Step S132: Select work simulation images whose structural similarity index is above a predetermined threshold.

[0038] In this embodiment, existing ground truth work images can be obtained, and these ground truth work images are images of workpieces in their stacked state at the site. First, each workpiece simulation image and the ground truth work image are converted into grayscale images, and a structural similarity index is calculated between each workpiece simulation image and the ground truth work image in the grayscale image format. The similarity between each workpiece simulation image and the ground truth work image can be evaluated using image brightness information, contrast information, image pattern information, etc.

[0039] A higher structural similarity index indicates a greater degree of similarity between two images. Therefore, the structural similarity index corresponding to each work simulation image is compared with a predetermined threshold. If the structural similarity index is equal to or greater than the predetermined threshold, the work simulation image is retained; otherwise, the corresponding work simulation image is deleted.

[0040] By using the method described above, it is possible to select work simulation images that closely match images of the stacked workpieces on site, based on the similarity of image structures, and to ensure the accuracy of the retained work simulation images.

[0041] As shown in Figure 3, in this embodiment, the step of calculating the structural similarity index between each work simulation image and an existing ground truth work image based on the structural similarity index created above can be specifically implemented as follows.

[0042] Step S1311: Each of the aforementioned work simulation images and existing ground truth work images are partitioned into multiple image windows.

[0043] Step S1312: Calculate the average image value and image variance for each image window of each work simulation image and each ground truth work image.

[0044] Step S1313: For each image window in each of the work simulation images, the covariance is calculated based on the mean image and image variance of the image window, and the mean image and image variance of the image window corresponding to the image window in the ground truth work image.

[0045] Step S1314: Calculate a structural similarity index based on the mean value of the image window, the image variance, and the covariance.

[0046] Step S1315: The final structural similarity index is calculated based on the multiple structural similarity indices of multiple image windows of the work simulation image.

[0047] In this embodiment, each work simulation image and the ground truth work image can be divided into multiple image windows. For example, they can be divided into 11x11 image windows.

[0048] Let I(x) be the image in each image window of the work simulation image, and J(x) be the image in each image window of the ground truth work image. First, we calculate the mean image and variance of each image window in the work simulation image, and the mean image and variance of each image window in the ground truth work image. The calculation formula is as follows:

number

[0049] μ I and μ J These represent the average image values ​​of image window I(x) and image window J(x), respectively, and σ I 2 and σ J 2 The terms I(x) and J(x) represent the image variance of image windows I(x) and J(x), respectively, and N represents the number of pixel points in the image window.

[0050] For each work simulation image, the covariance σ between a specific image window of the work simulation image and the corresponding image window of the ground truth work image is calculated. IJ It can be calculated. The formula is as follows:

number

[0051] Based on the mean, variance, and covariance of the image window, the structural similarity index SSIM(I,J) can be calculated using the following formula.

number

[0052] C1 and C2 are constants to avoid an initial value of zero, and usually take small values. The above calculation is repeated for the entire work simulation image using a sliding window method to obtain a structural similarity index for each image window. The average of the structural similarity indices of all image windows is calculated and used as the final structural similarity index. Typically, the final structural similarity index is in the range of [-1, 1], and the closer it is to 1, the higher the similarity between the two images. If a predetermined threshold is set to 0.5, work simulation images with a structural similarity index of 0.5 or higher can be selected, and the remaining work simulation images may be deleted.

[0053] The metal surface of a workpiece has high reflectivity to light irradiation, and this high reflectivity affects the image representation effect, specifically significantly influencing the low-frequency information in the image. Low-frequency information mainly refers to information about the main structure and slowly changing regions in the image, such as wide flat areas or gradual color changes. In other words, low-frequency information is usually represented as smooth, slowly changing regions in the image.

[0054] Based on the above, as shown in Figure 4, in this embodiment, the step of processing the target information contained in each of the selected work simulation images by wavelet transform processing can be implemented as follows.

[0055] Step S141: For each of the selected work simulation images, the low-frequency information contained in the work simulation image is extracted as target information.

[0056] Step S142: Histogram equalization processing is performed on the target information.

[0057] In this embodiment, low-frequency information in the work simulation image can be separated and obtained by wavelet transform processing. As shown in Figure 5, in this embodiment, this step can be specifically implemented as follows.

[0058] Step S1411: Process the work simulation image using a low-pass filter and a high-pass filter.

[0059] Step S1412: The work simulation image after processing is decomposed into low-frequency information and high-frequency information by performing downsampling on the processed work simulation image.

[0060] In this embodiment, the Harr wavelet transform is applied to the sorted work simulation image. The basic principle of the Harr wavelet transform is to displace the fundamental wavelet function and perform an inner product operation with the signal to be measured under different scaling factors. Here, the signal to be measured is the work simulation image after sorting. The principle of the transformation is expressed by the following equation.

number

[0061] α is the scaling factor, X(t) is the signal being measured, τ is the displacement, and φ * (·) is the basic wavelet.

[0062] The two-dimensional pyramid decomposition algorithm is a method for implementing the Harr wavelet transform, and by using the two-dimensional pyramid decomposition algorithm, work simulation images can be decomposed into high-frequency and low-frequency components. The decomposition formula is as follows:

number

[0063] x and y represent the horizontal and vertical coordinates of the pixel point, k and l are integers, and g and h represent the high-pass and low-pass filters, respectively.

number

[0064] The histogram equalization process for the separated low-frequency information mainly involves aggregating the pixel tonal values ​​for each pixel point in the low-frequency information portion of the image, calculating the average value of all pixel tonal values, determining the proportion of each tonal level and below that of all pixels using the histogram's cumulative distribution function, mapping the image's tonal values ​​to new tonal levels using the cumulative distribution function, and matching the average value of the new tonal distribution to the average value of the original image's pixel tonal values. Finally, the calculated new tonal values ​​are assigned to each pixel point in the image to obtain the processed image.

[0065] The above method can make the gradation distribution of the low-frequency information portion of the image more uniform, thereby improving the visual quality of the image. When the lighting is uneven or the contrast is low, and when the low-frequency information portion has high reflectivity to light illumination, the adverse effects of high reflectivity on image quality can be mitigated.

[0066] After obtaining work simulation images through the above processing, the constructed network model is trained using the work simulation images as training samples. In this embodiment, the constructed network model includes a feature extraction module, a feature fusion module, and a post-processing module. The feature extraction module includes a convolutional layer, a transformation layer, and a pooling layer, with a multiscale attention module further connected behind the transformation layer.

[0067] As shown in Figure 6, in this embodiment, the step of training the network model using work simulation images can be implemented as follows. Step S151: Each of the processed work simulation images is introduced into a feature extraction module included in the constructed network model, and the work simulation images are processed by the convolutional layer, transformation layer, pooling layer and multiscale attention module of the feature extraction module to obtain feature extraction results.

[0068] Step S152: The feature fusion module performs a fusion process on the feature extraction results to obtain the fusion processing result.

[0069] Step S153: The post-processing module performs post-processing on the fusion processing result to obtain the prediction result of the work simulation image.

[0070] In this embodiment, the training samples can be divided into a training set and a test set in a fixed ratio, for example, 7:3. The network model constructed using the work simulation images in the training set can be trained, and the model obtained through training can be tested using the work simulation images in the test set. This process can be repeated until the model satisfies the requirements.

[0071] In Figure 7, Backbone, Neck, and Head represent the feature extraction module, feature fusion module, and post-processing module, respectively. The feature extraction module includes multiple convolutional layers (Conv, where k, s, and p represent the size of the convolutional kernel, the stride when moving the convolutional kernel over the input data, and the padding size that adds zeros around the input data, respectively), multiple transformation layers (C2f), and a pooling layer (SPPF), with a multiscale attention module (EMA) further connected behind the transformation layers. For example, the second, third, and fourth transformation layers C2f in the feature extraction module are each followed by a multiscale attention module EMA. The multiscale attention module enhances image features by integrating information across different spatial dimensions and captures long-range and short-range dependencies in the image by calculating the similarity between global and local features. This improves the network model's ability to recognize and segment complex scenes.

[0072] The work simulation images introduced into the feature extraction module are processed by the convolutional layer, transformation layer, pooling layer, and multiscale attention module, respectively, to obtain feature extraction results. As can be seen in Figure 7, the feature extraction results obtained from the convolutional layer, transformation layer, pooling layer, and multiscale attention module are output to the feature fusion module. The feature fusion module fuses the feature extraction results output from the convolutional layer, transformation layer, pooling layer, and multiscale attention module of the feature extraction module, respectively, to obtain the fusion result.

[0073] The feature fusion module includes a convolutional layer (Conv), a transformation layer (C2f), a concatenation layer (Concat), an upsampling layer (Upsample), and an attention fusion module (UAFM). The attention fusion module is connected to the multiscale attention module in the feature extraction module and the transformation layer in the feature fusion module. It performs fusion processing on the output results of the multiscale attention module and the transformation layer, and outputs the fused result to the post-processing module. For example, the input terminal of the attention fusion module is connected to the first multiscale attention module in the feature extraction module and the first transformation layer of the feature fusion module, and the output terminal of the attention fusion module is connected to the post-processing module.

[0074] By adding an attention fusion module to the feature fusion module, detailed features from the lower layers and semantic features from the higher layers can be merged, highlighting the network's characteristics and improving the accuracy of network-based segmentation.

[0075] As can be seen from the diagram, the post-processing module includes a small-shot network (ProtoNet) and multiple heads (Segments). The small-shot network is connected to the attention fusion module of the feature fusion module, and each head is connected to the transformation layer of the feature fusion module. The small-shot network processes the feature fusion results output from the attention fusion module, and each head processes the feature fusion results received from the transformation layer. Finally, the processing results from the small-shot network and each head are fused to obtain the final output result.

[0076] In this embodiment, an EMA module is added after each transformation layer of the feature extraction module, effectively fusing global and local feature information through feature grouping and cross-dimensional interaction. This allows the model to capture the overall placement information and local feature pattern information of the workpiece within the deep container, improving the impact of changes in image lighting and workpiece entanglement, and enhancing the accuracy of workpiece detection. In the feature fusion module, the multiscale fusion portion is placed within the UAFM module, improving the accuracy of workpiece segmentation by the network while preserving the detailed features of the lower layers of the network in the feature extraction module and the abstract semantic features of the higher layers after feature fusion.

[0077] Based on the network model described above, the network model is trained using each work simulation image. Each work simulation image has a ground truth label that includes a ground truth class and a ground truth box. The ground truth class refers to the class of each work in the image, and the ground truth box refers to the frame surrounding each work.

[0078] When training the network model, each processed work simulation image is introduced into the constructed network model for processing, and the predicted class and predicted box of the work simulation image are output. The predicted class refers to the class of each work in the work simulation image recognized by the network model, and the predicted box refers to the frame surrounding each work in the segmented work simulation image output from the network model.

[0079] Based on the constructed loss function, the model parameters of the network model are adjusted, and the training is continued until the predetermined requirements are met, to obtain an instance segmentation model necessary for picking the work in the deep container.

[0080] The predetermined requirements are, for example, when the training time reaches the predetermined time, when the number of training iterations reaches the predetermined number, when the loss function converges and no longer changes, etc.

[0081] The loss function includes a class loss function, a predicted box regression loss function, and a segmentation loss function. The class loss function is constructed based on the correct class and predicted class of the work simulation image, and the predicted box regression function is constructed based on the correct box and predicted box of the work simulation image. The final loss function is obtained by superimposing the class loss function, the predicted box regression loss function, and the segmentation loss function with different weights.

[0082] In this embodiment, the class loss function f BCEL uses binary cross-entropy loss, and the calculation formula is as follows.

Equation

[0083] w represents the weight value of each class, and x n represents the predicted class of the work simulation image by the network model, and y nThis represents the correct class of the work simulation image.

[0084] The prediction box regression loss function consists of the bounding box regression loss function and the localization accuracy loss function. CIoUL It can be expressed by the following formula.

number

number

[0085] Note that the localization loss function f DFL (S i ,S i+1 The formula for calculating ) is as follows:

number

[0086] In conventional network models, the predicted target box coordinates correspond to a single ground truth target box coordinate, and the loss value is calculated accordingly. However, this calculation method is not applicable when the target boundary is unclear. The label range is y0 ≤ y ≤ y n And the estimated values ​​corresponding to a series of discrete correct values

number

number

number

number

[0087] The segmentation loss function employs a pixel-level cross-entropy loss function, and the segmentation loss function f SegL The formula for calculating this is as follows:

number

[0088] M is the mask image predicted by the network module, gt This is the ground truth mask image. BCE(·) represents the pixel-level binary cross-entropy operation on the prediction mask and the ground truth mask, and A gt This represents the area of ​​the smallest bounding box of the correct mask.

[0089] The final loss function is obtained by superimposing the above loss functions with different weights, and is specifically as follows:

number

[0090] λ1, λ2, λ3, and λ4 are the weight coefficients for each loss function, and can be set to, for example, 0.05, 0.15, 0.75, and 0.75, respectively.

[0091] To improve the generalization performance of the model, data augmentation processing can be performed during the training process. Data augmentation processing includes operations such as rotation, scaling, flipping, and color adjustment on work simulation images, and by simulating various complex scenarios, it ensures that the model obtained through training can be applied to actual application environments.

[0092] The workpiece picking method in deep containers according to this embodiment generates diverse virtual datasets based on virtual simulation technology, structural similarity indices, and methods for suppressing high reflectivity of metals using wavelet transforms. By covering various lighting, background, and workpiece entanglement scenarios, it is possible to ensure that the trained model possesses stronger generalization performance and robustness in industrial environments. Furthermore, by generating a large amount of high-quality training data, it is possible to reduce reliance on obtaining ground truth workpiece data and manual data labeling, thereby reducing the cost and time of data creation.

[0093] Furthermore, by introducing an attention mechanism and constructing a network model, the network model's ability to learn complex shapes and pattern features of workpieces was improved, enabling it to adaptively pay attention to workpiece features at different scales. This significantly improved the accuracy of detection and segmentation of stacked workpieces, tangled workpieces, and workpieces in complex orientations.

[0094] Based on the same inventive concept, as shown in Figure 8, an embodiment of the present invention further provides a schematic diagram of a functional module of a device for realizing workpiece picking in a deep container. In this embodiment, the device for realizing workpiece picking in a deep container can be divided into functional modules according to the above-described method embodiment. For example, each functional module may be divided according to each function, or two or more functions may be integrated into one processing module. The above-described integrated module may be implemented in hardware or in software functional module form. Furthermore, the module division according to the embodiment of the present invention is illustrative and merely a logical functional division, and a different division method may be adopted in actual implementation.

[0095] For example, if each functional module is divided according to its function, the device for realizing workpiece picking in a deep container shown in Figure 8 is merely a schematic diagram of the device. The device for realizing workpiece picking in a deep container may include a construction module, an acquisition module, a sorting module, a processing module, and a training module. The functions of each functional module of the device for realizing workpiece picking in a deep container will be described in detail below.

[0096] The construction module builds multiple three-dimensional workpiece models based on existing workpiece information.

[0097] The acquisition module stacks the multiple three-dimensional workpiece models inside a deep container model and acquires multiple workpiece simulation images under different simulation conditions.

[0098] The selection module selects multiple work simulation images from the multiple work simulation images based on the created structural similarity index.

[0099] The processing module processes the target information contained in each of the selected work simulation images using wavelet transform processing to reduce the high reflectivity of the work simulation images.

[0100] The training module uses each of the processed work simulation images to train the constructed network model and obtain an instance segmentation model necessary for picking workpieces in deep containers.

[0101] Furthermore, the above steps S11 to S15 can be executed using the construction module, acquisition module, selection module, processing module, and training module. For detailed implementation methods of the construction module, acquisition module, selection module, processing module, and training module, please refer to the relevant content of steps S11 to S15 above.

[0102] In a feasible form, the selection module calculates a structural similarity index between each work simulation image and an existing ground truth work image based on the created structural similarity index, and selects work simulation images whose structural similarity index is equal to or greater than a predetermined threshold.

[0103] In a feasible form, the sorting module specifically divides each of the work simulation images and existing ground truth work images into multiple image windows, calculates the average image value and image variance for each image window of each work simulation image and ground truth work image, calculates the covariance for each image window in each work simulation image based on the average image value and image variance of the image window and the average image value and image variance of the corresponding image window in the ground truth work image, calculates a structural similarity index based on the average image value, image variance and covariance of the image window, and calculates a final structural similarity index based on the multiple structural similarity indices of the multiple image windows of the work simulation image.

[0104] In a feasible form, the processing module extracts low-frequency information contained in each selected work simulation image as target information, and performs histogram equalization processing on the target information.

[0105] In a feasible form, the processing module processes the work simulation image using a low-pass filter and a high-pass filter, and then performs downsampling on the processed work simulation image to decompose the work simulation image into low-frequency information and high-frequency information.

[0106] In a feasible form, each work simulation image has a ground truth label including a ground truth class and a ground truth box. The training module processes each processed work simulation image by introducing it into a constructed network model, outputs a predicted class and a predicted box for the work simulation image, adjusts the model parameters of the network model based on a constructed loss function, and continues training until predetermined requirements are met, thereby obtaining an instance segmentation model necessary for picking workpieces in deep containers.

[0107] The loss function includes a class loss function, a predictive box regression loss function, and a segmentation loss function, wherein the class loss function is constructed based on the ground truth class and predicted class of the work simulation image, and the predictive box regression loss function is constructed based on the ground truth box and predicted box of the work simulation image.

[0108] In a feasible form, the network model includes a feature extraction module, a feature fusion module, and a post-processing module. The feature extraction module includes a convolutional layer, a transformation layer, and a pooling layer, with a multiscale attention module further connected behind the transformation layer.

[0109] The above training module is introduced into a feature extraction module included in the constructed network model, and the work simulation images are processed by the convolutional layer, transformation layer, pooling layer and multiscale attention module of the feature extraction module to obtain feature extraction results, the feature fusion module performs a fusion process on the feature extraction results to obtain fusion processing results, and the post-processing module performs post-processing on the fusion processing results to obtain prediction results for the work simulation images.

[0110] In a feasible form, the feature fusion module includes a conversion layer, a convolutional layer, a coupling layer, an upsampling layer, and an attention fusion module.

[0111] The attention fusion module is connected to the multiscale attention module in the feature extraction module and the conversion layer in the feature fusion module, performs fusion processing on the output results of the multiscale attention module and the conversion layer, and outputs the result after fusion processing to the post-processing module.

[0112] Figure 9 is a structural block diagram of an electronic device according to an embodiment of the present invention. This electronic device may be a computer system or the like. The electronic device includes a memory, a processor, and a communication module. Each element of the memory, processor, and communication module is electrically connected to each other directly or indirectly in order to enable data transfer and interaction. For example, these elements can be electrically connected to each other via one or more communication buses or signal lines.

[0113] Memory is a device that stores computer programs or data. Memory can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electric Erasable Programmable Read-Only Memory (EEPROM), etc.

[0114] The processor reads and writes data and programs stored in memory and executes a method for picking workpieces in a deep container according to any embodiment of the present invention.

[0115] A communication module establishes a communication connection between electronic devices and other communication terminals via a network, and transmits and receives data over the network.

[0116] Furthermore, the structure shown in Figure 9 is merely a schematic diagram of the electronic device's structure, and the electronic device may contain more or fewer components than those shown in Figure 9, or may have a different configuration from that shown in Figure 9.

[0117] Furthermore, embodiments of the present invention further provide a computer-readable storage medium that stores machine-executable commands that, when executed, realize a method for picking workpieces in a deep container according to the above embodiments.

[0118] Specifically, this computer-readable storage medium may be a general-purpose storage medium such as a removable disk or hard disk. When a computer program stored on this computer-readable storage medium is executed, the method for picking workpieces in the deep container described above can be performed. The process by which executable commands on the computer-readable storage medium are executed can be found in the relevant explanations in the above-described embodiment of the method, and is therefore omitted here.

[0119] As described above, the method, apparatus, and electronic equipment for picking workpieces in a deep container according to the embodiment of the present invention constructs multiple three-dimensional workpiece models based on existing workpiece information, stacks the multiple three-dimensional workpiece models in a deep container model, and acquires multiple workpiece simulation images under different simulation conditions. Based on the created structural similarity index, multiple workpiece simulation images are selected from the multiple workpiece simulation images, and the target information contained in each selected workpiece simulation image is processed by wavelet transform processing to reduce the high reflectivity of the workpiece simulation images. The constructed network model is trained using the processed workpiece simulation images to obtain an instance segmentation model necessary for picking workpieces in a deep container. In this application, the generalization performance and robustness of the model obtained by training are ensured by generating diverse simulation data with high accuracy.

[0120] In embodiments of the present invention, the described apparatus and methods can be implemented in other ways. The embodiments of the apparatus described above are illustrative only. For example, the unit divisions are merely logical functional divisions, and in actual implementation, different divisions may be used. For example, multiple units or components may be combined or integrated into another system, or some features may be omitted or not implemented. Furthermore, the mutual coupling, direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection via several interfaces, devices or units, and may be an electrical, mechanical or other type of connection.

[0121] Furthermore, units described as separate components may or may not be physically separate. Components shown as units may or may not be physical units; that is, they may be located in the same location or distributed across multiple network units. To achieve the objectives of this embodiment, it is possible to select some or all units according to actual requirements.

[0122] Furthermore, each functional module in each embodiment of the present invention may be a single independent part formed by integration, a single module, or a single independent part formed by integration of two or more modules.

[0123] Furthermore, the functionality is implemented in the form of a software function module, and when sold or used as an independent product, it can be stored on a computer-readable storage medium. Based on this understanding, the technical form of the present invention itself, or a part of the prior art that can contribute to it, or a part of the technical form, can be implemented in the form of a software product. This computer software product is stored on a storage medium and includes a plurality of commands for a computer device (such as a personal computer, server, or network device) to execute all or part of the steps of the above method in each embodiment of the present invention. The above-mentioned storage medium includes various media capable of storing program code, such as USB disks, portable hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0124] In this specification, relational terms such as "first" and "second" may be used solely to distinguish one entity or action from another entity or action, without necessarily requiring or implying any factual relationship or order between such entities or actions.

[0125] The above description is merely an example of the present invention and does not limit the scope of protection of the present invention. Those skilled in the art may have various modifications and changes to the present invention. Any modifications, equivalent substitutions, improvements, etc., made without departing from the spirit and principles of the present invention are within the scope of protection of the present invention.

Claims

1. Steps include constructing multiple 3D models of workpieces based on existing workpiece information, The steps include stacking the aforementioned three-dimensional work models inside a deep container model and obtaining multiple work simulation images under different simulation conditions, The steps include selecting multiple work simulation images from the multiple work simulation images based on the created structural similarity index, The steps include: processing the target information contained in each selected work simulation image by wavelet transform processing to reduce the high reflectivity of the work simulation image; The process includes the steps of: training a constructed network model using each of the processed work simulation images to obtain an instance segmentation model necessary for picking workpieces in a deep container; The step of selecting multiple work simulation images from the multiple work simulation images based on the created structural similarity index is: A step of calculating a structural similarity index between each work simulation image and an existing ground truth work image based on the created structural similarity index, The process includes the step of selecting work simulation images whose structural similarity index is equal to or greater than a predetermined threshold, The step of processing the target information contained in each selected work simulation image by wavelet transform processing is: For each of the selected work simulation images, the steps include extracting the low-frequency information contained in the work simulation image as target information, The step includes performing histogram equalization processing on the aforementioned target information. A method for picking workpieces in a deep container, characterized by the features described above.

2. The step of calculating a structural similarity index between each work simulation image and an existing ground truth work image based on the created structural similarity index is: The steps include dividing each of the aforementioned work simulation images and existing ground truth work images into multiple image windows, A step of calculating the average image value and image variance for each image window of each work simulation image and the ground truth work image, For each image window in each of the work simulation images, the covariance is calculated based on the image mean and image variance of the image window, and the image mean and image variance of the image window corresponding to the image window in the ground truth work image. The steps include: calculating a structural similarity index based on the average image value of the image window, the image variance, and the covariance; The process includes the step of calculating a final structural similarity index based on multiple structural similarity indices of multiple image windows of the work simulation image. The method for picking workpieces in a deep container according to feature 1.

3. The step of extracting low-frequency information contained in the aforementioned work simulation image is: The steps include processing the aforementioned work simulation image using a low-pass filter and a high-pass filter, The process includes the step of decomposing the processed work simulation image into low-frequency information and high-frequency information by performing downsampling on the processed work simulation image. The method for picking workpieces in a deep container according to feature 1.

4. Each of the aforementioned work simulation images has a ground answer label that includes a ground answer class and a ground answer box. The step of training the constructed network model using each of the processed work simulation images to obtain an instance segmentation model necessary for picking workpieces in a deep container is: The steps include: importing each processed work simulation image into the constructed network model for processing, and outputting the predicted class and predicted box of the work simulation image; The process includes the steps of: adjusting the model parameters of the network model based on the constructed loss function, continuing training until predetermined requirements are met, and obtaining an instance segmentation model necessary for picking workpieces in a deep container; The loss function includes a class loss function, a predictive box regression loss function, and a segmentation loss function, wherein the class loss function is constructed based on the ground truth class and predicted class of the work simulation image, and the predictive box regression loss function is constructed based on the ground truth box and predicted box of the work simulation image. The method for picking workpieces in a deep container according to feature 1.

5. The aforementioned network model includes a feature extraction module, a feature fusion module, and a post-processing module. The feature extraction module includes a convolutional layer, a transformation layer, and a pooling layer, with a multiscale attention module further connected behind the transformation layer. The step of training the network model constructed using each of the processed work simulation images is as follows: The process involves introducing each processed work simulation image into a feature extraction module included in the constructed network model, processing the work simulation image using the convolutional layer, transformation layer, pooling layer, and multiscale attention module of the feature extraction module, and obtaining feature extraction results. The steps include: performing a fusion process on the feature extraction results using the feature fusion module to obtain the fusion processing result; The post-processing module performs post-processing on the fusion processing result to obtain a prediction result for the work simulation image, including the step of obtaining a prediction result for the work simulation image. The method for picking workpieces in a deep container according to feature 1.

6. The aforementioned feature fusion module includes a conversion layer, a convolutional layer, a coupling layer, an upsampling layer, and an attention fusion module. The attention fusion module is connected to the multiscale attention module in the feature extraction module and the conversion layer in the feature fusion module, performs fusion processing on the output results of the multiscale attention module and the conversion layer, and outputs the result after fusion processing to the post-processing module. The method for picking workpieces in a deep container according to feature 5.

7. An apparatus for realizing the method of picking workpieces in a deep container according to any one of claims 1 to 6, wherein the method of picking workpieces in a deep container is realized, A construction module that constructs multiple 3D models of workpieces based on existing workpiece information, An acquisition module that stacks the aforementioned three-dimensional work models inside a deep container model and acquires multiple work simulation images under different simulation conditions, A selection module that selects multiple work simulation images from the multiple work simulation images based on the created structural similarity index, A processing module that processes the target information contained in each selected work simulation image by wavelet transform processing to reduce the high reflectivity of the work simulation image, The system includes a training module that trains a constructed network model using each of the processed work simulation images to obtain an instance segmentation model necessary for picking workpieces in deep containers. A device for realizing workpiece picking in a deep container, characterized by the above features.

8. The system includes a processor and memory that stores computer programs that can be executed by the processor, When the processor executes the computer program, it realizes the steps of the method according to any one of claims 1 to 6. An electronic device characterized by the following features.

Images