Parcel grasping method and related apparatus

Abstract

The application discloses a parcel grabbing method and related equipment, the method comprising: obtaining a target RGBD image corresponding to a parcel table; the target RGBD image contains three-dimensional point cloud data; obtaining each parcel existing in the target RGBD image by segmenting the target RGBD image; determining a parcel meeting optimal grabbing conditions from each parcel as a target parcel according to three-dimensional point cloud data corresponding to each parcel in the target RGBD image; the optimal grabbing conditions include that a physical attribute possessed matches a preset physical attribute requirement, and a space position represented by corresponding three-dimensional point cloud data meets a preset space position requirement; and grabbing the target parcel.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a parcel grabbing method and device, an electronic device, a storage medium, and a computer program product. BACKGROUND

[0002] In recent years, with the development of e-commerce, the express industry has also rapidly developed and gradually expanded. An express center usually includes a parcel feeding system and a parcel sorting system. The main function of the parcel feeding system is to sequentially take out parcels on a parcel feeding table and deliver them to the parcel sorting system. The main function of the parcel sorting system is to sort the parcels delivered by the parcel feeding system according to the commodity variety, commodity delivery location, or consignee category.

[0003] In actual scenarios, there is often a demand for "precise parcel taking" for the parcel feeding system, that is, while taking out a parcel from a parcel feeding table on which parcels are densely distributed and stacked in disorder, other parcels to be taken out cannot be interfered with.

[0004] Since there is currently no technical solution that meets this demand, in scenarios where this demand exists, "precise parcel taking" can only be achieved by relying on manual operation, and manual parcel taking has the problem of low parcel feeding efficiency. SUMMARY

[0005] Embodiments of the present application provide a training of an instance segmentation model and a parcel grabbing method and device, an electronic device, a storage medium, and a computer program product, to solve the problem of low parcel grabbing efficiency caused by the need to rely on manual operation when the existing parcel sorting system performs parcel grabbing.

[0006] Embodiments of the present application adopt the following technical solutions:

[0007] In a first aspect, the present application provides a parcel grabbing method, comprising:

[0008] obtaining a target RGBD image corresponding to a parcel feeding table; the target RGBD image containing three-dimensional point cloud data;

[0009] obtaining each parcel existing in the target RGBD image by segmenting the target RGBD image;

[0010] determining, as a target parcel, a parcel that meets optimal grabbing conditions from the each parcel according to the three-dimensional point cloud data corresponding to the each parcel in the target RGBD image; the optimal grabbing conditions including that a physical attribute possessed matches a preset physical attribute requirement, and a spatial position represented by the corresponding three-dimensional point cloud data meets a preset spatial position requirement;

[0011] grabbing the target parcel.

[0012] In a second aspect, the present application provides a model training method, comprising:

[0013] obtaining a plurality of RGB images on a parcel table and three-dimensional point cloud data corresponding to the plurality of RGB images;

[0014] based on the plurality of RGB images and the three-dimensional point cloud data corresponding to the plurality of RGB images, constructing RGBD images corresponding to the plurality of RGB images, and annotation information for indicating the external contour and category of the parcel to be grabbed corresponding to each of the RGBD images;

[0015] based on the RGBD images corresponding to the plurality of RGB images and the annotation information corresponding thereto, training a preset instance segmentation model to obtain the target instance segmentation model.

[0016] In a third aspect, the present application provides a parcel grabbing device, comprising:

[0017] an image acquisition module configured to obtain a target RGBD image corresponding to a parcel table; the target RGBD image includes three-dimensional point cloud data;

[0018] an image segmentation module configured to obtain each parcel present in the target RGBD image by segmenting the target RGBD image;

[0019] a parcel determination module configured to determine a parcel meeting optimal grabbing conditions from the each parcel as a target parcel according to the three-dimensional point cloud data corresponding to the each parcel in the target RGBD image; the optimal grabbing conditions include that the physical attributes possessed match preset physical attribute requirements, and a spatial position represented by the corresponding three-dimensional point cloud data meets preset spatial position requirements;

[0020] a parcel grabbing module configured to grab the target parcel.

[0021] In a fourth aspect, the present application provides a model training device, comprising:

[0022] a data acquisition module configured to obtain a plurality of RGB images on a parcel table and three-dimensional point cloud data corresponding to the plurality of RGB images;

[0023] an image construction module configured to construct RGBD images corresponding to the plurality of RGB images based on the plurality of RGB images and the three-dimensional point cloud data corresponding to the plurality of RGB images, and annotation information for indicating the external contour and category of the parcel to be grabbed corresponding to each of the RGBD images;

[0024] The model training module is configured to train a preset instance segmentation model based on the RGBD images corresponding to the plurality of RGB images and the corresponding label information, to obtain the target instance segmentation model.

[0025] In a fifth aspect, the present application provides an electronic device, comprising a memory and a processor;

[0026] The memory is configured to store a computer program.

[0027] The processor is coupled to the memory and is configured to execute the computer program, so as to:

[0028] obtaining a target RGBD image corresponding to a parcel table; the target RGBD image contains three-dimensional point cloud data; obtaining each parcel existing in the target RGBD image by segmenting the target RGBD image; determining a parcel meeting an optimal grabbing condition from the each parcel as a target parcel according to the three-dimensional point cloud data corresponding to the each parcel in the target RGBD image; the optimal grabbing condition includes that a physical attribute possessed matches a preset physical attribute requirement, and a spatial position represented by corresponding three-dimensional point cloud data meets a preset spatial position requirement; grabbing the target parcel; or

[0029] obtaining a plurality of RGB images on a parcel table and three-dimensional point cloud data corresponding to the plurality of RGB images; constructing RGBD images corresponding to the plurality of RGB images and label information indicating an external contour and a category of a parcel to be grabbed corresponding to each of the RGBD images based on the plurality of RGB images and the three-dimensional point cloud data corresponding to the plurality of RGB images; training a preset instance segmentation model based on the RGBD images corresponding to the plurality of RGB images and the corresponding label information, to obtain the target instance segmentation model.

[0030] In a sixth aspect, the present application provides a computer readable storage medium, characterized in that the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the parcel grabbing method of the first aspect or the training method of the instance segmentation model of the second aspect.

[0031] In a seventh aspect, the present application provides a computer program product, comprising a computer program, which is executed by a processor to implement the steps of the parcel grabbing method of the first aspect or the training method of the instance segmentation model of the second aspect.

[0032] The above at least one technical solution adopted by the embodiments of the present application can achieve the following beneficial effects:

[0033] By adopting the method provided in the embodiments of the present application, when the parcels on the parcel supply table are grabbed, the target RGBD image corresponding to the parcel supply table can be obtained, the target RGBD image contains three-dimensional point cloud data, the target RGBD image is segmented to obtain each parcel existing in the target RGBD image, and finally, the three-dimensional point cloud data corresponding to each parcel in the target RGBD image is taken as the basis to determine the parcel meeting the optimal parcel grabbing condition as the target parcel to be grabbed, and the target parcel is grabbed, in which process, the target parcel to be grabbed is determined without human intervention, and the parcel grabbing efficiency is effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0034] The accompanying drawings, which are included to provide a further understanding of the present application, form a part of the present application and illustrate the illustrative embodiments of the present application and together with the description serve to explain the present application. In the drawings:

[0035] Figure 1 An implementation flowchart of a parcel grabbing method provided in the embodiments of the present application is provided;

[0036] Figure 2 A network structure schematic diagram of a preset instance segmentation model constructed in the parcel grabbing method provided in the embodiments of the present application is provided;

[0037] Figure 3 A schematic diagram in which the parcel grabbing method provided in the embodiments of the present application is applied to an actual scene is provided;

[0038] Figure 4 A parcel instance segmentation effect schematic diagram in which the parcel grabbing method provided in the embodiments of the present application is applied to an actual scene is provided;

[0039] Figure 5 A flowchart in which the target parcel is determined based on the parcel category in the parcel grabbing method provided in the embodiments of the present application is provided;

[0040] Figure 6 A flowchart in which the parcel grabbing method provided in the embodiments of the present application is applied to an actual scene is provided;

[0041] Figure 7 An implementation flowchart of a model training method provided in the embodiments of the present application is provided;

[0042] Figure 8 A specific structure schematic diagram of a parcel grabbing device provided in the embodiments of the present application is provided;

[0043] Figure 9 A specific structure schematic diagram of a model training device provided in the embodiments of the present application is provided;

[0044] Figure 10A structural schematic diagram of an electronic device is provided in the embodiments of the present application. DETAILED DESCRIPTION

[0045] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced without the specific details. In other instances, well-known methods have not been described in detail in order not to unnecessarily obscure aspects of the present application.

[0046] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments of the present application. As used in this disclosure and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0047] It will be understood that, although the terms first, second, etc. can be used herein to describe various information, these terms are not intended to denote a temporal sequence, but are used only to distinguish one piece of information from another. For example, without departing from the scope of one or more embodiments of the present application, first can be termed second; likewise, second can be termed first. Depending on the context, the word "if' as used herein can be interpreted to mean "when" or "in response to determining."

[0048] First, the noun terms related to one or more embodiments of the present application are explained.

[0049] ROI, the full name of which is Region of Interest, is the region of interest. In deep learning, a common application is target detection using a convolutional neural network. When processing images or videos, the network needs to identify objects in the image, and in most cases, only some specific regions of the image are of interest, rather than the entire image, and this specific region can be called ROI.

[0050] Mask, in English, is a mask that uses selected images, patterns or objects to shield the processed image (all or part) to control the image processing area or processing process. The specific image or object used for covering is called mask or template. The mask is a binary image composed of 0 and 1. When the mask is applied in a certain function, the 1 value area is processed, and the shielded 0 value area is not included in the calculation. Image mask can be defined by specifying data value, data range, finite or infinite value, region of interest and annotation file, or any combination of the above options can be applied as input to establish the mask. Mask is a template for image filter. For example, in the machine vision scene of product quality detection, when extracting the surface defects of the product, a N*N matrix is used to filter the pixels of the image, and then the mark we need is highlighted. This matrix is a mask.

[0051] Image segmentation is a technique and process of dividing an image into several specific regions with unique properties and extracting the target of interest. It is a key step from image processing to image analysis. Existing image segmentation methods mainly include the following categories: threshold-based segmentation method, region-based segmentation method, edge-based segmentation method and segmentation method based on specific theory, such as deep learning. From a mathematical point of view, image segmentation is a process of dividing a digital image into mutually disjoint regions. The process of image segmentation is also a labeling process, that is, pixels belonging to the same region are assigned the same number.

[0052] Semantic segmentation: semantic segmentation is a classification at the pixel level. Pixels belonging to the same class are classified into the same class, so semantic segmentation is to understand the image at the pixel level. For example, in an image, pixels belonging to a person are classified into a class, pixels belonging to a motorcycle are also classified into a class, and in addition, background pixels are also classified into a class.

[0053] To solve the above problems existing in the prior art, the embodiment of the present application provides a parcel grabbing method. Specifically, in the present application, a parcel grabbing method and a training method of an instance segmentation model are provided, and the present application also relates to a parcel grabbing device and a training device of an instance segmentation model, an electronic device, a computer readable storage medium and a computer program product, which are described in detail one by one in the following embodiments.

[0054] The execution subject of the method can be various types of computing devices, or can be an application or application (APP) installed on a computing device, such as a parcel grabbing application. The computing device can be a user terminal such as a mobile phone, a tablet computer, a smart wearable device, etc., or a server, etc.

[0055] For ease of description, the embodiments of the present application take the server as an example to introduce the method. Those skilled in the art can understand that the embodiments of the present application take the server as an example to introduce the method, which is only an exemplary description and does not limit the protection scope of the claims corresponding to the present solution.

[0056] Figure 1 is a kind of package grabbing method provided by the embodiments of the present application applied to actual scene architecture schematic diagram.

[0057] Figure 1 The architecture shown can include a 3D structured light camera, a package supply table, a display, a computer host, a mechanical hand and a conveyor belt electrical controller. Among them, the package supply table is used to receive packages, and the received multiple packages are often stacked together in disorder. The 3D structured light camera is used to collect the RGB image and the corresponding three-dimensional point cloud data of the package in the ROI on the current package supply table. The target instance segmentation model is deployed in the computer host. The computer host is used to receive the RGB image and the corresponding three-dimensional point cloud data of the current package supply table collected by the 3D structured light camera, and based on the RGB image and the corresponding three-dimensional point cloud data of the current package supply table, construct the RGBD image corresponding to the RGB image on the current package supply table, and input the constructed RGBD image into the target instance segmentation model, and based on the RGBD image, the target instance segmentation model is segmented to obtain the mask area and the corresponding package category of each package on the current package supply table. The computer host determines the three-dimensional point cloud data corresponding to the mask area of each package on the current package supply table from the three-dimensional point cloud data corresponding to the current package supply table, and based on the three-dimensional point cloud data corresponding to the mask area of each package on the package supply table, the package meeting the preset grabbing condition for screening the optimal grabbing package is taken as the target package to be grabbed, and the display is used to display the processing result of the computer host. The computer host determines the grabbing point coordinates of the target package based on the three-dimensional point cloud data corresponding to the target package to be grabbed, and sends the grabbing point coordinates of the target package to the mechanical hand used for grabbing the package. The mechanical hand grabs the target package based on the grabbing point coordinates and puts it on the conveyor belt. The conveyor belt electrical controller transmits the target package to the shelf where the target package is stored based on the category of the target package. After receiving the package, the conveyor belt electrical controller sends an image acquisition signal to the 3D structured light camera, and the 3D structured light camera collects the RGB image and the three-dimensional point cloud data on the current package supply table again until the grabbing and transmission of all packages on the package supply table are completed.

[0058] Since the collected RGB image and three-dimensional point cloud data corresponding to the parcel table can be combined into a target RGBD image, and the target instance segmentation model is trained in advance to segment the target RGBD image, the mask area of each parcel on the parcel table is obtained, and finally, the three-dimensional point cloud data corresponding to the mask area of each parcel on the parcel table is determined as the basis for determining the parcel that best meets the preset grabbing condition, such as the parcel with the highest height on the parcel table or the parcel with the largest area on the parcel table, as the target parcel to be grabbed, so that the mechanical hand used to grab the parcel can grab the target parcel based on the three-dimensional point cloud data of the target parcel, without the need for manual operation to grab the parcel on the parcel table, thereby effectively improving the parcel grabbing efficiency.

[0059] Specifically, the implementation flowchart of the parcel grabbing method provided by one or more embodiments of the present application is shown in Figure 2 as shown, comprising:

[0060] In step 210, a target RGBD image corresponding to the parcel table is obtained; the target RGBD image contains three-dimensional point cloud data.

[0061] Specifically, the image acquisition device (for example, a structured light 3D camera) arranged above the surface of the parcel table can be used to collect the RGB image and three-dimensional point cloud data of the parcel table stacked with at least one parcel.

[0062] Optionally, since the RGB image and three-dimensional point cloud data corresponding to the parcel table are two types of data, in order to reduce the calculation resources consumed in subsequent image segmentation based on the RGB image and three-dimensional point cloud data corresponding to the parcel table, the present embodiment can increase a channel dimension based on the RGB image, and fill the three-dimensional point cloud data corresponding to the RGB image into the newly added channel dimension as a depth map, that is, the RGB image and three-dimensional point cloud data corresponding to the parcel table are processed as one image data, without the need for separate labeling and processing of the RGB image and three-dimensional point cloud data corresponding to the parcel table, thereby reducing the data types and data volume to be processed in the image segmentation process. Specifically, the target RGBD image corresponding to the parcel table is obtained, comprising:

[0063] The obtained three-dimensional point cloud data corresponding to the parcel table is mapped into a matrix with the same width and height as the RGB image corresponding to the parcel table, to obtain a depth map corresponding to the parcel table.

[0064] Based on the first distance between the image acquisition device and the parcel table, the depth map corresponding to the parcel table is updated; wherein the image acquisition device is used to collect the RGB image and three-dimensional point cloud data corresponding to at least one parcel on the parcel table.

[0065] The updated depth map corresponding to the parcel supply table is stored in a fourth channel of the RGB image corresponding to the parcel supply table to obtain a target RGBD image.

[0066] The depth map is a gray image, and each pixel value in the image represents a distance of a pixel point corresponding to the pixel value from the image acquisition device. To exclude the influence of the depth information of the non-parcel area on the parcel supply table, the ROI can be selected according to the parcel grabbing range, and the depth value of the parcel in the ROI on the parcel supply table is taken, and the depth value outside the ROI is set to zero.

[0067] Optionally, to facilitate determination of the topmost parcel on the parcel supply table, an embodiment of the present application can determine the distance between each parcel on the parcel supply table and the tabletop of the parcel supply table, and update the pixel value in the depth map based on the distance. Specifically, based on the obtained first distance of the image acquisition device from the parcel supply table, the depth map corresponding to the parcel supply table is updated, including:

[0068] determining the difference between the first distance of the image acquisition device from the parcel supply table and the depth value of each parcel in the depth map corresponding to the at least one parcel;

[0069] taking the difference between the first distance and the depth value of each parcel in the depth map corresponding to the parcel supply table as the depth value of each parcel in the depth map corresponding to the parcel supply table to update the depth map corresponding to the parcel supply table.

[0070] The first distance of the image acquisition device from the parcel supply table can be determined based on the obtained three-dimensional point cloud data corresponding to the parcel supply table when there is no parcel on the parcel supply table. The depth value of each parcel in the depth map corresponding to the at least one parcel on the parcel supply table can be directly obtained from the three-dimensional point cloud data corresponding to the parcel supply table. The embodiment of the present application prioritizes the determination of the topmost parcel on the parcel supply table. The height of each parcel from the tabletop of the parcel supply table can be obtained by subtracting the current depth value of each parcel from the first distance of the image acquisition device from the parcel supply table. Obviously, the closer the parcel is to the image acquisition device, the greater the depth value.

[0071] The updated depth map corresponding to the parcel supply table can also be normalized to obtain a normalized depth map corresponding to the parcel supply table on the parcel supply table. Specifically, each pixel value in the updated depth map corresponding to the parcel supply table can be normalized to 0-255. The normalized depth map is stored in the fourth channel of the RGB image, and as an example, since the png format image contains four channels, the four-channel image can be saved in png format.

[0072] In step 220, each parcel existing in the target RGBD image is obtained by segmenting the target RGBD image.

[0073] The target instance segmentation model can be used to segment the target RGBD image. Specifically, the training process of the target instance segmentation model can include:

[0074] At step 221, a training sample set is obtained, which is composed of RGBD images constructed based on RGB images and three-dimensional point cloud information corresponding to a plurality of parcel tables, and annotation information indicating the category and contour of the top layer parcel in each RGBD image.

[0075] The RGB images and three-dimensional point cloud information corresponding to the plurality of parcel tables can be obtained by an image acquisition device arranged above the parcel table, from a scene with different parcel morphologies stacked on one or more parcel tables. The RGBD image is constructed based on the RGB images and three-dimensional point cloud information corresponding to the plurality of parcel tables. Specifically, for the RGB image and three-dimensional point cloud data corresponding to each parcel table, the three-dimensional point cloud data can be mapped to a matrix with the same width and height as the corresponding RGB image to obtain a corresponding depth map. For each depth map, only the depth values within the pre-set parcel grabbing range (i.e. within the ROI) can be taken, and the depth values outside the ROI can be set to 0. At the same time, for each depth map, the distance between the image acquisition device and the parcel table can be subtracted from the current depth value of each parcel in the depth map to obtain the distance between each parcel and the parcel table in the depth map, and finally the depth values of each parcel in the depth map are updated based on the distance between each parcel and the parcel table in the depth map.

[0076] Each depth value in the updated depth map is then normalized to 0-255, and each depth value in the normalized depth map is stored in the fourth channel of the corresponding RGB image. The fourth channel is initially empty, and finally the four-channel image is saved as a png format to obtain the corresponding RGBD image. Since the value of the fourth channel of the png format saved by the RGBD image will cause the RGBD image to appear transparent, the image annotation tool labelme can be used to annotate the contour and category of the top layer parcel on the RGB image to obtain the annotation information of each parcel for training the model.

[0077] For NC parts in the RGB image (in the express industry, NC parts refer to special-shaped parts, which can be parts with oversized or particularly small sizes, or parts with special shapes that cannot be grasped by a mechanical hand, etc.), the embodiment of the present application provides the following two labeling methods: Method one, label the complete external contour and category of the NC part. This method is more accurate in contour labeling, but it is difficult to collect enough data due to the large number of NC part categories. Method two, do not label the external contour of the NC part, only label the background of the NC part, and then add the obtained background mask to the non-NC part package mask to obtain the remaining area as the mask of the NC part. This labeling process is relatively simple and can solve the problem of the diversity of NC parts, taking into account all types of NC parts, but it is not as accurate as method one.

[0078] Step 222, based on the YOLOv8 deep learning model, a preset instance segmentation model is constructed. Figure 2 A network structure diagram of the preset instance segmentation model constructed based on the YOLOv8 deep learning model.

[0079] Step 223, scale the size of each RGBD image in the training sample set to 640x640, and uniformly perform data augmentation methods such as mosaic, translation, rotation, mirroring, and normalization data processing methods. After the data processing of each RGBD image in the training sample set is completed, the RGB information in each RGBD image in the training sample set is separated and stored in another storage for visual inspection of the target instance segmentation model prediction effect.

[0080] Step 224, use each RGBD image in the training sample set after data processing as the input of the preset instance segmentation model, and modify the first layer convolution input channel of the preset instance segmentation model to 4, as shown in Figure 3 .

[0081] Step 225, to alleviate the sawtooth phenomenon of envelopes and cartons due to the bevel, which affects the accuracy of subsequent optimal package selection, two up-sampling 2 times can be used in the mask prototype branch used to generate the mask to predict the mask.

[0082] Specifically, after a convolution kernel of 3x3 and a step of 1, a feature map is obtained, and then a bilinear difference is used to up-sample the feature map. The above convolution and up-sampling are performed once to obtain a feature map up-sampled by 4 times, and then the feature map is mapped from 256 channels to 32 original masks, as shown in the following formula, and as shown in Figure 3 .

[0083] y = conv 3×3 (up(conv3×3 (up(conv 3×3 (x)))

[0084] wherein, conv 3×3 (·) includes a standard convolution with a convolution kernel of 3x3, and up(·) represents a bilinear interpolation up-sampling.

[0085] In step 226, a model loss function is obtained based on a linear combination of a classification loss, a bounding box regression loss, and a mask loss. The classification loss uses a cross-entropy loss function, the bounding box regression loss is obtained by combining a CIoU loss and a DFL (Distribution Focal Loss) loss, and the mask loss uses a binary cross-entropy loss function. The specific expression of the model loss function is as follows:

[0086] loss=λ cls L cls +λ box L CIoU +λ dfl L dfl +λ mask L mask

[0087] wherein, λ cls =0.5, λ box =7.5, λ dfl =1.5, λ mask =λ box / batch_size, and batch_size is a pre-set number of training batches.

[0088] In step 227, the above training sample set, pre-set instance segmentation model, and model loss function are trained using automatic mixed precision and an initial learning rate of 0.01 for 100 epochs (pre-set training rounds) to obtain a target instance segmentation model.

[0089] Optionally, the target instance segmentation model can also classify the package categories in the target RGBD image. Specifically, by segmenting the target RGBD image, each package present in the target RGBD image is obtained, including:

[0090] By segmenting the RGBD image of at least one package using the target instance segmentation model, the mask area of each package on the package table and the category information indicating the physical attributes of each package are obtained.

[0091] wherein, the process of segmenting the RGBD image of at least one package using the target instance segmentation model to obtain the mask area of each package on the package table and the category information can include:

[0092] Step 1, load the trained target instance segmentation model to the computer processor.

[0093] Step 2, obtain the RGB and 3D point cloud data corresponding to the current parcel table, and construct the RGBD image corresponding to the current parcel table based on the RGB and 3D point cloud data corresponding to the current parcel table.

[0094] Step 3, input the RGBD image corresponding to the current parcel table into the target instance segmentation model to output the mask region and category information of the top layer parcel in the current parcel table. Figure 4 In order to input the RGBD image corresponding to the current parcel table into the target instance segmentation model, the effect diagram of the obtained mask region of the top layer parcel in the current parcel table is output.

[0095] Step 4, delete the mask region and corresponding category information with low category confidence and large intersection over union by using the non-maximum value algorithm to remove some overlapping error information.

[0096] Step 5, obtain the mask region and category information predicted by the ROI model, and shield the mask region and category information outside the ROI region.

[0097] Step 230, according to the three-dimensional point cloud data corresponding to each parcel in the target RGBD image, determine the parcel meeting the optimal grabbing condition from each parcel as the target parcel; the optimal grabbing condition includes: the physical attributes possessed match the preset physical attribute requirement, and the spatial position represented by the corresponding three-dimensional point cloud data meets the preset spatial position requirement.

[0098] Wherein, the physical attributes possessed by the parcel can be represented by the category information. The spatial position represented by the corresponding three-dimensional point cloud data is used to indicate the size information of the parcel.

[0099] Wherein, the three-dimensional point cloud data corresponding to each parcel in the target RGBD image is the three-dimensional point cloud data of all parcels contained on the current parcel table. In order to determine the height information of each parcel on the parcel table, the three-dimensional point cloud data corresponding to the mask region of each parcel on the parcel table can be obtained from the three-dimensional point cloud data corresponding to the parcel table based on the mask region of each parcel on the current parcel table, that is, the three-dimensional point cloud data of each parcel on the parcel table is determined.

[0100] The optimal grabbing condition is used to indicate a condition of a package most easily grabbed on the current package table. As an example, to avoid the determined target package from touching other packages on the package table when grabbed, the optimal grabbing condition can be a package with the highest average height of the upper surface of the package from the table surface of the package table. Specifically, according to the three-dimensional point cloud data corresponding to each package in the target RGBD image, a package meeting the optimal grabbing condition is determined from each package as a target package, including:

[0101] determining an average height of each package based on the three-dimensional point cloud data corresponding to each package in the target RGBD image;

[0102] determining a package with the highest average height from each package based on the average height of each package;

[0103] determining a size of the package with the highest average height based on the point cloud data of the package with the highest average height;

[0104] when the physical attribute indicated by the category information of the package with the highest average height matches the preset physical attribute requirement, and the size of the package with the highest average height meets the preset spatial position requirement, the package with the highest average height is determined as the target package to be grabbed.

[0105] Optionally, to improve the accuracy of determining the size of the package with the highest average height and avoid interference from surrounding other packages when calculating the size of the package with the highest average height, the embodiments of the present application can remove impurity pixels in the mask area of the package with the highest average height by an erosion operator. Specifically, determining the size of the package with the highest average height based on the point cloud data of the package with the highest average height, including:

[0106] eroding impurity pixels in the mask area of the package with the highest average height that are different from other mask areas based on the erosion operator;

[0107] determining point cloud data corresponding to the eroded mask area from the point cloud data corresponding to at least one package;

[0108] determining the size of the package with the highest average height based on the point cloud data corresponding to the eroded mask area.

[0109] As an example, after determining the package with the highest average height, it can be determined according to the category of the package whether the package is a target package. If the target instance segmentation model predicts that the category of the package with the highest average height is a box, the package is taken as a target package. If the target instance segmentation model predicts that the category of the package with the highest average height is an envelope, it can be determined by the method of rectangularity whether the rectangularity of the envelope is higher than a preset threshold. If it is higher than the preset threshold, it indicates that the envelope does not exist in the case of being occluded, and then the package is taken as a target package. If the rectangularity of the current envelope is less than the threshold, it indicates that the envelope exists in the case of being occluded. If the envelope is determined as a target package, the other packages occluding the envelope will fall in the process of package grabbing. In this case, the target package needs to be determined again.

[0110] The rectangularity can reflect the fullness of the package in the image to its circumscribed rectangle. The higher the rectangularity, the smaller the area of the package occluded by other packages, and the greater the success rate of successfully grabbing the package and avoiding the falling of other packages. Figure 5 The flowchart for determining the target package based on the category information of the package is shown in FIG. 5. Figure 5 The flowchart can include: s51, determining the category of the package with the highest average height; s52, determining whether the package is a carton or a soft package; if the package is a carton or a soft package, s55 is executed, otherwise s53 is executed; s53, determining whether the package is an envelope; if the package is an envelope, s54 is executed, otherwise s55 is executed; s54, calculating the rectangularity of the package; s55, determining whether the rectangularity is greater than a preset threshold; if the rectangularity is greater than the preset threshold, s56 is executed, otherwise s57 is executed. s56, determining the package with the highest average height as a target package. s57, eliminating the information of the package with the highest average height.

[0111] After the target package is determined, the stray pixel points around the mask area of the target package can be eroded by an erosion operator, so that the size of the mask area of the target package is closer to the actual outer contour of the target package, thereby avoiding the problem that other packages on the package supply table are touched and fall when the target package is grabbed. Further, based on the point cloud data corresponding to the eroded mask area, the size of the target package is determined. If the size is within a preset range, the three-dimensional point cloud information of the target package can be sent to a mechanical hand for grabbing the package. The purpose of determining whether the size of the target package is within the preset range is to avoid the problem that the size of the target package exceeds the grabbable range of the mechanical hand for grabbing the package, resulting in the failure of grabbing the target package.

[0112] Step 240, grabbing the target package.

[0113] Optionally, to facilitate the mechanical hand for grabbing the package to accurately grab the target package, embodiments of the present application can construct a corresponding three-dimensional coordinate system based on the three-dimensional point cloud data of the target package, and determine the spatial coordinates of the grabbing point of the target package in the three-dimensional coordinate system. Specifically, the target package is grabbed, including:

[0114] Based on the three-dimensional point cloud data of the target package, the maximum fitting plane and the normal vector of the target package are determined.

[0115] Based on the maximum fitting plane and the normal vector of the target package, the spatial coordinates of the grabbing point of the target package are determined.

[0116] The spatial coordinates of the grabbing point of the target package are sent to the mechanical hand for grabbing the package, so that the mechanical hand grabs the target package based on the spatial coordinates of the grabbing point of the target package.

[0117] Wherein, based on the maximum fitting plane and the normal vector of the target package, the spatial coordinates of the grabbing point of the target package are determined, which can include:

[0118] The spatial coordinates of the center point of the maximum fitting plane of the target package are determined.

[0119] Based on the spatial coordinates of the center point of the maximum fitting plane of the target package and the normal vector, the spatial coordinates of the grabbing point of the target package are determined.

[0120] As an example, the target package is grabbed by the mechanical hand for grabbing the package based on the three-dimensional point cloud data of the target package, which can include:

[0121] Step 241, based on the three-dimensional point cloud data of the target package, the maximum fitting plane and its normal vector of the target package are determined by Ransac algorithm.

[0122] Wherein, the full name of Ransac algorithm is Random Sample Consensus, that is, random sample consensus algorithm, which is an iterative method for estimating the parameters of a mathematical model from a data set containing abnormal data (noise or outliers). The basic assumption of Ransac algorithm is that the data set contains both "inliers" that can be correctly described by the model and "outliers" that deviate from the normal range. The core idea of Ransac algorithm is to estimate the model parameters by randomly sampling a small part of the data, and then test all the data with this model. Among them, the data points that meet the model are regarded as inliers, and the data points that do not meet the model are regarded as outliers. Through iterative random sampling and estimation of model parameters, the model with the most inliers is finally obtained as the final estimation result.

[0123] Determining the maximum fitting plane of the target package and the normal vector thereof based on the three-dimensional point cloud data of the target package through the Ransac algorithm can generally include the following steps:

[0124] (1) Randomly selecting n spatial coordinates in the three-dimensional point cloud data of the target package (n is the minimum number of samples required for initializing the fitting plane), and fitting an initial plane with the samples.

[0125] (2) Using other spatial coordinates in the three-dimensional point cloud data of the target package, adding the spatial coordinates of the coordinate points falling in the initial plane into the inlier set.

[0126] (3) If the size of the inlier set exceeds a certain threshold, it is considered that the parameters of the correct fitting plane are obtained, and a new plane is re-fitted using the least squares method or the like with the inlier set.

[0127] (4) Repeating the above process until a certain number of sampling times are completed. If not enough inliers are found, the algorithm fails; otherwise, the plane formed by the largest inlier set obtained after sampling is selected as the final maximum fitting plane of the target package, and the normal vector of the maximum fitting plane is determined.

[0128] Step 242, calculating the spatial coordinates of the center point of the maximum fitting plane of the target package.

[0129] Step 243, constructing the x and y axes of the spatial coordinate system of the grabbing point of the target package according to the spatial coordinates of the center point of the maximum fitting plane of the target package and the maximum fitting plane, wherein the normal vector can be used as the z axis of the constructed spatial coordinate system.

[0130] Step 244, sending the spatial coordinate system of the grabbing point of the target package to the robot arm.

[0131] Step 245, after the robot arm carries the target package to the feeding belt, sending an image acquisition signal to the image acquisition device and a feeding instruction to the conveying belt.

[0132] Step 246, the image acquisition device determines whether there is only one package on the current feeding table through the three-dimensional point cloud data on the feeding table in response to the image acquisition signal, and if there is only one package, sends a feeding instruction to the conveying belt after the robot arm finishes grabbing. If it is determined that there are at least two packages on the current feeding table, an image acquisition instruction is sent to the image acquisition device.

[0133] If the current output of the target instance segmentation model has and only has the mask region and class information of one package, it is necessary to determine whether the package is the last package on the feeding table. Figure 6A process diagram for determining whether the package on the current package supply table is the last package in the package grabbing method provided by the embodiments of the present application. Specifically, it comprises: s61, sending an image acquisition signal to the image acquisition device; s62, the image acquisition device acquires the RGB image and the three-dimensional point cloud data on the current package supply table; s63, constructing the corresponding RGBD image of the current package supply table based on the RGB image and the three-dimensional point cloud data on the current package supply table; s64, inputting the RGBD image into the target instance segmentation model to output the mask area and the category of the package on the current package supply table; s65, determining whether the number of mask areas of the package on the current package supply table is 1; s66, determining whether the package on the current package supply table is the last package; specifically, a plurality of ROI internal images can be acquired when the package supply table is without a package, and the average value of the gray values of the plurality of ROI internal images is calculated; then, according to the mask area of the package on the current package supply table, the pixel of the package area in the RGB image on the current package supply table is filled with the average value of the gray values of the plurality of ROI internal images; finally, the average value of the gray values of the plurality of ROI internal images in the filled RGB image is calculated, and the average value of the gray values of the plurality of ROI internal images in the filled RGB image is compared with the average value of the gray values in the no-package state. If the difference between the two is within the preset threshold range, it is considered that the package on the current package supply table is the last package, and s67 is executed to grab the current package by the mechanical hand, and after the mechanical hand completes the grabbing, s68 is executed to send a feeding instruction to the conveyor belt electrical controller. If the difference between the two is not within the preset threshold range, s67 is executed to continue to execute s61 to send an image acquisition instruction to the image acquisition device after the current package is grabbed.

[0134] Step S247, if the target instance segmentation model only identifies the NC part, the package supply table can be rolled to scatter the packages on the package supply table, and after the rolling is completed, a collection signal is sent to the image acquisition device. If a package that can be grabbed is found, steps 110-140 are executed to grab the package, otherwise, if the NC part is still not found after rolling n times, a command for rejecting the NC part information is sent to the conveyor belt, and an image acquisition signal is sent to the image acquisition device.

[0135] By using the package grabbing method provided by the embodiments of the present application, when the packages on the package supply table are grabbed, the target RGBD image corresponding to the package supply table can be acquired, the target RGBD image contains three-dimensional point cloud data, and by segmenting the target RGBD image, each package existing in the target RGBD image is obtained. Finally, based on the three-dimensional point cloud data corresponding to each package in the target RGBD image, the package that meets the optimal package grabbing condition is determined as the target package to be grabbed, and the target package is grabbed. In this process, manual participation is not required to determine the target package to be grabbed, and the package grabbing efficiency is effectively improved.

[0136] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can also be executed by different devices as the execution subject. For example, the execution subject of steps 110 to 140 can be device A; for example, the execution subject of steps 210 to 220 can be device A, and the execution subject of steps 230 to 240 can be device B; and the like.

[0137] In addition, in some of the processes described in the above embodiments and the accompanying drawings, a plurality of operations appearing in a specific order are included, but it should be clearly understood that these operations can be executed or executed in parallel without the order in which they appear in this text. The serial numbers of the operations such as 210, 220, etc. are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes can include more or fewer operations, and the operations can be executed in sequence or in parallel.

[0138] Specifically, the implementation flowchart of the model training method provided by one or more embodiments of the present application is shown in Figure 7 as shown, comprising:

[0139] Step 710, obtaining a plurality of RGB images on the supply table and a plurality of RGB image corresponding three-dimensional point cloud data.

[0140] Specifically, the plurality of RGB images on the supply table and the plurality of RGB image corresponding three-dimensional point cloud data can be obtained by an image acquisition device arranged above the supply table from a scene in which one or more supply tables are stacked with different package forms.

[0141] Step 720, based on the plurality of RGB images and the plurality of RGB image corresponding three-dimensional point cloud data, constructing a plurality of RGBD images corresponding to the plurality of RGB images, and annotation information indicating the external contour and category of the package to be grabbed corresponding to each RGBD image.

[0142] Based on the plurality of RGB images and the corresponding three-dimensional point cloud information, a plurality of RGBD images are constructed. Specifically, for each RGB image and corresponding three-dimensional point cloud data, the three-dimensional point cloud data can be mapped to a matrix with the same width and height as the corresponding RGB image to obtain the corresponding depth map. For each depth map, only the depth map within the pre-set package grabbing range (i.e. within the ROI) can be taken, and the depth values outside the ROI can be set to 0. At the same time, for each depth map, the distance between the image acquisition device and the supply table can be subtracted from the current depth value of each package in the depth map to obtain the distance between each package in the depth map and the supply table, and finally the depth value of each package in the depth map is updated based on the distance between each package in the depth map and the supply table.

[0143] The depth values in the updated depth map are normalized to 0-255, and the normalized depth values in the depth map are stored in the fourth channel of the corresponding RGB image, which is initially empty. Finally, the four-channel image is saved as a png format, obtaining the corresponding RGBD image. Since the fourth channel of the png format saved by the RGBD image will cause the RGBD image to appear transparent, the image labeling tool labelme can be used to label the outline and category of the top layer package on the RGB image to obtain the labeling information of each package for training the model.

[0144] Among them, for the NC pieces (in the express industry, NC pieces refer to special-shaped pieces, which can be pieces with oversized or particularly small sizes, or pieces with special shapes that cannot be grasped by a mechanical hand, etc.) in the RGB image, the embodiments of the present application provide the following two labeling methods: method one, labeling the complete external contour and category of the NC piece, this method is more accurate in contour labeling, but since there are many types of NC pieces, it is difficult to collect enough data. Method two, do not label the external contour of the NC piece, only label the background of the NC piece, and the remaining area obtained by adding the obtained background mask and the non-NC piece package mask is used as the mask of the NC piece. This labeling process is relatively simple and can solve the problem of diversity of NC pieces, taking into account all types of NC pieces, but it is not as accurate as method one.

[0145] Step 730, training the preset instance segmentation model based on the RGBD images corresponding to the plurality of RGB images and the corresponding labeling information, to obtain a target instance segmentation model.

[0146] Among them, the first layer convolution input channel of the preset instance segmentation model is four channels.

[0147] Specifically, training the preset instance segmentation model based on the RGBD images corresponding to the plurality of RGB images and the corresponding labeling information, to obtain a target instance segmentation model, includes:

[0148] Step 731, constructing a preset instance segmentation model based on the YOLOv8 deep learning model. Figure 2 The network structure diagram of the preset instance segmentation model constructed based on the YOLOv8 deep learning model.

[0149] Step 732, scale the size of each RGBD image in the training sample set to 640x640, and uniformly perform data augmentation methods such as mosaic, translation, rotation, mirroring, and data processing methods such as normalization. After the data processing of each RGBD image in the training sample set is completed, the RGB information in each RGBD image in the training sample set is separated out and stored in another storage for visualizing the prediction effect of the target instance segmentation model.

[0150] Step 733, use each RGBD image in the training sample set after data processing as the input of the preset instance segmentation model, and modify the first layer convolution input channel of the preset instance segmentation model to 4, as shown in the following formula: Figure 2

[0151] Step 734, in order to alleviate the sawtooth phenomenon of envelopes and cartons due to the bevel, which affects the accuracy of subsequent optimal package selection, two 2x upsampling can be used in the branch mask prototype used to generate the mask to predict the mask.

[0152] Specifically, after a convolution kernel of 3x3 and a step of 1, a feature map is obtained, and then a bilinear difference value is used to upsample the feature map. The above convolution and upsample are executed once to obtain a feature map upsampled by 4, and then the feature map is mapped from 256 channels to 32 original masks, as shown in the following formula, and as shown in the following formula: Figure 2

[0153] y=conv 3×3 (up(conv 3×3 (up(conv 3×3 (x)))

[0154] wherein, conv 3×3 (·) includes a convolution kernel of 3x3 standard convolution, and up(·) represents bilinear difference upsample.

[0155] Step 735, based on the linear combination of classification loss, bounding box regression loss, and mask loss, the model loss function is obtained. Among them, the classification loss uses the cross-entropy loss function, the bounding box regression loss is obtained by combining the CioU loss and the DFL (Distribution Focal Loss, loss function related to the bounding box regression) loss, and the mask loss uses the binary cross-entropy loss function. The specific expression of the model loss function is as follows:

[0156] loss=λ cls L cls +λ box L CIoU +λ​​dfl L dfl +λ mask L mask

[0157] wherein, λ cls = 0.5, λ box = 7.5, λ dfl = 1.5, λ mask = λ box / batch_size, batch_size is the number of pre-set training batches.

[0158] Step 736, using the above training sample set, pre-set instance segmentation model and model loss function to train 100 epochs (pre-set training rounds) with automatic mixed precision, initial learning rate 0.01 and other parameters to obtain the target instance segmentation model.

[0159] By using the model training method provided in the embodiments of the present application, the RGBD images corresponding to the plurality of RGB images and the label information indicating the external contour and category of the package to be grabbed corresponding to each RGBD image can be constructed based on the plurality of RGB images and the three-dimensional point cloud data corresponding to the plurality of RGB images obtained on the package supply table, and the pre-set instance segmentation model can be trained based on the RGBD images corresponding to the plurality of RGB images and the label information corresponding thereto to obtain the target instance segmentation model. On the one hand, since the sample data of the target instance segmentation model obtained by training contains not only the RGB images of the packages on the package supply table, but also the depth maps of the packages on the package supply table, the problem of inaccurate image segmentation caused by the stacking of packages with similar colors due to segmentation based only on RGB images is avoided, and the accuracy of package image segmentation on the package supply table is effectively improved. On the other hand, the sample data used to train the target instance segmentation model only increases one channel dimension based on the RGB image, without increasing the complexity of the sample data, thereby reducing the consumption of computing resources in the model training process.

[0160] Figure 8 A structural schematic diagram of a package grabbing device 800 provided by an exemplary embodiment of the present application is shown in FIG. 8. As shown in the figure, the device 800 includes an image construction module 810, an image segmentation module 820, a package determination module 830 and a package grabbing module 840, wherein: Figure 8

[0161] The image acquisition module 810 is configured to acquire a target RGBD image corresponding to the package supply table, wherein the target RGBD image contains three-dimensional point cloud data.

[0162] The image segmentation module 820 is configured to obtain each package existing in the target RGBD image by segmenting the target RGBD image.​

[0163] The package determination module 830 is configured to determine, according to the three-dimensional point cloud data corresponding to each package in the target RGBD image, a package meeting an optimal grabbing condition as a target package from the packages; the optimal grabbing condition includes that a physical attribute possessed by the package matches a preset physical attribute requirement, and a spatial position represented by the corresponding three-dimensional point cloud data meets a preset spatial position requirement.

[0164] The package grabbing module 840 is configured to grab the target package.

[0165] With the device provided in the embodiments of the present application, when the packages on the package supply table are grabbed, the target RGBD image corresponding to the package supply table can be obtained, the three-dimensional point cloud data is included in the target RGBD image, each package existing in the target RGBD image is obtained by segmenting the target RGBD image, and finally the package meeting the optimal package grabbing condition is determined as the target package to be grabbed according to the three-dimensional point cloud data corresponding to each package in the target RGBD image, and the target package is grabbed. In this process, the target package to be grabbed does not need to be determined manually, and the package grabbing efficiency is effectively improved.

[0166] In some exemplary embodiments, when the image acquisition module 810 acquires the target RGBD image corresponding to the package supply table, the image acquisition module 810 is specifically configured to:

[0167] map the acquired three-dimensional point cloud data corresponding to the package supply table to a matrix with the same width and height as the RGB image corresponding to the package supply table, to obtain a depth map corresponding to the package supply table;

[0168] update the depth map corresponding to the package supply table based on the first distance between the image acquisition device and the package supply table, wherein the image acquisition device is configured to acquire the RGB image and the three-dimensional point cloud data corresponding to the at least one package on the package supply table;

[0169] store the updated depth map corresponding to the package supply table in a fourth channel of the RGB image corresponding to the package supply table, to obtain the target RGBD image.

[0170] In some exemplary embodiments, when the image acquisition module 810 updates the depth map corresponding to the package supply table based on the first distance between the image acquisition device and the package supply table, the image acquisition module 810 is specifically configured to:

[0171] determine a difference between the first distance between the image acquisition device and the package supply table and the depth value of each package in the depth map corresponding to the at least one package;

[0172] The difference between the first distance and the depth value of each parcel in the depth map corresponding to the parcel supply table is taken as the depth value of each parcel in the depth map corresponding to the parcel supply table to update the depth map corresponding to the parcel supply table.

[0173] In some example embodiments, the image segmentation module 820, when segmenting each parcel present in the target RGBD image, is specifically configured to:

[0174] segmenting the at least one parcel RGBD image through the target instance segmentation model to obtain the mask area of each parcel on the parcel supply table and the category information indicating the physical attribute of each parcel.

[0175] In some example embodiments, the parcel determination module 830, when determining a parcel meeting the optimal grabbing condition from the parcels as the target parcel according to the three-dimensional point cloud data corresponding to the parcels in the target RGBD image, is specifically configured to:

[0176] determining the average height of the parcels based on the three-dimensional point cloud data corresponding to the parcels in the target RGBD image;

[0177] determining the parcel with the highest average height from the parcels based on the average height of the parcels;

[0178] determining the size of the parcel with the highest average height based on the point cloud data of the parcel with the highest average height;

[0179] when the physical attribute indicated by the category information of the parcel with the highest average height matches the preset physical attribute requirement and the size of the parcel with the highest average height meets the preset spatial position requirement, taking the parcel with the highest average height as the target parcel to be grabbed.

[0180] In some example embodiments, the parcel determination module 830, when determining the size of the parcel with the highest average height based on the point cloud data of the parcel with the highest average height, is specifically configured to:

[0181] eroding the impurity pixels in the mask area of the parcel with the highest average height that are different from other mask areas based on an erosion operator;

[0182] determining the point cloud data corresponding to the eroded mask area from the point cloud data corresponding to the at least one parcel;

[0183] determining the size of the parcel with the highest average height based on the point cloud data corresponding to the eroded mask area.

[0184] In some example embodiments, when the package grabbing module 940 grabs the target package, specifically for:

[0185] Based on the three-dimensional point cloud data of the target package, determine the maximum fitting surface and normal vector of the target package;

[0186] Based on the maximum fitting surface and normal vector of the target package, determine the spatial coordinates of the grabbing point of the target package;

[0187] Send the spatial coordinates of the grabbing point of the target package to the mechanical hand for grabbing the package, so that the target package is grabbed by the mechanical hand based on the spatial coordinates of the grabbing point of the target package.

[0188] The package grabbing device 800 can implement the method of the method embodiment of Figures 1 to 6 , and specifically can refer to the package grabbing method of the example embodiment shown in Figures 1 to 6 , and details are not repeated.

[0189] Figure 9 A structural schematic diagram of a model training device 900 provided for an example embodiment of the present application is shown in FIG. 9. As shown in FIG. 9, the device 900 includes a data acquisition module 910, an image construction module 920, and a model training module 930, wherein: Figure 9

[0190] The data acquisition module 910 is configured to acquire a plurality of RGB images on a package table and three-dimensional point cloud data corresponding to the plurality of RGB images.

[0191] The image construction module 920 is configured to construct an RGBD image corresponding to the plurality of RGB images based on the plurality of RGB images and the three-dimensional point cloud data corresponding to the plurality of RGB images, and to indicate annotation information of an external contour and a category of a package to be grabbed corresponding to each RGBD image.

[0192] The model training module 930 is configured to train a preset instance segmentation model based on the RGBD image corresponding to the plurality of RGB images and the annotation information corresponding thereto, to obtain the target instance segmentation model.

[0193] The model training device 900 can implement the method of the method embodiment of Figure 7 , and specifically can refer to the model training method of the example embodiment shown in Figure 7 , and details are not repeated.

[0194] Figure 10 A structural schematic diagram of an electronic device provided for an example embodiment of the present application is shown in FIG. 10. As shown in FIG. 10, the device includes a memory 101 and a processor 102. Figure 10

[0195] ​​Memory 101 is used to store computer programs and can be configured to store various other data to support operation on the computing device. Examples of this data include instructions for any application or method operating on the computing device, contact data, phone book data, messages, pictures, videos, etc.

[0196] Processor 102, coupled to memory 101, is used to execute a computer program in memory 101 for: acquiring a target RGBD image corresponding to the package delivery station; the target RGBD image contains three-dimensional point cloud data; segmenting the target RGBD image to obtain each package present in the target RGBD image; determining, based on the three-dimensional point cloud data corresponding to each package in the target RGBD image, a package meeting optimal grasping conditions as the target package; the optimal grasping conditions include: possessing physical attributes that match preset physical attribute requirements, and the spatial position represented by the corresponding three-dimensional point cloud data meeting preset spatial position requirements; grasping the target package; or,

[0197] Acquire multiple RGB images on the package feeding platform and the corresponding 3D point cloud data of the multiple RGB images; based on the multiple RGB images and the corresponding 3D point cloud data of the multiple RGB images, construct RGBD images corresponding to the multiple RGB images, and annotation information for indicating the external contour and category of the package to be grabbed corresponding to each RGBD image; based on the RGBD images corresponding to the multiple RGB images and their corresponding annotation information, train a preset instance segmentation model to obtain the target instance segmentation model.

[0198] When using the electronic device provided in this application embodiment to grab packages on the package feeding platform, it can acquire the target RGBD image corresponding to the package feeding platform. The target RGBD image contains three-dimensional point cloud data. By segmenting the target RGBD image, each package present in the target RGBD image is obtained. Finally, based on the three-dimensional point cloud data corresponding to each package in the target RGBD image, the package that meets the optimal package grabbing conditions is determined as the target package to be grabbed, and the target package is grabbed. In this process, no manual intervention is required to determine the target package to be grabbed, which effectively improves the package grabbing efficiency.

[0199] Furthermore, such as Figure 10 As shown, the electronic device also includes other components such as a communication component 103, a display 104, a power supply component 105, and an audio component 106. Figure 10 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 10 The components shown. Additionally, depending on the implementation of the traffic playback device, Figure 10The components in the dashed box are optional components, not mandatory components. For example, when the electronic device is implemented as a terminal device such as a smartphone, a tablet computer, or a desktop computer, the electronic device can include Figure 10 The components in the dashed box; when the electronic device is implemented as a server device such as a conventional server, a cloud server, a data center, or a server array, the electronic device can not include Figure 10 The components in the dashed box.

[0200] Correspondingly, the embodiments of the present application also provide a computer readable storage medium storing a computer program, and the computer program is executed by a processor to enable the processor to implement the steps in the above method embodiments.

[0201] Correspondingly, the embodiments of the present application also provide a computer program product, which includes computer programs / instructions that can implement the steps performed by the electronic device in the above design diagram processing method embodiments when the computer programs / instructions are executed. Optionally, the computer program product, in addition to executing the above test method of the application software or the steps in the method embodiments.

[0202] The communication component in the above Figure 10 The communication component in the above

[0203] The memory in the above Figure 10 The memory in the above

[0204] The memory in the above Figure 10The display in the above-described

[0205] The above-described Figure 10 The power component in the above-described

[0206] The above-described Figure 1 The audio component in the above-described

[0207] Those skilled in the art will understand that embodiments of the present application can be provided as methods, systems, or computer program products. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) embodying computer readable program code.

[0208] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0209] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0210] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0211] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks ​ means for functionally implementing the steps listed in the flowchart block or blocks.

[0212] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0213] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory, non-volatile memory, etc. in the form of computer-readable media. The memory is an example of computer-readable media.

[0214] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0215] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not only include those elements, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0216] The above only describes the embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A package grasping method characterized by, The method comprises: obtaining a target RGBD image corresponding to a parcel delivery table; the target RGBD image contains three-dimensional point cloud data, and the target RGBD image is obtained by filling the three-dimensional point cloud data as a depth map into a new channel dimension of an RGB image corresponding to the parcel delivery table; performing segmentation on the target RGBD image through a target instance segmentation model to obtain a mask area of each parcel existing in the target RGBD image and category information for indicating a physical attribute of each parcel, the target instance segmentation model being trained by a training sample set composed of RGBD images constructed based on a plurality of RGB images corresponding to parcel delivery tables and three-dimensional point cloud information, and annotation information for indicating a category and an outline of a top layer parcel in each RGBD image; determining a parcel that meets an optimal grabbing condition from the parcels as a target parcel according to three-dimensional point cloud data and a mask area corresponding to the parcels in the target RGBD image, the optimal grabbing condition including a parcel with the highest average height of an upper surface of the parcel from a table surface of the parcel delivery table; grabbing the target parcel.

2. The method of claim 1, wherein, The method comprises: mapping three-dimensional point cloud data corresponding to a parcel delivery table obtained to a matrix with a width and a height consistent with an RGB image corresponding to the parcel delivery table to obtain a depth map corresponding to the parcel delivery table; updating the depth map corresponding to the parcel delivery table based on a first distance of an image acquisition device from the parcel delivery table, wherein the image acquisition device is used to acquire an RGB image and three-dimensional point cloud data corresponding to at least one parcel on the parcel delivery table; storing the updated depth map corresponding to the parcel delivery table to a fourth channel of the RGB image corresponding to the parcel delivery table to obtain the target RGBD image.

3. The method of claim 2, wherein, The method comprises: determining the first distance of the image acquisition device from the parcel delivery table and a depth value of each parcel in the depth map corresponding to the at least one parcel; taking a difference between the first distance and the depth value of each parcel in the depth map corresponding to the parcel delivery table as the depth value of each parcel in the depth map corresponding to the parcel delivery table to update the depth map corresponding to the parcel delivery table.

4. The method of claim 1, wherein, The method comprises: determining an average height of each parcel based on three-dimensional point cloud data corresponding to the parcels in the target RGBD image; determining a parcel with the highest average height from the parcels based on the average heights of the parcels; determining a size of the parcel with the highest average height based on point cloud data of the parcel with the highest average height; when a physical attribute indicated by category information of the parcel with the highest average height matches a preset physical attribute requirement and the size of the parcel with the highest average height meets a preset spatial position requirement, taking the parcel with the highest average height as the target parcel to be grabbed.

5. The method of claim 4, wherein, determining the size of the average height highest package based on the point cloud data of the average height highest package, comprising: eroding impurity pixels in the mask area of the average height highest package based on an erosion operator, which are different from other mask areas; determining the point cloud data corresponding to the eroded mask area from the point cloud data corresponding to at least one package; determining the size of the average height highest package based on the point cloud data corresponding to the eroded mask area.

6. The method of claim 1, wherein, the target package comprises: determining the maximum fitting surface and normal vector of the target package based on the three-dimensional point cloud data of the target package; determining the spatial coordinates of the grabbing point of the target package based on the maximum fitting surface and normal vector of the target package; sending the spatial coordinates of the grabbing point of the target package to a mechanical hand for grabbing packages, so that the target package is grabbed by the mechanical hand based on the spatial coordinates of the grabbing point of the target package.

7. A model training method, comprising: The model is the target instance segmentation model in claim 1, and the method comprises: obtaining a plurality of RGB images on a package table and three-dimensional point cloud data corresponding to the plurality of RGB images; based on the plurality of RGB images and the three-dimensional point cloud data corresponding to the plurality of RGB images, constructing RGBD images corresponding to the plurality of RGB images, and annotation information for indicating the external contour and category of each package to be grabbed corresponding to the RGBD images; training a preset instance segmentation model based on the RGBD images corresponding to the plurality of RGB images and the corresponding annotation information, to obtain the target instance segmentation model.

8. A package gripping device, characterized by comprise: an image acquisition module for acquiring a target RGBD image corresponding to a package table; the target RGBD image contains three-dimensional point cloud data, and the target RGBD image is obtained by filling the three-dimensional point cloud data as a depth map into a new channel dimension of an RGB image corresponding to the package table; an image segmentation module for segmenting the target RGBD image by a target instance segmentation model to obtain the mask area of each package existing in the target RGBD image and category information for indicating the physical properties of each package, wherein the target instance segmentation model is trained by a training sample set composed of RGBD images constructed based on a plurality of RGB images and three-dimensional point cloud information corresponding to the package table, and annotation information for indicating the category and contour of the top package in each RGBD image; a package determination module for determining a package meeting optimal grabbing conditions as a target package from the packages according to the three-dimensional point cloud data and mask area corresponding to the packages in the target RGBD image, wherein the optimal grabbing conditions include a package with the highest average height of the upper surface of the package from the table surface of the package table; a package grabbing module for grabbing the target package.

9. A model training apparatus characterized by comprising: The model is the target instance segmentation model in claim 8, and the model training device comprises: a data acquisition module for acquiring a plurality of RGB images on a package table and three-dimensional point cloud data corresponding to the plurality of RGB images; An image construction module is configured to construct an RGBD image corresponding to each of the plurality of RGB images based on the plurality of RGB images and the three-dimensional point cloud data corresponding to the plurality of RGB images, and to indicate the label information of the external contour and the category of the package to be grabbed corresponding to each of the RGBD images. A model training module is configured to train a preset instance segmentation model based on the RGBD images corresponding to the plurality of RGB images and the label information corresponding thereto, to obtain the target instance segmentation model.

10. An electronic device, comprising: The method comprises the following steps: a memory and a processor; the memory is configured to store a computer program; the processor is coupled to the memory and is configured to execute the computer program, so as to: obtain a target RGBD image corresponding to a package table; the target RGBD image comprises three-dimensional point cloud data, and the target RGBD image is obtained by filling the three-dimensional point cloud data as a depth map into a new channel dimension of an RGB image corresponding to the package table; perform segmentation on the target RGBD image by using a target instance segmentation model to obtain each package existing in the target RGBD image, the target instance segmentation model is trained based on an RGBD image constructed based on a plurality of RGB images corresponding to a plurality of package tables and three-dimensional point cloud information, and a training sample set composed of label information indicating the category and contour of a top layer package in each RGBD image; determine a package meeting an optimal grabbing condition from the packages as a target package according to the three-dimensional point cloud data corresponding to the packages in the target RGBD image; the optimal grabbing condition comprises: a package with the highest average height of the upper surface of the package from the table surface of the package table; grab the target package; or obtain a plurality of RGB images on a package table and three-dimensional point cloud data corresponding to the plurality of RGB images; construct an RGBD image corresponding to each of the plurality of RGB images based on the plurality of RGB images and the three-dimensional point cloud data corresponding to the plurality of RGB images, and indicate the label information of the external contour and the category of the package to be grabbed corresponding to each of the RGBD images; train a preset instance segmentation model based on the RGBD images corresponding to the plurality of RGB images and the label information corresponding thereto, to obtain the target instance segmentation model.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps in the method of any one of claims 1 to 7.

12. A computer program product, characterised in that, The computer program is executed by the processor to implement the steps in the method of any one of claims 1 to 7.

Images