Method and device for determining parcel grabbing, parcel grabbing device and mechanical arm

Abstract

The invention discloses a method and device for determining parcel grabbing, a device for grabbing parcels and a mechanical arm, and aims to solve the problem of how to reasonably determine which parcel / parcels should be grabbed preferentially by a mechanical arm so as to prevent the parcels from sliding off and being damaged as far as possible. The method for determining parcel grabbing comprises the following steps: acquiring an image containing a parcel to be grabbed; segmenting a first mask image which is partially shielded and wrapped from the image; segmenting a second mask image of each visible parcel except the partially shielded parcel from the image; according to the first mask image and the second mask image, determining the occlusion relationship between the partially occluded parcels and the visible parcels in the image; and on the basis of the shielding relationship, determining the parcels which are grabbed preferentially from the parcels to be grabbed. The preferentially grabbed parcels determined on the basis of the shielding relation are parcels which are located on the upper layer and are not shielded, and therefore the problem that the parcels slide down and are damaged due to the fact that the shielded parcels are preferentially grabbed is solved.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a method and device for determining to grab a package, and a device and a robotic arm for grabbing a package. Background Art

[0002] Using a robot to grab parcels on the logistics production line is a common parcel sorting method in the logistics industry.

[0003] In actual parcel grabbing scenarios, parcels on the logistics production line are often stacked together, leaving some parcels partially exposed and others obscured by other parcels. In this case, if the robot grabs the obscured parcel as the target parcel, other parcels obstructing the target parcel may slip and be damaged.

[0004] In this scenario, how to reasonably determine which package(s) the robot should grab first to avoid the packages from slipping and getting damaged is a problem that needs to be solved urgently in the existing technology. Summary of the invention

[0005] The embodiments of the present application provide a method and device for determining to grab a package, and a device and a robotic arm for grabbing a package, so as to solve the problem existing in the prior art of how to reasonably determine which package(s) the robotic arm should grab first, so as to avoid the package from slipping and being damaged as much as possible.

[0006] The present application embodiment adopts the following technical solutions: A method for determining a grab package, comprising: Acquire an image containing a package to be grasped; Segmenting a first mask image partially occluded and wrapped from the image; the first mask image includes: a mask image region corresponding to a visible portion partially occluded and wrapped in the image, and a mask image region inferred from an invisible portion occluded and wrapped in the image; Segmenting a second mask image of each visible package except the partially blocked package from the image; the second mask image includes a mask image area corresponding to a visible portion of each visible package in the image; Determining, according to the first mask image and the second mask image, an occlusion relationship between the partially occluded package and each visible package in the image; Based on the occlusion relationship, a package to be grasped with priority is determined from the packages to be grasped.

[0007] A device for determining to grab a package, comprising: An image acquisition unit, used to acquire an image containing a package to be grasped; A first image segmentation unit is used to segment a first mask image that is partially occluded and wrapped from the image; the first mask image includes: a mask image area corresponding to a visible part that is partially occluded and wrapped in the image, and a mask image area inferred for an invisible part that is occluded and wrapped in the image; A second image segmentation unit is configured to segment a second mask image of each visible package except the partially blocked package from the image; the second mask image includes a mask image region corresponding to a visible portion of each visible package in the image; an occlusion relationship determining unit, which determines an occlusion relationship between the partially occluded package and each visible package in the image according to the first mask image and the second mask image; The grab package determination unit determines a package to be grabbed with priority from the packages to be grabbed based on the occlusion relationship.

[0008] A robotic arm for grabbing a package, the robotic arm determines an optimal moving path of the robotic arm based on the determined occlusion relationship and the position of the package to be grabbed preferentially.

[0009] A device for grabbing a package, the device for grabbing a package comprising a mechanical arm, a communication module and a processor; A processor, configured to execute the steps of the above method to obtain the occlusion relationship and the position of the package to be grabbed first; A communication module, used for sending the occlusion relationship obtained by the processor and the location of the package to be grabbed preferentially to the robotic arm; The robotic arm determines an optimal moving path of the robotic arm based on the received occlusion relationship and the location of the package to be grasped preferentially.

[0010] A computing device, comprising: a memory and a processor, wherein: The memory is used to store computer programs; The processor is coupled to the memory and is used to execute the computer program stored in the memory to perform the above method.

[0011] A computer-readable storage medium storing a computer program, wherein the computer program can implement the above method when executed by a computer.

[0012] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: Since the first mask image includes the mask image area corresponding to the visible part of the partially obscured package and the mask image area inferred from the invisible part, and the second mask image includes the mask image area corresponding to each visible package except the partially obscured package, the occlusion relationship between each visible package can be determined based on the first mask image and the second mask image. The package to be grabbed first based on the occlusion relationship is the package on the upper layer that is not obscured, thereby avoiding the problem of the package slipping and being damaged due to the obscured package being grabbed first. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A specific implementation flow chart of a method for determining to grab a package provided in an embodiment of the present application; Figure 2 Schematic diagram of the dual model training process; Figure 3 Schematic diagram of the dual model prediction process; Figure 4 A flowchart of a specific implementation of the method for determining to grab a package based on a dual model provided in an embodiment of the present application; Figure 5 An example of an image obtained containing a package to be grabbed; Figure 6 Based on Figure 5 Schematic diagram of the occlusion relationship matrix determined by the image; Figure 7 A schematic diagram of a specific structure of a device for determining to grab a package provided in an embodiment of the present application; Figure 8 A schematic diagram of the structure of a device for grabbing a package provided in an embodiment of the present application; Fig. 9 A schematic diagram of the specific structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0015] Those skilled in the art may appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0016] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or device comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or devices.

[0017] In order to solve the problem in the prior art of how to reasonably determine which package(s) a robot should grab first so as to avoid the packages from slipping and being damaged as much as possible, an embodiment of the present application provides a method for determining which packages to grab.

[0018] The executor of the method may be any computing device that can implement the method, such as a server, a mobile phone, a personal computer, a smart wearable device, a smart robot, and the like.

[0019] In addition, the embodiments of the present application do not limit the execution order of different steps. When using the method provided in the embodiments of the present application, the execution order of different steps can be adjusted according to actual needs.

[0020] For ease of description, the method provided in the embodiment of the present application is described in detail below by taking a device for determining to grab a package as the execution subject of the method as an example.

[0021] like Figure 1 As shown, it is a specific implementation flow chart of a method for determining to grab a package provided in an embodiment of the present application, including the following steps 11 to 15: Step 11, obtaining an image containing the package to be grasped.

[0022] The packages to be grabbed refer to the packages that are on the logistics production line (or logistics conveyor belt) waiting to be grabbed; the image of the packages to be grabbed can be collected by a 2D camera placed above the logistics production line.

[0023] The device for grabbing the package obtains the image of the package waiting to be grabbed on the logistics production line captured by the 2D camera.

[0024] Step 12: Segment the first mask image that is partially blocked and wrapped from the image.

[0025] A partially occluded package refers to a visible package in an image (a package that is not completely occluded) whose part is blocked by other packages and the blocked part is not visible in the image.

[0026] In the field of image processing and computer vision, a mask image usually refers to a binary image or Boolean image of the same size as the original image, which is used to specify one or several target areas in the image, and usually marks the pixel value of the target area as 1 (or true), and the pixel value of other areas as 0 (or false). The target area of ​​the first mask image in this step is partially occluded.

[0027] The first mask image includes: a mask image region corresponding to the visible portion that is partially occluded and wrapped in the image, and a mask image region inferred for the invisible portion that is occluded and wrapped in the image. That is, in the first mask image, the pixel values ​​of the regions corresponding to the visible portion that is occluded and wrapped in the image and the inferred invisible portion are marked as 1, and the pixel values ​​of other regions in the image are marked as 0.

[0028] In a specific implementation, segmenting the first mask image partially blocked and wrapped from the image specifically includes: Based on a first instance segmentation model pre-trained using a sample image, a first mask image partially occluded and wrapped is segmented from the image.

[0029] The sample image used has a label, which is a mask image that marks the partially occluded and wrapped portion of the sample image. The mask image includes: a mask image region corresponding to the visible portion of the sample image that is partially occluded and wrapped in the sample image, and a mask image region corresponding to the invisible portion of the sample image that is partially occluded and wrapped in the sample image.

[0030] The first instance segmentation model can be a lightweight convolutional neural network model, wherein the lightweight of the model in this embodiment is achieved by reducing the number of convolutional layers in the neural network.

[0031] Training the lightweight convolutional neural network model to obtain a first instance segmentation model includes the following steps 121 to 122: Step 121: Build a dataset.

[0032] Use high-resolution industrial smart cameras (2D cameras) to capture 2D images of packages on the logistics production line. The captured images must have sufficient clarity and details to facilitate subsequent segmentation and recognition.

[0033] Use 2D data annotation software to annotate the objects in the image. The annotated images are used as sample images. The annotated data is converted into txt format, and the annotated sample images are divided into training sets and validation sets. During the annotation process, the accuracy of the mask corresponding to each package can be checked to improve the segmentation accuracy of the model.

[0034] The collected images can also be further processed, such as data enhancement, which can include cropping, rotating, folding, etc., to improve the model's ability to recognize packages with different postures.

[0035] Step 122, such as Figure 2 As shown, the model is trained.

[0036] The sample image processed in step 121 is used as the input of the instance segmentation model to be trained.

[0037] Performing a specified operation in each round of iterative training until the instance segmentation model to be trained converges, and using the instance segmentation model to be trained after convergence as the first instance segmentation model; The above specified operations include: Input a sample image to the instance segmentation model to be trained, determine the value of the loss function based on the instance segmentation result output by the instance segmentation model to be trained and the real instance segmentation result; adjust the parameters of the instance segmentation model to be trained based on the principle of minimizing the value of the loss function. The real instance segmentation result is the label of the sample, specifically including: the mask image that is partially occluded and wrapped in the sample image.

[0038] like Figure 2 In the training part of the first instance segmentation model: in each round of iterative training, a lightweight convolution layer is used to extract the features of the input sample image to obtain the intermediate layer feature map; then a feature map of the same size as the input sample image is obtained through a linear layer and an upsampling layer; the feature map and the label (GT, GroundTruth) of the input sample image are used to calculate the loss function, and the parameters of the instance segmentation model to be trained are adjusted based on the principle of minimizing the value of the loss function. The adjustment of the parameters can be specifically achieved by backpropagating the gradient obtained by the loss function calculation back into the network to update the weights, and finally obtaining the trained first instance segmentation model.

[0039] In the above loss calculation phase, the feature map extracted by the instance segmentation model to be trained is compared with the actual instance segmentation result to calculate the loss. The loss value is back-propagated to update the model weights, ensuring that the lightweight model can accurately restore the original state of the occluded target while maintaining high efficiency.

[0040] like Figure 3As shown, an image containing a package to be grabbed, which is collected in real time from a logistics production line, is input into the trained first instance segmentation model, so that the first instance segmentation model outputs a first mask image of a package that is partially obscured in the image. Figure 3 The package M in the input image is partially blocked by other packages. After being processed by the first instance segmentation model, a first mask image corresponding to the package M is obtained.

[0041] In step 12, a mask image containing a visible part and an occluded part that is partially occluded and wrapped in the image is segmented through a lightweight first instance segmentation model, thereby realizing the display of the mask image of the occluded part that is partially occluded and wrapped in the image, so that the original state of the partially occluded and wrapped part can be obtained based on the first instance segmentation model.

[0042] Step 13, segmenting from the image a second mask image of each visible package except the partially obscured package; the second mask image includes a mask image area corresponding to the visible portion of each visible package in the image.

[0043] In one implementation, segmenting the second mask image of each visible package except the partially blocked package from the image includes steps 131 and 132: Step 131 : segmenting the mask images of the visible packages from the image based on the pre-trained second instance segmentation model.

[0044] The labels of the sample images for training the second instance segmentation model are: mask images that annotate the visible parts of each visible package in the sample image.

[0045] The second instance segmentation model may also be a convolutional neural network model. The training process of the second instance segmentation model specifically includes the following steps 1311 to 1312: Step 1311: Build a data set.

[0046] Use a high-resolution industrial smart camera (2D camera) to capture 2D images of packages on the logistics production line. The captured images (original images) must have sufficient clarity and details to facilitate subsequent segmentation and recognition.

[0047] Use 2D data annotation software to annotate the objects in the image. The annotated images are used as sample images. The annotated data is converted into txt format, and the annotated sample images are divided into training sets and validation sets. During the annotation process, the accuracy of the mask corresponding to each package can be checked to improve the segmentation accuracy of the model.

[0048] The collected images can also be further processed, such as data enhancement, which can include cropping, rotating, folding, etc., to improve the model's ability to recognize packages with different postures.

[0049] The original image used in step 1311 may be the same or different from the original image used in step 121, and this application does not limit this. Step 1311 is different from step 121 in that different annotations are added to the image to form different sample image labels. In step 1311, the visible part of each visible package in the original image is annotated to form a mask image of the visible part of each visible package corresponding to the original image.

[0050] Step 1312, such as Figure 2 As shown, the model is trained.

[0051] The sample image processed in step 1311 is used as the input of the instance segmentation model to be trained.

[0052] Performing specified operations in each round of iterative training until the instance segmentation model to be trained converges, and using the instance segmentation model to be trained after convergence as the second instance segmentation model; The above specified operations include: Input a sample image to the instance segmentation model to be trained, determine the value of the loss function based on the instance segmentation result output by the instance segmentation model to be trained and the real instance segmentation result; adjust the parameters of the instance segmentation model to be trained based on the principle of minimizing the value of the loss function. The real instance segmentation result is the label of the sample, specifically including: a mask image of the visible part of each visible packet in the sample image.

[0053] like Figure 2 In the training part of the second instance segmentation model: in each round of iterative training, multiple convolutional layers are used to extract the features of the input sample image to obtain the intermediate layer feature map; then the linear layer and the upsampling layer are used to obtain the feature map of the same size as the input sample image; the loss function is calculated for the feature map and the label (GT) of the input sample image, and the parameters of the trained instance segmentation model are adjusted based on the principle of minimizing the value of the loss function.

[0054] based on Figure 2 For the input sample image shown in the figure, when training the second instance segmentation model, all visible packages of the input sample image are marked in the label, the background pixel values ​​in the label are processed to 0, and the pixel values ​​of other targets (the visible parts of the visible packages in the image) are processed to 1, and the input sample image is converted into a binary image. Figure 2If there are 3 targets in the dataset, a binary image containing 3 targets will be generated in the label. If the dataset contains 100 sample images, a binary image containing 100 targets will be generated.

[0055] like Figure 3 As shown, the image containing the package to be grabbed collected in real time from the logistics production line is input into the trained second instance segmentation model, so that the second instance segmentation model outputs a mask image of each visible package in the image. Figure 3 The visible packages in the input image include package M, package S1 and package S2. After being processed by the second instance segmentation model, a mask image corresponding to the visible parts of packages M, S1 and S2 is obtained.

[0056] The first instance segmentation model and the second instance segmentation model above both use automatic mixed precision technology during the training process to improve the efficiency of the training process. In the initial stage, the learning rate of the model can be set to 0.01, and the model weights are continuously optimized through 100 epochs (rounds, an epoch represents the cycle in which the forward propagation and backpropagation processes of the entire training set are traversed and an update is completed) training cycles to achieve optimal performance.

[0057] Step 132: Remove the mask image area corresponding to the partially blocked package in the image from the mask image of each visible package, and obtain a second mask image of each visible package other than the partially blocked package.

[0058] Depend on Figure 3 It can be seen that the mask image of each visible package obtained in step 131 includes both the mask image area of ​​the visible part of the partially obscured package (package M) and the mask image area of ​​the unobstructed packages (package S1, package S2); therefore, by removing the mask image area corresponding to the partially obscured package from the mask image of each visible package obtained, the mask image of the unobstructed package can be obtained, that is, the second mask image of each visible package other than the partially obscured package.

[0059] Step 14: Determine, based on the first mask image and the second mask image, an occlusion relationship between the partially occluded package and each visible package in the image.

[0060] Since the first mask image includes the mask image area corresponding to the original state that is partially blocked and wrapped (such as Figure 3 The second mask image contains the mask image area of ​​each visible package in the image except the partially blocked package (such as Figure 3If the mask image area included in the first mask image overlaps with the mask image area in the second mask image, it means that the visible package in the corresponding second mask image in the overlapping area obscures the partially obscured package in the first mask image. Figure 3 In the figure, the mask area of ​​package S1 in the second mask image overlaps with the mask area of ​​package M in the first mask image, indicating that package S1 obscures package M.

[0061] Step 15: Determine a package to be grasped with priority from the packages to be grasped based on the occlusion relationship.

[0062] According to the occlusion relationship determined in step 14, the unoccluded packages are determined as the packages to be grabbed first. Figure 3 In step 14, it is determined that package S1 blocks package M; package S2 does not block other packages, nor is it blocked by other packages; therefore, in Figure 3 In the image shown, package S1 and package S2 are determined as the packages to be grabbed first.

[0063] In a specific implementation, the method for determining to grab a package in this embodiment further includes the following step S16: Step 16: Generate an occlusion relationship matrix based on the occlusion relationship.

[0064] Each row and each column of the occlusion relationship matrix corresponds to each visible package in the image, and each element in the occlusion relationship matrix represents a representation value of the mutual occlusion relationship between two different packages in the visible packages.

[0065] The characterization value of the mutual occlusion relationship can be used to characterize the mutual occlusion relationship between two different packages by a preset value. In this embodiment, the preset value is set to 0, 1, and -1, where the characterization value of 0 indicates that there is no occlusion between the two packages; the characterization value of 1 indicates that the package corresponding to the row where the current element is located occludes the package corresponding to the column where it is located; the characterization value of -1 indicates that the package corresponding to the row where the current element is located is occluded by the package corresponding to the column where it is located.

[0066] The occlusion relationship matrix is ​​used to determine the grabbing order of each visible package in the image.

[0067] In a feasible implementation, the order of grabbing each visible package is determined based on the number of preset characterization values ​​in each row of the occlusion relationship matrix. The preset characterization value can be 1 or -1. Taking the preset characterization value of -1 as an example, the process of determining the order of grabbing each visible package in the image based on the occlusion relationship matrix includes the following steps 161 to 162: Step 161, based on the occlusion relationship matrix, check the values ​​of each row of elements in the matrix, and take the packages corresponding to the rows in which the values ​​of the entire row of elements in the matrix are not -1 as the packages to be grabbed in the first order. Because the values ​​of the entire row of elements are not -1, it means that the packages corresponding to the row are not occluded by other packages.

[0068] Then, in step 162, the subsequent packages to be grabbed are determined according to the number of elements in the entire row with a value of -1 in the matrix. A specific implementation method includes: checking the row in the matrix with the least number of elements in the entire row with a value of -1, if the packages corresponding to the column where the elements in the row with a value of -1 are located are in the first order of grabbing packages, then the row with the least number of elements in the entire row with a value of -1 is used as the second order of grabbing packages. Because the packages corresponding to the columns where the elements in the entire row with a value of -1 are located are in the first order of grabbing packages, it means that after the packages corresponding to the columns are grabbed, the packages corresponding to the current row become unobstructed packages, and when grabbing, there will be no situation where the packages slip and cause damage to the packages.

[0069] like Figures 4 to 6 FIG. 4 is a flow chart of a specific implementation method of this embodiment. In the figure, the method of grabbing a package includes steps 41 to 49: Step 41, obtaining an image containing the package to be captured. Figure 5 As shown, the acquired image contains the packages to be grabbed, and the packages visible in the image are package 1 to package 4 respectively.

[0070] Step 42 , input the image acquired in step 41 into the first instance segmentation model, so that the first instance segmentation model performs the operation of step 44 .

[0071] Step 43 , input the image acquired in step 42 into the second instance segmentation model, so that the second instance segmentation model performs the operation of step 45 .

[0072] Step 44: The first instance segmentation model outputs a first mask image wrapped by the occlusion. Figure 5 In the image shown, the package 4 is partially blocked, so the first mask image of the package 4 is output in this step.

[0073] Step 45: The second instance segmentation model outputs a mask image of each visible package. Figure 5 In the image shown, Package 1 to Package 4 are all visible packages, so in this step, a mask image containing the visible parts of each package 1 to Package 4 is output.

[0074] Step 46, further processing the mask image outputted in step 45, removing the mask image area that is partially blocked and wrapped in the mask image, and obtaining a second mask image of each visible wrapper other than the partially blocked wrapper. Figure 5In the image shown, package 4 is partially blocked, and the other packages are not blocked, so this step obtains a second mask image containing packages 1 to 3.

[0075] Step 47, based on the first mask image obtained in step 44 and the second mask image obtained in step 46, determine the occlusion relationship of each visible package in the image, and based on the occlusion relationship, determine the occlusion relationship matrix. Figure 5 In the image shown, package 4 is blocked by package 2, and package 1 and package 3 do not block other packages, nor are they blocked by other packages.

[0076] based on Figure 5 The image shown is determined as Figure 6 The occlusion relationship matrix diagram is shown in FIG. The occlusion relationship matrix is ​​represented in a table in the diagram, and the first row and the first column of the table represent the visible packages in the image. It can be seen from the figure that package 2 occludes package 4, and there is no occlusion relationship between the other packages.

[0077] Step 48, determining the coordinates of the center point of each visible package. The coordinates of the center point of each visible package can be determined by the first instance segmentation model and the second instance segmentation model.

[0078] The first instance segmentation model can be based on an existing instance segmentation model, such as the Mask R-CNN model, and is trained by using the mask image corresponding to the partially occluded package in the sample image as a label. The Mask R-CNN model is an instance segmentation model commonly used in existing computer vision tasks. The mask image it outputs includes the bounding box of each target area in the input image and the coordinates of the bounding box. Therefore, the first instance segmentation model based on this embodiment can determine the coordinates of the center point of the partially occluded package.

[0079] The second instance segmentation model can use the Mask R-CNN model, so it is possible to determine the coordinates of the center points of each visible package in the second mask image.

[0080] Step 49, based on the occlusion relationship matrix and the center point coordinates of each visible package, determine the center point coordinates of the package to be grasped.

[0081] Based on the above steps 41 to 49, the package to be captured is determined.

[0082] After determining the package to be grasped, this embodiment may further include: obtaining the occlusion relationship matrix; determining the optimal movement path of the robotic arm for grasping the package according to the occlusion relationship matrix and the position (center point coordinates) of each visible package in the image.

[0083] The optimal moving path can be determined by: determining the grabbing order of each visible package in the image based on the occlusion relationship matrix; determining the optimal moving path of the robotic arm based on the position of each visible package in the image and the determined grabbing order and taking the minimization of the moving path of the robotic arm as the principle.

[0084] In the method for determining the package to be grabbed in this embodiment, since the first mask image includes the mask image area corresponding to the visible part of the partially blocked package and the mask image area inferred from the invisible part, and the second mask image includes the mask image area corresponding to each visible package except the partially blocked package, the blocking relationship between the visible packages can be determined based on the first mask image and the second mask image. The package to be grabbed first based on the blocking relationship is the package on the upper layer that is not blocked, thereby avoiding the problem of the package slipping and being damaged due to the blocked package being grabbed first.

[0085] In order to solve the problem in the prior art of how to reasonably determine which package(s) the robot should grab first so as to avoid the packages from slipping and being damaged as much as possible, based on the same inventive concept as the above-mentioned embodiments of the present application, this embodiment provides a device for determining the packages to be grabbed.

[0086] The specific structural diagram of the device is shown in Figure 7 As shown, it includes the following functional units 71-75: The image acquisition unit 71 is used to acquire an image containing the package to be captured.

[0087] The first image segmentation unit 72 is used to segment a first mask image that is partially blocked and wrapped from the image.

[0088] The first mask image includes: a mask image region corresponding to a visible portion that is partially occluded and wrapped in the image, and a mask image region inferred from an invisible portion that is occluded and wrapped in the image.

[0089] The first image segmentation unit 72 is specifically used to segment a first mask image that is partially occluded and wrapped from the image based on a first instance segmentation model pre-trained using a sample image; the sample image has a label, and the label is the mask image that is partially occluded and wrapped in the sample image.

[0090] The first instance segmentation model is trained in the following way: Performing a specified operation in each round of iterative training until the instance segmentation model to be trained converges, and using the converged instance segmentation model to be trained as the first instance segmentation model; The specified operation includes: Inputting the sample image into the instance segmentation model to be trained, determining the value of the loss function based on the instance segmentation result output by the instance segmentation model to be trained and the actual instance segmentation result; adjusting the parameters of the instance segmentation model to be trained based on the principle of minimizing the value of the loss function; The actual instance segmentation result includes: a mask image that is partially occluded and wrapped in the sample image. The second image segmentation unit 73 segments a second mask image of each visible package except the partially blocked package from the image; the second mask image includes a mask image area corresponding to the visible part of each visible package in the image.

[0091] The second image segmentation unit 73 is specifically used for: Segmenting mask images of the visible packages from the image based on a pre-trained second instance segmentation model; From the mask images of the visible packages, the mask image regions corresponding to the partially blocked packages in the image are removed to obtain second mask images of the visible packages other than the partially blocked packages.

[0092] The occlusion relationship determining unit 74 determines the occlusion relationship between the partially occluded package and each visible package in the image according to the first mask image and the second mask image.

[0093] The grab package determining unit 75 determines a package to be grabbed with priority from the packages to be grabbed based on the occlusion relationship.

[0094] The package grabbing determination unit 75 is further used to determine the occlusion relationship matrix between the visible packages based on the occlusion relationship; and determine the grabbing order of the packages based on the occlusion relationship matrix and the positions of the visible packages in the image. In the device for determining the grabbing of a package of this embodiment, since the first mask image includes the mask image area corresponding to the visible part of the partially blocked package and the mask image area inferred from the invisible part, and the second mask image includes the mask image area corresponding to each visible package except the partially blocked package, the occlusion relationship between each visible package can be determined based on the first mask image and the second mask image. The package to be grabbed first based on the occlusion relationship is the package on the upper layer that is not blocked, thereby avoiding the problem of the package slipping and being damaged due to the blocked package being grabbed first.

[0095] On the basis of the above embodiments, this embodiment provides a robotic arm for grabbing packages. The robotic arm is installed around a logistics production line and is used to grab packages on the logistics production line.

[0096] The robotic arm of this embodiment determines the optimal movement path of the robotic arm based on the occlusion relationship or occlusion relationship matrix determined in the above embodiment and the location of the package to be grasped preferentially.

[0097] The optimal movement path of the robot arm is determined based on the principle of minimizing the movement distance of the robot arm.

[0098] On the basis of the above-mentioned embodiments, this embodiment provides a device for grabbing a package, which can be installed around a logistics production line to realize the whole process of grabbing a package on the logistics production line.

[0099] like Figure 8 As shown, the device for grabbing a package in this embodiment includes a robotic arm, a communication module and a processor; Processor 81, used for the steps of the above method embodiment to obtain the occlusion relationship or occlusion relationship matrix, and the location of the determined priority package; A communication module 82, configured to send the occlusion relationship or occlusion relationship matrix obtained by the processor and the location of the package to be grabbed preferentially to the robotic arm; The robotic arm 83 determines the optimal moving path of the robotic arm based on the received occlusion relationship and the location of the package to be grabbed preferentially.

[0100] The device for grabbing a package of this embodiment is based on the above-mentioned method for determining the grabbing package. Since the first mask image includes the mask image area corresponding to the visible part of the partially obscured package and the mask image area inferred from the invisible part, and the second mask image includes the mask image area corresponding to each visible package except the partially obscured package, the occlusion relationship between each visible package can be determined based on the first mask image and the second mask image. The package to be grabbed first based on the occlusion relationship is the package on the upper layer that is not obscured, thereby avoiding the problem of the package slipping and being damaged due to the priority grabbing of the obscured package.

[0101] Based on the same inventive concept as the aforementioned embodiments of the present application, an embodiment of the present application further provides a computing device.

[0102] like Fig. 9As shown, the computing device includes: a memory 91 and a processor 92. The memory 91 can be configured to store various other data to support operations on the electronic device. Examples of these data include instructions for any application or method operating on the electronic device. The memory 91 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk or an optical disk.

[0103] The processor 92 is coupled to the memory 91 and is used to execute the computer program stored in the memory 91 to execute the method for determining to grab a package described in the above embodiment.

[0104] When the processor 92 executes the computer program to execute the method for determining the package to be grabbed, since the first mask image includes the mask image area corresponding to the visible part of the partially blocked package and the mask image area inferred from the invisible part, and the second mask image includes the mask image area corresponding to each visible package except the partially blocked package, the blocking relationship between the visible packages can be determined based on the first mask image and the second mask image. The package to be grabbed first based on the blocking relationship is the package on the upper layer that is not blocked, thereby avoiding the problem of the package slipping and being damaged due to the blocked package being grabbed first.

[0105] When the processor 92 executes the computer program in the memory 91, in addition to the above functions, it can also realize other functions, and the details can be found in the description of the previous embodiments.

[0106] Further, such as Fig. 9 As shown, the computing device also includes: a display 94, a communication component 93, a power component 95, an audio component 96 and other components. Fig. 9 Only some components are shown schematically, and it does not mean that the computing device only includes Fig. 9 Components shown.

[0107] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a computer, the methods provided in the above embodiments can be implemented.

[0108] An embodiment of the present application also provides a computer program product, including a computer program, which implements the methods provided in the above embodiments when executed by a processor.

[0109] The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a processor, the methods provided in the above embodiments are implemented.

[0110] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0111] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for determining to grab a package, characterized in that: include: Acquire an image containing a package to be grasped; Segmenting a first mask image wrapped by the partial occlusion from the image; The first mask image includes: a mask image region corresponding to a visible portion that is partially occluded and wrapped in the image, and a mask image region inferred from an invisible portion that is occluded and wrapped in the image; Segmenting a second mask image of each visible package except the partially blocked package from the image; the second mask image includes a mask image area corresponding to a visible portion of each visible package in the image; Determining, according to the first mask image and the second mask image, an occlusion relationship between the partially occluded package and each visible package in the image; Based on the occlusion relationship, a package to be grasped with priority is determined from the packages to be grasped.

2. The method according to claim 1, characterized in that Segmenting a first mask image partially blocked and wrapped from the image specifically includes: Based on a first instance segmentation model pre-trained using a sample image, segmenting a first mask image that is partially occluded and wrapped from the image; The sample image has a label, and the label is a mask image that is partially blocked and wrapped in the sample image; The mask image includes: a mask image region corresponding to a visible portion of the sample image that is partially occluded and wrapped in the sample image, and a mask image region corresponding to an invisible portion of the sample image that is partially occluded and wrapped in the sample image.

3. The method according to claim 2, characterized in that The first instance segmentation model is trained in the following way: Performing specified operations in each round of iterative training until the instance segmentation model to be trained converges, and using the converged instance segmentation model to be trained as the first instance segmentation model; The specified operation includes: Inputting the sample image into the instance segmentation model to be trained, determining the value of the loss function based on the instance segmentation result output by the instance segmentation model to be trained and the actual instance segmentation result; adjusting the parameters of the instance segmentation model to be trained based on the principle of minimizing the value of the loss function; The actual instance segmentation result includes: a mask image that is partially occluded and wrapped in the sample image.

4. The method according to any one of claims 1 to 3, characterized in that: Segmenting a second mask image of each visible package except the partially blocked package from the image, comprising: Segmenting mask images of the visible packages from the image based on a pre-trained second instance segmentation model; From the mask images of the visible packages, the mask image regions corresponding to the partially blocked packages in the image are removed to obtain second mask images of the visible packages other than the partially blocked packages.

5. The method according to claim 1, characterized in that The method further comprises: Based on the occlusion relationship, an occlusion relationship matrix is ​​generated; each row and each column of the occlusion relationship matrix respectively corresponds to each visible package in the image, and each element in the occlusion relationship matrix represents a representation value of the mutual occlusion relationship between two different packages in each visible package; The occlusion relationship matrix is ​​used to determine the grabbing order of each visible package in the image.

6. The method according to claim 5, characterized in that The method comprises: Obtaining the occlusion relationship matrix; An optimal moving path of a robotic arm for grabbing the package is determined based on the occlusion relationship matrix and the position of each visible package in the image.

7. A device for determining to grab a package, characterized in that: include: An image acquisition unit, used to acquire an image containing a package to be grasped; A first image segmentation unit, used for segmenting a first mask image partially blocked and wrapped from the image; The first mask image includes: a mask image region corresponding to a visible portion that is partially occluded and wrapped in the image, and a mask image region inferred from an invisible portion that is occluded and wrapped in the image; A second image segmentation unit is configured to segment a second mask image of each visible package except the partially blocked package from the image; the second mask image includes a mask image region corresponding to a visible portion of each visible package in the image; an occlusion relationship determining unit, which determines an occlusion relationship between the partially occluded package and each visible package in the image according to the first mask image and the second mask image; The grab package determination unit determines a package to be grabbed with priority from the packages to be grabbed based on the occlusion relationship.

8. A robotic arm for grabbing a package, characterized in that: The robotic arm determines the optimal moving path of the robotic arm based on the occlusion relationship determined in any one of claims 1 to 6 and the location of the package to be grabbed preferentially.

9. A device for grabbing a package, characterized in that: The device for grabbing a package includes a mechanical arm, a communication module and a processor; A processor, configured to execute the steps of the method according to any one of claims 1 to 6 to obtain the occlusion relationship and the position of the package to be grabbed first; A communication module, used for sending the occlusion relationship obtained by the processor and the location of the package to be grabbed preferentially to the robotic arm; The robotic arm determines an optimal moving path of the robotic arm based on the received occlusion relationship and the location of the package to be grasped preferentially.

10. A computing device, characterized in that: include: Memory and processor, wherein: The memory is used to store computer programs; The processor is coupled to the memory and is used to execute the computer program stored in the memory to execute the method described in any one of claims 1 to 6.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a computer, it can implement the method described in any one of claims 1 to 6.

12. A computer program product, characterized in that It includes a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 6.

Images