Object grabbing and sorting method, system and equipment and medium

Abstract

The invention provides an object grabbing sorting method, system and device and a medium, and the sorting method comprises the steps: obtaining an object image recording a plurality of to-be-grabbed objects, and a depth image corresponding to the object image; inputting the object image into an optimized detection model to obtain a target category score and a target plane position output by the optimized detection model; according to the target plane position, point cloud segmentation is conducted on the depth image, target instance point cloud data are obtained, and the target instance point cloud data are used for representing a set of depth point clouds of the corresponding to-be-grabbed object within the target plane position range; and according to the target instance point cloud data, capturing and sorting the corresponding target category scores to obtain a capturing and sorting result. According to the sorting method, computing resources required for object grabbing sorting can be reduced, and the quality of the generated object grabbing sequence is improved. The invention relates to the technical field of computer vision.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, system, device and medium for grasping and sorting objects. Background Art

[0002] At present, traditional robot grasping and sorting methods can be mainly divided into two categories. The first method mainly uses target detection technology to detect the center point coordinates of multiple objects to be grasped, and determines the grasping order of the objects based on the center point coordinates. The quality of the object grasping order determined by this method is not high. When grasping objects, the robot is easily affected by the tilt state of the object and the physical topological relationship between the objects, causing the robot to fall off and the object grasping success rate is not high.

[0003] The second method mainly uses an image detector to locate the object positions of multiple objects to be grasped, and then builds an operation relationship prediction model between objects based on the located object positions, and determines the object grasping order through the generated operation relationship tree. This method requires more computing resources, which makes the real-time performance of object grasping unsatisfactory.

[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0005] The purpose of the present invention is to solve one of the technical problems existing in the related art to at least a certain extent.

[0006] The main purpose of the embodiments of the present application is to propose an object grasping and sorting method, which can reduce the computing resources required for object grasping and sorting, improve the quality of the generated object grasping sequence, and thus improve the success rate and real-time performance of object grasping.

[0007] To achieve the above purpose, an embodiment of the present application provides an object grasping and sorting method, which includes:

[0008] Acquire an object image recording a plurality of objects to be grasped, and a depth image corresponding to the object image;

[0009] Inputting the object image into the optimized detection model to obtain the target category score and target plane position output by the optimized detection model;

[0010] According to the target plane position, the depth image is segmented into point clouds to obtain target instance point cloud data, wherein the target instance point cloud data is used to represent a set of depth point clouds of the corresponding object to be grasped within the target plane position range;

[0011] According to the target instance point cloud data, the corresponding target category scores are captured and sorted to obtain a capture and sorting result.

[0012] In some embodiments, performing point cloud segmentation on the depth image according to the target plane position to obtain target instance point cloud data includes:

[0013] Determining, according to the depth image, regional point cloud data corresponding to all the objects to be grasped;

[0014] According to the target plane position, the regional point cloud data is screened to obtain the target instance point cloud data.

[0015] In some embodiments, the step of performing point cloud screening on the regional point cloud data according to the target plane position to obtain the target instance point cloud data includes:

[0016] According to the target plane position, determining the corresponding target point cloud range;

[0017] According to the target point cloud range, the regional point cloud data is segmented into instance point cloud to obtain the target instance point cloud data.

[0018] In some embodiments, the method further comprises:

[0019] Performing point cloud filtering on the target instance point cloud data to obtain filtered target instance point cloud data;

[0020] The filtered target instance point cloud data is clustered and segmented to obtain clustered target instance point cloud data.

[0021] In some embodiments, the step of crawling and sorting the corresponding target category scores according to the target instance point cloud data to obtain a crawling and sorting result includes:

[0022] Acquire a first point cloud value and a second point cloud value of each target instance point cloud data, wherein the first point cloud value is used to indicate an average point cloud density of the target instance point cloud data, and the second point cloud value is used to indicate an average point cloud height of the target instance point cloud data;

[0023] According to the first point cloud value and the second point cloud value, logarithmic transformation and exponential weighting processing are performed on the corresponding target category scores to obtain a plurality of object weighted scores, each of which corresponds to one of the objects to be grasped;

[0024] The grasping sorting result is obtained according to the weighted scores of all the objects.

[0025] In some embodiments, according to the first point cloud value and the second point cloud value, logarithmic transformation and exponential weighting processing are performed on the corresponding target category score to obtain the object weighted score, including:

[0026] Obtaining a first weight corresponding to the first point cloud value, a second weight corresponding to the second point cloud value, and a third weight corresponding to the target category score;

[0027] Performing a first logarithmic transformation on the first point cloud value to obtain a first logarithmic value, and performing a second logarithmic transformation on the second point cloud value to obtain a second logarithmic value;

[0028] Performing a third logarithmic transformation on the target category score to obtain a category logarithmic value;

[0029] The first logarithmic value, the second logarithmic value, and the category logarithmic value are exponentially weighted according to the first weight, the second weight, and the third weight to obtain the object weighted score.

[0030] In some embodiments, the optimized detection model includes a feature extraction module and a feature fusion module, the feature extraction module includes a plurality of cascaded bottleneck residual blocks, and the object image is input into the optimized detection model to obtain the target category score and target plane position output by the optimized detection model, including:

[0031] Performing multi-level feature extraction on the object image through the multiple cascaded bottleneck residual blocks to obtain object features of multiple different scales;

[0032] The feature fusion module performs feature pyramid fusion on the object features of the multiple different scales to obtain the target category score and the target plane position.

[0033] To achieve the above purpose, another aspect of the embodiment of the present application provides an object grasping and sorting system, including:

[0034] An acquisition unit, used to acquire an object image recording a plurality of objects to be grasped, and a depth image corresponding to the object image;

[0035] A processing unit, used for inputting the object image into the optimized detection model to obtain the target category score and target plane position output by the optimized detection model;

[0036] A segmentation unit, configured to perform point cloud segmentation on the depth image according to the target plane position to obtain target instance point cloud data, wherein the target instance point cloud data is used to represent a set of depth point clouds of the corresponding object to be grasped within the target plane position range;

[0037] The sorting unit is used to grab and sort the corresponding target category scores according to the target instance point cloud data to obtain a grab and sort result.

[0038] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0039] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0040] The embodiments of the present application include at least the following beneficial effects:

[0041] The present application provides an object grasping and sorting method, system, device and medium, wherein the sorting method obtains an object image recording a number of objects to be grasped, and a depth image corresponding to the object image; the object image is input into an optimized detection model to obtain a target category score and a target plane position output by the optimized detection model; according to the target plane position, the depth image is segmented into point clouds to obtain target instance point cloud data, and the target instance point cloud data is used to characterize a set of depth point clouds of the corresponding objects to be grasped within the target plane position range; according to the target instance point cloud data, the corresponding target category scores are grasped and sorted to obtain a grasping and sorting result. This sorting method segments the depth image based on the target plane position output by the detection model, which can avoid the step of segmenting instance point cloud data one by one from all objects to be grasped in the depth image, effectively reducing the required computing resources, thereby improving the real-time performance of object grasping; in addition, this sorting method also grasps and sorts the target category scores output by the optimized detection model according to the target instance point cloud data, and can comprehensively consider the grasping order of several objects to be grasped at multiple scales, effectively improving the quality of the generated object grasping order, thereby improving the success rate of object grasping. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flowchart of a method for grasping and sorting objects provided in an embodiment of the present application;

[0043] Figure 2 is a detailed flow chart of step S120 provided in an embodiment of the present application;

[0044] Figure 3 It is a schematic diagram of a framework of an optimized detection model provided in an embodiment of the present application;

[0045] Figure 4 is a detailed flow chart of step S130 provided in an embodiment of the present application;

[0046] Figure 5 is a detailed flow chart of step B2 provided in an embodiment of the present application;

[0047] Figure 6 is an optional flow chart of an object grabbing and sorting method provided in an embodiment of the present application;

[0048] Figure 7 is a detailed flow chart of step S140 provided in an embodiment of the present application;

[0049] Figure 8 is a detailed flow chart of step D2 provided in an embodiment of the present application;

[0050] Fig. 9 It is a schematic diagram of the framework of an object grasping and sorting system provided in an embodiment of the present application;

[0051] Fig.10 It is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices / equipment and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0053] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0054] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0056] Traditional industrial inspection and grasping scenes are mostly unstructured complex environments, often accompanied by cluttered backgrounds and changing light, making industrial-level grasping and picking operations very difficult. Specifically, in industrial scenes, objects to be picked are often stacked, squeezed or blocked, and are highly similar in appearance, texture, and position, and have intricate physical topological relationships in space. In related technologies, traditional robot grasping and sorting methods can be mainly divided into two categories. The first method mainly detects the center point coordinates of multiple objects to be grasped through target detection technology, and determines the grasping order of objects by comparing the size relationship between the center point coordinates of each object to be grasped. However, due to the influence of the tilt state of the object and the physical topological relationship between the objects, the center point of the object is not necessarily the center of mass of the object when the robot grasps it. The robot is easy to fall off when grasping the object. The quality of the object grasping order determined by this method is not high, and the object grasping success rate is not high.

[0057] The second method mainly uses an image detector to locate the object positions of multiple objects to be grasped, and then builds an operation relationship prediction model between objects based on the located object positions. The object grasping order is determined by generating an operation relationship tree. In scenes with a large number of objects, this method needs to calculate the operation relationship between each pair of objects separately, which requires more computing resources, making the real-time performance of object grasping unsatisfactory.

[0058] It should be noted that in the related art, after detecting the coordinates of the center point of the object, the center point coordinates are mapped into three-dimensional space to obtain the height of the center point, and the grasping and sorting are performed according to the height of the center point of each object to be grasped. This method has a small scope of application, and is mainly used for grasping and sorting regular objects such as cardboard boxes. In the scene where irregular objects are stacked, this method often cannot accurately obtain the height information of each object, and the quality of grasping and sorting is poor. In addition, there is also a method in the related art to obtain the local features and global features of the object through a pre-built feature extractor, and then directly input the obtained local features and global features into the sequential prediction network for prediction, so as to obtain the grasping and sorting. In the scene where objects are placed in a disorderly manner, this method often makes it difficult for the feature extractor to obtain the complete local features of a single object, resulting in low grasping and sorting quality predicted by the sequential prediction network, and the cost of training a sequential prediction network alone is high, and the inference time is long.

[0059] In view of this, the present application provides an object grasping and sorting method, system, device and medium, wherein the sorting method segments the depth image based on the target plane position output by the detection model, which can avoid the step of segmenting instance point cloud data one by one from all objects to be grasped in the depth image, effectively reducing the required computing resources, and thus improving the real-time performance of object grasping; in addition, the sorting method also grasps and sorts the target category scores output by the optimized detection model according to the target instance point cloud data, and can comprehensively consider the grasping order of several objects to be grasped at multiple scales, effectively improving the quality of the determined object grasping order, and thus improving the success rate of object grasping.

[0060] The object grasping and sorting method provided in the embodiment of the present application can be applied to industrial application scenarios (such as garbage sorting, workpiece selection, product classification and placement, etc.). In industrial application scenarios, industrial manufacturers can use the object grasping and sorting method provided in the embodiment of the present application to sort multiple objects to be grasped by the robot, and grasp them based on the obtained grasping and sorting, which can reduce the computing resources required for object grasping and sorting, effectively improve the quality of object grasping and sorting, and further improve the success rate and real-time performance of object grasping, and effectively avoid the collision, squeezing, etc. of objects during the grasping process.

[0061] The object grabbing and sorting method provided in the embodiment of the present application can be applied to a terminal, a server, or a software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the method, etc., but is not limited to the above forms.

[0062] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0063] Reference Figure 1 , Figure 1 is an optional flow chart of an object grabbing and sorting method provided in an embodiment of the present application. Figure 1 The sorting method in may include but is not limited to steps S110 to S140.

[0064] Step S110, obtaining an object image recording a plurality of objects to be grasped, and a depth image corresponding to the object image;

[0065] In the embodiment of the present application, an object image and a depth image in a certain scene environment can be obtained by an RGB-D camera, or an object image and a depth image in a certain scene environment can be obtained by an RGB camera and a depth camera respectively. The scene environment can be an unstructured environment, and there are multiple objects to be grasped in the scene environment. In addition, the object image in the example of the present application can be a RGB image, which is used to record the position information of several objects to be grasped in the scene environment, and the depth image is used to record the depth information corresponding to the several objects to be grasped in the scene environment.

[0066] Step S120, inputting the object image into the optimized detection model to obtain the target category score and target plane position output by the optimized detection model;

[0067] Reference Figure 2 and Figure 3 In some embodiments, the optimized detection model includes a feature extraction module and a feature fusion module, the feature extraction module includes a plurality of cascaded bottleneck residual blocks, and the step 120 of inputting the object image into the optimized detection model to obtain the target category score and target plane position output by the optimized detection model includes:

[0068] A1. performing multi-level feature extraction on the object image through the multiple cascaded bottleneck residual blocks to obtain object features of multiple different scales;

[0069] A2. Performing feature pyramid fusion on the object features of the multiple different scales through the feature fusion module to obtain the target category score and the target plane position.

[0070] In the embodiment of the present application, the target category score is used to characterize the category and confidence score of the object to be grasped detected by the detection model; the target plane position is used to characterize the plane position of the corresponding object to be grasped.

[0071] It can be understood that the feature extraction module in the embodiment of the present application includes multiple cascaded bottleneck residual blocks (BottleNeck). Specifically, Figure 3 The feature extraction module in consists of a base layer (CBH), fifteen cascaded bottleneck residual blocks, and a spatial pyramid pooling layer (SPPF). Each bottleneck residual block contains a depth-separable convolution and an inverse residual structure. In addition, Figure 3 The feature fusion module in can be implemented by the feature pyramid structure of the YOLO model, which includes a base layer (CBH), a depthwise separable convolution layer (Depthwise Separable Convolution, DSC), upsampling (UpSample) and concatenation (Concat).

[0072] It should be noted that, in an embodiment of the present application, step A1 may be inputting the object image into the feature extraction module in the optimized detection model, and the sixth cascaded bottleneck residual block, the thirteenth cascaded bottleneck residual block and the spatial pyramid pooling layer in the feature extraction module respectively output object features of different scales; step A2 may be upsampling object features of different scales to the same resolution for fusion through a feature fusion module, and outputting the target category score and target plane position based on the corresponding classification detection head.

[0073] It is worth mentioning that the embodiment of the present application is based on a feature extraction module composed of multiple cascaded bottleneck residual blocks, which can reduce the number of parameters of the network structure and the amount of calculation for feature extraction of object images, and can reduce the information loss of the input object image during the transformation process, thereby improving the real-time and accuracy of generating target category scores and target plane positions.

[0074] Step S130: performing point cloud segmentation on the depth image according to the target plane position to obtain target instance point cloud data, wherein the target instance point cloud data is used to represent a set of depth point clouds of the corresponding object to be grasped within the target plane position range;

[0075] Reference Figure 4 In some embodiments, the step S130 of performing point cloud segmentation on the depth image according to the target plane position to obtain target instance point cloud data includes:

[0076] B1. Determine, based on the depth image, regional point cloud data corresponding to all the objects to be grasped;

[0077] In an embodiment of the present application, a programming language (such as Python) or an image processing library (such as OpenCV) can be used to read the depth image, and then the coordinates and pixel values ​​of each pixel are converted into coordinates in 3D space according to the pixel values ​​in the depth image, so as to obtain regional point cloud data corresponding to all objects to be grasped.

[0078] It can be understood that the regional point cloud data in the embodiment of the present application can be a collection of point cloud data corresponding to the depth image, or it can be point cloud data within the overall object contour range of all objects to be captured, and the present application does not impose any restrictions on this.

[0079] B2. According to the target plane position, the regional point cloud data is screened to obtain the target instance point cloud data.

[0080] Reference Figure 5 Further, the step B2, based on the target plane position, performs point cloud screening on the regional point cloud data to obtain the target instance point cloud data, includes:

[0081] B21. Determine the corresponding target point cloud range according to the target plane position;

[0082] B22. According to the target point cloud range, perform instance point cloud segmentation on the regional point cloud data to obtain the target instance point cloud data.

[0083] In an embodiment of the present application, for a certain object to be grasped, if the depth image and the object image are acquired through an RGB-D camera, the depth image and the object image are usually registered and aligned. At this time, step B21 can directly determine the corresponding target point cloud range based on the target plane position (x1, y1, x2, y2) output by the detection model. The target point cloud range is used to characterize the set of point cloud data corresponding to the object to be grasped in the regional point cloud data; or, if the depth image is acquired by a depth camera and the object image is acquired by an RGB camera, at this time, step B21 can obtain the registration and alignment relationship between the object image coordinate system and the point cloud data coordinate system, and then determine the target point cloud range corresponding to the object to be grasped based on the registration and alignment relationship and the target plane position. As for the target point cloud range of the remaining objects to be grasped, it can be simply deduced based on the above content, and this application will not elaborate on it here.

[0084] It can be understood that for a certain object to be grasped, step B22 can be based on the target point cloud range corresponding to the object to be grasped, and the regional point cloud data can be accurately segmented to obtain the target instance point cloud data corresponding to the object to be grasped. The same is true for the other objects to be grasped, which can be simply deduced.

[0085] Reference Figure 6 In some embodiments, the method further comprises:

[0086] C1. performing point cloud filtering on the target instance point cloud data to obtain filtered target instance point cloud data;

[0087] C2. Cluster and segment the filtered target instance point cloud data to obtain clustered target instance point cloud data.

[0088] In an embodiment of the present application, after obtaining the target instance point cloud data of each object to be grasped, point cloud filtering and cluster segmentation operations can be performed on each target instance point cloud data respectively to remove noise points of each target instance point cloud data and improve the point cloud quality of the target instance point cloud data.

[0089] It can be understood that the point cloud filtering in step C1 can adopt any one of the methods such as straight-through filtering, voxel filtering, statistical filtering, radius filtering, Gaussian filtering, conditional filtering, etc. Specifically, the embodiment of the present application takes the point cloud filtering using radius filtering as an example. For a certain target instance point cloud data, step C1 can first obtain a preset filtering threshold, and the specific value of the filtering threshold can be flexibly set according to the actual situation, and then calculate each data point in the target instance point cloud data and the distance between other data points; then compare the size relationship between the filtering threshold and the calculated distance. If a certain calculated distance is greater than the filtering threshold, remove a certain other data point corresponding to the calculated distance; or, if a certain calculated distance is less than or equal to the filtering threshold, retain a certain other data point corresponding to the calculated distance, thereby obtaining the filtered target instance point cloud data, and the same is true for the remaining target instance point cloud data, which can be simply deduced.

[0090] It should be noted that, for a certain filtered target instance point cloud data, the clustering segmentation in step C2 can be implemented based on K-means clustering. Specifically, it can be based on a preset number of cluster clusters, randomly selecting data points from the filtered target instance point cloud data as cluster cluster centers, and then calculating the distance between each data point of the filtered target instance point cloud data and the cluster cluster center, and based on the calculated minimum distance, assigning each data point to the corresponding cluster cluster; then, for each cluster cluster, the cluster cluster center is updated by calculating the average value of each cluster cluster, and it is determined whether the clustering segmentation meets the convergence condition. If the convergence condition is met, the clustered target instance point cloud data is determined according to the latest cluster cluster; or, if the convergence condition is not met, return to the step of calculating the distance between each data point of the filtered target instance point cloud data and the cluster cluster center.

[0091] It is worth mentioning that the convergence condition of cluster segmentation can be whether the degree of change of the cluster center of the same cluster cluster in two adjacent cycles is less than or equal to a preset change threshold, or it can be whether the number of cluster segmentation cycles is greater than or equal to the maximum cycle threshold. The specific change threshold and maximum cycle threshold can be set according to actual conditions, and this application does not limit this. The same applies to the other filtered target instance point cloud data.

[0092] Step S140: According to the target instance point cloud data, the corresponding target category scores are captured and sorted to obtain a captured and sorted result.

[0093] Reference Figure 7 In some embodiments, the step S140, based on the target instance point cloud data, the corresponding target category scores are captured and sorted to obtain the captured and sorted results, including:

[0094] D1. Obtain a first point cloud value and a second point cloud value of each target instance point cloud data, wherein the first point cloud value is used to indicate an average point cloud density of the target instance point cloud data, and the second point cloud value is used to indicate an average point cloud height of the target instance point cloud data;

[0095] In an embodiment of the present application, the target instance point cloud data may be the target instance point cloud data obtained in step B2, or the clustered target instance point cloud data obtained in step C2. The first point cloud value is used to indicate the average point cloud density of the target instance point cloud data, that is, the average density of all data points of the target instance point cloud data within the target plane position range. The larger the first point cloud value, the larger the exposure surface of the corresponding object to be grasped, the closer its distance to the camera for acquiring the depth image and / or the object image, and the greater the possibility that it is in a higher-level position among all objects to be grasped; conversely, the smaller the first point cloud value, the smaller the exposure surface of the corresponding object to be grasped, the farther its distance from the camera for acquiring the depth image and / or the object image, and the smaller the possibility that it is in a higher-level position among all objects to be grasped.

[0096] It can be understood that the second point cloud value is used to indicate the average point cloud height of the target instance point cloud data, that is, the average position of all data points of the target instance point cloud data in the vertical direction within the target plane position range.

[0097] It should be noted that the first point cloud value in the embodiment of the present application can be simply calculated based on the coordinate data of each data point in the target instance point cloud data, and the second point cloud value can be simply calculated based on the coordinate data of each data point in the target instance point cloud data in the vertical direction. There are many specific implementation methods, which will not be elaborated in this application.

[0098] D2. Performing logarithmic transformation and exponential weighting processing on the corresponding target category scores according to the first point cloud value and the second point cloud value to obtain a plurality of object weighted scores, each of which corresponds to one of the objects to be grasped;

[0099] Reference Figure 8 Further, in step D2, according to the first point cloud value and the second point cloud value, the corresponding target category score is logarithmically transformed and exponentially weighted to obtain an object weighted score, including:

[0100] D21, obtaining a first weight corresponding to the first point cloud value, a second weight corresponding to the second point cloud value, and a third weight corresponding to the target category score;

[0101] D22, performing a first logarithmic transformation on the first point cloud value to obtain a first logarithmic value, and performing a second logarithmic transformation on the second point cloud value to obtain a second logarithmic value;

[0102] D23, performing a third logarithmic transformation on the target category score to obtain a category logarithmic value;

[0103] D24. Perform exponential weighting on the first logarithmic value, the second logarithmic value, and the category logarithmic value according to the first weight, the second weight, and the third weight to obtain the object weighted score.

[0104] In an embodiment of the present application, for the first point cloud value, the second point cloud value and the target category score corresponding to a certain object to be grasped, the specific values ​​of the corresponding first weight, second weight and third weight can be set in advance. For example, the first weight can be 0.2, the second weight can be 0.3, and the third weight can be 0.5.

[0105] It can be understood that the first logarithmic transformation in step D22 can be based on a preset logarithmic function to calculate a first logarithmic value corresponding to the first point cloud value. The equivalent expression of the first logarithmic value can be:

[0106] D = log(1+d)

[0107] Among them, d is the first point cloud value; D is the first logarithmic value.

[0108] It should be noted that the second logarithmic transformation in step D22 and the third logarithmic transformation in step D23 are similar to the aforementioned contents about the first logarithmic transformation and can be simply deduced. As for step D24, the exponential transformation forms of the first logarithmic value, the second logarithmic value and the category logarithmic value can be obtained respectively, and then the corresponding exponential transformation forms are weighted based on the first weight, the second weight and the third weight to obtain the object weighted score of the object to be grasped, and the object weighted scores of the other objects to be grasped are similar.

[0109] Exemplarily, the equivalent expression of the exponential weighting in step D24 may be:

[0110]

[0111] Among them, α is the third weight, e S is the exponential transformation form corresponding to the category logarithm S; β is the second weight, e H is the exponential transformation form corresponding to the second logarithmic value H; γ is the first weight, e H It is the exponential transformation form corresponding to the first logarithmic value D.

[0112] D3. Obtain the grasping sorting result according to the weighted scores of all the objects.

[0113] In an embodiment of the present application, after obtaining the object weighted scores of all objects to be grasped, step D3 can be performed by comparing the size relationships of the weighted scores of all objects and sorting the obtained size relationships in order to obtain the grasping sorting results corresponding to all objects to be grasped; or, step D3 can also be performed based on a normalization function to convert the object weighted score r into a score distribution R in the range of [0, 1] and with a sum of 1, and then determine the grasping sorting results based on the score distribution.

[0114] For example, an equivalent expression of the score distribution R may be:

[0115]

[0116] Among them, r i is the weighted score of the object corresponding to the i-th object to be grasped; r j is the weighted score of the object corresponding to the jth object to be grasped; n is the total number of objects to be grasped; softmax() is the softmax normalization function; R is the score distribution.

[0117] It is worth mentioning that the embodiment of the present application can smooth the extreme values ​​of the target category score, the first point cloud value or the second point cloud value of all objects to be grasped based on the logarithmic transformation; then the influence of important factors is amplified by exponential weighting, and the important factors can specifically be the confidence score in the target category score, the average point cloud density of the instance point cloud data and the average point cloud height of the instance point cloud data, so that the obtained object weighted score will not be dominated by extreme values, and can reasonably reflect the influence of important factors; in addition, the logarithmic transformation and exponential weighting processing of the embodiment of the present application can suppress the negative impact brought by the scale differences between different important factors, thereby improving the quality of the subsequently generated grasping and sorting results.

[0118] See also Fig. 9 The present application also provides an object grasping and sorting system, which includes:

[0119] An acquisition unit 810 is used to acquire an object image recording a plurality of objects to be grasped, and a depth image corresponding to the object image;

[0120] A processing unit 820 is used to input the object image into the optimized detection model to obtain the target category score and target plane position output by the optimized detection model;

[0121] A segmentation unit 830 is used to perform point cloud segmentation on the depth image according to the target plane position to obtain target instance point cloud data, wherein the target instance point cloud data is used to represent a set of depth point clouds of the corresponding object to be grasped within the target plane position range;

[0122] The sorting unit 840 is used to grab and sort the corresponding target category scores according to the target instance point cloud data to obtain a grab and sort result.

[0123] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0124] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0125] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0126] See also Fig.10 , Fig.10 The hardware structure of an electronic device of an embodiment is illustrated, and the electronic device includes:

[0127] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0128] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 902, and the processor 901 calls and executes the methods of the embodiments of this application;

[0129] Input / output interface 903, used to implement information input and output;

[0130] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);

[0131] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0132] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0133] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and the computer program implements the above method when executed by a processor.

[0134] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0135] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0136] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0137] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0138] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0139] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0140] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0141] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0142] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0143] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0144] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0145] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0146] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for grasping and sorting objects, characterized in that: include: Acquire an object image recording a plurality of objects to be grasped, and a depth image corresponding to the object image; Inputting the object image into the optimized detection model to obtain the target category score and target plane position output by the optimized detection model; According to the target plane position, the depth image is segmented into point clouds to obtain target instance point cloud data, wherein the target instance point cloud data is used to represent a set of depth point clouds of the corresponding object to be grasped within the target plane position range; According to the target instance point cloud data, the corresponding target category scores are captured and sorted to obtain a capture and sorting result.

2. The sorting method according to claim 1, characterized in that: The step of performing point cloud segmentation on the depth image according to the target plane position to obtain target instance point cloud data includes: Determining, according to the depth image, regional point cloud data corresponding to all the objects to be grasped; According to the target plane position, the regional point cloud data is screened to obtain the target instance point cloud data.

3. The sorting method according to claim 2, characterized in that: The step of performing point cloud screening on the regional point cloud data according to the target plane position to obtain the target instance point cloud data includes: According to the target plane position, determining the corresponding target point cloud range; According to the target point cloud range, the regional point cloud data is segmented into instance point cloud to obtain the target instance point cloud data.

4. The sorting method according to claim 1, characterized in that: The method further comprises: Performing point cloud filtering on the target instance point cloud data to obtain filtered target instance point cloud data; The filtered target instance point cloud data is clustered and segmented to obtain clustered target instance point cloud data.

5. The sorting method according to claim 1, characterized in that: The step of crawling and sorting the corresponding target category scores according to the target instance point cloud data to obtain a crawling and sorting result includes: Acquire a first point cloud value and a second point cloud value of each target instance point cloud data, wherein the first point cloud value is used to indicate an average point cloud density of the target instance point cloud data, and the second point cloud value is used to indicate an average point cloud height of the target instance point cloud data; According to the first point cloud value and the second point cloud value, logarithmic transformation and exponential weighting processing are performed on the corresponding target category scores to obtain a plurality of object weighted scores, each of which corresponds to one of the objects to be grasped; The grasping sorting result is obtained according to the weighted scores of all the objects.

6. The sorting method according to claim 5, characterized in that: According to the first point cloud value and the second point cloud value, logarithmic transformation and exponential weighting processing are performed on the corresponding target category score to obtain an object weighted score, including: Obtaining a first weight corresponding to the first point cloud value, a second weight corresponding to the second point cloud value, and a third weight corresponding to the target category score; Performing a first logarithmic transformation on the first point cloud value to obtain a first logarithmic value, and performing a second logarithmic transformation on the second point cloud value to obtain a second logarithmic value; Performing a third logarithmic transformation on the target category score to obtain a category logarithmic value; The first logarithmic value, the second logarithmic value, and the category logarithmic value are exponentially weighted according to the first weight, the second weight, and the third weight to obtain the object weighted score.

7. The sorting method according to claim 1, characterized in that: The optimized detection model includes a feature extraction module and a feature fusion module, the feature extraction module includes a plurality of cascaded bottleneck residual blocks, the object image is input into the optimized detection model, and the target category score and target plane position output by the optimized detection model are obtained, including: Performing multi-level feature extraction on the object image through the multiple cascaded bottleneck residual blocks to obtain object features of multiple different scales; The feature fusion module performs feature pyramid fusion on the object features of the multiple different scales to obtain the target category score and the target plane position.

8. An object grasping and sorting system, characterized in that: include: An acquisition unit, used to acquire an object image recording a plurality of objects to be grasped, and a depth image corresponding to the object image; A processing unit, used for inputting the object image into the optimized detection model to obtain the target category score and target plane position output by the optimized detection model; A segmentation unit, configured to perform point cloud segmentation on the depth image according to the target plane position to obtain target instance point cloud data, wherein the target instance point cloud data is used to represent a set of depth point clouds of the corresponding object to be grasped within the target plane position range; The sorting unit is used to grab and sort the corresponding target category scores according to the target instance point cloud data to obtain a grab and sort result.

9. An electronic device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images