Image data processing method, device, electronic device and storage medium

Abstract

The present application discloses an image data processing method, device, electronic device and storage medium. The image data processing method includes: receiving image data containing an object to be processed; identifying the object to be processed from the image data and generating a mask of the object to be processed; performing morphological processing on the generated mask of the object to be processed; and further processing the mask after morphological processing to obtain a corrected mask and / or grasping point association information of the object to be processed. The present invention proposes a method for correcting a mask and obtaining two-dimensional grasping point information, so that when it is impossible to obtain an accurate object mask, an inaccurate mask can be corrected to obtain an accurate mask as much as possible and further obtain accurate grasping point information, effectively avoiding the problem of inaccurate grasping points caused by inaccurate masks, which in turn leads to inaccurate grasping or falling during grasping.

Description

Technical Field

[0001] The present application relates to the field of automatic control and program control B25J of a robotic arm or fixture. Background Art

[0002] The rapid rise of industries like e-commerce and express delivery in recent years has created promising development opportunities for the logistics industry. With the maturation of technologies like artificial intelligence and machine vision, automation is increasingly being deployed in fundamental logistics functions such as warehousing, handling, and sorting. The trend is for industrial robots to replace manual labor. As a crucial component of industrial robots, grippers are increasingly being used in logistics, sorting, the consumer electronics industry, and food loading.

[0003] Currently, automated control of robotic arms or grippers requires point cloud data of the object to be grasped. A common approach involves first capturing point cloud data using a 3D camera. This data is then used to determine the trajectory points or gripping points for the robotic arm's motion. Based on this information, the robotic arm is then controlled to perform the grasping operation at the appropriate location and along the appropriate trajectory. A robotic arm operates in three dimensions, and its motion trajectory is a three-dimensional trajectory in length, width, and height. Therefore, the trajectory points or gripping points should also have information in the X, Y, and Z axes. The point cloud data captured by a 3D camera includes this information in three dimensions. However, in some industrial scenarios, obtaining clear point cloud information of the grasped object is difficult. For example, in the cosmetics industry, glass, particularly black glass bottles, are used to hold liquids. These glass bottles have weak intrinsic light signals, and their reflective material is easily affected by multiple reflections from surrounding objects. Furthermore, the transparency of glass produces a high level of diffuse and multiple reflections, making it difficult to capture appropriate point cloud data. Specifically, in such industrial scenarios, the point cloud of the object to be grasped may not be available, the collected point cloud may be missing, or the point cloud may be poorly defined. This can cause the robot to either fail to identify the object to be grasped, or to calculate and use incorrect grasping points based on an inaccurate point cloud, resulting in a missed grasp or even a dropped bottle. Currently, there is no existing solution for automatically controlling robot grasping based on robot vision in industrial scenarios with missing point clouds. Summary of the Invention

[0004] In view of the above problems, the present invention is proposed to overcome the above problems or at least partially solve the above problems. Specifically, firstly, the present invention can obtain the grasping point information of the object to be grasped with the help of other image data such as color pictures when the point cloud of the object to be grasped cannot be obtained, so that the robot or clamp can directly rely on the grasping point information without the help of the point cloud of the object to grasp the object, effectively solving the problem of object grasping in an environment where point clouds are missing; secondly, the present invention proposes a method for correcting the mask and obtaining two-dimensional grasping point information, so that when the accurate object mask cannot be obtained, the inaccurate mask can be corrected to obtain the most accurate mask and further obtain Accurate grasping point information effectively avoids inaccurate grasping points caused by inaccurate masks, which in turn leads to problems such as inaccurate grasping or falling during grasping; thirdly, the present invention proposes a method for a robot to automatically convert input two-dimensional grasping point information into three-dimensional grasping point information. This method can automatically obtain reference information that can convert two-dimensional grasping points into three-dimensional grasping points based on the environmental characteristics of the object to be grasped, and obtain grasping point information based on the reference information for the robot to grasp. This solution enables the robot to obtain complete grasping point information based on existing information when grasping point information is missing, and reduces manual intervention.

[0005] All the solutions disclosed in the claims and description of this application have one or more of the above-mentioned innovations and, accordingly, can solve one or more of the above-mentioned technical problems. Specifically, this application provides an image data processing method, apparatus, electronic device, and storage medium.

[0006] The image data processing method according to the embodiment of the present application includes:

[0007] receiving image data containing an object to be processed;

[0008] Identifying an object to be processed from the image data and generating a mask of the object to be processed;

[0009] Performing morphological processing on the generated mask of the object to be processed;

[0010] The morphologically processed mask is further processed to obtain the corrected mask and / or grasping point association information of the object to be processed.

[0011] In certain embodiments, the morphological manipulation comprises a morphological dilation manipulation.

[0012] In some embodiments, further processing the morphologically processed mask to obtain a corrected mask and / or grip point association information of the object to be processed includes:

[0013] Obtain the bounding rectangle of the expanded mask;

[0014] Based on the circumscribed rectangle of the mask, generating an inscribed circle of the circumscribed rectangle;

[0015] A correction mask and / or gripping point association information of the object to be processed is obtained based on the inscribed circle.

[0016] In some embodiments, obtaining the bounding rectangle of the expanded mask includes: generating four corner points of the bounding rectangle based on the expanded mask, and then generating the bounding rectangle based on the corner points.

[0017] In some embodiments, further processing the morphologically processed mask to obtain a corrected mask and / or grip point association information of the object to be processed includes:

[0018] Use circle detection algorithm to process the dilated mask;

[0019] The correction mask and / or grasping point association information of the object to be processed is obtained based on the processing result of the circle detection algorithm.

[0020] In some embodiments, the circle detection algorithm includes: a circular Hough transform algorithm, a random Hough transform algorithm and / or a random circle detection algorithm.

[0021] In some embodiments, further processing the morphologically processed mask to obtain a corrected mask and / or grip point association information of the object to be processed includes:

[0022] Based on the pre-saved template of the object to be processed, the expanded mask is processed using a template matching algorithm;

[0023] The correction mask and / or grasping point association information of the object to be processed is obtained based on the processing result of the template matching algorithm.

[0024] In some embodiments, the matching algorithm comprises a shape-based matching algorithm.

[0025] In some embodiments, identifying the object to be processed from the image data and generating a mask of the object to be processed includes: processing the image data based on deep learning to identify the object to be processed and generating a mask of the object to be processed.

[0026] An image data processing device according to an embodiment of the present application includes:

[0027] An image data receiving module, configured to receive image data containing an object to be processed;

[0028] a mask generation module, configured to identify an object to be processed from image data and generate a mask of the object to be processed;

[0029] A mask processing module, used for performing morphological processing on the generated mask of the object to be processed;

[0030] The processing module is used to further process the mask after morphological processing to obtain the corrected mask and / or grasping point association information of the object to be processed.

[0031] In certain embodiments, the morphological manipulation comprises a morphological dilation manipulation.

[0032] In certain embodiments, the processing module is specifically configured to:

[0033] Obtain the bounding rectangle of the expanded mask;

[0034] Based on the circumscribed rectangle of the mask, generating an inscribed circle of the circumscribed rectangle;

[0035] A correction mask and / or gripping point association information of the object to be processed is obtained based on the inscribed circle.

[0036] In some embodiments, obtaining the bounding rectangle of the expanded mask includes: generating four corner points of the bounding rectangle based on the expanded mask, and then generating the bounding rectangle based on the corner points.

[0037] In certain embodiments, the processing module is specifically configured to:

[0038] Use circle detection algorithm to process the dilated mask;

[0039] The correction mask and / or grasping point association information of the object to be processed is obtained based on the processing result of the circle detection algorithm.

[0040] In some embodiments, the circle detection algorithm includes: a circular Hough transform algorithm, a random Hough transform algorithm and / or a random circle detection algorithm.

[0041] In certain embodiments, the processing module is specifically configured to:

[0042] Based on the pre-saved template of the object to be processed, the expanded mask is processed using a template matching algorithm;

[0043] The correction mask and / or grasping point association information of the object to be processed is obtained based on the processing result of the template matching algorithm.

[0044] In certain embodiments, the matching algorithm comprises a shape-based matching algorithm.

[0045] In some embodiments, identifying the object to be processed from the image data and generating a mask of the object to be processed includes: processing the image data based on deep learning to identify the object to be processed and generating a mask of the object to be processed.

[0046] The electronic device of an embodiment of the present application includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the image data processing method of any of the above embodiments is implemented.

[0047] The computer-readable storage medium of the embodiment of the present application stores a computer program thereon, and when the computer program is executed by a processor, the image data processing method of any of the above embodiments is implemented.

[0048] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become obvious from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0050] Figure 1 It is a flowchart of a method for obtaining grasping point information in a poor point cloud scenario according to certain embodiments of the present application;

[0051] Figure 2 is a flowchart of a method for obtaining correction mask and grip point association information in certain embodiments of the present application;

[0052] Figure 3 is a flowchart of a method for converting two-dimensional grasping point information into three-dimensional grasping point information in certain embodiments of the present application;

[0053] Figure 4 is a schematic diagram of an object to be grasped and an object mask obtained using a non-dedicated deep learning network in certain embodiments of the present application;

[0054] Figure 5 It is a structural diagram of a device for acquiring grasping point information in a scenario with poor point cloud quality according to certain embodiments of the present application;

[0055] Figure 6 is a schematic structural diagram of an apparatus for obtaining correlation information of a correction mask and gripping points according to certain embodiments of the present application;

[0056] Figure 7 It is a schematic structural diagram of an apparatus for converting two-dimensional grasping point information into three-dimensional grasping point information according to certain embodiments of the present application;

[0057] Figure 8 It is a schematic structural diagram of an electronic device according to certain embodiments of the present application. DETAILED DESCRIPTION

[0058] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0059] Figure 1 FIG. 1 shows a flow chart of a method for obtaining item grabbing point information according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0060] Step S100, obtaining a non-point cloud image containing the object to be grasped;

[0061] Step S110, processing the non-point cloud image to obtain a mask of the object to be grasped;

[0062] Step S120, processing the mask of the object to be grasped to obtain grasping point association information, wherein the grasping point association information includes information used to calculate the grasping point information;

[0063] Step S130: Based on the grasping point association information, grasping point information for controlling the robot to grasp the object to be grasped is obtained.

[0064] Regarding step S100, in industrial scenarios, robots are often required to grasp a large number of objects, so there may be multiple objects to grasp, not just one. In a preferred embodiment, the objects to be grasped may be a group of black cosmetic bottles placed in a material basket. The robot needs to grasp the cosmetic bottles from the basket and transport them to another location. Because black glass has a weak optical signal and is easily affected by multiple reflections from surrounding objects, resulting in a large amount of diffuse and multiple reflection data, it is likely that point clouds will not be obtained when captured using industrial cameras, making it difficult to collect the 3D point cloud data typically required for robot control. Therefore, the method of the present invention is particularly suitable for use in such scenarios, using image data other than point cloud data to identify the objects to be grasped, calculate grasping point information, and control the robot to perform the grasping. In this embodiment, the non-point cloud image data can preferably be a 2D color image of the object. Unlike a point cloud, a 2D color image can more clearly identify the objects contained therein. Even objects such as black transparent glass bottles, which cannot generate point clouds, can still be clearly captured and identified. An industrial camera can be used to take pictures, and the group of objects to be grasped can be placed under the visual sensor for shooting to obtain image data of the objects to be grasped.

[0065] For step S110, the mask of the object can be calculated using any existing method. Preferably, the mask of the object can be obtained based on a deep learning network. Image recognition is a conventional application of deep learning networks, and there are already general deep learning networks in the prior art that can perform image recognition. As a specific embodiment, an existing deep learning network for identifying objects or extracting object masks can be used to identify objects and extract masks. In order to improve the accuracy of recognition, the network can be trained in advance. The acquired 2D color image is input into the deep learning network, and the color image is processed by the deep learning network to identify the image interest area for which the mask is to be generated, generate an image mask in this area, and use the image mask to cover the original image area. In this way, the mask of the object to be grasped can be obtained based on the color image.

[0066] In step S120, after obtaining the object's mask, the object's grasping point association information can be calculated within the mask area. This so-called grasping point association information lacks some necessary information required for grasping point information and is not complete grasping point information. Therefore, it cannot be directly used by the robot to perform grasping. However, the grasping point association information can be used to calculate grasping point information. In a specific embodiment, the grasping point association information can be two-dimensional information of the grasping point, such as X-axis and Y-axis coordinate information. Assuming that the current control system requires determining the grasping point's three-dimensional information (X, Y, and Z) before controlling the robot's grasping position (depending on the gripper used, actual industrial scenarios may require more information, such as the gripper's rotation angle). After obtaining the two-dimensional grasping point information, grasping cannot be performed based on this information. Subsequent steps will require converting the two-dimensional grasping point information into three-dimensional grasping point information. The specific location of the grasping point depends on the gripper used and the object to be grasped. For example, if the object to be grasped is a black cosmetic bottle, the center point of the bottle mouth can be selected as the grasping point.

[0067] A non-customized deep learning network that can be used in various situations to identify objects. When processing captured photos, identifying objects in the photos, and extracting object masks, the generated masks usually do not fit perfectly with the photographed objects. Sometimes they may slightly exceed the graspable area of ​​the object, and sometimes they may be slightly smaller than the graspable area of ​​the object. In addition, the shape of the mask is usually inconsistent with the graspable area of ​​the object. Figure 4 The figure shows an object to be grasped and a typical mask of the object to be grasped generated by a deep learning network. Specifically, Figure 4 The picture shows a scenario where the object to be grasped is the cosmetic bottle mentioned above. The circular part is the bottle mouth area of ​​the glass bottle to be grasped, that is, the grasping area of ​​the robotic arm. The shaded part is the mask extracted by a general deep learning network. It can be seen that in this case, the mask of the object is inaccurate, and the grasping points calculated from it are naturally inaccurate. Inaccurate grasping points may lead to problems such as not being able to grasp the bottle or the bottle falling during grasping. One solution is to design a deep learning network specifically for this scenario and repeatedly train it to improve the accuracy of the processing results after inputting the image into the network. In this way, after inputting the image into the deep learning network, an accurate mask and grasping points can be obtained. However, this solution is expensive and is dedicated to specific scenarios, and has low flexibility. Another solution is to process the inaccurate mask and obtain the grasping point information from the processed mask.

[0068] The present invention preferably uses the latter solution. Specifically, the applicant has developed a low-cost and more universal mask correction and gripping point acquisition method, which is also one of the key points of the present invention.

[0069] Figure 2 FIG. 1 is a flow chart showing an image data processing method for mask correction and acquisition of grip point association information according to an embodiment of the present invention. Figure 2 As shown, the method includes:

[0070] Step S200, receiving image data containing an object to be processed;

[0071] Step S210, identifying an object to be processed from the image data and generating a mask of the object to be processed;

[0072] Step S220, performing morphological processing on the generated mask of the object to be processed;

[0073] Step S230 : further processing the morphologically processed mask to obtain a corrected mask and / or gripping point association information of the object to be processed.

[0074] Regarding step S200, this embodiment does not limit the type of image data. Any type of image data, as long as it can be used to identify objects contained therein and generate a mask using existing deep learning algorithms, is applicable to this embodiment. Preferably, this embodiment can use a method similar to step S100 to obtain image data, and the image data can be a 2D color image.

[0075] For step S210 , a mask of the object to be processed may be identified and generated in a manner similar to step S120 .

[0076] For step S220, Figure 4 As shown in Figure 2, the object mask obtained by the general deep learning network usually does not perfectly fit the object contour ( Figure 4The grasping area in the image is the bottle mouth. It's easy to see that the mask differs significantly from the actual bottle mouth, resulting in a skewed appearance and possibly numerous holes. This isn't a significant issue for general object recognition applications, but in industrial applications involving grasping objects, where high precision is required, such errors are unacceptable. Therefore, the acquired mask needs to be processed. Morphological processing is first performed to alter the graphical form of the mask area. In one embodiment, this morphological processing can be dilation. After acquiring the 2D image information, dilation is performed on the image to fill in defects such as missing or irregular images. For example, for each pixel on the mask, a certain number of surrounding points, for example, 8-25 points, can be set to the same color as the pixel. This step is equivalent to filling in the area surrounding each pixel. Therefore, if there are any missing areas in the object mask, this operation will completely fill in the missing areas. After this process, the object mask becomes complete, free of missing areas. At the same time, the mask as a whole becomes slightly "fatter" due to the dilation. Appropriate dilation facilitates subsequent image processing operations.

[0077] For step S230 , as an example, the present invention discloses three processing methods to obtain the correction mask of the object and obtain the grasping point association information.

[0078] The first approach includes:

[0079] Step S240, obtaining the circumscribed rectangle of the mask after the expansion process;

[0080] Step S241, generating an inscribed circle of the circumscribed rectangle based on the circumscribed rectangle of the mask;

[0081] Step S242: obtaining the correction mask and / or gripping point association information of the object to be processed based on the inscribed circle.

[0082] In step S240, any bounding rectangle algorithm can be used to obtain the bounding rectangle of the mask. As a specific implementation method, the X coordinate value and Y coordinate value of each pixel point in the mask can be calculated, and the minimum X value, the minimum Y value, the maximum X value, and the maximum Y value can be selected respectively; then, the four values ​​are combined into the coordinates of the point, that is, the minimum X value and the minimum Y value form the coordinates (X min , Y min ), the maximum X value and Y value form the coordinate (X max , Y max ), the minimum X value and the maximum Y value form the coordinate (X min , Y max ), and the maximum X value and the minimum Y value form the coordinates (X max , Y min ). Take point (Xmin , Y min ), (X max , Y max ), (X min , Y max ), (X max , Y min ) as the four corner points of the circumscribed rectangle and connecting them, the circumscribed rectangle is obtained.

[0083] For step S241, the key of the present invention is to use the inscribed circle algorithm as a part of calculating the correction mask and the grab point information, without making any improvement to the inscribed circle algorithm. Therefore, the specific inscribed circle algorithm is not limited, and any inscribed circle algorithm can be used in the present invention.

[0084] In step S242, after obtaining the inscribed circle, the portion of the mask enclosed by the inscribed circle can be calculated and used as the correction mask for the object to be processed. The contour of the obtained correction mask has the same shape and size as the inscribed circle. The position of the center of the inscribed circle correction mask is calculated and the center information is obtained. This center information serves as the grasping point association information. Specifically, the two-dimensional position information of the center, for example, the X-axis and Y-axis information of the center, can be used as the X-axis and Y-axis position information of the grasping point.

[0085] The shape of the corrected mask obtained in this way is consistent with the shape of the bottle mouth, but the covered area may still be different from the actual area of ​​the bottle mouth.

[0086] The second approach includes:

[0087] Step S250 , processing the mask after the expansion process using a circle detection algorithm;

[0088] Step S251 : obtaining the correction mask and / or gripping point association information of the object to be processed based on the processing result of the circle detection algorithm.

[0089] In step S250, the circle detection algorithm, also known as the circle finding algorithm, can be used to detect circular features in irregular shapes and locate circles within them. Common algorithms include the circular Hough transform algorithm, the randomized Hough transform algorithm, and the randomized circle detection algorithm. This embodiment focuses on using a circle detection algorithm to find circles within a morphologically processed mask, without limiting the specific circle detection algorithm used. Since the bottle mouth itself is circular, and the collected mask contains some features of this shape, the circle found within the mask area roughly corresponds to the location of the bottle mouth.

[0090] In step S251, after a circle is found within the mask using a circle detection algorithm, the portion of the mask enclosed by the circle is used as the correction mask for the object to be processed. The center of the circle is then calculated, and this information is used as the grasping point association information. Similar to method 1, the grasping point association information can be the two-dimensional position information of the circle center, which is used as the X-axis and Y-axis position information of the grasping point.

[0091] The second method does not need to calculate the circumscribed rectangle and inscribed circle that do not exist. It only needs to find the circular part from the existing mask area. The calculation accuracy of this method is higher than that of the first method.

[0092] The third approach includes:

[0093] Step S260 , processing the expanded mask using a template matching algorithm based on a pre-stored template of the object to be processed;

[0094] Step S261 : obtaining the correction mask and / or gripping point association information of the object to be processed based on the processing result of the template matching algorithm.

[0095] In step S260, the template of the object to be processed can be a template of the entire object to be processed, or a template of the grasping area of ​​the object to be processed. For example, if the object to be grasped is a black glass cosmetic bottle and the gripper needs to grasp the bottle mouth, the template of the object to be processed can be a three-dimensional template of the entire black glass cosmetic bottle, or a template of only the graspable area of ​​the bottle mouth. After obtaining the uncorrected mask area, a matching algorithm is used to match the template within the mask area based on the pre-stored template. Simply put, the template is equivalent to a known small image, and the template matching algorithm is equivalent to searching for a target within a large image that includes the small image. Given that the target is in the image and has the same size, orientation, and image elements as the template, the template matching algorithm can find the target (i.e., the small image) in the image and determine its position and posture. This embodiment does not limit the specific matching algorithm. Since the mask itself loses color information, the focus is on shape matching rather than color matching. Therefore, the present invention preferably uses a shape-based matching algorithm for matching. Furthermore, considering both matching efficiency and accuracy, a successful match is considered when the shape similarity reaches 70-95%. The specific value can be selected and adjusted based on the needs of the actual application scenario.

[0096] For step S261, after finding a shape that matches the pre-stored template in the mask, the mask enclosed by the shape can be used as a correction mask, and the grasping point association information can be further calculated. Since the template is used, no matter what shape the object to be grasped is presented in the area to be grasped, the object can be matched and grasped, and it is not limited to the scenario where the grasping area of ​​the object to be grasped is circular. Accordingly, when grasping different objects, the position of the grasping point is also different. In one embodiment, the grasping point can be the center point of the correction mask of the object to be grasped, and the grasping point association information can be the two-dimensional position information of the center point, that is, the X-axis position information and Y-axis position information of the grasping point.

[0097] In engineering practice, the inventors found that the third method can achieve higher standards in terms of both accuracy and computing speed, and is the best among the three implementation methods of the present invention. In addition, the first two methods are actually only applicable to industrial scenarios where the area to be grasped is circular, while the third method can be used to grasp any object. Therefore, the third implementation method is also one of the focuses of the present invention.

[0098] In step S130, as previously described, the grasping point association information lacks some necessary information required for grasping point information and is not complete grasping point information. Therefore, it cannot be directly used by the robot to perform grasping. However, the grasping point association information can be used to calculate grasping point information. In one embodiment, the grasping point association information may be two-dimensional grasping point information, while the grasping point information required by the robot is three-dimensional. To facilitate robot use, the two-dimensional grasping point association information should be converted into three-dimensional information, thereby grasping the object to be grasped based on the three-dimensional information. If three-dimensional grasping point data cannot be directly obtained, one method for converting the two-dimensional data into three-dimensional data is to manually input the data of the third dimension of the object to be grasped. For example, if the object to be grasped is a black glass cosmetic bottle for which three-dimensional grasping point information is unavailable, specific height information can be pre-entered manually. In this way, after processing the two-dimensional image and obtaining the two-dimensional grasping point information through the aforementioned scheme, the two-dimensional grasping point information can be further converted into three-dimensional grasping point information based on the height information for use by the robot. This method requires manual input of height information before grasping. However, many robot-based applications aim to reduce manual intervention. Furthermore, manual height input requires the bottle rack to always be in the same position. If the rack is placed in a different position, the height information will change, requiring re-entry, otherwise correct grasping will fail. Therefore, a more automated approach is desired.

[0099] In order to realize this automated processing mode, the applicant has proposed a method for converting two-dimensional grasping point information into three-dimensional grasping point information without manual intervention, which is also one of the focuses of the present invention.

[0100] Figure 3 FIG. 1 is a flow chart showing a method for converting two-dimensional grasping point information into three-dimensional grasping point information according to an embodiment of the present invention. Figure 3 As shown, the method includes:

[0101] Step S300, obtaining reference object information of the object to be grasped;

[0102] Step S310, obtaining two-dimensional grasping point information of the object to be grasped;

[0103] Step S320: Processing the reference object information of the object to be grasped to obtain reference information, wherein the reference information includes information not included in the two-dimensional grasping point information and is capable of converting the two-dimensional information into three-dimensional information;

[0104] Step S330 : generating three-dimensional grasping point information of the object to be grasped based on the reference information and the two-dimensional grasping point information of the object to be grasped.

[0105] Regarding step S300, the present invention is applicable to scenarios where the point cloud of the object to be grasped is poor. Therefore, the reference object should be an object with a qualified point cloud. A qualified point cloud in this invention refers to a point cloud that can provide information on the dimensions missing from the object's two-dimensional grasping point information. For example, if the X-axis and Y-axis information of the grasping point are available, an object with recognizable Z-axis information can serve as a reference object. The reference object can be an object relatively close to the object to be grasped, or it can be another similar object to be grasped placed with the object to be grasped. Specifically, in industrial scenarios where a large number of objects to be grasped, such as cosmetic bottles, are placed in a material frame, the frame's point cloud is typically complete, and as long as the entire frame is not significantly deformed, its height is consistent at all locations. That is, at all locations, the height of the material frame is the same as the height of the object to be grasped, or there is a fixed height difference between the heights of the object to be grasped and the height of the object to be grasped. Therefore, if a point cloud of the object to be grasped cannot be obtained in this scenario, 2D color image data of the object and point cloud data of the recognition frame can be collected to proceed to subsequent steps. In other embodiments, when using a camera to capture multiple objects to be grasped at different locations, the point cloud quality of each object to be grasped in the overall point cloud data captured at a specific location may vary. Specifically, it may be impossible to obtain appropriate point cloud data for some objects to be grasped, but appropriate point cloud data for others may be obtained. In this case, the object to be grasped with the better point cloud can be selected as a reference for the other objects to be grasped.

[0106] Point cloud information can be obtained through a 3D industrial camera. A 3D industrial camera is generally equipped with two lenses, each capturing the group of objects to be grasped from different angles. After processing, a three-dimensional image of the object can be displayed. The group of objects to be grasped is placed below the visual sensor, and the two lenses shoot simultaneously. Based on the relative posture parameters of the two images obtained, a universal binocular stereo vision algorithm is used to calculate the X, Y, and Z coordinate values ​​of each point of the object to be filled and the coordinate orientation of each point, and then convert it into point cloud data of the group of objects to be grasped. In specific implementation, laser detectors, visible light detectors such as LEDs, infrared detectors, and radar detectors can also be used to generate point clouds. The present invention does not limit the specific implementation method.

[0107] As an example, a two-dimensional color image corresponding to a three-dimensional object region and a depth map corresponding to the two-dimensional color image can also be acquired along a depth direction perpendicular to the object. The two-dimensional color image corresponds to an image of a planar region perpendicular to a preset depth direction; each pixel in the depth map corresponding to the two-dimensional color image corresponds one-to-one with each pixel in the two-dimensional color image, and the value of each pixel is the depth value of that pixel. In one embodiment, the acquired reference object information can be a point cloud of the reference object or a depth map of the reference object.

[0108] Regarding step S310, this embodiment needs to obtain two-dimensional grasping point information, but the focus is not on the acquisition method, and therefore the specific method of obtaining this information is not limited. Preferably, the method for obtaining grasping point association information in any of the above embodiments can be used to obtain two-dimensional grasping point information.

[0109] For step S320, taking an industrial scenario where a plurality of black glass cosmetic bottles to be grasped are arranged in a material frame as an example, after obtaining the point cloud of the entire group of items, the point cloud of a relatively clear reference object can be further identified in the obtained overall point cloud. For example, the point cloud of the material frame or the point cloud of the bottle mouth with relatively clear point clouds can be identified from the overall point cloud. The identified point cloud is then processed to extract the height information therein as reference information. Although this embodiment takes the example of the missing information of the two-dimensional grasping point information being height information, those skilled in the art will understand that when the missing information is not height information, the reference information may also not be height information.

[0110] In step S330, if a point cloud of the bottle mouth is used, since the bottles in the material frame are of the same type, the height of this point cloud is the same as the bottle mouth height of all bottles. After obtaining the bottle mouth height information, it is combined with the two-dimensional grasping point information to obtain three-dimensional grasping point information. The gripper can then perform grasping based on this three-dimensional grasping point information. If a point cloud of the material frame is used, the height obtained from the point cloud may be the same as or different from that of the bottle. If the heights are different, an adjustment value can be preset based on the height difference between the two. After obtaining the height information of the material frame and the two-dimensional grasping point information, the three-dimensional grasping point information can be determined by combining this adjustment value. For example, if the frame height is 10 cm and the adjustment value is -2 cm, the bottle mouth height can be calculated as 10-2 = 8 cm. This is then combined with the X-axis and Y-axis information of the grasping point to obtain the three-dimensional grasping point information.

[0111] The robots or fixtures mentioned in the above embodiments may include various types of general-purpose fixtures. General-purpose fixtures refer to fixtures with standardized structures and a wide range of applications, such as three-jaw and four-jaw chucks for lathes, and flat-nose pliers and dividing heads for milling machines. For example, according to the clamping power source used by the fixture, the fixture can be divided into manual clamping fixtures, pneumatic clamping fixtures, hydraulic clamping fixtures, gas-liquid clutches, electromagnetic clamps, vacuum clamps, etc., or other bionic devices that can pick up objects. The present invention does not limit the specific type of fixture, as long as it can achieve the object grabbing operation.

[0112] In addition, it should be noted that although each embodiment of the present invention has a specific combination of features, further combinations and cross-combinations of these features between the embodiments are also feasible.

[0113] According to the above embodiments, firstly, the present invention can obtain the grasping point information of the object to be grasped by means of other image data such as color pictures when the point cloud of the object to be grasped cannot be obtained, so that the robot or clamp can directly rely on the grasping point information without the need for the point cloud of the object to grasp the object, effectively solving the object grasping problem in an environment where the point cloud is missing; secondly, the present invention proposes three methods for correcting the mask and obtaining the two-dimensional grasping point information, so that when the accurate object mask cannot be obtained, the inaccurate mask can be corrected to obtain the most accurate mask possible. And obtain the grasping point information, effectively avoiding the inaccurate grasping point caused by inaccurate mask, which in turn leads to the problem of inaccurate grasping or falling during grasping; thirdly, the present invention proposes a method for the robot to automatically convert the input two-dimensional grasping point information into three-dimensional grasping point information. This method can automatically obtain reference information that can convert two-dimensional grasping points into three-dimensional grasping points according to the environmental characteristics of the object to be grasped, and obtain grasping point information based on the reference information for the robot to grasp. This solution enables the robot to obtain complete grasping point information based on existing information when grasping point information is missing, and reduces manual intervention.

[0114] Figure 5 FIG. 4 shows a device for acquiring grasping point information according to another embodiment of the present invention, the device comprising:

[0115] An image acquisition module 400 is used to acquire a non-point cloud image containing the object to be grasped, i.e., to implement step S100;

[0116] A mask generation module 410 is used to process the non-point cloud image to obtain a mask of the object to be grasped, i.e., to implement step S110;

[0117] The mask processing module 420 is used to process the mask of the object to be grasped to obtain grasping point association information, that is, to implement step S120;

[0118] The grasping point information generating module 430 is used to obtain grasping point information for controlling the robot to grasp the object to be grasped based on the grasping point association information, that is, to implement step S130.

[0119] Figure 6 FIG. 4 shows an image data processing device according to another embodiment of the present invention, the device comprising:

[0120] An image data receiving module 500 is used to receive image data containing the object to be processed, i.e., to implement step S200;

[0121] The mask generation module 510 is used to identify the object to be processed from the image data and generate a mask of the object to be processed, that is, to implement step S210;

[0122] The mask processing module 520 is used to perform morphological processing on the generated mask of the object to be processed, i.e., to implement step S220;

[0123] The processing module 530 is used to further process the mask after morphological processing to obtain the corrected mask and / or gripping point association information of the object to be processed, that is, to implement step S230.

[0124] Figure 7 FIG. 4 shows a device for acquiring grasping point information according to another embodiment of the present invention, the device comprising:

[0125] The reference object information acquisition module 600 is used to obtain the reference object information of the object to be grasped, that is, to implement step S300;

[0126] A two-dimensional information acquisition module 610 is used to obtain two-dimensional grasping point information of the object to be grasped, that is, to implement step S310;

[0127] A reference information acquisition module 620 is configured to process the reference object information of the object to be grasped and acquire reference information, wherein the reference information includes information not contained in the two-dimensional grasping point information and is capable of converting the two-dimensional information into three-dimensional information, i.e., for implementing step S320;

[0128] The grasping point information generating module 630 is configured to generate three-dimensional grasping point information of the object to be grasped based on the reference information and the two-dimensional grasping point information of the object to be grasped, that is, to implement step S330 .

[0129] above Figure 5-Figure 7 In the device embodiment shown, only the main functions of the modules are described. All functions of each module correspond to the corresponding steps in the method embodiment. The working principle of each module can also refer to the description of the corresponding steps in the method embodiment, and will not be repeated here. In addition, although the above embodiment defines the correspondence between the functions of the functional modules and the methods, those skilled in the art will understand that the functions of the functional modules are not limited to the above correspondence, that is, a specific functional module can also implement other method steps or part of the method steps. For example, the above embodiment describes the method of grasping point information generation module 630 for implementing step S330, but according to the needs of the actual situation, the grasping point information generation module 630 can also be used to implement the method or part of the method of step S300, S310 or S320.

[0130] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method of any of the above-mentioned embodiments. It should be noted that the computer program stored in the computer-readable storage medium of the embodiment of the present application can be executed by the processor of the electronic device. In addition, the computer-readable storage medium can be a storage medium built into the electronic device, or a storage medium that can be plugged into the electronic device. Therefore, the computer-readable storage medium of the embodiment of the present application has high flexibility and reliability.

[0131] Figure 8 A schematic structural diagram of an electronic device according to an embodiment of the present invention is shown. The electronic device may be a control system / electronic system configured in an automobile, a mobile terminal (e.g., a smart mobile phone), a personal computer (PC, such as a desktop computer or a notebook computer), a tablet computer, a server, etc. The specific embodiment of the present invention does not limit the specific implementation of the electronic device.

[0132] like Figure 8 As shown, the electronic device may include: a processor (processor) 1202 , a communication interface (Communications Interface) 1204 , a memory (memory) 1206 , and a communication bus 1208 .

[0133] in:

[0134] The processor 1202 , the communication interface 1204 , and the memory 1206 communicate with each other via a communication bus 1208 .

[0135] The communication interface 1204 is used to communicate with other devices such as clients or other servers.

[0136] The processor 1202 is configured to execute the program 1210 , and specifically may execute the relevant steps in the above method embodiment.

[0137] Specifically, the program 1210 may include program codes, which include computer operation instructions.

[0138] Processor 1202 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The one or more processors included in an electronic device may be processors of the same type, such as one or more CPUs, or processors of different types, such as one or more CPUs and one or more ASICs.

[0139] The memory 1206 is used to store the program 1210. The memory 1206 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage.

[0140] The program 1210 may be downloaded and installed from a network through the communication interface 1204 and / or installed from a removable medium. When the program is executed by the processor 1202, the processor 1202 may perform various operations in the above method embodiments.

[0141] In summary, the present invention includes:

[0142] A method for obtaining grasping point information, comprising:

[0143] Obtain a non-point cloud image containing the object to be grasped;

[0144] Processing the non-point cloud image to obtain a mask of the object to be grasped;

[0145] Process the mask of the object to be grasped to obtain the grasping point association information;

[0146] Based on the grasping point association information, grasping point information for controlling the robot to grasp the object to be grasped is obtained.

[0147] Optionally, the non-point cloud image includes a color image.

[0148] Optionally, the processing of the non-point cloud image includes processing the non-point cloud image based on deep learning.

[0149] Optionally, the processing of the non-point cloud image based on deep learning includes pre-building a deep learning network and inputting the non-point cloud image into the deep learning network for processing.

[0150] Optionally, the processing of the mask of the object to be grasped includes performing morphological processing on the mask of the object to be grasped.

[0151] Optionally, the grasping point association information includes two-dimensional information of the grasping point.

[0152] Optionally, the grasping point includes a center point of a graspable area of ​​the item.

[0153] Optionally, the grasping point information includes three-dimensional information of the grasping point.

[0154] Optionally, based on the grasping point association information, grasping point information for controlling the robot to grasp the object to be grasped is obtained, including dimension information where the pre-set grasping point association information is missing. Based on the grasping point association information and the dimension information where the pre-set grasping point association information is missing, grasping point information for controlling the robot to grasp the object to be grasped is obtained.

[0155] A device for acquiring grasping point information, comprising:

[0156] An image acquisition module is used to acquire non-point cloud images containing objects to be grasped;

[0157] a mask generation module, configured to process the non-point cloud image to obtain a mask of the object to be grasped;

[0158] The mask processing module is used to process the mask of the object to be grasped to obtain the grasping point association information;

[0159] The grasping point information generation module is used to obtain grasping point information for controlling the robot to grasp the object to be grasped based on the grasping point association information.

[0160] Optionally, the non-point cloud image includes a color image.

[0161] Optionally, the mask generation module processes the non-point cloud image based on deep learning.

[0162] Optionally, a deep learning network is pre-built, and the mask generation module inputs the non-point cloud image into the deep learning network for processing.

[0163] Optionally, the mask processing module performs morphological processing on the mask of the object to be grasped.

[0164] Optionally, the grasping point association information includes two-dimensional information of the grasping point.

[0165] Optionally, the grasping point includes a center point of a graspable area of ​​the item.

[0166] Optionally, the grasping point information includes three-dimensional information of the grasping point.

[0167] Optionally, dimension information of the grasping point association information that is missing is pre-set, and the grasping point information generation module obtains the grasping point information used to control the robot to grasp the object to be grasped based on the grasping point association information and the pre-set dimension information of the grasping point association information that is missing.

[0168] A method for obtaining grasping point information, comprising:

[0169] Get the reference information of the object to be grasped;

[0170] Obtain the two-dimensional grasping point information of the object to be grasped;

[0171] Processing the reference object information of the object to be grasped to obtain reference information, wherein the reference information includes information not included in the two-dimensional grasping point information and is capable of converting the two-dimensional information into three-dimensional information;

[0172] Three-dimensional grasping point information of the object to be grasped is generated based on the reference information and the two-dimensional grasping point information of the object to be grasped.

[0173] Optionally, the reference object information includes a point cloud and / or a depth map of the reference object.

[0174] Optionally, the reference object has a qualified point cloud.

[0175] Optionally, the reference objects include other objects to be grasped and / or material frames.

[0176] Optionally, the two-dimensional grasping point information includes X-axis information and Y-axis information of the grasping point.

[0177] Optionally, the reference information includes Z-axis information.

[0178] Optionally, the three-dimensional grasping point information of the item to be grasped is generated based on the reference information and the two-dimensional grasping point information of the item to be grasped, including a preset reference information adjustment value. After adjusting the reference information using the reference information adjustment value, the three-dimensional grasping point information of the item to be grasped is generated based on the adjusted reference information and the two-dimensional grasping point information of the item to be grasped.

[0179] Optionally, the grasping point includes a center point of a graspable area of ​​the item.

[0180] A device for acquiring grasping point information, comprising:

[0181] A reference object information acquisition module is used to obtain reference object information of the object to be grasped;

[0182] A two-dimensional information acquisition module is used to obtain two-dimensional grasping point information of the object to be grasped;

[0183] a reference information acquisition module, configured to process the reference object information of the object to be grasped, acquire reference information, wherein the reference information includes information not contained in the two-dimensional grasping point information, and is capable of converting the two-dimensional information into three-dimensional information;

[0184] The grasping point information generating module is configured to generate three-dimensional grasping point information of the object to be grasped based on the reference information and the two-dimensional grasping point information of the object to be grasped.

[0185] Optionally, the reference object information includes a point cloud and / or a depth map of the reference object.

[0186] Optionally, the reference object has a qualified point cloud.

[0187] Optionally, the reference objects include other objects to be grasped and / or material frames.

[0188] Optionally, the two-dimensional grasping point information includes X-axis information and Y-axis information of the grasping point.

[0189] Optionally, the reference information includes Z-axis information.

[0190] Optionally, a reference information adjustment value is preset, and the grasping point information generation module uses the reference information adjustment value to adjust the reference information, and then generates three-dimensional grasping point information of the object to be grasped based on the adjusted reference information and the two-dimensional grasping point information of the object to be grasped.

[0191] Optionally, the grasping point includes a center point of a graspable area of ​​the item.

[0192] A method for processing image data, comprising:

[0193] receiving image data containing an object to be processed;

[0194] Identifying an object to be processed from the image data and generating a mask of the object to be processed;

[0195] Performing morphological processing on the generated mask of the object to be processed;

[0196] The morphologically processed mask is further processed to obtain the corrected mask and / or grasping point association information of the object to be processed.

[0197] Optionally, the morphological processing includes morphological dilation processing.

[0198] Optionally, the morphologically processed mask is further processed to obtain a corrected mask and / or gripping point association information of the object to be processed, including:

[0199] Obtain the bounding rectangle of the expanded mask;

[0200] Based on the circumscribed rectangle of the mask, generating an inscribed circle of the circumscribed rectangle;

[0201] A correction mask and / or gripping point association information of the object to be processed is obtained based on the inscribed circle.

[0202] Optionally, obtaining the circumscribed rectangle of the expanded mask includes: generating four corner points of the circumscribed rectangle based on the expanded mask, and then generating the circumscribed rectangle based on the corner points.

[0203] Optionally, the morphologically processed mask is further processed to obtain a corrected mask and / or gripping point association information of the object to be processed, including:

[0204] Use circle detection algorithm to process the dilated mask;

[0205] The correction mask and / or grasping point association information of the object to be processed is obtained based on the processing result of the circle detection algorithm.

[0206] Optionally, the circle detection algorithm includes: a circular Hough transform algorithm, a random Hough transform algorithm and / or a random circle detection algorithm.

[0207] Optionally, the morphologically processed mask is further processed to obtain a corrected mask and / or gripping point association information of the object to be processed, including:

[0208] Based on the pre-saved template of the object to be processed, the expanded mask is processed using a template matching algorithm;

[0209] The correction mask and / or grasping point association information of the object to be processed is obtained based on the processing result of the template matching algorithm.

[0210] Optionally, the matching algorithm includes a shape-based matching algorithm.

[0211] Optionally, identifying the object to be processed from the image data and generating a mask of the object to be processed includes: processing the image data based on deep learning to identify the object to be processed and generating a mask of the object to be processed.

[0212] An image data processing device, comprising:

[0213] An image data receiving module, configured to receive image data containing an object to be processed;

[0214] a mask generation module, configured to identify an object to be processed from image data and generate a mask of the object to be processed;

[0215] A mask processing module, used for performing morphological processing on the generated mask of the object to be processed;

[0216] The processing module is used to further process the mask after morphological processing to obtain the corrected mask and / or grasping point association information of the object to be processed.

[0217] Optionally, the morphological processing includes morphological dilation processing.

[0218] Optionally, the processing module is specifically configured to:

[0219] Obtain the bounding rectangle of the expanded mask;

[0220] Based on the circumscribed rectangle of the mask, generating an inscribed circle of the circumscribed rectangle;

[0221] A correction mask and / or gripping point association information of the object to be processed is obtained based on the inscribed circle.

[0222] Optionally, obtaining the circumscribed rectangle of the expanded mask includes: generating four corner points of the circumscribed rectangle based on the expanded mask, and then generating the circumscribed rectangle based on the corner points.

[0223] Optionally, the processing module is specifically configured to:

[0224] Use circle detection algorithm to process the dilated mask;

[0225] The correction mask and / or grasping point association information of the object to be processed is obtained based on the processing result of the circle detection algorithm.

[0226] Optionally, the circle detection algorithm includes: a circular Hough transform algorithm, a random Hough transform algorithm and / or a random circle detection algorithm.

[0227] Optionally, the processing module is specifically configured to:

[0228] Based on the pre-saved template of the object to be processed, the expanded mask is processed using a template matching algorithm;

[0229] The correction mask and / or grasping point association information of the object to be processed is obtained based on the processing result of the template matching algorithm.

[0230] Optionally, the matching algorithm includes a shape-based matching algorithm.

[0231] Optionally, identifying the object to be processed from the image data and generating a mask of the object to be processed includes: processing the image data based on deep learning to identify the object to be processed and generating a mask of the object to be processed.

[0232] A method for processing image data, comprising:

[0233] receiving image data containing an object to be processed;

[0234] Identifying an object to be processed from the image data and generating a mask of the object to be processed;

[0235] Performing morphological processing on the generated mask of the object to be processed;

[0236] Based on the pre-saved template of the object to be processed, the morphologically processed mask is processed using a template matching algorithm;

[0237] The correction mask and / or grasping point association information of the object to be processed is obtained based on the processing result of the template matching algorithm.

[0238] Optionally, the morphological processing includes morphological dilation processing.

[0239] Optionally, the matching algorithm includes a shape-based matching algorithm.

[0240] Optionally, the grasping point includes a center point of a graspable area of ​​the item.

[0241] Optionally, identifying the object to be processed from the image data and generating a mask of the object to be processed includes: processing the image data based on deep learning to identify the object to be processed and generating a mask of the object to be processed.

[0242] Optionally, the grasping point association information includes two-dimensional information of the grasping point.

[0243] An image data processing device, comprising:

[0244] An image data receiving module, configured to receive image data containing an object to be processed;

[0245] a mask generation module, configured to identify an object to be processed from image data and generate a mask of the object to be processed;

[0246] A mask processing module, used for performing morphological processing on the generated mask of the object to be processed;

[0247] The processing module is used to process the morphologically processed mask using a template matching algorithm based on a pre-saved template of the object to be processed, and obtain the corrected mask and / or grasping point association information of the object to be processed based on the processing result of the template matching algorithm.

[0248] Optionally, the morphological processing includes morphological dilation processing.

[0249] Optionally, the matching algorithm includes a shape-based matching algorithm.

[0250] Optionally, the grasping point includes a center point of a graspable area of ​​the item.

[0251] Optionally, identifying the object to be processed from the image data and generating a mask of the object to be processed includes: processing the image data based on deep learning to identify the object to be processed and generating a mask of the object to be processed.

[0252] Optionally, the grasping point association information includes two-dimensional information of the grasping point.

[0253] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with the embodiment or example is included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0254] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0255] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processing module, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Furthermore, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing it in another suitable manner if necessary, and then storing it in a computer memory.

[0256] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0257] It should be understood that various parts of the embodiments of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0258] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0259] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0260] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0261] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are illustrative and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above implementation methods within the scope of the present application.

Claims

1. A method for processing image data, characterized in that: include: receiving image data containing an object to be processed; Identifying an object to be processed from the image data and generating a mask of the object to be processed; Performing morphological processing on the generated mask of the object to be processed; The morphologically processed mask is further processed to obtain the corrected mask and grasping point association information of the object to be processed; Based on the grasping point association information, acquiring grasping point information for controlling the robot to grasp the object to be grasped; The morphological processing includes morphological expansion processing; The further processing includes shape correction processing.

2. The image data processing method according to claim 1, wherein: The morphologically processed mask is further processed to obtain the corrected mask and grasping point association information of the object to be processed, including: Obtain the bounding rectangle of the expanded mask; Based on the circumscribed rectangle of the mask, generating an inscribed circle of the circumscribed rectangle; The correction mask and grasping point association information of the object to be processed are obtained based on the inscribed circle.

3. The image data processing method according to claim 2, wherein: The obtaining of the circumscribed rectangle of the expanded mask includes: generating four corner points of the circumscribed rectangle based on the expanded mask, and then generating the circumscribed rectangle based on the corner points.

4. The image data processing method according to claim 1, wherein: The morphologically processed mask is further processed to obtain the corrected mask and grasping point association information of the object to be processed, including: Use circle detection algorithm to process the dilated mask; The correction mask and grasping point association information of the object to be processed are obtained based on the processing results of the circle detection algorithm.

5. The image data processing method according to claim 4, wherein: The circle detection algorithm includes: a circular Hough transform algorithm, a random Hough transform algorithm and / or a random circle detection algorithm.

6. The image data processing method according to claim 1, wherein: The morphologically processed mask is further processed to obtain the corrected mask and grasping point association information of the object to be processed, including: Based on the pre-saved template of the object to be processed, the expanded mask is processed using a template matching algorithm; The correction mask and grasping point association information of the object to be processed are obtained based on the processing results of the template matching algorithm.

7. The image data processing method according to claim 6, wherein: The matching algorithm includes a shape-based matching algorithm.

8. The image data processing method according to any one of claims 1 to 7, characterized in that: The identifying the object to be processed from the image data and generating a mask of the object to be processed includes: processing the image data based on deep learning to identify the object to be processed and generating a mask of the object to be processed.

9. An image data processing device, characterized in that: include: An image data receiving module, configured to receive image data containing an object to be processed; a mask generation module, configured to identify an object to be processed from image data and generate a mask of the object to be processed; A mask processing module, used for performing morphological processing on the generated mask of the object to be processed; A processing module, configured to further process the mask after morphological processing to obtain the corrected mask and grasping point association information of the object to be processed; A grasping point information generation module, configured to obtain grasping point information for controlling the robot to grasp an object to be grasped based on the grasping point association information; The morphological processing includes morphological expansion processing; The further processing includes shape correction processing.

10. The image data processing device according to claim 9, wherein: The processing module is specifically used for: Obtain the bounding rectangle of the expanded mask; Based on the circumscribed rectangle of the mask, generating an inscribed circle of the circumscribed rectangle; The correction mask and grasping point association information of the object to be processed are obtained based on the inscribed circle.

11. The image data processing device according to claim 10, wherein: The obtaining of the circumscribed rectangle of the expanded mask includes: generating four corner points of the circumscribed rectangle based on the expanded mask, and then generating the circumscribed rectangle based on the corner points.

12. The image data processing device according to claim 9, wherein: The processing module is specifically used for: Use circle detection algorithm to process the dilated mask; The correction mask and grasping point association information of the object to be processed are obtained based on the processing results of the circle detection algorithm.

13. The image data processing device according to claim 12, wherein: The circle detection algorithm includes: a circular Hough transform algorithm, a random Hough transform algorithm and / or a random circle detection algorithm.

14. The image data processing device according to claim 9, wherein: The processing module is specifically used for: Based on the pre-saved template of the object to be processed, the expanded mask is processed using a template matching algorithm; The correction mask and grasping point association information of the object to be processed are obtained based on the processing results of the template matching algorithm.

15. The image data processing device according to claim 14, wherein: The matching algorithm includes a shape-based matching algorithm.

16. The image data processing device according to any one of claims 9 to 15, characterized in that: The identifying the object to be processed from the image data and generating a mask of the object to be processed includes: processing the image data based on deep learning to identify the object to be processed and generating a mask of the object to be processed.

17. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the image data processing method according to any one of claims 1 to 8 when executing the computer program.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image data processing method according to any one of claims 1 to 8 is implemented.

Images