Component Identification Method and System

Abstract

An embodiment of the present application provides a component recognition method and a component recognition system. The components are arranged in at least two trays, and the at least two trays are stacked. The method includes: obtaining point cloud information of the components; obtaining a depth image of the components according to the point cloud information of the components; performing cutting and layering on the depth image of the components to obtain cutting and layering information of the components; and grouping the components according to the cutting and layering information to obtain grouping information of the components. This method can use three-dimensional machine vision to recognize components, improve the accuracy of component recognition, and enable the robotic arm to accurately grasp the components to be grasped.

Description

Technical Field

[0001] This application relates to the field of robot technology, and particularly to a method and system for identifying components. Background Art

[0002] With the progress of technology, customized production is becoming more and more popular. When performing customized production, due to the large differences in the shapes of components, the robotic arm needs to identify the shapes of components to pick up the correct components.

[0003] However, in the current related technologies, the robotic arm identifies components through two-dimensional machine vision. Since the components involved in customized production are complex and the components may be stacked, there may be misidentifications when using two-dimensional machine vision to identify components. Summary of the Invention

[0004] In view of this, it is necessary to provide a method and system for identifying components. The method and system can use three-dimensional machine vision to identify components, improve the accuracy of component identification, and enable the robotic arm to accurately grasp the components to be grasped.

[0005] An embodiment of this application provides a method for identifying components. The components are arranged in at least two trays, and the at least two trays are stacked. The method includes:

[0006] Obtaining the point cloud information of the components;

[0007] Obtaining the depth image of the components according to the point cloud information of the components;

[0008] Cutting and layering the depth image of the components to obtain the cutting and layering information of the components;

[0009] Grouping the components according to the cutting and layering information to obtain the grouping information of the components.

[0010] According to some embodiments of this application, the method for identifying components further includes:

[0011] Calculating the maximum plane of the components;

[0012] Calculating the grasping points of the components to be grasped according to the normal vector of the maximum plane, and grasping the components to be grasped according to the grasping points.

[0013] According to some embodiments of this application, the calculating the maximum plane of the components includes:

[0014] Obtaining the point cloud information of the components located in the same cutting and layering; wherein the point cloud information of the components can be obtained from the three-dimensional image;

[0015] Obtain the plane with the most z-axis values based on the point cloud information of the component, and set the plane with the most z-axis values as the maximum plane.

[0016] According to some embodiments of the present application, the maximum plane of the component is calculated through the grouping information and the Hessian normal form of the component.

[0017] According to some embodiments of the present application, the included angle between the placements of the components in adjacent layers is 180 degrees.

[0018] According to some embodiments of the present application, the components located in the same cutting layer are different.

[0019] According to some embodiments of the present application, the component recognition method further includes:

[0020] Grasp the components located in the same cutting layer;

[0021] Adjust the grasping angle according to the center point and Euler angles of the maximum plane.

[0022] According to some embodiments of the present application, the component recognition method further includes:

[0023] Grasp the components located in the same cutting layer;

[0024] Adjust the grasping angle according to the center point and Euler angles of the component to be grasped.

[0025] According to some embodiments of the present application, the component recognition method further includes:

[0026] Obtain the curvature value of the component from the depth image of the component;

[0027] Identify the component according to the point cloud information, the grouping information and the curvature value of the component.

[0028] An embodiment of the present application provides a component recognition system. The components are arranged in at least two trays, and the at least two trays are stacked. The system includes:

[0029] A three-dimensional camera for obtaining a three-dimensional image of the component;

[0030] A point cloud calculation module connected to the three-dimensional camera for obtaining the point cloud information of the component according to the three-dimensional image of the component;

[0031] A depth image module connected to the point cloud calculation module for obtaining the depth image of the component according to the point cloud information of the component;

[0032] The grouping module, connected to the depth image module, is used to cut and layer the depth image of the component to obtain the cutting and layering information of the component; the grouping module is also used to group the components according to the cutting and layering information to obtain the grouping information of the components.

[0033] The component recognition method and system provided by the embodiments of the present application can use 3D machine vision to recognize components, improve the accuracy of component recognition, and enable the robotic arm to accurately grasp the components to be grasped. Description of the Drawings

[0034] Figure 1 It is a 2D recognition image of components in the related art.

[0035] Figure 2 It is a schematic diagram of a component recognition system provided by an embodiment of the present application.

[0036] Figure 3 It is a schematic diagram of a component recognition system provided by another embodiment of the present application.

[0037] Figure 4 It is a schematic diagram of a component recognition system provided by another embodiment of the present application.

[0038] Figure 5 It is a 3D image of a component provided by an embodiment of the present application.

[0039] Figure 6 It is a cutting image of a component provided by an embodiment of the present application.

[0040] Figure 7 It is a grouping image of components provided by an embodiment of the present application.

[0041] Figure 8 It is a flowchart of a component recognition method provided by an embodiment of the present application.

[0042] Description of the Main Element Symbols

[0043] Component recognition system 10;

[0044] Pallet 20;

[0045] Production line 30;

[0046] Robotic arm 100;

[0047] 3D camera 200;

[0048] Point cloud computing module 300;

[0049] Depth image module 400;

[0050] Grouping module 500;

[0051] Plane calculation module 510;

[0052] Grasping calculation module 520;

[0053] Separation module 530;

[0054] The first group of components 610;

[0055] The second group of components 620. Specific implementation mode

[0056] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments.

[0057] It should be noted that "at least one" in the embodiments of the present application means one or more, and multiple means two or more. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application.

[0058] It should be noted that in the embodiments of the present application, terms such as "first" and "second" are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order. The features defined with "first" and "second" may explicitly or implicitly include one or more of the said features. In the description of the embodiments of the present application, words such as "exemplary" or "for example" are used to represent examples, illustrations or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Exactly speaking, using words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner.

[0059] Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts are within the scope of protection of the present application.

[0060] With the progress of technology, customized production is becoming more and more popular. When carrying out customized production, there are great differences in the shapes of components, and the robotic arm needs to identify the shapes of components to pick up the correct components.

[0061] In the current related technologies, parts are recognized through two-dimensional machine vision. Since the parts involved in customized production are complex and the parts may be stacked, and the trays for placing the parts in the factory are usually transparent, when using a two-dimensional camera for image recognition, it may misrecognize the parts on other layers, resulting in the robotic arm being unable to grasp the correct parts, which may cause the robotic arm to report an error and may pose a safety hazard.

[0062] The following describes the part recognition system and the part recognition method provided by the embodiments of the present application in conjunction with the accompanying drawings.

[0063] Figure 1 is a two-dimensional recognition image of parts in the related technology. As Figure 1 shown, the parts are placed above the trays, and the trays are stacked between different trays. It can be understood that when the tray is transparent, it is difficult to use two-dimensional images for recognition, and it may misrecognize the parts in other layer trays, thereby causing the robotic arm to be unable to correctly grasp the parts.

[0064] Figure 2 is a schematic diagram of a part recognition system 10 provided by an embodiment of the present application. As Figure 2 shown, the part recognition system 10 includes a robotic arm 100 and a three-dimensional camera 200, and the robotic arm 100 is connected to the three-dimensional camera 200. Figure 2 The tray 20 and the production line 30 are also shown in

[0065] In the embodiment of the present application, the connection method between the robotic arm 100 and the three-dimensional camera 200 is a fixed connection. When the robotic arm 100 moves above the tray 20, it will drive the three-dimensional camera 200 to move synchronously above the tray 20, and the tray 20 is used to accommodate the parts. The robotic arm 100 is used to move the parts from the tray 20 to the production line 30.

[0066] In the embodiment of the present application, the three-dimensional camera 200 includes a three-dimensional lidar (3D Rader), a three-dimensional lidar (3D Lidar), a stereo camera (Stereo Camera), and a time-of-flight camera (Time-of-flight Camera). The three-dimensional camera 200 is used to capture three-dimensional images.

[0067] In the embodiment of the present application, the placement angle of the parts on adjacent layer trays 20 can be set to 180 degrees. It can be understood that in the embodiment of the present application, by setting the placement angle of the parts on adjacent layer trays 20 to 180 degrees, the part recognition system 10 can easily distinguish the parts on different layers and avoid the part recognition system 10 from misrecognizing the parts on other layer trays 20.

[0068] In the embodiments of the present application, the components within the component group may be different. It can be understood that the components within the same tray 20 may be different. The component identification system 10 may control the robotic arm 100 to pick up components from different trays 20.

[0069] Figure 3 It is a schematic diagram of the component identification system 10 provided by another embodiment of the present application. As Figure 3 shown, compared with Figure 2 , the component identification system 10 further includes: a point cloud computing module 300, a depth image module 400, and a grouping module 500. The three-dimensional camera 200 is connected to the robotic arm 100, the point cloud computing module 300 is connected to the three-dimensional camera 200, the depth image module 400 is connected to the point cloud computing module 300, the grouping module 500 is connected to the depth image module 400, and the robotic arm is connected to the grouping module 500.

[0070] In the embodiments of the present application, the three-dimensional camera 200 is connected to the robotic arm 100. The three-dimensional camera 200 can move with the robotic arm 100 to adjust the shooting position and angle of the three-dimensional camera 200. It can be understood that the connection method between the three-dimensional camera 200 and the robotic arm 100 may be a fixed connection or a movable connection that can move within a certain range, which is not limited herein. The three-dimensional camera 200 is used to shoot three-dimensional images of components. The three-dimensional images include images of components and other facilities on the production line 30.

[0071] It can be understood that before shooting the three-dimensional images of the components, the three-dimensional camera 200 first adjusts its position to make the three-dimensional camera 200 as parallel to the components as possible, so as to reduce the computational complexity during subsequent position calculations.

[0072] In the embodiment of the present application, the point cloud computing module 300 is configured to obtain the point cloud information of the component according to the three-dimensional image of the component captured by the three-dimensional camera 200. It can be understood that the three-dimensional camera 200 first captures a three-dimensional reference photo, and the point cloud computing module 300 obtains the point cloud information of the component according to the reference photo. The point cloud information includes point cloud data (Point Cloud Data). The point cloud data is a set of vectors in a three-dimensional coordinate system, and the point cloud information is the set of all the point cloud data. Among them, the three-dimensional coordinate system includes (x, y, z, Rx, Ry, Rz) coordinates, where x, y, z respectively represent the x-axis, y-axis and z-axis coordinates, and Rx, Ry, Rz respectively represent the rotation angles of the component around the x, y, z axes, that is, the Euler angles (Eular). The point cloud computing module 300 inputs the point cloud information to the depth image module 400.

[0073] In the embodiment of the present application, the depth image module 400 is configured to generate a depth image of the component according to the point cloud information of the component. The depth image module 400 sorts all the (x, y, z) coordinate points of the components, selects the point closest to the three-dimensional camera 200, sets this point as the reference point, and sets the z-axis value of the reference point as Z1 to generate reference point information. The depth image module 400 re-adjusts the coordinates in the point cloud information with the reference point set as the origin of the coordinate axis to form a depth image. The depth image module 400 transmits the depth image of the component and the information of the reference point to the grouping module 500.

[0074] In the embodiment of the present application, please refer to Figure 7 , Figure 7The component grouping image provided by an embodiment of the present application. It can be understood that the grouping module 500 is used to cut and layer the depth image of the components to obtain component grouping. Specifically, the depth image module 400 first obtains the three-dimensional coordinates (x, y, z) of the components. Subsequently, the depth image module 400 sets the depth D according to the interval between the z-axes of the component trays 20. It can be understood that in industrial production, components are usually placed on the trays 20, and different trays 20 are often stacked, so the depth D can also be set according to the thickness of the tray 20 or the interval between the trays 20. Taking the z-axis value of the component as Z1 as an example, the grouping module 500 selects the components with z-axis values between [Z1, Z1 + D] according to the depth image of the components, the information of the reference point, and the depth D. It can be understood that the depth D can also be set according to the thickness of the tray 20 and the placement rules of the components. For example, when the depth of the tray 20 is 50 cm, the depth D of the depth image can be set to 50 cm or 55 cm, etc., and the present application does not limit this here. The grouping module 500 obtains all the components with z-axis values between [Z1, Z1 + D], and transmits the information of the components to the robotic arm 100.

[0075] In an embodiment of the present application, the robotic arm 100 is used to grasp the components to be grasped according to the component grouping. It can be understood that the robotic arm 100 can sort the components and grasp the corresponding components according to the sorting. After all the components are grasped, the robotic arm 100 can move the tray 20 to expose the components on the next layer of the tray 20, and drive the 3D camera 200 to a position parallel to the components to start the next round of component recognition and grasping.

[0076] Figure 4 It is a schematic diagram of the component recognition system 10 provided by another embodiment of the present application. Compared with Figure 3 the grouping module 500 further includes: a plane calculation module 510, a grasping calculation module 520, and a separation module 530.

[0077] In an embodiment of the present application, the plane calculation module 510 is arranged inside the grouping module 500 and is used to calculate the maximum plane of the component grouping according to the component grouping and the Hessian Normal Form. The maximum plane is the plane that contains the most components.

[0078] In the embodiment of the present application, the plane calculation module 510 can perform maximum plane calculation by using the built-in Application Programming Interface (API) in the Point Cloud Library (PCL). It can be understood that the plane calculation module 510 first calculates the plane with the most z-axis values based on the components whose z-axis values are all within [Z1, Z1 + D] to obtain a maximum plane. The plane calculation module 510 then re-establishes a new three-dimensional coordinate system based on the maximum plane and sets the maximum plane coordinate values (x1, y1, z1) of the component according to the new three-dimensional coordinate system. The plane calculation module 510 transmits the maximum plane coordinate values (x1, y1, z1) of the component to the grasping calculation module 520.

[0079] In the embodiment of the present application, the grasping calculation module 520 calculates the maximum plane Euler angles (Rx1, Ry1, Rz1) of the component according to the maximum plane coordinate values (x1, y1, z1) of the component and the normal vector of the maximum plane. Among them, the maximum plane coordinate values of the component and the maximum plane Euler angles of the component can form a component grasping point. The grasping calculation module 520 transmits the component grasping point to the robotic arm 100. The robotic arm 100 then adjusts the grasping angle according to the component grasping point (i.e., the maximum plane coordinate values and the maximum plane Euler angles of the component) to improve the grasping accuracy of the robotic arm 100.

[0080] It can be understood that since there may be multiple components placed in the tray 20, the plane of the tray 20 is not absolutely horizontal. Therefore, if the coordinates of the components are calculated based on the assumption that the tray 20 is horizontal, there will be errors, resulting in the robotic arm 100 being unable to accurately grasp the components. Therefore, using the plane calculation module 510 to calculate the maximum plane and calculating the grasping points of the components according to the grasping calculation module 520 can enable the robotic arm 100 to grasp the components more accurately.

[0081] In the embodiment of the present application, the grouping module 500 further includes a separation module 530. The separation module 530 is connected to the grasping calculation module 520 and is used to obtain the curvature values in the depth image of the component. The separation module 530 sorts the curvature values and differentiates the components according to the point cloud size and curvature values corresponding to the components, and separates the images of the components until all similar components are separated. It can be understood that since the surface features of the components will bring different curvature changes, the components and their contours can be found by matching the point cloud size and curvature features. It can be understood that the separated components can be the same or different.

[0082] In an embodiment of the present application, after the grouping module 500 calculates the positions and contours of all components, it recalculates the plane center and normal vector of the average coordinate values of the components after grouping by the grouping module 500, and sends the plane center coordinate value and normal vector information to the robotic arm 100, so that the robotic arm 100 moves to the plane center.

[0083] It can be understood that when the tray 20 is made of a transparent material and the components are all placed on the transparent tray 20, the transparent material of the tray 20 will cause a large error in the calculation of the point cloud data in the depth image of the components after multiple calculations. Therefore, the grouping module 500 can also obtain the point cloud information from the 3D camera 200, recalculate the coordinate values of the components according to the point cloud information, and perform center calculation according to the coordinate values of the components. It can be understood that the center calculation is the center position of the components calculated according to the coordinate values of the components. The robotic arm 100 adjusts the grasping position and angle according to the center position and Euler angle of the components to accurately grasp the components.

[0084] Please refer to Figures 5 to 6 , Figure 5 which is a three-dimensional image of a component provided by an embodiment of the present application. Figure 6 which is a cutting image of a component provided by an embodiment of the present application.

[0085] In an embodiment of the present application, as Figures 5 to 6 shown, when the tray 20 is transparent, the component recognition system 10 provided by the embodiment of the present application can accurately recognize the position and contour of the component, and recognize the coordinate value and Euler angle of the component according to the position and contour of the component, so that the robotic arm 100 can adjust the grasping position and angle according to the coordinate value and Euler angle of the component, and achieve accurate grasping of the component.

[0086] In an embodiment of the present application, the component recognition system 10 groups the components. Among them, the first group of components 610 and the second group of components 620 are components located in two different trays. Since the included angle between the components placed between adjacent trays 20 is 180 degrees, the component recognition system 10 can accurately recognize the first group of components 610 and the second group of components 620. When the component recognition system 10 controls the robotic arm 100 to grasp the components, it will not attempt to grasp the second group of components 620, avoiding errors reported by the robotic arm 100 and potential safety hazards.

[0087] Figure 8 is a flowchart of a component recognition method provided by an embodiment of the present application. As Figure 8 shown, the component recognition method is applied to the component recognition system 10.

[0088] S100: Obtain the three-dimensional image of the component part.

[0089] In the embodiment of the present application, the three-dimensional image of the component part from the three-dimensional camera 200 can be obtained through the point cloud computing module 300. The three-dimensional image includes a reference photo, and the reference photo is the three-dimensional image taken by the three-dimensional camera 200 when it is parallel to the tray 20 for placing the component part.

[0090] S200: Obtain the point cloud information of the component part according to the three-dimensional image of the component part.

[0091] In the embodiment of the present application, the point cloud information of the component part can be obtained by the point cloud computing module 300 according to the reference photo. The point cloud information includes point cloud data, and the point cloud data includes the points of the component part and their (x, y, z, Rx, Ry, Rz) coordinates.

[0092] S300: Obtain the depth image of the component part according to the point cloud information of the component part.

[0093] In the embodiment of the present application, the depth image module 400 is used to obtain the depth image of the component part according to the point cloud information. Specifically, the depth image module 400 sorts all the (x, y, z) coordinate points of the component part, selects the point closest to the three-dimensional camera 200, sets this point as the reference point, and sets the z-axis value of the reference point as Z1. The depth image module 400 transmits the generated depth image of the component part and the information of the reference point to the grouping module 500.

[0094] S400: Cut and layer the depth image of the component part to obtain the cut and layer information of the component part.

[0095] In the embodiment of the present application, the grouping module 500 is used to cut and layer the depth image of the component part to obtain the grouping of the component part. Specifically, the depth image module 400 first sets the depth D according to the thickness information of the tray 20. The grouping module 500 selects the component parts with z-axis values between [Z1, Z1 + D] according to the depth image of the component part, the information of the reference point, and the depth D. It can be understood that the depth D can be set according to the thickness of the tray 20 and the placement rule of the component part. For example, the depth D can be set to 50 cm or 55 cm, etc., and the present application does not limit this here. The grouping module 500 obtains all the component parts with z-axis values between [Z1, Z1 + D], and transmits the information of the component parts to the robotic arm 100.

[0096] S500: Group the component parts according to the cut and layer information to obtain the grouping information of the component parts.

[0097] In an embodiment of the present application, the grouping module 500 is configured to select components with z-axis values between [Z1, Z1 + D] according to the depth image of the component, the information of the reference point, and the depth D, so as to complete the grouping of the components.

[0098] In an embodiment of the present application, the robotic arm 100 is configured to grasp the components to be grasped according to the component grouping. It can be understood that the robotic arm 100 can sort the components and grasp the corresponding components according to the sorting. After all the components are grasped, the robotic arm 100 can move the tray 20 to expose the components on the next layer of the tray 20, and drive the 3D camera 200 to a position parallel to the components to start the next round of component recognition.

[0099] Those of ordinary skill in the art in this technical field should recognize that the above embodiments are only used to illustrate the present application, rather than to limit the present application. As long as appropriate changes and variations made to the above embodiments fall within the scope of the spirit of the present application, they fall within the scope of protection required by the present application.

Claims

1. A method for identifying components, where the components are arranged in at least two trays, and the at least two trays are stacked. Characterized in that, The method includes: Obtaining the point cloud information of the components; Obtaining the depth image of the components according to the point cloud information of the components; Cutting and stratifying the depth image of the components to obtain the cutting and stratifying information of the components; Grouping the components according to the cutting and stratifying information to obtain the grouping information of the components; Calculating the maximum plane of the components, where the maximum plane is the plane that contains the most components; Calculating the grasping points of the components to be grasped according to the coordinate values of the maximum plane of the components and the normal vector of the maximum plane, so as to grasp the components to be grasped according to the grasping points; Grasping the components located in the same cutting and stratifying layer; Adjusting the grasping angle according to the center point and Euler angle of the maximum plane.

2. The method for identifying components according to claim 1, Characterized in that, The calculating the maximum plane of the components includes: Obtaining the point cloud information of the components located in the same cutting and stratifying layer; where the point cloud information of the components located in the same cutting and stratifying layer can be obtained from the three-dimensional image; Obtaining the plane with the most z-axis values according to the point cloud information of the components located in the same cutting and stratifying layer, and setting the plane with the most z-axis values as the maximum plane.

3. The method for identifying components according to claim 1, Characterized in that, The maximum plane of the components is calculated through the grouping information of the components and the Hessian normal form.

4. The method for identifying components according to claim 1, Characterized in that, The included angle between the components in adjacent stratifications is 180 degrees.

5. The method for identifying components according to claim 3, Characterized in that, The components located in the same cutting and stratifying layer are different.

6. The method for identifying components according to claim 1, Characterized in that, The method for identifying components further includes: Obtaining the curvature value of the components from the depth image of the components; Identifying the components according to the point cloud information, the grouping information and the curvature value of the components.

7. A component identification system, where the components are arranged in at least two trays, and the at least two trays are stacked. Characterized in that, It includes: A three-dimensional camera for obtaining the three-dimensional image of the components; A point cloud calculation module connected to the three-dimensional camera for obtaining the point cloud information of the components according to the three-dimensional image of the components; A depth image module connected to the point cloud calculation module for obtaining the depth image of the components according to the point cloud information of the components; A grouping module connected to the depth image module for cutting and stratifying the depth image of the components to obtain the cutting and stratifying information of the components; The grouping module is further configured to group the components according to the cutting and layering information to obtain the grouping information of the components; calculate the maximum plane of the components, where the maximum plane is the plane containing the most components; calculate the grasping points of the components to be grasped according to the coordinate values of the maximum plane of the components and the normal vector of the maximum plane, so as to grasp the components to be grasped according to the grasping points; grasp the components located in the same cutting layer; and adjust the grasping angle according to the center point and Euler angle of the maximum plane.

Images