Robot system and control method

Abstract

The present invention provides a robot system and a control method capable of obtaining the thickness of an object with high accuracy. The robot system comprises: a photographing unit mounted on the robot, which photographs a two-dimensional image of the object; an image processing unit, which obtains distance information of the object based on the two-dimensional image; a distance image generating unit, which generates a distance image based on the distance information; and a thickness calculating unit, which calculates the thickness of the object based on the distance image, wherein the photographing unit photographs the two-dimensional image of the object regardless of the positional relationship between the side surface of the object and the photographing unit.

Description

Technical Field

[0001] The invention relates to a robot system and a control method. Background Art

[0002] In the past, various technologies have been proposed to measure the distance to an object having a three-dimensional shape (for example, see Patent Document 1). The three-dimensional measuring device described in Patent Document 1 includes: a shape measuring unit that obtains distance information for each point on the surface of the object existing in the field of view; and an indicator that is exposed and arranged on a sensor housing and indicates the positional relationship between the surface of the object calculated based on the distance information obtained by the shape measuring unit and a reference point in the three-dimensional measuring device.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2019-207152 Summary of the invention

[0006] Problems to be solved by the invention

[0007] When a robot takes out stacked objects such as corrugated cardboard boxes and stacks them in another place (pallet, etc.), a robot system including the robot needs to accurately obtain the thickness of the object. Therefore, it is desired to obtain the thickness of the object with high accuracy.

[0008] Means for solving problems

[0009] The robot system disclosed in the present invention comprises: a photographing unit, which photographs a two-dimensional image of an object; an image processing unit, which obtains distance information of the object based on the two-dimensional image; a distance image generating unit, which generates a distance image based on the distance information; and a thickness calculating unit, which calculates the thickness of the object based on the distance image, and the photographing unit photographs the two-dimensional image of the object regardless of the positional relationship between the side surface of the object and the photographing unit.

[0010] The control method of the robot system disclosed in the present invention comprises: a step of capturing a two-dimensional image of an object; a step of obtaining distance information of the object based on the two-dimensional image; a step of generating a distance image based on the distance information; and a step of calculating the thickness of the object based on the distance image, wherein a capturing unit that captures the two-dimensional image of the object captures the two-dimensional image of the object regardless of the positional relationship between the side surface of the object and the capturing unit.

[0011] Effects of the Invention

[0012] According to the present invention, the thickness of an object can be acquired with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 An overview of the robot system according to this embodiment is shown.

[0014] Figure 2 This section shows the structure of the imaging unit.

[0015] Figure 3 This section explains the process of obtaining the parallax.

[0016] Figure 4 Indicates the processing for the range image.

[0017] Figure 5 An example showing a distance image and the thickness of an object.

[0018] Figure 6 This shows a distance image containing five objects stacked together.

[0019] Figure 7 This example shows an example in which the area of ​​a blob of an object is set as a feature amount and the set feature amount is detected.

[0020] Figure 8 This shows an example in which the minor axis length of a plaque of an object is set as a feature amount and the set feature amount is detected.

[0021] Fig. 9 is a flowchart showing the processing of the robot system. DETAILED DESCRIPTION

[0022] Hereinafter, an example of an embodiment of the present invention will be described.

[0023] Figure 1 The schematic diagram of the robot system 100 according to the present embodiment is shown in FIG. Figure 1 As shown, the robot system 100 includes a robot 10, an imaging unit 11, a robot control device 20, and an image control device 30. The robot system 100 performs operations such as processing an object W by the robot 10 based on an image captured by the imaging unit 11.

[0024] A hand or a tool is mounted on the front end of the arm of the robot 10. The robot 10 performs operations such as processing the object W under the control of the robot control device 20. In addition, the imaging unit 11 is mounted on the front end of the arm of the robot 10. In addition, the imaging unit 11 may not be mounted on the robot 10, for example, it may be set at a predetermined position.

[0025] The imaging unit 11 is mounted on the front end portion of the arm of the robot 10. The imaging unit 11 captures a distance image and a two-dimensional image of the object W.

[0026] Figure 2 The configuration of the imaging unit 11 is shown.

[0027] like Figure 2 As shown, the imaging unit 11 includes an internal camera 111 and a projector 112. The imaging unit 11 captures a two-dimensional image of the object W. The two-dimensional image is an image composed of grayscale.

[0028] The internal camera 111 has two cameras and captures an object W (subject) irradiated with patterned light such as a stripe pattern by the projector 112. The relative positions of the two cameras are predetermined, and the optical axes of the two cameras are arranged parallel to each other.

[0029] The projector 112 functions as a light source and irradiates the object W with pattern light such as a stripe pattern.

[0030] return Figure 1 The robot control device 20 is connected to the robot 10 to control the movement of the robot 10.

[0031] The image control device 30 is connected to the imaging unit 11 and controls the imaging unit 11. In addition, the image control device 30 executes a predetermined process on the image captured by the imaging unit 11.

[0032] Furthermore, the image control device 30 includes an image processing unit 301 , a distance image generating unit 302 , an image recognizing unit 303 , and a thickness calculating unit 304 .

[0033] The image processing unit 301 acquires distance information of the object W based on the two-dimensional image of the object W captured by the imaging unit 11 .

[0034] The distance image generating unit 302 generates a distance image based on the distance information acquired by the image processing unit 301 .

[0035] The thickness calculation unit 304 calculates the thickness of the object W based on the generated distance image.

[0036] Figure 3 This is used to explain the process of finding the parallax. Figure 3 In the figure, images 1 and 2 are two-dimensional images captured by two internal cameras 111.

[0037] The image processing unit 301 searches for an image in the image 2 that is the same as the small area (image range) in the image 1 , and calculates the parallax between the image 1 and the image 2 .

[0038] The difference in pixel position between the two images 1 and 2 is called parallax. The longer the distance from the internal camera 111 to the object W is, the smaller the parallax is. Conversely, the closer the distance from the internal camera 111 to the object W is, the larger the parallax is.

[0039] For example, in Figure 3 In the example, the position of the small area in image 1 is (X=200, Y=150), and the position of the small area in image 2 corresponding to image 1 is (X=200, Y=300), so the parallax for Y is 300-150=150. In addition, Figure 3 X and Y in represent pixels of the internal camera 111.

[0040] The image processing unit 301 converts the parallax between the two two-dimensional images into a distance to obtain distance information.

[0041] The conversion from parallax to distance is performed using the following equation (1).

[0042] Z=B×F / S (1)

[0043] Here, Z is the distance (mm), B is the distance between two cameras (mm), F is the focal length (mm), and S is the parallax (mm).

[0044] Then, the distance image generation unit 302 generates a distance image using the acquired distance information. That is, the distance image is obtained by imaging the distance information from the imaging unit 11 (internal camera 111) to the object W. Therefore, in the distance image, a place close to the imaging unit 11 appears bright on the image, and conversely, a place far from the imaging unit 11 appears dark on the image.

[0045] Figure 4 Indicates the processing for the range image. Figure 5 An example showing a distance image and the thickness of an object.

[0046] The image recognition unit 303 connects three-dimensional points that are close to each other in the range image, and gives features to the set of three-dimensional points by area, angle, etc. The image recognition unit 303 detects the object W by detecting the set of three-dimensional points (plaque) given the features. Then, the thickness calculation unit 304 calculates the thickness of the detected object W.

[0047] exist Figure 4 In the example shown, in the distance image M1 , the search range is set using the plaque major axis length and the plaque minor axis length as feature quantities, thereby detecting 11 objects.

[0048] Figures 6 to 8 This is used to explain the process of detecting an object from a plaque. Figures 6 to 8 In the example shown, the detection process of five stacked objects B1 to B5 will be described.

[0049] Figure 6First, when teaching the detection of an object, the image recognition unit 303 selects one or more feature quantities (e.g., major axis length, minor axis length, etc.) so that all objects can be detected, and sets a search range for searching one or more feature quantities according to the operation of the operation unit (not shown) by the teaching staff.

[0050] The image recognition unit 303 detects the plaque existing in the set search range as the object. In order to reduce false detection and non-detection, the image recognition unit 303 repeatedly performs detection according to the operation of the operation unit by the instructor, thereby adjusting the search range for searching the feature amount.

[0051] Figure 7 This example shows an example in which the area of ​​a patch of an object is set as a feature quantity and the set feature quantity is detected. Figure 7 In the example shown, images M4 and M5 are shown as examples of executing object detection. In images M4 and M5, since the areas of patches of all objects are within the set search range, the image recognition unit 303 can detect all objects B1 to B5.

[0052] Figure 8 This shows an example in which the minor axis length of the plaque of the object is set as a feature quantity and the set feature quantity is detected. Figure 8 In the example shown, images M6 and M7 are shown as examples of executing object detection.

[0053] In image M6, since the minor axis lengths of the plaques of all objects are within the set search range, the image recognition unit 303 can detect all objects B1 to B5. On the other hand, in image M7, since the minor axis length of the plaque of object B2 is not within the set search range, the image recognition unit 303 cannot detect object B2.

[0054] Thus, when the search range is inappropriate, the image recognition unit 303 may misdetect or fail to detect the object. In the above example, the area and the minor axis length of the object are used as the search range of the feature quantity to be adjusted, but the feature quantity is not limited to this. The feature quantity may also be, for example, the major axis length, angle, etc.

[0055] The thickness calculation unit 304 may calculate the thickness of the object W recognized by the image recognition unit 303 before the robot 10 takes out the object W. In addition, when a plurality of objects W are placed on a pallet for placing goods, the thickness calculation unit 304 may calculate the thickness of the plurality of objects W for each pallet and calculate the average value of the thicknesses of the plurality of objects W to thereby calculate the thickness of the object W.

[0056] In addition, the thickness calculation unit 304 can calculate the thickness of the object W recognized by the image recognition unit 303 using, for example, the following methods (1) to (4).

[0057] (1) The length of the major axis / minor axis of the plaque of the object W is used.

[0058] (2) Using the average value of the major axis / minor axis lengths of multiple plaques.

[0059] (3) Use the length between the centers of gravity of adjacent patches in the height direction.

[0060] (4) The average value of the lengths between the centroids of a plurality of patches adjacent to each other in the height direction is used.

[0061] By performing such processing, Figure 5 As shown in FIG. 1 , even when a plurality of corrugated cardboard boxes or the like are stacked as the object W, the thickness calculation unit 304 can calculate the thickness of the object W. Figure 5 In particular, the calculation Figure 1 The thickness of the object W at the four locations shown.

[0062] Fig. 9 : is a flowchart showing the processing of the robot system 1 .

[0063] In step S1, the imaging unit 11 captures a two-dimensional image of the object W. Here, the imaging unit 11 moves along with the movement of the robot 10, and captures a two-dimensional image of the object regardless of the positional relationship between the side surface of the object W and the imaging unit 11. That is, the imaging unit 11 captures a two-dimensional image of the object regardless of whether the side surface of the object W faces the imaging unit 11 directly or not.

[0064] In step S2 , the image processing unit 301 acquires distance information of the object W based on the two-dimensional image.

[0065] In step S3 , the distance image generating unit 302 generates a distance image based on the distance information.

[0066] In step S4 , the image recognition unit 303 recognizes the object W based on the range image.

[0067] In step S5 , the thickness calculation unit 304 calculates the thickness of the object W based on the distance image in which the object W is recognized.

[0068] As described above, the robot system 100 of the present embodiment includes: the imaging unit 11 that captures a two-dimensional image of the object W; the image processing unit 301 that obtains distance information of the object W based on the two-dimensional image; the distance image generating unit 302 that generates a distance image based on the distance information; and the thickness calculating unit 304 that calculates the thickness of the object W based on the distance image. The imaging unit 11 moves along with the movement of the robot 10, and captures a two-dimensional image of the object W regardless of the positional relationship between the side surface of the object W and the imaging unit 11.

[0069] The prior art uses a three-dimensional sensor to measure the thickness of an object based on the difference between the three-dimensional information (e.g., height, shape, etc.) of a table or floor serving as a reference and the three-dimensional information of an object (e.g., stacked goods). Therefore, the prior art can measure the thickness of an object only when the objects do not overlap. In addition, the prior art requires the three-dimensional sensor to face the upper surface of the object.

[0070] On the other hand, the robot system 100 of the present embodiment calculates the thickness of the object W based on the distance image, and therefore can obtain the thickness of the object W with high accuracy regardless of whether the side surface of the object W faces the imaging unit 11 or not. In addition, unlike the prior art, the robot system 100 can obtain the thickness of the object W even when the objects W are stacked. In addition, the robot system 100 calculates the thickness of the object W based on the distance image, and therefore does not need to measure the reference position as in the prior art.

[0071] In addition, the thickness calculation unit 304 calculates the thickness of the object W by calculating the average value of the thicknesses of the plurality of objects W based on the distance image. Thus, the robot system 100 can obtain the thickness of the object W with high accuracy even when objects W having substantially the same shape (such as corrugated cardboard boxes) are stacked.

[0072] In addition, the thickness calculation unit 304 connects three-dimensional points that are close to each other in the range image, adds features to the set of three-dimensional points, detects the set of three-dimensional points (patch), and calculates the thickness of the detected object W. Thus, the robot system 100 can obtain the thickness of the object W with high accuracy regardless of whether the side surface of the object W faces the imaging unit 11 directly or not.

[0073] The above is a description of the embodiments of the present invention. The robot system 100 can be implemented by hardware, software, or a combination thereof. In addition, the control method performed by the robot system 100 can also be implemented by hardware, software, or a combination thereof. Here, implementation by software means that the program is read in and executed by a computer.

[0074] The program can be stored using various types of non-transitory computer-readable media and provided to the computer. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., hard disk drives), optical magnetic recording media (e.g., optical magnetic disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (random access memory)).

[0075] In addition, the above-mentioned embodiments are preferred embodiments of the present invention, but the scope of the present invention is not limited to the above-mentioned embodiments, and the present invention can be implemented in various modified forms within the scope not departing from the gist of the present invention.

[0076] Description of Reference Numerals

[0077] 1Robotic system

[0078] 10. Robot

[0079] 11 Filming Department

[0080] 20Robot control device

[0081] 30 Image control unit

[0082] 301 Image Processing Department

[0083] 302 distance image generation unit

[0084] 303 Image Recognition Department

[0085] 304 thickness calculation unit.

Claims

1. A robot system, characterized in that: have: an imaging unit for capturing a two-dimensional image of an object; an image processing unit for acquiring distance information of the object based on the two-dimensional image; a distance image generating unit configured to generate a distance image based on the distance information; an image recognition unit configured to recognize the object based on the distance image; as well as a thickness calculation unit that calculates the thickness of the object recognized by the image recognition unit based on the distance image, the imaging unit captures the two-dimensional image of the object regardless of the positional relationship between the side surface of the object and the imaging unit, The image recognition unit selects one or more feature quantities based on the operation of the instructor to detect the object, sets a search range for searching the one or more feature quantities, and adjusts the search range for searching the feature quantities by repeatedly detecting the object.

2. The robot system according to claim 1, characterized in that: The thickness calculation unit calculates the thickness of the object by calculating an average value of thicknesses of the plurality of objects based on the distance image.

3. The robot system according to claim 1 or 2, characterized in that: The thickness calculation unit connects three-dimensional points that are close to each other in the range image, adds features to a set of the three-dimensional points, detects the object by detecting the set of the three-dimensional points, and calculates the thickness of the detected object.

4. A control method for a robot system, characterized in that: have: The step of capturing a two-dimensional image of an object; A step of obtaining distance information of the object based on the two-dimensional image; A step of generating a distance image based on the distance information; a step of recognizing the object based on the range image; as well as a step of calculating the thickness of the object based on the distance image, the imaging unit that captures the two-dimensional image of the object captures the two-dimensional image of the object regardless of the positional relationship between the side surface of the object and the imaging unit, The step of identifying the object includes: selecting one or more feature quantities according to the operation of the teaching staff so as to detect all of the objects, and setting a search range for retrieving the one or more feature quantities; repeatedly detecting the object according to the operation of the teaching staff, thereby adjusting the search range for retrieving the feature quantities.

Images