Robot system and imaging method

Abstract

Provided are a robot system and an image capturing method capable of solving problems that may occur when a program for image capturing by an auto-focus camera is used under different conditions. A robot system comprises a camera with an auto-focus lens, and a robot for performing work on the basis of an image capturing result of the camera, and can set a plurality of programs for using the camera. The robot system further comprises a user interface that can set a focus position of the auto-focus lens in each of the programs, and a storage unit that stores focus values set in each of the plurality of programs. The user interface is configured to be able to execute, during execution of a program, a first image capturing mode for performing image capturing using an auto-focus function of the auto-focus lens, and a second image capturing mode for performing image capturing using the focus value stored in the storage unit, without using the auto-focus function.

Description

[Technical Field]

[0001] The present invention relates to a robot system having a user interface for capturing an image of an object with an autofocus camera, and an imaging method using the user interface. [Background technology]

[0002] Systems that capture an object with a camera, perform detection processing on the image, and operate a robot based on the detection results are well known. Conventionally, cameras used in such systems typically used lenses that required manual focus adjustment (manual focus lenses). However, in recent years, improvements in autofocus technology and falling prices have led to an increasing use of lenses that can automatically adjust the focus (autofocus lenses).

[0003] For example, a technology is known in which an image of a person is captured by a CCD camera with an autofocus function, the distance between the robot and the person is calculated, and the robot's operation is controlled (see, for example, Patent Document 1). Also known is a technology in which an imaging unit with an autofocus mechanism is mounted on a robot arm, an image of an object is captured by the imaging unit, and the robot is controlled based on the obtained image (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2002-219676 [Patent Document 2] Japanese Patent Application Publication No. 2019-125056 Summary of the Invention [Problem to be solved by the invention]

[0005] The use of a camera with an autofocus lens has the advantage that it allows workers without specialized knowledge to obtain reasonably accurate images and eliminates the need for manual focus adjustment. However, when capturing images using autofocus, even if the position and orientation of the object being captured relative to the camera are exactly the same, the automatically adjusted focus position (focal length) may differ depending on lighting and other conditions. Therefore, for example, the focus position obtained by autofocus may differ between when a detection program is taught and when the taught detection program is applied to the production site. In such cases, the taught detection program is executed at the production site to perform detection processing on images captured with a focus different from that at the time of teaching, which may result in the desired detection accuracy not being achieved.

[0006] Furthermore, even if there is no problem with the accuracy of the focus adjustment, the focus adjustment takes a certain amount of time, and therefore performing the focus adjustment while the production line is operating at the production site increases the takt time of the robot system. [Means for solving the problem]

[0007] One aspect of the present disclosure is a robot system that includes a camera equipped with an autofocus lens and a robot that performs work based on the image capture results of the camera, and that is capable of setting multiple programs for using the camera, and that includes a user interface that allows the focus position of the autofocus lens to be set in each of the programs, and a memory unit that stores the focus values ​​set in each of the multiple programs, and that is configured to be able to execute, when the program is executed, a first image capture mode that uses the autofocus function of the autofocus lens to capture an image, and a second image capture mode that does not use the autofocus function but uses the focus value stored in the memory unit to capture an image.

[0008] Another aspect of the present disclosure is an imaging method for a robot system having a camera equipped with an autofocus lens and a robot that performs work based on the imaging results of the camera, and in which multiple programs for using the camera can be set, the imaging method including: setting a focus position of the autofocus lens in each of the programs; saving the focus value set in each of the multiple programs; and executing, during execution of the program, a first imaging mode in which imaging is performed using the autofocus function of the autofocus lens, and a second imaging mode in which imaging is performed using the saved focus value without using the autofocus function. [Effects of the Invention]

[0009] According to the present disclosure, manual focus adjustment is no longer necessary, while the variation in focus position that may occur when using a program for capturing images with an autofocus camera under different conditions can be eliminated or reduced. Furthermore, by using the second imaging mode that does not utilize the autofocus function, the focusing operation of the camera is reduced, thereby extending the life of the camera. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a schematic configuration diagram of a robot system according to an embodiment. [Figure 2] FIG. 10 is a diagram showing an example of setting an imaging program using a user interface. [Figure 3] 10A and 10B are diagrams illustrating other setting examples of the imaging program using a user interface. [Figure 4] FIG. 4 is a diagram showing an example in which different focus values ​​are set in the example of FIG. 3. [Figure 5] 10 is a flowchart illustrating an example of processing related to an imaging program. DETAILED DESCRIPTION OF THE INVENTION

[0011] 1 is a schematic diagram of a robot system according to a preferred embodiment. The robot system 10 includes at least one robot 12, a robot control device 14 that controls the robot 12, and a camera 20 equipped with an autofocus lens 18 for capturing an image of a detection target (workpiece) 16. The robot 12, the control device 14, and the camera 20 are connected to each other via wire or wirelessly so as to be able to communicate with each other.

[0012] The robot 12 is, for example, an industrial articulated robot, and is configured to be able to perform various operations such as processing and removing a workpiece based on commands sent from a robot control device 14 and on the results of imaging the workpiece 16, which will be described later. The robot control device 14 includes a processor and a storage unit (memory, etc.), and controls the camera 20 according to a detection program prepared in advance, and can adjust the focus of the lens 18 when imaging the workpiece 16.

[0013] The robot control device 14 can store and execute the above-mentioned detection program in addition to the robot program for controlling the robot 12. Alternatively, the detection program may be stored in a computer such as a personal computer (PC) 22 having a processor and a storage unit (memory, etc.) connected to the control device 14 or the camera 20 by wire or wirelessly, and the PC 22 may control the camera 20. At least one of the control device 14 and the PC 22 has an input unit such as a keyboard 24 or a touch panel and a display unit such as a display 26, and the input unit and the display unit form a user interface, which will be described later.

[0014] 1, the camera 20 is fixed to and supported on a stand 27 or the like that is independent of the robot 12, and this type of configuration is suitable, for example, when the workpiece 16 is transported by a conveyor (not shown) or the like and always passes within the field of view of the camera. Alternatively, the camera 20 may be provided on a movable part such as the robot arm 28 of the robot 12, and this type of configuration is suitable, for example, when the workpiece is relatively large and it is desirable to move the camera to capture images of different parts of the workpiece.

[0015] Example 1 FIG. 2 shows an example of a user interface used when capturing an image of a connecting rod-shaped workpiece 16a using the camera 20. Reference numeral 30 denotes a teaching screen displayed on the display 26 of the control device 14 or the PC 22. In this embodiment, the teaching screen 30 is a user interface for configuring settings related to capturing the workpiece and for setting parameters for a detection program that uses the workpiece image. The teaching screen 30 includes an image display area 32 for the captured workpiece, a tree area 34 for displaying a configuration tree of tools included in the detection program, and a setting area 36 for a tool selected as a setting target within the configuration tree. In this embodiment, as shown in the tree area 34, the detection program using the camera 20 includes an imaging program_1 for capturing an image of the workpiece 16a using the camera 20 and a template matching tool_1 for matching the image of the workpiece 16a obtained by executing the imaging program_1 with a pre-prepared template.

[0016] Next, the procedure for capturing an image of a workpiece using a user interface will be described. First, when a worker operates (e.g., touches or presses) the image capture button 40, for example, when teaching the robot 12, the camera 20 captures an image of the workpiece 16a, as shown in FIG. 1. This image capture is usually performed using the autofocus function of the camera 20, but if the focus is fixed, it can also be captured at a fixed focus. This method of capturing an image of a workpiece using the autofocus function of the camera 20 is also referred to as the first image capture mode.

[0017] The focus position (the distance between the camera 20 and the workpiece 16a) automatically adjusted and set by the autofocus function is displayed in the focus position display section 42 in the setting area 36. Alternatively, the focus position display section 42 may display the focus position used in a previous (e.g., immediately preceding) photograph.

[0018] If the image of the workpiece 16a captured at the focus position (770 mm in this case) displayed on the focus position display unit 42 and displayed in the display area 32 is determined to be appropriate, the operator can operate (e.g., touch or press) the focus lock button 44 to lock the focus position (and save it in the memory of the control device 14 or the PC 22). Alternatively, if the image of the workpiece 16a is not appropriate (e.g., if it is clearly out of focus and unclear, or if it is somewhat clear but is determined to be inappropriate for performing the above-mentioned template matching), the operator can correct the focus position via the user interface (specifically, by operating the focus position adjustment button 46) to obtain an appropriate image, and then operate the focus lock button 44 to save the focus position.

[0019] In this way, during the robot teaching stage, etc., the saving (fixing) of the focus position in one detection program is completed using the first imaging mode. Next, when the robot system 10 is moved and applied to a production site such as a production line and an image of the workpiece 16a is taken with the camera 20 in the same manner as in the teaching stage, the autofocus function of the camera 20 is not used and the image of the workpiece 16a is taken using the focus position fixed in the first imaging mode. However, the relative positional relationship between the camera 20 and the workpiece 16a (the position and orientation of the workpiece 16a relative to the camera 20) is assumed to be the same between the teaching stage and the production stage. This method of taking an image of the workpiece using the focus value saved in the first imaging mode without using the autofocus function of the camera 20 is also referred to as the second imaging mode.

[0020] 2, various settings can be made in addition to displaying and fixing (saving) the focus position. For example, as shown in the setting area 36, ​​regarding the execution of the imaging program, settings such as whether to reduce the image (reference symbol 50), setting the exposure mode (fixed or automatically adjusted) (reference symbol 52), setting the exposure time (reference symbol 54), setting the number of multiple exposures (reference symbol 56), setting whether to use LED lighting (reference symbol 58), and setting the image display mode (whether to display only the captured image or to display the captured image with the detection results or teaching model superimposed thereon, etc.) (reference symbol 60) can be made.

[0021] In the example of Figure 2, as shown in the tree area 34, the detection program includes an imaging program and a matching tool, and the focus position is set and saved within the imaging program, but the present disclosure is not limited to this. For example, if the imaging program and detection program are prepared separately and included in the robot's teaching program, the focus position can also be saved and set within the detection program or in a program part of the teaching program other than the imaging program and detection program. Thus, in the present disclosure, the program for using the camera is not limited to the imaging program.

[0022] Example 2 3 shows an example of a user interface used when using the camera 20 to capture an image of a workpiece 16ba that is configured from a gear-shaped rotary cutter 62 and a flange 64 that rotatably supports the rotary cutter 62. In the second embodiment, mainly the parts that are different from the first embodiment will be described, and parts that may be the same as those in the first embodiment will be assigned the same reference numerals as in the first embodiment, and detailed description thereof will be omitted.

[0023] In this embodiment, as shown in the tree area 34, the detection program using the camera 20 includes an imaging program_2 for imaging the workpiece 16b using the camera 20, and a template matching tool_2 for matching the image of the workpiece 16b obtained by executing the imaging program_2 with a template prepared in advance.

[0024] Next, the procedure for capturing an image of a workpiece using a user interface will be described. First, when an operator operates (e.g., touches or presses) the image capture button 40, for example, during teaching of the robot 12, the camera 20 captures an image of the workpiece 16b, as shown in FIG. 1. However, in the second embodiment, it is assumed that multiple different portions of the workpiece 16b must be captured and detected. First, as shown in FIG. 3, the flange 64 (particularly, the hole formed in the flange 64) is captured in the first image capture mode (autofocus function). Whether or not to use the autofocus function can be set, for example, by operating the autofocus setting button 66 in the setting area 36. In the image of the workpiece 16b obtained in this manner, as shown in the display area 32, the flange 64 is in focus, while the rotary cutter 62, which is significantly different in distance from the camera 20 than the flange 64, is out of focus and somewhat blurred.

[0025] The focus position (the distance between the camera 20 and the flange 64) automatically adjusted and set by the autofocus function is displayed in the focus position display section 42 within the setting area 36. Alternatively, the focus position display section 42 may display the focus position used in a previous (e.g., immediately preceding) photograph.

[0026] If the image of the workpiece 16b captured at the focus position (800 mm in this case) displayed on the focus position display unit 42 and displayed in the display area 32 is determined to be appropriate, the operator can operate (e.g., touch or press) the focus lock button 44 to lock the focus position (and save it in the memory of the control device 14 or the PC 22). Alternatively, if the image of the workpiece 16b is not appropriate (e.g., if it is clearly out of focus and unclear, or if it is somewhat clear but is determined to be inappropriate for performing the above-mentioned template matching), the operator can operate the focus position adjustment button 46 to adjust the focus position at which an appropriate image is obtained, and then operate the focus lock button 44 to save the focus position.

[0027] Next, as shown in Fig. 4, the rotary cutter 62 is imaged in the first imaging mode (autofocus function). Whether or not to use the autofocus function can be set by, for example, operating the autofocus setting button 66 in the setting area 36. In the image of the workpiece 16b obtained in this manner, as shown in the display area 32, the rotary cutter 62 is in focus, while the flange 64 is out of focus and slightly blurred.

[0028] The focus position (the distance between the camera 20 and the rotary cutter 62) automatically adjusted and set by the autofocus function is displayed in the focus position display section 42 in the setting area 36. Alternatively, the focus position display section 42 may display the focus position used in a previous (e.g., immediately preceding) photograph.

[0029] If the image of the workpiece 16b captured at the focus position (here, 1100 mm) displayed on the focus position display unit 42 and displayed in the display area 32 is determined to be appropriate, the operator can operate (for example, touch or press) the focus lock button 44 to lock the focus position (and save it in the memory of the control device 14 or PC 22). Alternatively, if the image of the workpiece 16b is not appropriate (for example, if it is clearly out of focus and unclear, or if it is somewhat clear but is determined to be inappropriate for performing the above-mentioned template matching), the operator can operate the focus position adjustment button 46 to adjust the focus position at which an appropriate image is obtained, and then operate the focus lock button 44 to save the focus position.

[0030] As described above, in Figures 3 and 4, the focus positions obtained by the autofocus function are different, and the focus positions are saved for each. In other words, in Example 2, even if the workpiece is the same, the imaging locations are different, and therefore the focus positions are also different, so the respective detection programs are also treated as different. In this way, even if the imaging target (workpiece) is the same, if the imaging location is different, the optimal focus position will also be different, so it is preferable to prepare different detection programs and save and fix the focus position for each. Needless to say, when the imaging targets are different, as in Example 1 and Example 2, different detection programs are used.

[0031] In this way, during the robot teaching stage or the like, the first imaging mode completes saving (fixing) of the focus position for each of the multiple (here, two) detection programs. Next, when the robot system 10 is applied to a production site such as a production line and an image of the workpiece 16b is taken with the camera 20, as in the teaching stage, the autofocus function of the camera 20 is not used, and the image of the workpiece 16b, more specifically, the image of the rotary cutter 62 and the image of the flange 64 are taken separately using the fixed focus position in the first imaging mode. However, the relative positional relationship between the camera 20 and the workpiece 16b (the position and orientation of the workpiece 16b relative to the camera 20) is assumed to be the same between the teaching stage and the production stage. This method of taking an image of the workpiece using the focus value saved in the first imaging mode without using the autofocus function of the camera 20 is also referred to as the second imaging mode.

[0032] According to the above-described embodiment, even workers unfamiliar with manual focus operation can easily learn how to focus. Since the autofocus function is not used at the production site, the time required to adjust the focus position is reduced, thereby shortening the cycle time of the entire system. Furthermore, since no focusing operation is performed at the production site, the number of focusing operations is reduced compared to when the autofocus function is used during both teaching and production line operation, thereby extending the camera's lifespan. Furthermore, even if the relative positional relationship between the camera and the workpiece is the same, with autofocus, the focus position can vary from one detection process to the next due to lighting conditions and external disturbances (such as insects or scattered foreign objects), especially at the production site. This embodiment can also avoid such variations.

[0033] In the above-described embodiment, the first imaging mode is executed when teaching the robot, and the second imaging mode is executed when the robot is applied to a production site, but the present disclosure is not limited to this. For example, instead of teaching at a location other than the production site, a simulation can be performed using a PC 22 or the like, and the first imaging mode can be executed in the simulation to save the focus position. In the simulation, by inputting the positional relationship between the camera and the workpiece, an image that would be obtained when actually capturing an image of the workpiece with the camera can be created and displayed by calculation. Therefore, the focus position can be set, adjusted, and saved using the same operations as in the above-described embodiment.

[0034] FIG. 5 is a flowchart showing the processing flow in the above-described embodiment. First, in step S1, an image of a workpiece is captured using autofocus in a first imaging mode (for example, during robot teaching). In the next step S2, the focus position captured using autofocus is saved. This focus position is saved for each detection program. In other words, a different focus position value is set and saved for each detection program. In the next step S3, for example, when operating a robot at a production site, an image of the workpiece is captured using the focus position saved in the first imaging mode (for example, during teaching) without using the autofocus function when executing the detection program as a second imaging mode. [Explanation of symbols]

[0035] 10 Robot Systems 12. Robot 14 Robot control device 16, 16a, 16b Work 18 Lenses 20 Camera 22 PC 24 Input section 26 Display 27 Mounting stand 28 Robot Arm 30 Teaching screen 32 Image display area 34 Tree Area 36 Settings area 40 Capture button 42 Focus position display 44 Focus lock button 46 Focus position adjustment button 62 cutter 64 flange 66 Autofocus mode setting section

Claims

1. A robot system having a camera equipped with an autofocus lens and a robot that performs work based on the image captured by the camera, wherein a plurality of programs for using the camera can be set, a user interface that allows the user to set the focus position of the autofocus lens in each of the programs; a storage unit for storing focus values ​​set by each of the plurality of programs, the user interface is configured to be able to execute, during execution of the program, a first imaging mode in which imaging is performed using an autofocus function of the autofocus lens, and a second imaging mode in which imaging is performed using the focus value stored in the memory unit without using the autofocus function.

2. The robot system according to claim 1 , wherein a relative positional relationship between the camera and the object to be imaged by the camera is the same in the first imaging mode and the second imaging mode.

3. 3. The robot system according to claim 2, wherein the focus value stored in the memory unit is a focus value obtained by using the autofocus function in the first imaging mode, or a focus value obtained by correcting the focus value obtained by using the autofocus function via the user interface.

4. The robot system according to claim 2 or 3, wherein the first imaging mode is executed when the robot is being taught, and the second imaging mode is executed when the robot is being used at a production site.

5. 5. The robot system according to claim 1, wherein the program is prepared for each different workpiece or each different imaging site of the same workpiece.

6. 1. An imaging method for a robot system having a camera equipped with an autofocus lens and a robot that performs work based on the imaging results of the camera, wherein a plurality of programs for using the camera can be set, setting a focus position of the autofocus lens in each of the programs; storing focus values ​​set in each of the plurality of programs; and executing, during execution of the program, a first imaging mode in which imaging is performed using an autofocus function of the autofocus lens, and a second imaging mode in which imaging is performed using the saved focus value without using the autofocus function.

7. The imaging method according to claim 6 , wherein the first imaging mode is executed when the robot is being taught, and the second imaging mode is executed when the robot is being used at a production site.

Images