Calibration method

Abstract

Focusing on improving camera accuracy, this provides an easier way to calibrate cameras. [Solution] A calibration method performed in a system comprising a robot that performs a predetermined machining operation on a workpiece, a camera capable of imaging the robot or the workpiece, a plurality of markers M arranged in a workspace where the robot and camera are installed, and a computer, the method comprising: an imaging step in which the camera images at least one of the plurality of markers M; and a determination step in which the computer determines the position and orientation of the camera based on the image captured by the camera, wherein the plurality of markers M are uniquely identifiable, and the imaging surfaces of the plurality of markers M are arranged in a direction intersecting the horizontal plane.

Description

Technical Field

[0001] The present invention relates to a calibration method.

Background Art

[0002] Patent Document 1 describes that an image of a robot captured by a camera and a 3D model of the robot reproduced in a virtual space are superimposed and displayed, and when the user moves the 3D model to a predetermined position by dragging or the like, the robot also moves to the predetermined position.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] According to this invention, the labor and man-hours generated in robot operation can be reduced.

[0005] By the way, there has been an increasing demand for more accurately superimposing the image of the robot and the 3D model with respect to this invention.

[0006] Therefore, in view of the above problems, the present invention focuses on improving the accuracy of the camera and aims to provide a method for more easily calibrating the camera.

Means for Solving the Problems

[0007] The object of the present invention is achieved by the following means. The reference numerals in parentheses are those of the embodiments described later, but the present invention is not limited thereto.

[0008] The calibration method according to claim 1 is A robot that performs a predetermined machining operation on a workpiece (for example, robot 2 shown in Figure 2), The robot (for example, robot 2 shown in Figure 2) or a camera capable of imaging the workpiece (for example, camera 3 shown in Figure 2), Multiple signs (e.g., signs M shown in Figures 6-8) are placed within the workspace where the robot (e.g., robot 2 shown in Figure 2) and the camera (e.g., camera 3 shown in Figure 2) are installed, A computer (for example, computer 4 shown in Figure 2), A calibration method performed in a system equipped with (for example, system 1 shown in Figure 2), The camera (for example, camera 3 shown in Figure 2) performs an imaging step (for example, step S102 shown in Figure 4) in which it images at least one of the plurality of signs (for example, signs M shown in Figures 6 to 8), The computer (for example, computer 4 shown in Figure 2) includes a determination step (for example, step S103-8 shown in Figure 4) in which it determines the position and orientation of the camera (for example, camera 3 shown in Figure 2) based on the image captured by the camera (for example, camera 3 shown in Figure 2), The aforementioned multiple markers (for example, marker M shown in Figures 6-8) are uniquely identifiable, The imaging surfaces of the plurality of markers (for example, markers M shown in Figures 6-8) are arranged in a direction that intersects with the horizontal plane (for example, the mounting surface TF shown in Figure 2).

[0009] The calibration method described in claim 2 is the calibration method described in claim 1, The system (for example, system 1 shown in Figure 2) further includes a first auxiliary device (for example, first auxiliary device 7 shown in Figures 6 and 7) positioned so that the camera (for example, camera 3 shown in Figure 2) can take images. The first auxiliary device (for example, the first auxiliary device 7 shown in Figures 6 and 7) has a side surface located in a direction intersecting the horizontal plane (for example, the mounting surface TF shown in Figure 2), The marker (for example, marker M shown in Figures 6 and 7) is characterized by being positioned on the side surface of the first auxiliary device (for example, the first auxiliary device 7 shown in Figures 6 and 7).

[0010] The calibration method described in claim 3 is the calibration method described in claim 2, The first auxiliary device (for example, the first auxiliary device 7 shown in Figures 6 and 7) comprises a plurality of columnar members (for example, columnar members 71, 72, 73, and 74 shown in Figure 6) extending in a direction intersecting the horizontal plane (for example, the mounting surface TF shown in Figure 2), The aforementioned columnar members (for example, the columnar members 71, 72, 73, and 74 shown in Figure 6) are erected with space between them, The side surface of the first auxiliary device is characterized by being the side surface of the plurality of columnar members (for example, the columnar members 71, 72, 73, and 74 shown in Figure 6).

[0011] The calibration method described in claim 4 is the calibration method described in claim 1, The system (for example, system 1 shown in Figure 2) further includes a second auxiliary tool (for example, second auxiliary tool 8 shown in Figure 8) provided on a work tool (for example, welding torch 24 shown in Figure 8) attached to the robot (for example, robot 2 shown in Figure 2), The second auxiliary device (for example, the second auxiliary device 8 shown in Figure 8) has sides (for example, sides 81, 82 shown in Figure 8) that are positioned in a direction intersecting the horizontal plane (for example, the mounting surface TF shown in Figure 2), The sign (for example, sign M shown in Figure 8) is characterized by being positioned on the side surface (for example, side surfaces 81, 82 shown in Figure 8) of the second auxiliary device (for example, second auxiliary device 8 shown in Figure 8).

[0012] The calibration method described in claim 5 is the calibration method described in claim 4 above, The second auxiliary device (for example, the second auxiliary device 8 shown in Figure 8) is characterized in that it is attached to the work tool (for example, the welding torch 24 shown in Figure 8) using a mounting mechanism for a position adjustment gauge for the work tool (for example, a conventionally known mounting mechanism for a position adjustment gauge 26 provided at the location indicated by the dashed line on the arm 2 shown in Figure 8).

[0013] The display method described in claim 6 is: The aforementioned camera (for example, camera 3 shown in Figure 2) is a plurality of cameras, The computer (for example, computer 4 shown in Figure 2) performs a storage step (for example, step S104 shown in Figure 4) which records the position and orientation of the plurality of cameras (for example, camera 3 shown in Figure 2) determined by the calibration method described in claim 1, 2, or 4 above, The computer (for example, computer 4 shown in Figure 2) acquires and displays captured images (for example, thumbnails 51a shown in Figure 9(a)) from the multiple cameras (for example, camera 3 shown in Figure 2) to create a list (for example, list 51 shown in Figure 9(a)), and performs a list display step (for example, steps S204, 208 shown in Figure 5), The robot image acquisition step (for example, step S205 shown in Figure 5) involves the computer (for example, computer 4 shown in Figure 2) acquiring an image (for example, image 53 shown in Figure 9(a)) from one camera (for example, camera 31 shown in Figure 2) selected from the list (for example, list 51 shown in Figure 9(a)), Step (for example, step S206 shown in Figure 5) of acquiring an image (for example, image 63A shown in Figure 9(b)) of the robot reproduced in the virtual space from a simulator (for example, computer 4 shown in Figure 2) that reproduces the robot (for example, robot 2 shown in Figure 2) in a virtual space, as viewed from the position and orientation of one camera (for example, camera 31 shown in Figure 2) selected from the list, The computer (for example, computer 4 shown in FIG. 2) overlays and displays an imaging image (for example, image 53 shown in FIG. 9(a)) of one camera (for example, camera 31 shown in FIG. 2) selected from the list and an image (for example, image 63B shown in FIG. 9(a)) obtained from the simulator (for example, simulator 4 shown in FIG. 2) (for example, step S207 shown in FIG. 5). comprising The imaging image (for example, thumbnail 51a shown in FIG. 9(a)) for creating the list is characterized by being displayed in a lower image quality than the imaging image (for example, image 53 shown in FIG. 9(a)) of the one selected camera.

Effect of the Invention

[0014] Next, the effects of the present invention will be described with reference numerals in the drawings. The reference numerals in parentheses are those of the embodiments described later, but the present invention is not limited thereto.

[0015] According to the invention according to claim 1, calibration can be easily performed regardless of the position and orientation of the camera.

[0016] That is, conventionally, the calibration board is placed on a horizontal surface (for example, mounting surface TF shown in FIG. 2), and the imaging surface becomes horizontal. Further, the position of the calibration board cannot be detected unless all the feature points on the calibration board are imaged and detected. For this reason, if the camera (for example, camera 3 shown in FIG. 2) faces the direction of the horizontal surface (for example, mounting surface TF shown in FIG. 2) on which the calibration board is placed and cannot image the entire calibration board, the camera (for example, camera 3 shown in FIG. 2) cannot capture a calibration-enabled image.

[0017] However, according to the invention according to claim 1, since the imaging surface of the marker (for example, the marker M shown in FIGS. 6 to 8) is arranged in a direction intersecting with the horizontal plane (for example, the mounting surface TF shown in FIG. 2), regardless of the position and orientation of the camera (for example, refer to the cameras 31 and 39 shown in FIG. 2), there is a high possibility of being able to image the marker (for example, the marker M shown in FIGS. 6 to 8). Moreover, the marker (for example, the marker M shown in FIGS. 6 to 8) is uniquely identifiable. Therefore, unlike the conventional case, it is not necessary to image and detect all the markers (for example, the marker M shown in FIGS. 6 to 8).

[0018] Therefore, by arranging the uniquely identifiable marker (for example, the marker M shown in FIGS. 6 to 8) in a direction intersecting with the horizontal plane (for example, the mounting surface TF shown in FIG. 2), the possibility of being able to image a calibratable image is increased regardless of the position and orientation of the camera (for example, the camera 3 shown in FIG. 2).

[0019] Therefore, according to the present invention, calibration can be easily performed regardless of the position and orientation of the camera.

[0020] According to the invention according to claim 2, the marker (for example, the marker M shown in FIGS. 6 and 7) is arranged on the side surface of the first auxiliary tool (for example, the first auxiliary tool 7 shown in FIGS. 6 and​​​According to the invention of claim 3, the camera (for example, camera 3 shown in Figure 2) can image not only the marker (for example, marker M shown in Figure 6) placed on the side of the columnar member (for example, columnar members 71 and 72 shown in Figure 6) that is closer to the camera (for example, camera 3 shown in Figure 2) of the first auxiliary device (for example, first auxiliary device 7 shown in Figures 6 and 7), but also the marker (for example, marker M shown in Figure 6) on the columnar member further back (for example, columnar members 73 and 74 shown in Figure 6). Therefore, since more markers can be imaged, higher accuracy calibration can be performed. Furthermore, since the marker (for example, marker M shown in Figure 6) on the columnar member further back (for example, columnar members 73 and 74 shown in Figure 6) can also be imaged, the calibration accuracy in the depth direction can be improved.

[0022] According to the invention of claim 4, by moving the arm, the work tool (for example, the welding torch 24 shown in Figure 8) can be moved, and the position of the marker M on the second auxiliary device (for example, the second auxiliary device 8 shown in Figure 8) can be changed, while the second auxiliary device can be imaged by the camera (for example, the camera 3 shown in Figure 2). Therefore, since more markers can be imaged, higher-precision calibration can be performed.

[0023] According to the invention of claim 5, a second auxiliary tool (for example, the second auxiliary tool 8 shown in Figure 8) can be easily attached using a mechanism that has been conventionally provided on a work tool (for example, the welding torch 24 shown in Figure 8).

[0024] According to the invention of claim 6, the image capture (for example, the thumbnail 51a shown in Figure 9(a)) used to create the list (for example, the list 51 shown in Figure 9(a)) is of relatively low quality, thus reducing the processing burden on the computer. [Brief explanation of the drawing]

[0025] [Figure 1] This is an overall diagram of the system according to the present invention. [Figure 2] This is an overall schematic diagram of the system according to the same embodiment. [Figure 3](a) is an explanatory diagram illustrating a database for storing the position coordinates of a sign according to the same embodiment, and (b) is an explanatory diagram illustrating a database for storing the position coordinates of a camera according to the same embodiment. [Figure 4] This is a flowchart illustrating the calibration process according to the same embodiment. [Figure 5] This is a flowchart illustrating the overlay process according to the same embodiment. [Figure 6] This is a diagram showing the first auxiliary device according to the same embodiment. [Figure 7] This figure shows an image captured by a camera of the first auxiliary device according to the same embodiment. [Figure 8] This is an explanatory diagram showing a second auxiliary device according to the same embodiment. [Figure 9] (a) is an explanatory diagram illustrating the screen for synthesis according to the same embodiment, and (b) is an explanatory diagram illustrating the screen for the simulator according to the same embodiment. [Modes for carrying out the invention]

[0026] Hereinafter, an embodiment of the system according to the present invention will be specifically described with reference to the drawings. In the following description, when the directions up, down, left, and right are indicated, in drawings where the direction is indicated by an arrow, it refers to the direction indicated by the arrow, and in drawings where the direction is not indicated by an arrow, it refers to up, down, left, and right as viewed from the front of the drawing.

[0027] <Overview of the entire system> First, an overview of System 1 in this embodiment will be described using Figure 1. System 1 in this embodiment mainly consists of a turntable T on which a workpiece (not shown) is placed, a robot 2 that performs predetermined processing operations such as welding and polishing, and multiple cameras 3 that capture images of the robot 2 and other equipment. As shown in Figure 1, the workspace K in which these devices are arranged is surrounded by a safety fence D.

[0028] Next, we will explain the components of System 1 in more detail using Figure 2.

[0029] <Robot Description> Robot 2 is a conventionally known industrial robot. More specifically, as shown in Figure 2, Robot 2 comprises a base 20 fixed to the floor of a workshop, an arm 22 whose base end is rotatably connected to the base 20, and a work tool attached to the flange 22a at the tip of the arm 22 (see also Figure 8). In this embodiment, a welding torch 24 is shown as the work tool, but it is not limited to this, and various work tools can be used to perform operations such as welding, sealing, painting, cutting, polishing, and handling on a workpiece (not shown) placed on a turntable T. Also, in this embodiment, as shown in Figure 2, a 6-axis articulated general-purpose robot is shown as Robot 2, but it is not limited to this, and any robot can be used.

[0030] Robot 2 is connected to computer 4 via network N through a programmable logic controller (hereinafter referred to as PLC29).

[0031] Furthermore, in this specification, the three-dimensional coordinate system with the center of the base 20 as the origin (see Figure 2) will be referred to as the robot coordinate system R.

[0032] <Camera Description> Cameras 31-39 (Camera 3 shown in Figure 1) are conventionally known cameras capable of capturing images within the workspace K (see Figure 1). In this embodiment, cameras 31-39 are network cameras that transmit video based on the NDI method and transmit video to computer 4 via network N.

[0033] The calibration of cameras 31-39 is performed by having cameras 31-39 capture images of the marker M (see Figures 6-8).

[0034] <Explanation of the first assistive device> Here, the first auxiliary device 7 used to calibrate cameras 31-39 shown in Figure 2 will be explained using Figure 6.

[0035] Figure 6 shows the first auxiliary device 7 placed on the mounting surface TF of the turntable T (see Figure 2), as viewed from the robot 2 side (see Figure 2).

[0036] As shown in Figure 6, the first auxiliary device 7 is formed by erecting columnar members 71 to 74 at the four corners of a roughly square base member 75 and fixing them by welding or the like.

[0037] Multiple markers M are placed on each of the four sides of these columnar members 71-74 (the four sides extending in the direction intersecting the bottom member 75). The markers M are uniquely identifiable markers (e.g., ArUco markers).

[0038] To explain in more detail using the columnar member 71 as an example, as shown in Figure 6, two markers M are arranged vertically spaced apart on one side 71a of the columnar member 71 on the robot 2 side (the side closer to the viewer). Then, as shown in Figure 6, three markers M are arranged vertically spaced apart on the side 71b adjacent to side 71a. The two markers M on side 71a and the three markers M on the adjacent side 71b are arranged in a staggered pattern, separated by the corner formed by sides 71a and 71b. The remaining two sides of the columnar member 71, although not shown in Figure 6, also have markers M arranged in the same way as sides 71a and 71b.

[0039] As shown in Figure 6, similar to the columnar member 71, multiple markers M are also placed on each of the four sides of the columnar members 72 to 74.

[0040] On the other hand, as shown in Figure 6, multiple markers M are arranged horizontally on the bottom member 75 in a staggered pattern with spacing between them.

[0041] The method of using this first auxiliary device 7 (see Figure 6) will be described later.

[0042] <Computer Description> The computer 4 shown in Figure 2 is an information processing device, such as a PC (Personal Computer). Specifically, as shown in Figure 2, it consists of a CPU 40, an input unit 41 that can input predetermined data to the computer 4 from an external source using a mouse, keyboard, touch panel, USB connector, Lightning connector, etc., a ROM 42 consisting of a writable flash ROM that stores predetermined application programs, etc., a RAM 43 that functions as a work area and buffer memory, an image output unit 44 that can output images based on a format suitable for transmission standards such as HDMI (registered trademark), and a communication unit 45 that can connect to a network N using communication means such as wireless LAN, wired LAN, dial-up, etc.

[0043] The ROM 42 stores a calibration program (see Figure 4), a program for displaying superimposed images (see Figure 9(a)) (see Figure 5), a simulator program (not shown), and a 3D model 61 of robot 2 used in the simulator program (see Figure 9(b)).

[0044] Furthermore, ROM42 also stores databases. More specifically, ROM42 stores the marker DB and camera DB shown in Figure 3. The recording of data into these marker DB and camera DB is performed in relation to the calibration of cameras 31-39 (see Figure 4), and will be explained in more detail later.

[0045] <Explanation of the synthesis screen and simulator screen> The composite screen 5 displays a screen generated by a program executed by computer 4 (see Figure 5) (see Figure 9(a)).

[0046] Here, we will briefly explain the screens displayed on the synthesis screen 5 and the simulator screen 6 using Figure 9(a).

[0047] First, the image composition screen 5 displays a list 51 of cameras 31-39 on the left and a detailed display section 52 in the center. The list 51 on the left displays the images captured by each camera 31-39 as thumbnails 51a. Meanwhile, the detailed display section 52 in the center displays the image 53 captured by camera 1, which was selected in the list 51. This image 53 is overlaid with image 63B (see Figure 9(a)), which is obtained from image 63A (see Figure 9(b)), which includes the 3D model 61 of robot 2 (see Figure 9(a)).

[0048] On the other hand, the simulator screen 6 displays a three-dimensional model 61 of the robot 2 (see Figure 2) generated by a simulator program (not shown) executed by the computer 4 (see Figure 9(b)). Since this simulator program (not shown) is a well-known technology, a detailed explanation is omitted in this embodiment and will be explained as necessary.

[0049] As shown in Figure 2, the synthesis screen 5 and the simulator screen 6 are each independent liquid crystal monitors, both connected to the computer 4 by monitor cables such as HDMI® cables.

[0050] <Explanation of the calibration process> Next, referring to the flowchart shown in Figure 4, we will explain the processing performed by computer 4 when calibrating cameras 31-39 shown in Figure 2.

[0051] First, the operator places the first auxiliary tool 7 on the mounting surface TF of the turntable T (see Figure 2) (see Figure 6). Next, the operator clicks an icon (not shown) on the synthesis screen 5 using a mouse or the like. In response, the CPU 40 of the computer 4 (see Figure 2; hereinafter referred to as computer 4) starts the calibration program (see Figure 4) stored in the ROM 42 (see Figure 2). As a result, computer 4 starts the process shown in Figure 4.

[0052] First, the computer 4 stores the three-dimensional coordinates in the robot coordinate system R (see Figure 2) of the four corners of each marker M of the columnar members 71-74 and the bottom member 75 of the first auxiliary device 7 (see Figure 6) (step S101). More specifically, the computer 4 obtains the three-dimensional coordinates in the robot coordinate system R of the four corners of the marker M of the first auxiliary device 7 (see Figure 6) by any means and stores them in the coordinate fld12 of the marker DB (see Figure 3(a)). Therefore, by executing step S101, the marker DB (see Figure 3(a)) will have the same number of records as there are markers M (see Figure 6), and each record will have four three-dimensional coordinates in the robot coordinate system R recorded in it.

[0053] The method by which the computer 4 obtains the 3D coordinates of the four corners of the marker M (see Figure 6) in the robot coordinate system R can be carried out by conventionally known methods. For example, the computer 4 may display a screen on the synthesis screen 5 that accepts input from the operator, and have the operator input using an input unit 41 (see Figure 2), such as a keyboard. Alternatively, the computer 4 may display a message on the synthesis screen 5 prompting the operator to point the four corners of the marker M (see Figure 6) with the welding torch 24 of the robot 2. In that case, the computer 4 may obtain the 3D coordinates of the tool center point (hereinafter referred to as TCP) of the welding torch 24 in the robot coordinate system R from the PLC 29.

[0054] Next, the computer 4 acquires images of the marker M (see Figure 6) captured by cameras 31-39 shown in Figure 2 (step S102). More specifically, the computer 4 captures the video from cameras 31-39 received via the network N and temporarily records it in RAM 43 (see Figure 2). Therefore, by executing step S102, the RAM 43 (see Figure 2) will contain the same number of captured images as there are cameras 31-39.

[0055] In this embodiment, since the first auxiliary device 7 (see Figure 6) is used, the likelihood of being able to image the marker M regardless of the position and orientation of the cameras 31-39 increases. This point will be explained.

[0056] First, if the camera is positioned above the turntable T, as in camera 32 (see Figure 2), and is positioned to look down on the mounting surface TF (see Figure 2), it is possible to image the marker M placed on the bottom member 75 (see Figure 6). In that case, the marker M placed on the bottom member 75 (see Figure 6) can be used for calibration.

[0057] On the other hand, in calibration, there are cases where the marker M placed on the bottom member 75 (see Figure 6) cannot be used. For example, a camera like camera 31 (see Figure 2), which is installed at almost the same height as the mounting surface TF (see Figure 2) of the turntable T and is positioned to look upwards over the mounting surface TF (see Figure 2), cannot image the marker M placed on the bottom member 75 (see Figure 6). However, as shown in Figure 6, the marker M is also placed on the sides of the columnar members 71-74. Therefore, a camera installed in a position and orientation like camera 31 (see Figure 2) can image the marker M on the columnar members 71-74, for example, as shown in Figure 7(a).

[0058] Similarly, a camera like camera 39 (see Figure 2), which is positioned above the turntable T but not facing the mounting surface TF (see Figure 2), cannot image the marker M placed on the bottom member 75 (see Figure 6). However, it can image the marker M placed on the sides of the columnar members 71-74. For this reason, a camera installed in a position and orientation like camera 39 (see Figure 2) can image the marker M on the upper side of the columnar members 71-74, as shown in Figure 7(b).

[0059] Thus, by using the first auxiliary device 7 (see Figure 6), the likelihood of being able to image the labels M on the columnar members 71-74 (see Figure 6) increases, regardless of the position and orientation of the cameras 31-39 (see Figure 2).

[0060] Let's return to the explanation of the process shown in Figure 4. Next, the computer 4 determines the position and orientation of each camera 31-39 (see Figure 2). More specifically, the computer 4 performs the following processing on each captured image recorded in RAM 43 (see Figure 2) (step S103-1).

[0061] First, computer 4 recognizes the marker M on a single captured image temporarily stored in RAM 43 (see Figure 2). Then, it performs the following processing on all of the recognized marker M (step S103-2).

[0062] First, the computer 4 temporarily records the 2D coordinates of the four corners of the recognized sign M in the coordinate system G on the captured image (see Figure 7(a)) in the RAM 43 (see Figure 2) (step S103-3). For example, when the computer 4 recognizes the sign M10 shown in Figure 7(a), it temporarily records the 2D coordinates (x1,y1), (x2,y2), (x3,y3), and (x4,y4) of the four corners of the sign M10 in the coordinate system G on the captured image (see Figure 7(a)) in the RAM 43 (see Figure 2).

[0063] Next, computer 4 searches the marker database (see Figure 3(a)) and reads the 3D coordinates of the four corners of the recognized marker M in the robot coordinate system R (step S103-4). For example, if computer 4 recognizes the marker M10 shown in Figure 7(a), it searches the marker database (see Figure 3(a)) for the corresponding record and reads the 3D coordinates of its four corners (mx101,my101,mz101), (mx102,my102,mz102), (mx103,my103,mz103), and (mx104,my104,mz104) from coordinate fld12. Since marker M is uniquely identifiable, computer 4 can search for the record corresponding to the recognized marker M.

[0064] Next, the computer 4 temporarily records the 3D coordinates in the robot coordinate system R that it has read, linked to the 2D coordinates on the captured image (see Figure 7(a)), in the RAM 43 (step S103-5).

[0065] Once the computer 4 has finished the above process for all the markers M recognized in one captured image (step S103-6), it performs the same process for all the markers M recognized in the next captured image.

[0066] Once the above processing is completed for all captured images to be recorded in RAM43 (see Figure 2) (i.e., for all cameras 31-39 shown in Figure 2) (step S103-7), proceed to the next step.

[0067] Next, the computer 4 estimates a transformation matrix for each camera 31-39 to transform the 2D coordinates on the captured image (see Figure 7(a)) recorded in RAM 43 to the 3D coordinates in the robot coordinate system R (step S103-8). Then, for each camera 31-39, the position and orientation are determined from the estimated transformation matrix (step S103-9). When estimating the transformation matrix and determining the position and orientation of the cameras, conventionally known mathematical camera models, such as the pinhole camera model, can be used.

[0068] Furthermore, when using a mathematical camera model, the more combinations there are of 2D coordinates on the captured image (see Figure 7(a)) and 3D coordinates in the robot coordinate system R, the higher the accuracy of the calibration. In other words, the more markers M can be captured, the higher the accuracy of the calibration.

[0069] In this regard, the columnar members 71-74 are installed with space between them, as shown in Figure 6. Therefore, when imaging the first auxiliary device 7, in addition to the markers M on the columnar members closer to the front (columnar members 71 and 72 in Figure 6), the markers M on the columnar members further back (columnar members 73 and 74 in Figure 6) can also be imaged. As a result, more markers M can be imaged, enabling higher accuracy calibration. Furthermore, since the markers M on the columnar members further back (columnar members 73 and 74 in Figure 6) can also be imaged, the calibration accuracy in the depth direction can be improved.

[0070] Next, the computer 4 stores the position and orientation of each camera (step S104). Specifically, for each of the cameras 31 to 39 shown in Figure 2, the determined position and orientation are stored in the camera DB, position fld22 and orientation fld23 (see Figure 3(b)).

[0071] As described above, computer 4 performs calibration of cameras 31-39 shown in Figure 2.

[0072] <Explanation of displaying superimposed images> Next, referring to the flowchart shown in Figure 5, we will explain the processes that computer 4 performs when displaying the composite screen 5 (see Figure 2) (see Figure 9(a)).

[0073] First, the operator removes the first auxiliary tool 7 (see Figure 6) from the turntable T and places the workpiece (not shown) on it. Next, the operator uses a mouse or the like to click an icon (not shown) to activate the program (see Figure 5) stored in the ROM 42 (see Figure 2) of the computer 4. As a result, the computer 4 starts the process shown in Figure 5.

[0074] At this time, computer 4 also starts a simulator program (not shown) that generates the screen to be displayed on simulator screen 6 (see Figure 9(b)).

[0075] Next, computer 4 searches for cameras (step S201). More specifically, it notifies each camera 31-39 (see Figure 2) by broadcasting a message indicating that it is searching for cameras to the network N (see Figure 2).

[0076] Next, computer 4 creates a camera list (step S202). More specifically, computer 4 creates a camera list (not shown) based on the responses received from cameras 31-39 (see Figure 2). The camera list is used by computer 4 to determine which camera's video is received from network N, and includes camera information such as IP addresses.

[0077] Computer 4 can identify cameras 31, 32, etc., stored in the camera database shown in Figure 3(b), from the camera information in the camera list. Any method of identification is acceptable; for example, camera information such as the IP address of each camera may be recorded in both the camera database (see Figure 3(b)) and the camera list.

[0078] Next, computer 4 notifies a simulator program (not shown) running within computer 4 of the position and orientation of camera 1 (step S203). More specifically, computer 4 reads the position fld22 and orientation fld23 from the record of a predetermined camera (e.g., "camera 31") in the camera DB (see Figure 3(b)) and notifies the simulator program (not shown) of the read position and orientation. The notification may be made by any method, but in this embodiment, a shared area is provided in RAM 43, and the read position and orientation are stored in that shared area. In addition, computer 4 temporarily records in RAM 43 which camera's position and orientation it notified the simulator program of (e.g., "camera 31" is recorded).

[0079] Furthermore, upon receiving notification, the simulator program (not shown) displays an image 63A on the simulator screen 6, showing a 3D model 61 of robot 2 reproduced in a virtual space, viewed from the notified position and in the notified posture (see Figure 9(b)).

[0080] Next, computer 4 captures the video from each camera and creates a list of images (step S204). More specifically, computer 4 captures the video from cameras 31-39 received via network N at low resolution. The captured images from cameras 31-39 are temporarily stored in RAM 43.

[0081] Next, computer 4 captures an image for superimposition from the video of camera 1 (step S205). More specifically, computer 4 reads the camera whose position and orientation it previously notified to the simulator program (not shown) from RAM 43 (for example, it reads "camera 31"). It then captures the video from the camera received via network N in high resolution, uses it as the superimposition image 53 (see Figure 9(a)), and temporarily records it in RAM 43.

[0082] Next, the computer 4 captures the simulator screen 6 and masks everything except the robot (step S206). More specifically, it captures the image 63A (see Figure 9(b)) that the simulator program (not shown) displays on the simulator screen 6. Then, it temporarily saves the image with everything except the 3D model 61 of the robot (see Figure 9(b)) masked from the captured image as the simulator image 63B (see Figure 9(a)) in RAM 43.

[0083] Next, the computer 4 overlays the overlay image 53 (see Figure 9(a)) and the simulator image 63B (see Figure 9(a)) and displays them (step S207). More specifically, it overlays the overlay image 53 (see Figure 9(a)) and the simulator image 63B (see Figure 9(a)) that were previously recorded in the RAM 43 and displays them.

[0084] Next, the computer 4 displays a list of images 51 (step S208). More specifically, the computer 4 displays the low-resolution images from the captured cameras 31-39, which were temporarily recorded in the RAM 43 earlier, as thumbnails 51a on the list 51 of the composite screen 5 shown in Figure 9(a). When the computer 4 detects that a thumbnail 51a has been clicked, it temporarily records the camera corresponding to the clicked thumbnail 51a in the RAM 43 (for example, it records "camera 32").

[0085] Thus, the thumbnails 51a used in list 51 are captured at low resolution, which reduces the processing load on computer 4 and the usage of RAM 43, etc. Here, "low resolution" can be any resolution that reduces the processing load on computer 4, etc., but for example, it could be a resolution that results in a file size of 2MB or less.

[0086] Next, the computer 4 checks whether another camera has been selected in the image list 51 (step S209). More specifically, it checks whether the camera corresponding to the clicked thumbnail 51a is recorded in the RAM 43 (see Figure 2). If it is recorded (step S209:Y), the computer 4 searches for the recorded camera in the camera DB (see Figure 3(b)). The computer 4 then reads the position fld22 and orientation fld23 from the retrieved record and notifies the simulator program (not shown). The computer 4 also temporarily records in the RAM 43 which camera's position and orientation it notified (for example, "camera 32").

[0087] On the other hand, if thumbnail 51a has not been clicked (step S209:N), return to step S204.

[0088] Computer 4 repeats the processing described in steps S204 to S210 to achieve a predetermined frame rate (for example, 3 to 5 fps).

[0089] The user can operate the robot 2 (see Figure 2) on the simulator screen 6 using an input unit 41 such as a mouse (see Figure 2). More specifically, the user first moves the 3D model 61 (see Figure 9(b)) to a predetermined position on the simulator screen 6 by dragging or other means (see 3D model 62 shown in Figure 9(b)). The computer 4 repeats the processing in steps S204 to S210 (see Figure 5) at a predetermined frame rate (for example, 3 to 5 fps), so the moved 3D model 62 (see Figure 9(b)) is superimposed and reflected on the synthesis screen 5 (see 3D model 62 shown in Figure 9(a)). Therefore, the user can check whether it is OK to move the robot 2 in the same way as the 3D model 62 (see Figure 9(a)) while looking at the detailed display unit 52 of the synthesis screen 5. Once it is confirmed that there is no problem, the user clicks the move button 64 (see Figure 9(b)) displayed on the simulator screen 6. In response, computer 4 will instruct robot 2 (see Figure 2) to move.

[0090] According to the embodiment described above, calibration can be easily performed regardless of the position or orientation of the cameras 31 to 39.

[0091] In other words, conventionally, the calibration board is placed on the mounting surface TF (see Figure 2), and the imaging surface is horizontal. Furthermore, unless all feature points on the calibration board are captured and detected, the position of the calibration board cannot be detected. For this reason, if camera 3 (see Figure 2) is not facing the mounting surface TF and can not capture the entire calibration board, camera 3 (see Figure 2) cannot capture a calibration-ready image.

[0092] However, according to this embodiment, since the imaging surface of the marker M (see Figures 6-8) is positioned in a direction that intersects with the mounting surface TF (see Figure 2), there is a high probability that the marker M (see Figures 6-8) can be imaged regardless of the position and orientation of the camera 3 (see cameras 31 and 39 shown in Figure 2). Moreover, each marker M (see Figures 6-8) is uniquely identifiable (if one marker M can be imaged, the positions of the four corners of the marker M can be identified using the marker DB (see Figure 3(a))). Therefore, it is not necessary to image and detect all of the markers M (see Figures 6-8) as in the conventional method.

[0093] Therefore, by placing a uniquely identifiable marker M on the first auxiliary device 7 (see Figure 6) and using it, the likelihood of acquiring a calibrable image regardless of the camera's position and orientation increases.

[0094] Based on the above, according to this embodiment, calibration can be easily performed regardless of the position or orientation of the cameras 31 to 39.

[0095] <Explanation of variations> It should be noted that System 1 shown in this embodiment is merely an example, and various modifications and changes are possible within the scope of the gist of the present invention as described in the claims.

[0096] For example, in this embodiment, the imaging surface of the marker M is positioned in a direction intersecting the mounting surface TF of the turntable T. However, it is not always necessary to use the turntable T. Instead of using the turntable T, the imaging surface of the marker M can be positioned in a direction intersecting the horizontal plane (for example, the floor surface FL shown in Figure 1) by installing the first auxiliary device 7 on a predetermined horizontal plane (for example, the floor surface FL shown in Figure 1).

[0097] Furthermore, in this embodiment, each of the four sides of the columnar members 71 to 74 is erected perpendicular to the bottom member 75 (i.e., the mounting surface TF). However, each of the four sides of the columnar members 71 to 74 may be inclined at a predetermined angle.

[0098] Furthermore, in this embodiment, the first auxiliary device 7 is configured to have columnar members erected at the four corners of the base member 75, as shown in Figure 6, but it is not limited to this configuration. For example, the number and shape of the columnar members may be changed. Also, the first auxiliary device 7 may be in the shape of a cube, cylinder, triangular prism, or the like.

[0099] Furthermore, although the first auxiliary device 7 is used in this embodiment, the sign M may be placed in the work space K (see Figure 1) instead, or in addition to it. For example, it may be installed by leaning it against the base 20 (see Figure 2) of the robot 2 (see Figure 7(c)). Alternatively, it may be hung from the safety fence D (see Figure 1).

[0100] Even with the above modifications, the imaging surface of the marker M can be positioned in a direction that intersects the mounting surface TF, allowing for easy camera calibration regardless of the camera's position or orientation.

[0101] Furthermore, in this embodiment, the cameras 31 to 39 shown in Figure 2 are calibrated, and the superimposed screen is displayed as a result (see Figure 5). However, the use of the camera position and viewpoint information obtained by the calibration method according to the present invention (see Figure 3(b)) is not limited to the display of the superimposed screen shown in Figure 5. For example, when generating a multi-view image (an image of a subject captured from a desired position and viewpoint), it can be used to determine which camera's image will be used to generate the multi-view image. In another example, when reproducing real space as a three-dimensional space, the position and viewpoint information of each camera (see Figure 3(b)) can be used to determine which part of the three-dimensional space each camera's image will be used to reproduce.

[0102] <Explanation of the second assistive device> Furthermore, in this embodiment, calibration was performed using the first auxiliary tool 7 shown in Figure 6, but the second auxiliary tool 8 shown in Figure 8 may also be used. The second auxiliary tool 8 will be described below.

[0103] As shown in Figure 8, the second auxiliary device 8 has a cubic sign placement member 8b attached to the lower end of a rod-shaped support member 8a. On the other hand, the upper end is connected to the welding torch 24 via a disc-shaped plate 8a1. Also, as shown in Figure 8, a uniquely identifiable sign M is placed on each side of this sign placement member 8b.

[0104] The method of connecting the second auxiliary device 8 to the welding torch 24 can be anything; for example, the mounting mechanism 26 for position adjustment gauges (see Figure 8) conventionally provided on the welding torch 24 may be used. The mounting mechanism 26 for position adjustment gauges is a mechanism for attaching a position adjustment gauge that checks whether the TCP of the welding torch 24 is deviating from a predetermined position. The mounting mechanism 26 for position adjustment gauges is a conventionally known mechanism and is therefore not shown in the figure, but for example, if the mounting mechanism 26 for position adjustment gauges is a bolt hole, a bolt hole should be provided at the corresponding location on the plate 8a1 of the second auxiliary device 8. Then, the bolt hole provided on the plate 8a1 and the bolt hole of the mounting mechanism 26 for position adjustment gauges should be aligned and a bolt (not shown) should be fastened from below to fix the second auxiliary device 8 to the mounting mechanism 26 for position adjustment gauges. In this way, the second auxiliary device 8 can be easily attached to the welding torch 24 using the mounting mechanism 26 for position adjustment gauges conventionally provided on the welding torch 24.

[0105] Calibration using the second auxiliary tool 8 differs from the calibration process using the first auxiliary tool 7 (see Figure 4) in step S101, and will be explained below.

[0106] As preparation before performing step S101, the computer 4 first obtains the coordinates of the four corners of the marker M of the second auxiliary tool 8 in the TCP coordinate system TZ (see Figure 8), with the TCP of the welding torch 24 as the origin, and temporarily stores them in RAM 43. Any method of obtaining the coordinates is acceptable; for example, the computer 4 may display a screen on the synthesis screen 5 that accepts input from the operator, and have the operator input using an input unit 41 (see Figure 2), such as a keyboard. Alternatively, the computer 4 may accept input from the operator regarding the dimensions of the second auxiliary tool 8 and the length and width of the marker M, and calculate the coordinates from these values.

[0107] Furthermore, since the second auxiliary device 8 is attached to the welding torch 24, the positional relationship between the TCP of the welding torch 24 and the marker M of the second auxiliary device 8 on the TCP coordinate system TZ (see Figure 8) remains unchanged regardless of the orientation of the arm 22.

[0108] Then, as part of step S101, the computer 4 stops the arm 22 in a predetermined position via the PLC 29. The computer 4 then obtains the 3D coordinates of the TCP of the welding torch 24 in the robot coordinate system R from the PLC 29.

[0109] Then, the computer 4 uses the acquired 3D coordinates of the TCP in the robot coordinate system R to convert the coordinates of the four corners of the marker M in the TCP coordinate system TZ (see Figure 8) recorded in the RAM 43 into 3D coordinates in the robot coordinate system R. This conversion may also be performed, for example, by calculating the posture of the welding torch 24 using parameters such as the joint angles of each joint of the arm 22.

[0110] The 3D coordinates of the four corners of the marker M in the robot coordinate system R, calculated in this way, are stored in the coordinates fld12 of the marker DB (see Figure 3(a)). Therefore, by performing step S101 using the second auxiliary device 8, the marker DB (see Figure 3(a)) will contain the same number of records as the number of markers M placed on the second auxiliary device 8, and each record will contain the 3D coordinates of four robot coordinate systems R.

[0111] Steps S102 and beyond can be performed in the same manner as when using the first assistive device 7.

[0112] Furthermore, if steps S102 onwards are performed as follows, calibration can be performed with greater accuracy.

[0113] In other words, for each of the cameras 31 to 39, once the computer 4 has linked and stored the 2D coordinates of the four corners of each marker M recognized by the images captured by each camera 31 to 39 of the second auxiliary device 8 (see Figure 8) with the 3D coordinates of the four corners of the marker M in the robot coordinate system R (step S103-7), it returns to step S101 as shown above as a modified example. The computer 4 then stops the arm 22 (see Figure 8) in a different posture, recalculates the 3D coordinates of the four corners of the marker M of the second auxiliary device 8 (see Figure 8) in the robot coordinate system R in that posture, and updates the record in the marker DB (see Figure 3(a)) with the recalculated 3D coordinates. Next, it performs the processing from step S102 onwards again, links and stores the 2D coordinates and 3D coordinates (step S103-7), and returns once again to step S101 as shown above as a modified example. After performing this iterative process several times, the transformation matrix is ​​estimated for each camera (step S103-8), and the camera's position and orientation are determined (step S103-9).

[0114] In this way, by moving the arm 22 (see Figure 8), the welding torch 24 (see Figure 8) is moved, and the position of the marker M on the second auxiliary device 8 is changed, allowing each of the cameras 31-39 to image the second auxiliary device 8. This makes it possible to image more markers M. Therefore, higher-precision calibration becomes possible.

[0115] <Explanation of the use of unrecognizable signs> By the way, the marker Mx shown in Figure 7(b) cannot be uniquely identified because part of it is hidden by the columnar member 72 in the foreground. Therefore, the 3D coordinates of the marker Mx in the robot coordinate system R cannot be obtained from the identification DB of ROM 42 (see Figure 3(a)).

[0116] However, in such cases, the position of a recognizable marker M (see Figure 7(b)) may be used to estimate which marker Mx (see Figure 7(b)) corresponds to on the first auxiliary device 7. The three-dimensional coordinates of the estimated marker M in the robot coordinate system R may then be obtained from the identification DB (see Figure 3(a)).

[0117] For example, it is possible to estimate which marker M corresponds to marker Mx (see Figure 7(b)) as follows.

[0118] First, for example, when manufacturing the first assistive device 7, a three-dimensional coordinate system (not shown, hereinafter referred to as the three-dimensional coordinate system on the first assistive device 7) is set with a predetermined position on the first assistive device 7 as the origin, and the three-dimensional coordinates of the four corners of multiple markers M are obtained. The obtained three-dimensional coordinates are recorded in the ROM 42.

[0119] Next, the 3D coordinates of the marker M, which was recognized from the image captured by camera 3 (see Figure 7(b)), are read from ROM 42 in the 3D coordinate system on the first assistive device 7. Then, the 3D coordinate system on the first assistive device 7 is reconstructed using the read-out 3D coordinates.

[0120] Next, the 3D coordinates in the 3D coordinate system on the first auxiliary device 7 that have not yet been read from the ROM 42 (i.e., the 3D coordinates of the unrecognized marker M) are read from the ROM 42 and virtually placed on the reconstructed 3D coordinate system. These virtually placed markers M are designated as candidates for marker Mx.

[0121] Next, we estimate which of the candidate markers Mx is marker Mx. Various factors can be considered in this estimation. For example, candidates can be narrowed down based on the positional relationship between the recognized marker M and marker Mx in the image (see Figure 7(b)), or by determining the distance in the depth direction from the size ratio of the recognized marker M and marker Mx. Furthermore, in performing such estimation, a neural network can be used that has been trained using images of marker Mx partially hidden by columnar members 71-74 (e.g., Figure 7(b)) as training images, and which marker Mx corresponds to which marker M as a target signal.

[0122] In this way, by using the positions of recognizable markers M, it is possible to estimate which marker M an unrecognizable marker Mx (see Figure 7(b)) is. The estimated 3D coordinates of the marker M in the robot coordinate system R can be obtained from the identification DB in ROM42 (see Figure 2(a)). Therefore, for the corners p of the marker Mx that are not hidden by columnar members, etc. (see Figure 7(b)), a combination of the 2D coordinates on the image (see Figure 7(b)) and the 3D coordinates in the robot coordinate system R can be created. Thus, even unrecognizable markers Mx can be used for calibration.

[0123] In particular, by using the first auxiliary device 7 and the second auxiliary device 8, the position of each marker M can be measured and determined in advance when manufacturing these first and second auxiliary devices 7 and 8. Therefore, it becomes easy to use unrecognizable markers Mx (see Figure 7(b)) for calibration. [Explanation of Symbols]

[0124] 1 System 2 Robots 24 Welding Torch (Work Tool) 26. Mounting mechanism for position adjustment gauge 3, 31, 32, 39 Camera 4 Computers 5 Synthesis screen (display section) 7 First aid 71-74 Columnar members 8 Second aid TF mounting surface (plane) K work space M sign

Claims

1. A robot that performs a predetermined machining operation on a workpiece, A camera capable of imaging the robot or the workpiece, Multiple signs are placed within the workspace where the robot and the camera are installed, Computers and, A calibration method performed in a system equipped with, The camera performs an imaging step of capturing an image of at least one of the plurality of signs, The computer includes a determination step of determining the position and orientation of the camera based on the image captured by the camera, The aforementioned plurality of markers are uniquely identifiable, The imaging surfaces of the plurality of markers are arranged in a direction that intersects the horizontal plane. Calibration method.

2. The system further includes a first auxiliary device positioned so that the camera can take images, The first auxiliary device has a side surface located in a direction intersecting the horizontal plane, The calibration method according to claim 1, wherein the marker is positioned on the side surface of the first auxiliary device.

3. The first auxiliary device comprises a plurality of columnar members extending in a direction intersecting the horizontal plane, The aforementioned plurality of columnar members are erected with space between them, The calibration method according to claim 2, wherein the side surface of the first auxiliary device is the side surface of the plurality of columnar members.

4. The system further comprises a second auxiliary device provided on a work tool attached to the robot, The second auxiliary device has a side surface that is positioned in a direction intersecting the horizontal plane, The calibration method according to claim 1, wherein the marker is positioned on the side surface of the second auxiliary device.

5. The calibration method according to claim 4, wherein the second auxiliary device is provided on the work tool using a mounting mechanism for the position adjustment gauge of the work tool.

6. The camera is a plurality of cameras, The computer includes a storage step of recording the positions and orientations of the plurality of cameras determined by the calibration method described in claim 1, 2, or 4, The computer performs a list display step in which it acquires and displays captured images from the multiple cameras in order to create a list of the multiple cameras, The robot image acquisition step involves the computer acquiring an image captured by one camera selected from the list from that camera, The steps include: obtaining an image from a simulator that reproduces the robot in a virtual space, showing the robot as viewed from the position and orientation of one camera selected from the list; The computer performs a superimposed display step in which it superimposes and displays an image captured by one camera selected from the list and an image acquired from the simulator. Equipped with, The image captured for creating the aforementioned list is of lower quality than the image captured by the selected camera, in a display method.

Images