Methods, systems, and storage media for visualizing calibration status of a camera

Abstract

The present invention provides a method, system, and storage medium for visualizing the calibration state of a camera, enabling confirmation of whether calibration parameters are correct. The method of the present disclosure includes: (a) a process of acquiring calibration parameters of a first camera; (b) a process of finding the relative position of the first camera and a second camera using the second camera; and (c) a process of superimposing a specific image showing the calibration state of the first camera on an image taken by the second camera, to display on a display device according to the position and posture of the value change of the calibration parameters.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a method, a system, and a storage medium that visualize a calibration state of a camera. BACKGROUND

[0002] In various devices such as robots, a technology that achieves an interaction action with an object by recognizing a position of the object using a camera is utilized. In a case where a camera is used in such a use, a calibration parameter of the camera is set by performing a pre-correction (calibration). The calibration parameter includes an internal parameter that represents a performance of a lens, a relationship between the lens and a pixel, and an external parameter that represents a relative position of the camera and an external device. In a case where a camera is used in a robot, by disposing the camera at a position where a relative position to the robot is fixed and a work area is entirely a field of view range, an external parameter between the camera and the robot can be calculated. A technology that thus calculates the external parameter is disclosed in Patent Literature 1.

[0003] Patent Literature 1: Japanese Patent Application Publication No. 2021-24056

[0004] However, the calibration parameter of the camera sometimes deviates with time. Therefore, there is a demand to confirm whether the calibration parameter is correct. SUMMARY

[0005] According to a first aspect of the present disclosure, a method of visualizing a calibration state of a camera is provided. The method includes: (a) a step of acquiring a calibration parameter of a first camera; (b) a step of calculating a relative position of the first camera and a second camera using the second camera; and (c) a step of superimposing a specific image that shows the calibration state of the first camera on an image captured by the second camera to display a position and posture that varies depending on a value of the calibration parameter on a display device.

[0006] According to a second aspect of the present disclosure, a system of visualizing a calibration state of a camera is provided. The system includes: a first camera that captures an image using a camera using device; a second camera for confirmation of a calibration state of the first camera; a display device that displays an image captured by the second camera; and a control device that is connected to the first camera, the second camera, and the display device. The control device performs: (a) a process of acquiring a calibration parameter of the first camera; (b) a process of calculating a relative position of the first camera and the second camera using the second camera; and (c) a process of superimposing a specific image that shows the calibration state of the first camera on an image captured by the second camera to display a position and posture that varies depending on a value of the calibration parameter on the display device.

[0007] According to a third aspect of the present disclosure, there is provided a storage medium storing a computer program causing a processor to execute a process of visualizing a calibration state of a camera. The computer program causes the processor to execute: (a) a process of acquiring a calibration parameter of a first camera; (b) a process of finding a relative position of the first camera and a second camera using the second camera; and (c) a process of superimposing a specific image showing the calibration state of the first camera on an image captured by the second camera to display on a display device in a position and posture varying according to a value of the calibration parameter. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 is an explanatory diagram showing the configuration of a robot system in an embodiment.

[0009] Figure 2 is a schematic diagram showing the relationship of various coordinate systems in the first embodiment.

[0010] Figure 3 is a functional block diagram of a control device in an embodiment.

[0011] Figure 4 is a flowchart showing the processing sequence of the first embodiment.

[0012] Figure 5 is an explanatory diagram showing a method of determining the relative position of the first camera and the second camera.

[0013] Figure 6 is an explanatory diagram showing an example of display of a specific image showing the calibration state.

[0014] Figure 7 is an explanatory diagram showing another example of display of a specific image showing the calibration state.

[0015] Figure 8 is an explanatory diagram showing an example of an adjustment window of a calibration parameter.

[0016] Figure 9 is a schematic diagram showing the relationship of various coordinate systems in the second embodiment.

[0017] Figure 10 is a flowchart showing the processing sequence of the second embodiment.

[0018] BRIEF DESCRIPTION OF DRAWINGS

[0019] 100 … robot; 110 … base; 120 … robot arm; 121 … first link; 122 … second link; 123 … working shaft; 150 … robot hand; 200 … robot controller; 300 … control device; 310 … processor; 312 … camera correction execution section; 314 … correction state visualization section; 320 … memory; 330 … interface circuit; 340 … input device; 350 … display device; 360 … wireless communication device; 410 … first camera; 420 … second camera; 430 … third camera; 440 … VR glasses; 500 … workbench; 501 … XY plane; 510 … corner frame; 520 … marker plate; 530 … reference marker. DETAILED DESCRIPTION

[0020] A. First Embodiment

[0021] Figure 1 is an explanatory diagram showing an example of a robot system in an embodiment. The robot system is provided with: a robot 100 as a camera utilization device, a robot controller 200 that controls the robot 100, a control device 300, three cameras 410, 420, 430, VR glasses 440 (Virtual Reality Glass), and a workbench 500. The control device 300 is, for example, a personal computer. The control device 300 transmits a control instruction to the robot controller 200. The control device 300 can also be referred to as an “upper-level control device”.

[0022] The robot 100 is provided with: a base 110, a robot arm 120. The robot arm 120 has a first link 121, a second link 122, and a working shaft 123. At the front end of the robot arm 120, a robot hand 150 as an end effector is installed. The robot hand 150 can be realized as a gripper that can hold a workpiece, a suction pad. In the vicinity of the front end portion of the robot hand 150, a TCP (Tool Center Point) as a control point of the robot 100 is provided. Note that the control point TCP can be provided at an arbitrary position.

[0023] The robot arm 120 is sequentially connected by four joints J1 to J4. In the present embodiment, a four-axis robot is exemplified, but a robot provided with an arbitrary robot arm mechanism having a plurality of joints can be used. In addition, the robot 100 of the present embodiment is a horizontal multi-joint robot, but a vertical multi-joint robot can also be used.

[0024] The first camera 410 is used to take an image of a workpiece that is a work object of the robot 100, and to calculate a three-dimensional position and attitude of the workpiece. As the first camera 410, an RGB camera is preferably used, but an RGBD camera, a monochrome camera can also be used. The RGBD camera is a camera having an RGB camera and a D camera (depth camera). In Figure 1 In the example, the first camera 410 is fixed at a position above the work table 500. However, the first camera 410 can be disposed at a position at which a relative position of the robot coordinate system and the first camera 410 can be determined, regardless of the attitude of the robot arm 120. For example, the first camera 410 can also be fixed to the second link 122 of the robot 100. The robot 100 is a camera-using device that uses the result of imaging by the first camera 410. However, as the camera-using device, any kind of device other than the robot 100 can be used.

[0025] The second camera 420 is used to take an image of a work environment including the work table 500, to confirm the calibration state of the first camera 410. As the second camera 420, an RGB camera is preferably used, but a monochrome camera can also be used, and in addition, a stereo camera that is not a single-lens reflex camera can also be used. In the case where the second camera 420 is a stereo camera, the marker only needs to be able to recognize the positions of specific reference points in left and right images, respectively, and can not be able to recognize a three-dimensional position and attitude from one image. In Figure 1 In the example, the second camera 420 is fixed to the VR glasses 440 mounted by the user. However, the second camera 420 can also be disposed at a fixed position within the work environment.

[0026] The third camera 430 is used to calculate depth information in the image taken by the first camera 410. As the third camera 430, for example, a camera that functions as a stereo camera in common with the first camera 410 can also be used. Alternatively, as the third camera 430, a depth camera that measures depth can also be used. In Figure 1 In the example, the third camera 430 is fixed in common with the first camera 410 at a position above the work table 500. However, the third camera 430 can be disposed at a position at which a relative position of the first camera 410 and the third camera 430 can be determined, regardless of the attitude of the robot arm 120. The third camera 430 can also be omitted.

[0027] The VR glasses 440 are a display device that displays an image taken by the second camera 420. The VR glasses 440 also display a specific image that indicates the calibration state of the first camera 410 as virtual reality. However, as the display device, any kind of display device other than the VR glasses 440 can be used.

[0028] The robot 100 is placed on the work table 500. A corner frame 510 showing a work area is provided at the four corners of the work table 500. A marker plate 520 is also depicted above the work table 500. The marker plate 520 is placed on the surface of the work table 500 in a horizontal state, but is depicted in a tilted posture above the work table 500 in the figure for ease of illustration. Figure 1

[0029] The marker plate 520 is used to find the relative positions of the first camera 410 and the second camera 420, and is a flat object on which a pattern for identifying the position and posture thereof is depicted on the surface. The surface pattern of the marker plate 520 is constituted by a marker that enables the position and posture of the marker plate 520 to be identified by analyzing an image thereof. Instead of the marker plate 520, a marker can also be provided at a position fixed relative to the first camera 410. For example, a marker can also be fixed to the frame of the first camera 410 or the robot 100. Note that in the case where the first camera 410 and the second camera 420 constitute a stereo camera, the marker need only enable the positions of specific reference points to be identified in the left and right images, respectively, and can not enable a three-dimensional position and posture to be identified from one image.

[0030] Figure 2 is a schematic diagram showing the relationship between various coordinate systems in the first embodiment. In this example, the origin positions of the respective coordinate systems are depicted at positions deviating from actual positions for ease of illustration, and the robot 100 is also depicted smaller than actual size. The marker plate 520 is provided at a predetermined position on the XY plane 501 of the work table 500. On the horizontal XY plane 501 of the work table 500, a reference marker 530 showing a reference position at which a specific image showing a calibration state is arranged is provided.

[0031] Figure 2 The coordinate systems depicted in the above are as follows.

[0032] (1) Robot coordinate system Σr

[0033] The robot coordinate system Σr is an orthogonal three-dimensional coordinate system taking a predetermined position of the robot 100 as the coordinate origin.

[0034] (2) Work table coordinate system Σt

[0035] The work table coordinate system Σt is an orthogonal three-dimensional coordinate system taking a predetermined position of the XY plane 501 of the work table 500 as the coordinate origin.

[0036] (3) Marker coordinate system Σm

[0037] The marker coordinate system Σm is an orthogonal three-dimensional coordinate system taking a predetermined position of the marker plate 520 as the coordinate origin.

[0038] ​(4) First camera coordinate system Σc

[0039] The first camera coordinate system Σc is an orthogonal three-dimensional coordinate system that takes a predetermined position of the first camera 410 as a coordinate origin.

[0040] (5) Second camera coordinate system Σg

[0041] The second camera coordinate system Σg is an orthogonal three-dimensional coordinate system that takes a predetermined position of the second camera 420 as a coordinate origin. The VR glasses 440 are display devices that display images captured by the second camera 420, and thus pixel coordinates of the VR glasses 440 are obtained by transforming three-dimensional coordinates of the second camera coordinate system Σg by a projection transformation matrix of the second camera 420.

[0042] (6) Third camera coordinate system Σd

[0043] The third camera coordinate system Σd is an orthogonal three-dimensional coordinate system that takes a predetermined position of the third camera 430 as a coordinate origin.

[0044] When the first camera 410 is calibrated with respect to the robot 100, the relative positions of the first camera coordinate system Σc and the robot coordinate system Σr become known. That is, each three-dimensional coordinate value in the first camera coordinate system Σc and the robot coordinate system Σr can be transformed using extrinsic parameters included in the calibration parameters of the first camera 410. It is assumed that the relative position of the robot coordinate system Σr and the table coordinate system Σt is also known. It is assumed that the relative position of the first camera coordinate system Σc and the marker coordinate system Σm is also known. In addition, it is assumed that the relative position of the first camera coordinate system Σc and the third camera coordinate system Σd is also known.

[0045] Instead of using the marker plate 520, a pattern serving as a marker can be fixed to the first camera 410 and the robot 100. In this case, the relative position of the first camera coordinate system Σc and the marker coordinate system Σm also becomes known. In addition, the relative position between the first camera coordinate system Σc and the marker coordinate system Σm is preferably calculated using extrinsic parameters of the first camera 410.

[0046] Figure 3 is a block diagram showing the functions of the control device 300. The control device 300 has a processor 310, a memory 320, and an interface circuit 330. The input device 340, the display device 350, and the wireless communication device 360 are connected to the interface circuit 330, and in addition, the robot controller 200, the first camera 410, and the third camera 430 are also connected to the interface circuit 330. The second camera 420 and the VR glasses 440 are wirelessly connected to the wireless communication device 360 of the control device 300 via a wireless communication device possessed by the VR glasses 440. However, the second camera 420 and the VR glasses 440 can also be connected to the control device 300 by wire.

[0047] The processor 310 has a function as the camera correction execution section 312 and the correction state visualization section 314. The camera correction execution section 312 performs a process of determining the calibration parameters of the first camera 410 by performing the calibration of the first camera 410 to the robot 100. The correction state visualization section 314 performs a process of displaying a specific image showing the calibration state of the first camera 410. The functions of the camera correction execution section 312 and the correction state visualization section 314 are realized by the processor 310 executing the computer program stored in the memory 320, respectively. However, a part or all of the functions of the camera correction execution section 312 and the correction state visualization section 314 can also be realized by hardware circuitry.

[0048] In the memory 320, the calibration parameters CP of the first camera 410 and the robot control program RP are stored. The calibration parameters CP include the intrinsic parameters IP and the extrinsic parameters OP. The intrinsic parameters IP are the inherent parameters of the lens system of the first camera 410, including a projection transformation matrix, a distortion parameter. The extrinsic parameters OP are parameters used when calculating the relative position between the first camera 410 and the robot 100, including a coordinate transformation matrix between the first camera coordinate system Sc and the robot coordinate system Sr. For other cameras, calibration parameters are also stored in the memory 320 as needed. The robot control program RP is composed of a plurality of commands that cause the robot 100 to act.

[0049] The correction state visualization section 314, in order to visualize the calibration state of the first camera 410, superimposes a specific image showing the calibration state of the first camera 410 on the image taken by the second camera 420. As the specific image, for example, a grid-shaped image indicating the position of the XY plane 501 of the worktable 500, a specific marker image can be used. In addition, a stereoscopic image can also be used as the specific image. The position at which the specific image is displayed is set at a position in which the three-dimensional coordinates in the first camera coordinate system Sc of a specific position observed from the first camera 410 are converted into the three-dimensional coordinates of the second camera coordinate system Sc. At this time, the three-dimensional coordinates of the specific position are calculated using the calibration parameters of the first camera 410. Therefore, the specific image is superimposed on the image taken by the second camera 420 to be displayed in a position and posture that varies according to the values of the calibration parameters. In the present embodiment, as the specific position at which the specific image is displayed, the XY plane 501, which is the horizontal surface of the worktable 500, is used. More specifically, the specific position is set at the reference position indicated by the reference marker 530 shown in FIG. 6. Hereinafter, a process of superimposing and displaying the specific image showing the calibration state of the first camera 410 on the position of the XY plane 501 of the worktable 500 taken by the second camera 420 will be described. Figure 2

[0050] Figure 4 ​is a flowchart showing a processing sequence in the first embodiment. In step S110, the calibration parameter CP of the first camera 410 is acquired. For example, in a case where the calibration processing is performed before step S110, the correction state visualization section 314 acquires by reading out the calibration parameter CP from the memory 320. On the other hand, in a case where the calibration processing is not performed, the camera correction execution section 312 performs the calibration processing of the first camera 410, and generates the calibration parameter CP. The calibration processing can be performed with a calibration plate in a known method. Note that, in a case where the surface of the marker plate 520 is drawn with the same calibration pattern as the calibration plate, the calibration processing of the first camera 410 can also be performed using the marker plate 520. At this time, in order to perform the photographing of the marker plate 520 at a plurality of angles, the marker plate 520 can also be held by the robot 100, and the photographing can be performed while changing the angle thereof.

[0051] The processing after step S120 is processing of visualizing the calibration parameter acquired in step S110. In a case where the processing after step S120 is performed after a considerable time has passed from the calibration processing, by the visualization processing, it is possible to confirm whether or not a deviation occurs in the calibration parameter over time. In addition, it is also possible to adjust the calibration parameter as necessary.

[0052] In step S120, the correction state visualization section 314 acquires the depth information of the specific position observed from the first camera 410. As described above, in the present embodiment, the "specific position" is the XY plane 501 of the horizontal of the workbench 500. As a method of acquiring the depth information, any one of various methods exemplified below can be used.

[0053] Method 1 of acquiring depth information

[0054] In the acquisition method 1, the depth of the specific position is measured using the third camera 430 whose relative position to the first camera 410 is known as a depth camera. In this method, it is possible to convert the depth measured by the third camera 430 into the depth of the first camera coordinate system Σc using the known coordinate transformation matrix between the first camera coordinate system Σc and the third camera coordinate system Σd.

[0055] Method 2 of acquiring depth information

[0056] In the acquisition method 2, the first camera 410 and the third camera 430 are used as a stereo camera, and the depth of the specific position is measured by stereo matching.

[0057] Method 3 of acquiring depth information

[0058] In the acquisition method 3, a pattern capable of identifying a three-dimensional position is arranged on the XY plane 501 of the work table 500, and the position is identified from a pattern image captured by the first camera 410 to determine the specific position, that is, the depth of the XY plane 501 of the work table 500. As the pattern, for example, the marker plate 520, the reference marker 530 can be used. In the case of using the marker plate 520, the depth of the XY plane 501 of the work table 500 is a value obtained by adding the thickness of the marker plate 520 to the depth of the surface position of the marker plate 520. In the case of using the acquisition method 3, the third camera 430 can be omitted.

[0059] Acquisition method 4 of depth information

[0060] In the acquisition method 4, in the case where the distance between the specific position, that is, the XY plane 501 of the work table 500 and the first camera 410 is known, the distance is used as the depth of the XY plane 501. In the case where the coordinate transformation matrix between the first camera coordinate system Sc and the work table coordinate system St is known, the depth information of the XY plane 501 can be acquired from the coordinate transformation matrix. As Figure 2 As described above, the coordinate transformation matrix between the first camera coordinate system Sc and the work table coordinate system St can be calculated using the external parameters of the first camera 410.

[0061] The acquisition methods 1 and 2 described above are the same in that the depth information is calculated using the third camera 430. In addition, in the acquisition methods 1 and 2, the depth related to an arbitrary shape can be acquired. Therefore, a specific object having a three-dimensional characteristic shape can be arranged in the field of view of the first camera 410, and the position of the specific object can be used as the specific position. In this case, if a specific image in which a three-dimensional specific object showing the calibration state of the first camera 410 is overlaid at the position of the specific object is displayed in the image of the second camera 420 and displayed on the display device, it has an advantage that the calibration state can be easily confirmed.

[0062] In step S130, the calibration state visualization unit 314 calculates the three-dimensional coordinate value of the specific position in the first camera coordinate system Sc using the depth information obtained in step S120 and the calibration parameters of the first camera 410. The calculation can be performed, for example, as follows.

[0063] The matrix K and the distortion parameters D included in the internal parameters of the first camera 410 are represented by the following formula.

[0064] [Math. 1]

[0065] [Math. 1]

[0066]

[0067] D = (k1 k2 p1 p2 k3 k4 k5 k6) (2)

[0068] Here, f x , f y is a focal distance, c x , c y is a principal point position, and k1 to k6 are radius direction distortion coefficients, and p1, p2 are circumferential direction distortion coefficients. The matrix K represents a projection transformation between a three-dimensional camera coordinate and a two-dimensional pixel coordinate.

[0069] If the pixel coordinates (u, v) of the first camera 410 are used in the above (1), (2), the distorted camera homogeneous coordinates (x d , y d ) can be calculated according to the following equations.

[0070] [Math. 2]

[0071] [Math. 2]

[0072]

[0073] In addition, by solving the following equation, the camera homogeneous coordinates (x c , y c ) from which the distortion is removed can be calculated.

[0074] [Math. 3]

[0075] [Math. 3]

[0076]

[0077]

[0078] The three-dimensional coordinate values (X c , Y c , Z c ) of the specific position in the first camera coordinate system Σc are calculated using the camera homogeneous coordinates (x c , y c ) from which the distortion is removed and the Z coordinate value Z c showing the depth of the specific position acquired in step S120, by the following equation.

[0079] [Math. 4]

[0080] [Math. 4]

[0081]

[0082] The three-dimensional coordinate values (X c , Y c , Zc ) using the calibration parameters of the first camera 410, as a position at which a specific image showing the calibration state of the first camera 410 is displayed.

[0083] Note that the three-dimensional coordinate values (X c , Y c , Z c ) obtained by the above (1) to (6) are not limited to the XY plane 501 of the worktable 500, and can calculate arbitrary positions within the imaging range of the first camera 410. Therefore, using the above (1) to (6), three-dimensional coordinate values can be calculated for other arbitrary specific positions such as the position of the surface of the marker plate 520, the position of a three-dimensional object, and the like.

[0084] In step S140, the calibration state visualization section 314 acquires the relative position of the first camera 410 and the marker plate 520. As Figure 2 explained above, in the first embodiment, the relative position of the first camera coordinate system Σc and the marker coordinate system Σm is known. That is, the coordinate transformation matrix [R|t]cm between the first camera coordinate system Σc and the marker coordinate system Σm is known. However, the coordinate transformation matrix [R|t]cm can also be obtained by multiplying the known coordinate transformation matrices [R|t]cr, [R|t]rt, [R|t]tm. [R|t]cr represents the coordinate transformation between the first camera coordinate system Σc and the robot coordinate system Σr, [R|t]rt represents the coordinate transformation between the robot coordinate system Σr and the worktable coordinate system Σt, and [R|t]tm represents the coordinate transformation between the worktable coordinate system Σt and the marker coordinate system Σm. The coordinate transformation matrix [R|t]cr between the first camera coordinate system Σc and the robot coordinate system Σr is an external parameter included in the calibration parameters of the first camera 410.

[0085] In step S150, the calibration state visualization section 314 captures the marker plate 520 by the second camera 420, and determines the relative position of the second camera 420 and the marker plate 520. In step S160, the calibration state visualization section 314 determines the relative position of the first camera 410 and the second camera 420 using the results of step S140 and step S150.

[0086] Figure 5is an explanatory diagram showing a determination method of the relative position of the first camera 410 and the second camera 420 in steps S140-S160. The marker plate 520 has a marker pattern capable of identifying a three-dimensional position of a marker coordinate system Σm. Therefore, by capturing the marker plate 520 by the second camera 420 to acquire a marker image, and analyzing the marker image, it is possible to determine the coordinate transformation matrix [R|t]mg between the marker coordinate system Σm and the second camera coordinate system Σg. Then, by multiplying the coordinate transformation matrices [R|t]cm, [R|t]mg, it is possible to find the coordinate transformation matrix [R|t]cg between the first camera coordinate system Σc and the second camera coordinate system Σg. In this way, in the first embodiment, the marker plate 520 is captured by the second camera 420, and the relative position of the first camera 410 and the second camera 420 is determined using the result of analyzing the image.

[0087] In Figure 4 Step S170, the correction state visualization section 314 converts the three-dimensional coordinate value of the first camera reference of the specific position into the three-dimensional coordinate value of the second camera reference. This transformation is a process of converting the three-dimensional coordinate value (X c , Y c , Z c ) of the specific position obtained in the above step S130 into the three-dimensional coordinate value in the second camera coordinate system Σg using the coordinate transformation matrix [R|t]cg obtained in step S160.

[0088] In step S180, the correction state visualization section 314 displays a specific image in which the imaging image of the second camera 420 is superimposed with the calibration state of the first camera 410, and displays it on the VR glasses 440 as a display device.

[0089] Figure 6is an explanatory diagram showing an example of a specific image showing the calibration state in step S180. In this example, the specific image SM1 is a lattice-shaped image showing the three-dimensional position of the XY plane 501 of the work table 500. The pixel coordinate value of the display position of the specific image SM1 is obtained by converting the three-dimensional coordinate value of the second camera coordinate system g obtained in step S170 into a two-dimensional pixel coordinate value by the projection transformation matrix of the second camera 420. In the processing of steps S120 to S170, since the calibration parameters of the first camera 410 are used, the specific image SM1 is superimposed on the image captured by the second camera 420 to be displayed in a position and posture that varies according to the value of the calibration parameters. The user can confirm whether the calibration state of the first camera 410 is correct by comparing the display position of this specific image SM1 with the display position of the XY plane 501 of the work table 500. In particular, in the present embodiment, since the reference mark 530 showing the reference position of the specific image SM1 is provided on the XY plane 501, the calibration state of the first camera 410 can be easily confirmed by comparing the specific image SM1 and the reference mark 530. In addition, in the present embodiment, since the display of the specific image SM1 is performed using the VR glasses 440 equipped with the second camera 420, when the user changes the position of the second camera 420, the display position of the XY plane 501 of the work table 500 and the specific image SM1 are changed in correspondence therewith. Therefore, there is an advantage that it is easy to confirm whether the position of the specific image SM1 coincides with the XY plane 501 of the work table 500.

[0090] Figure 7 is an explanatory diagram showing another display example of a specific image showing the calibration state. Here, an example in which a plurality of specific images SM1, SM2 can be displayed is shown according to a plurality of calibration parameter sets different from each other. As the calibration parameters of the first camera 410, a plurality of parameter sets are sometimes generated according to the difference in the pixel area of the first camera 410, the difference in the depth. In such a case, it is preferable to display a plurality of specific images corresponding to a plurality of parameter sets.

[0091] In Figure 7 the example, a drop-down menu PM for selecting a parameter set of the calibration parameters is displayed on the display screen of the VR glasses 440 as the display device. When the user selects one parameter set using the drop-down menu PM, a specific image corresponding to the parameter set is selectively displayed. Note that a plurality of specific images SM1, SM2 corresponding to a plurality of parameter sets can also be displayed at the same time. In this way, if a plurality of specific images SM1, SM2 corresponding to a plurality of parameter sets are selectively or simultaneously displayed, there is an advantage that the user can confirm which one of the plurality of parameter sets is accurate.

[0092] In Figure 4In step S190, the user observes the image displayed in step S180, and determines whether the calibration state of the first camera 410 is good. In the case where the calibration state is not good, the process proceeds to step S200, and adjustment of the calibration parameters is performed.

[0093] Figure 8 FIG. 19 is an explanatory diagram illustrating an example of an adjustment window W1 in which the calibration parameters displayed in step S200 are displayed on the display device. In this adjustment window W1, a pull-down menu PM for selecting a parameter set of the calibration parameters, and sliders SL1 to SL6 for adjusting the position and posture of the specific image SM1 are provided. The user can adjust the position and posture of the specific image SM1 using these sliders SL1 to SL6. Note that adjustment tools other than the sliders SL1 to SL6 can also be used. The correction state visualization section 314 adjusts the calibration parameters of the first camera 410 according to the adjustment amount of the specific image SM1 by the user. The adjustment target is mainly the extrinsic parameters. However, the intrinsic parameters can also be included in the adjustment target. Note that the pull-down menu PM can be omitted.

[0094] As the adjustment tools, adjustment tools that directly adjust the values of the calibration parameters of the first camera 410 can also be used. That is, at least one of the adjustment tools for adjusting the position and posture of the specific image SM1 and the adjustment tools for adjusting the values of the calibration parameters can be displayed on the display device together with the specific image SM1. If the values of the calibration parameters are adjusted according to the adjustment by the user, in the case where the calibration state illustrated by the specific image SM1 is not appropriate, the calibration parameters can be adjusted to appropriate values.

[0095] After the adjustment of the calibration parameters in step S200, the process returns to step S120, and the above-described processes of steps S120 to S190 are repeated. However, in the case where the adjustment of the calibration parameters does not affect the depth information, the process of step S120 is skipped. In step S190, in the case where the calibration state is determined to be good, the process of FIG. 18 ends. Figure 4

[0096] As described above, in the above-described first embodiment, the specific image SM1 illustrating the calibration state of the first camera 410 is superimposed on the image captured by the second camera 420, and is displayed in the position and posture that vary according to the values of the calibration parameters, and thus the calibration state of the first camera 410 can be visually confirmed.

[0097] ​Note that, as the specific image SM1 showing the calibration state, an image showing a stereoscopic object can also be displayed instead of a flat image. For example, an image showing a stereoscopic marker plate 520 can also be displayed as the specific image. In this way, by the user observing the relationship between this specific image and the position of the actual marker plate 520, it is possible to confirm whether the calibration state of the first camera 410 is correct. Note that the position at which the specific image showing the calibration state is displayed is not limited to the XY plane 501 of the worktable 500, and can be set at any position within the imaging range of the first camera 410.

[0098] B. Second Embodiment

[0099] Figure 9 is a diagram showing the relationship between various coordinate systems in the second embodiment. Note that the configuration of the device is the same as that of the first embodiment shown in Figure 1 and Figure 3 . In the second embodiment, the marker plate 520 is disposed at any position that can be imaged by the first camera 410 and the second camera 420.

[0100] In the second embodiment, as in the first embodiment, the relative position of the first camera coordinate system Sc and the robot coordinate system Sr is known, and the coordinate transformation matrix [R | t]cr thereof is included in the external parameters of the first camera 410. In addition, the coordinate transformation matrix [R | t]rt of the robot coordinate system Sr and the worktable coordinate system St is also known. On the other hand, the relative position of the first camera coordinate system Sc and the marker coordinate system Sm is unknown.

[0101] Figure 10 is a flowchart showing the processing sequence of the second embodiment. The point of difference from the processing sequence of the first embodiment shown in Figure 4 is that step S140 is replaced by step S145, and the other steps are substantially the same as Figure 4 .

[0102] In step S145, the correction state visualizing section 314 acquires a marker image by the first camera 410 capturing the marker plate 520, and determines the coordinate transformation matrix [R|t]cm between the first camera coordinate system Sc and the marker coordinate system Sm by analyzing the marker image. In step S150, as in the first embodiment, the marker plate 520 is captured by the second camera 420 and a marker image is acquired, and the coordinate transformation matrix [R|t]mg between the marker coordinate system Sm and the second camera coordinate system Sg is determined by analyzing the marker image. In step S160, using the results of step S145 and step S150, the relative positions of the first camera 410 and the second camera 420 are determined. More specifically, by multiplying the coordinate transformation matrices [R|t]cm, [R|t]mg, the coordinate transformation matrix [R|t]cg between the first camera coordinate system Sc and the second camera coordinate system Sg is determined.

[0103] The processes of steps S145, S150, S160 in the second embodiment are the same as the processes of steps S140, S150, S160 in the first embodiment in that the coordinate transformation matrix [R|t]cg representing the relative positions of the first camera 410 and the second camera 420 is determined using a marker image. The processes after step S170 are omitted from the description because they are the same as in the first embodiment.

[0104] Note that in step S120 of the second embodiment, as explained in the first embodiment as the acquisition method 4 of depth information, it is preferable to acquire the depth information of the XY plane 501 from the known coordinate transformation matrix [R|t]ct between the first camera coordinate system Sc and the table coordinate system St. This coordinate transformation matrix [R|t]ct can be obtained by multiplying the known coordinate transformation matrix [R|t]cr between the first camera coordinate system Sc and the robot coordinate system Sr and the known coordinate transformation matrix [R|t]rt between the robot coordinate system Sr and the table coordinate system St. The coordinate transformation matrix [R|t]cr between the first camera coordinate system Sc and the robot coordinate system Sr corresponds to the extrinsic parameters of the first camera 410. Therefore, the three-dimensional coordinate values of the second camera reference at the specific position calculated in steps S120 to S170 in the second embodiment are calculated using the extrinsic parameters of the first camera 410.

[0105] As described above, in the second embodiment, instead of using the known relative position of the first camera 410 and the marker, the relative position of the first camera 410 and the marker plate 520 is determined by performing the process of analyzing the marker image captured by the first camera 410, and the relative position of the first camera 410 and the second camera 420 is determined. In either case of the first embodiment and the second embodiment, any pattern that can determine the relative position with the first camera 410 can be used as the marker. The phrase "can determine the relative position with the first camera 410" includes both the case where the relative position with the first camera 410 is known as in the first embodiment and the case where the position with the first camera 410 can be determined by analyzing the image of the marker captured by the first camera 410 as in the second embodiment.

[0106] As described above, in the second embodiment, like the first embodiment, the specific image SM1 showing the calibration state of the first camera 410 is overlaid on the image captured by the second camera 420 to be displayed in the position and posture that varies depending on the value of the calibration parameter, and thus the calibration state of the first camera 410 can be visually confirmed.

[0107] Other modes:

[0108] The present disclosure is not limited to the above-described embodiments and can be implemented in various ways without departing from the scope of the gist thereof. For example, the present disclosure can also be implemented by the following aspects. The technical features in the above-described embodiments corresponding to the technical features in each of the following aspects can be appropriately replaced, combined, or deleted in order to solve part or all of the technical problems of the present disclosure or to achieve part or all of the effects of the present disclosure. Note that the technical features can also be appropriately deleted if it is not necessary for solving the technical problems.

[0109] (1) According to a first aspect of the present disclosure, there is provided a method of visualizing a calibration state of a camera. The method includes: (a) a process of acquiring a calibration parameter of a first camera; (b) a process of using a second camera to find a relative position of the first camera and the second camera; and (c) a process of overlaying a specific image showing the calibration state of the first camera on an image captured by the second camera to be displayed in a position and posture that varies depending on a value of the calibration parameter on a display device.

[0110] According to the method, the specific image showing the calibration state of the first camera is overlaid on the image captured by the second camera to be displayed in the position and posture that varies depending on the value of the calibration parameter, and thus the calibration state of the first camera can be visually confirmed.

[0111] (2) In the above method, the process (b) can also include: (bl) a process of determining a first three-dimensional coordinate value in a reference coordinate system of the first camera with respect to a specific position in an image capturing range of the first camera; (b2) a process of acquiring a marker image by capturing a marker capable of determining a relative position with the first camera by the second camera; and (b3) a process of determining a coordinate transformation matrix representing the relative position of the first camera and the second camera using the marker image. In addition, the process (c) can also include: (cl) a process of converting the first three-dimensional coordinate value in the reference coordinate system of the first camera to a second three-dimensional coordinate value in a reference coordinate system of the second camera using the coordinate transformation matrix; and (c2) a process of setting a position of the specific image in an image captured by the second camera at a position indicated by the second three-dimensional coordinate value.

[0112] According to this method, the relative position of the first camera and the second camera can be determined using a marker, and the position of the display object image can be determined.

[0113] (3) In the above method, the process (bl) can also include: a process of obtaining depth information of the specific position in the image captured by the first camera using a third camera; and a process of calculating the first three-dimensional coordinate value of the specific position using a pixel coordinate value of the first camera related to the specific position, the depth information, and an internal parameter included in the calibration parameter.

[0114] According to this method, the three-dimensional coordinate value of the specific position in the reference coordinate system of the first camera can be obtained using depth information obtained by the third camera.

[0115] (4) In the above method, the process (bl) can also include: a process of obtaining depth information of the specific position in the image captured by the first camera using a pattern capable of recognizing a three-dimensional position; and a process of calculating the first three-dimensional coordinate value of the specific position using a pixel coordinate value of the first camera related to the specific position, the depth information, and an internal parameter included in the calibration parameter.

[0116] According to this method, the three-dimensional coordinate value of the specific position in the reference coordinate system of the first camera can be obtained using depth information obtained from a pattern capable of recognizing a three-dimensional position.

[0117] (5) In the above method, the specific position is a horizontal XY plane, and the specific image can also be an image representing the XY plane.

[0118] According to this method, the calibration state of the first camera can be visually confirmed by observing the relationship between the XY plane and the specific image displayed on the display device.

[0119] (6) In the above method, the calibration parameter includes a plurality of sets of calibration parameters, and the process (c) can also include a process of displaying, on the display device, a plurality of the specific images corresponding to the plurality of sets of calibration parameters, respectively.

[0120] According to this method, the plurality of specific images corresponding to the plurality of sets of calibration parameters can be visually confirmed.

[0121] (7) The above method can also further include: (d) a process of displaying, on the display device together with the specific image, an adjustment tool for adjusting at least one of a position and posture of the specific image and a value of the calibration parameter; and (e) a process of adjusting the value of the calibration parameter according to adjustment by a user using the adjustment tool.

[0122] According to this method, the calibration parameter can be adjusted to an appropriate value in a case where the calibration state shown by the specific image is not appropriate.

[0123] (8) According to a second aspect of the present disclosure, there is provided a system for visualizing a calibration state of a camera. The system includes a first camera configured to capture an image using a camera capturing device, a second camera configured to confirm a calibration state of the first camera, a display device configured to display an image captured by the second camera, and a control device connected to the first camera, the second camera, and the display device. The control device is configured to perform: (a) a process of obtaining a calibration parameter of the first camera; (b) a process of calculating a relative position of the first camera and the second camera using the second camera; and (c) a process of superimposing a specific image showing the calibration state of the first camera on the image captured by the second camera to display on the display device in a position and posture that varies according to a value of the calibration parameter.

[0124] (9) According to a third aspect of the present disclosure, there is provided a computer program for causing a processor to perform a process for visualizing a calibration state of a camera. The computer program causes the processor to perform: (a) a process of obtaining a calibration parameter of a first camera; (b) a process of calculating a relative position of the first camera and a second camera using the second camera; and (c) a process of superimposing a specific image showing the calibration state of the first camera on an image captured by the second camera to display on a display device in a position and posture that varies according to a value of the calibration parameter.

[0125] The present disclosure can also be realized in various other forms than those described above. For example, it can be realized in the form of a robot system that includes a robot and a robot control device, a computer program for realizing the functions of the robot control device, a non-transitory storage medium that records the computer program, and the like.

Claims

1. A method of visualizing a calibration status of a camera, characterized in that, comprising: (a) a process of acquiring calibration parameters of a first camera; (b) a process of finding a relative position of the first camera and a second camera using the second camera; and (c) a process of superimposing a specific image showing a calibration state of the first camera on an image taken by the second camera in order to visualize the calibration state of the first camera, to display a position and posture of the first camera on a display device in accordance with a change in a value of the calibration parameters.

2. The method according to claim 1, wherein the process (b) comprises: (b1) a process of determining a first three-dimensional coordinate value in a reference coordinate system of the first camera with respect to a specific position within an image-capturing range of the first camera; (b2) a process of acquiring a marker image by taking a marker capable of determining a relative position with the first camera by the second camera; and (b3) a process of determining a coordinate transformation matrix representing the relative position of the first camera and the second camera using the marker image, the process (c) comprises: (c1) a process of converting the first three-dimensional coordinate value in the reference coordinate system of the first camera into a second three-dimensional coordinate value in a reference coordinate system of the second camera using the coordinate transformation matrix; and (c2) a process of setting a position of the specific image in the image taken by the second camera at a position shown by the second three-dimensional coordinate value.

3. The method according to claim 2, wherein the process (bl) comprises: a process of finding depth information of the specific position in an image taken by the first camera using a third camera; and a process of calculating the first three-dimensional coordinate value of the specific position using a pixel coordinate value of the first camera with respect to the specific position, the depth information, and an internal parameter included in the calibration parameters.

4. The method according to claim 2, wherein the process (bl) comprises: a process of finding depth information of the specific position in an image taken by the first camera using a pattern capable of recognizing a three-dimensional position; and a process of calculating the first three-dimensional coordinate value of the specific position using a pixel coordinate value of the first camera with respect to the specific position, the depth information, and an internal parameter included in the calibration parameters.

5. The method according to any one of claims 2 to 4, wherein the specific position is a horizontal XY plane, and the specific image is an image representing the XY plane.

6. The method according to claim 1, wherein the calibration parameters include a plurality of sets of calibration parameters, and the process (c) comprises: a process of displaying a plurality of the specific images corresponding to the plurality of sets of calibration parameters, respectively, on the display device. Further comprising: (d) a process of displaying an adjustment tool for adjusting at least one of a position and posture of the specific image and a value of the calibration parameters on the display device together with the specific image; and (e) a process of adjusting the value of the calibration parameters in accordance with an adjustment by a user using the adjustment tool. provided with: ​ ​ ​ ​ ​ ​ 7. The method of claim 1, wherein, ​ ​ ​ 8. A system for visualizing the calibration status of a camera, characterized in that ​ a first camera for taking images with a camera using device; a second camera for the confirmation of the calibration status of the first camera; a display device for displaying images taken by the second camera; and a control device connected with the first camera, the second camera and the display device, the control device performing: (a) a process for obtaining calibration parameters of the first camera; (b) a process for finding the relative position of the first camera and the second camera using the second camera; and (c) a process for visualizing the calibration status of the first camera by superimposing a specific image showing the calibration status of the first camera on an image taken by the second camera in a position and attitude that varies according to the value of the calibration parameters on the display device. a computer program for storing, which causes a processor to perform a process for visualizing the calibration status of a camera, the processor performing: (a) a process for obtaining calibration parameters of a first camera; 9. A storage medium, characterized by (b) a process for finding the relative position of the first camera and a second camera using the second camera; and (c) a process for visualizing the calibration status of the first camera by superimposing a specific image showing the calibration status of the first camera on an image taken by the second camera in a position and attitude that varies according to the value of the calibration parameters on a display device. ​ ​ ​

Images