Industrial camera calibration operation support device

The support device assists in determining optimal image capture for industrial camera calibration by superimposing calibration pattern positions and effects, improving accuracy and reducing user effort through visual guidance.

JP7583249B2Active Publication Date: 2024-11-14DENSO WAVE INC
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2020184029
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-12-05
Filing Date
2020-11-03
Publication Date
2024-11-14
Estimated Expiration
2040-11-03

AI Technical Summary

Technical Problem

Existing calibration methods for industrial cameras lack guidance on determining the appropriate images to capture for accurate calibration, as they primarily focus on matching camera movement with the displayed calibration pattern without providing reference for image selection.

Method used

A support device that includes a camera, display device, and processing unit to superimpose the position and distribution of calibration patterns on predetermined images, aiding users in selecting optimal capture positions and improving calibration accuracy by displaying the effect of calibration patterns in two- and three-dimensional spaces.

Benefits of technology

Facilitates accurate placement of calibration patterns for improved image capture, reducing user effort by guiding optimal image selection and enhancing calibration accuracy through visual feedback on pattern positioning and effect distribution.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007583249000001
    Figure 0007583249000001
  • Figure 0007583249000002
    Figure 0007583249000002
  • Figure 0007583249000003
    Figure 0007583249000003
Patent Text Reader

Abstract

To support a user in determining a captured image to be captured in a support device that supports the user in operation of calibrating an industrial camera.SOLUTION: A calibration operation support device for an industrial camera is applied to a system (10) that includes a camera (11) that captures a predetermined area (20) set in advance, a display device (13) that displays a predetermined image (25) that is an image of a predetermined area being captured by a camera, and a processing device (12) that calibrates the camera on the basis of a plurality of captured images of a calibration pattern (30) taken by the camera, and the position of the calibration pattern in the plurality of captured images captured as an image used for calibration is superimposed on the predetermined image and displayed by the display device.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a support device that supports a user in performing calibration operations for an industrial camera. [Background technology]

[0002] Conventionally, there is a calibration operation support device that detects the orientation of a calibration pattern in captured image data obtained by capturing an image of the calibration pattern, and rotates and displays the captured image so that the detected orientation coincides with a predetermined orientation (see Patent Document 1). According to Patent Document 1, since the calibration pattern can always be displayed in a constant orientation, the movement of the hand moving the camera matches the movement of the displayed calibration pattern, making it easy to perform calibration operations. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2008-135996 A Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, in order to perform accurate camera calibration, it is necessary to capture the calibration pattern at various angles using as many parts of the captured image as possible. In Patent Document 1, although the movement of the hand moving the camera matches the movement of the displayed calibration pattern, this does not provide any reference for the user to determine what kind of captured image should be captured.

[0005] The present invention has been made to solve these problems, and its main purpose is to assist a user in determining what kind of captured images should be captured in an assistance device that assists a user in calibrating an industrial camera. [Means for solving the problem]

[0006] The first means for solving the above problem is 1. A calibration operation support device for an industrial camera, which is applied to a system including a camera that captures an image of a preset predetermined area, a display device that displays a predetermined image that is an image of the predetermined area being captured by the camera, and a processing device that calibrates the camera based on a plurality of captured images of a calibration pattern captured by the camera, The position of the calibration pattern in the plurality of captured images captured as images to be used for the calibration is superimposed on the predetermined image and displayed by the display device.

[0007] According to the above configuration, the camera captures an image of a predetermined area that has been set in advance. The display device displays a predetermined image that is an image of the predetermined area being captured by the camera. The processing device calibrates the camera based on a plurality of captured images of the calibration pattern captured by the camera.

[0008] Here, the calibration operation support device for industrial cameras displays the positions of the calibration patterns in the multiple captured images captured as images used for calibration on a display device by superimposing them on a predetermined image. This allows the user to know which positions in the predetermined area of ​​the captured images in which the calibration patterns have been placed have already been captured. This makes it easy for the user to place the calibration patterns in positions where they have not yet been placed and capture the captured images, thereby helping the user determine what kind of captured images should be captured.

[0009] The second method is 1. A calibration operation support device for an industrial camera, which is applied to a system including a camera that captures an image of a preset predetermined area, a display device that displays a predetermined image that is an image of the predetermined area being captured by the camera, and a processing device that calibrates the camera based on a plurality of captured images of a calibration pattern captured by the camera, The calibration pattern in the plurality of captured images captured as images to be used for the calibration has an effect of improving the accuracy of the calibration in the specified region, and the distribution of this effect is superimposed on the specified image and displayed on the display device.

[0010] According to the above configuration, the calibration operation support device for an industrial camera displays, on a display device, a distribution of the effect of the calibration patterns in the multiple captured images captured as images used for calibration on improving the accuracy of the calibration in a specified area (hereinafter referred to as "accuracy improvement effect"), superimposed on the specified image. This allows the user to understand to what extent the accuracy improvement effect of the calibration patterns in the multiple captured images is exerted on which parts of the specified area. This makes it easy for the user to place the calibration patterns in positions where the accuracy improvement effect is insufficient and capture the captured images, thereby supporting the user in determining what kind of captured images should be captured.

[0011] In a third aspect, the calibration pattern includes a plurality of feature points, and the distribution of the effect is a distribution of total values ​​at each position, the total value being a predetermined value that becomes smaller the further away from each feature point, with the position of each feature point as a vertex. With this configuration, the distribution of the accuracy improvement effect in a predetermined region can be easily expressed using a predetermined value that is largest at the position of each feature point and becomes smaller the further away from each feature point. Furthermore, since the distribution of the accuracy improvement effect is expressed as a distribution of total values ​​being a predetermined value at each position, it is possible to prevent the process of obtaining the distribution of the accuracy improvement effect from becoming complicated even when the number of captured images is large.

[0012] Specifically, as in the fourth means, a configuration can be adopted in which the predetermined value is a value based on each normal distribution with the position of each feature point as a peak. With this configuration, the accuracy improvement effect exerted by each feature point in the predetermined region can be reflected even in positions far from each feature point. Therefore, it becomes easy to quantitatively grasp to what extent the accuracy improvement effect is exerted on which part of the predetermined region.

[0013] In a fifth aspect, the predetermined value is a value greater than 0 throughout the entire predetermined region. This configuration makes it possible to prevent the occurrence of any part of the predetermined region where the accuracy improvement effect is 0. This makes it easier to quantitatively grasp the extent to which the accuracy improvement effect extends to which part of the predetermined region.

[0014] In a sixth method, the position of the calibration pattern in the specified region where the sum of the respective total values ​​in the region corresponding to the calibration pattern is smallest is determined as the position where the calibration pattern should be placed in the specified region, and is displayed by the display device superimposed on the specified image.

[0015] According to the above configuration, the position of the calibration pattern in the specified region can be determined as the position where the calibration pattern should be placed, where the sum of the total values ​​obtained by adding up the predetermined values ​​at each position within the region corresponding to the calibration pattern is smallest, that is, the position of the calibration pattern where the effect of improving accuracy of the calibration pattern in the already captured captured image is smallest. Then, the position where the calibration pattern should be placed in the specified region is displayed by the display device, superimposed on the specified image. Therefore, when photographing the calibration pattern with a camera, the user can easily understand where the calibration pattern should be placed.

[0016] If the camera has not been calibrated, the calibration operation support device cannot accurately calculate the position of each feature point of the calibration pattern in three-dimensional space based on the captured image, and therefore cannot accurately grasp the distribution of the accuracy improvement effect in a predetermined area in three-dimensional space.

[0017] In this regard, in the seventh means, the predetermined area is regarded as a two-dimensional plane, the position of each feature point is regarded as a position on the two-dimensional plane, and the distribution of the effect is superimposed on the predetermined image and displayed by the display device. With this configuration, even if the camera calibration has not been performed, the calibration operation support device can grasp and display the distribution of the accuracy improvement effect in the predetermined area as a two-dimensional plane. Therefore, it becomes easy for the user to determine what kind of captured image should be captured when performing the calibration.

[0018] As described above, if the camera has not been calibrated, the calibration operation support device cannot accurately grasp the distribution of the accuracy improvement effect in a predetermined area as a three-dimensional space, and therefore it is difficult to display the accurate position where the calibration pattern should be placed in the predetermined area as a three-dimensional space by superimposing it on the predetermined image.

[0019] In this regard, based on the seventh aspect, the eighth aspect causes the display device to display the positions at which the calibration pattern should be placed in the predetermined area as the two-dimensional plane, superimposed on the predetermined image. With this configuration, even if camera calibration has not been performed, the calibration operation support device can grasp the distribution of accuracy improvement effect in the predetermined area as a two-dimensional plane, and can therefore display the positions at which the calibration pattern should be placed, superimposed on the predetermined image.

[0020] After the camera calibration is performed, the camera parameters acquired by the calibration can be used to accurately calculate the position of each feature point of the calibration pattern in three-dimensional space based on the captured image.

[0021] Therefore, in the ninth aspect, after the calibration is performed, the predetermined area is regarded as a three-dimensional space using the parameters related to the camera acquired by the calibration, the positions of the feature points are regarded as positions in the three-dimensional space, and the distribution of the effect is superimposed on the predetermined image and displayed by the display device. With this configuration, the calibration operation support device for an industrial camera can accurately grasp and display the distribution of the accuracy improvement effect in the predetermined area as a three-dimensional space. Therefore, it becomes easy for a user to place a calibration pattern in a position where the accuracy improvement effect is insufficient in the predetermined area as a three-dimensional space and capture a captured image.

[0022] A tenth aspect of the present invention is to display, on the display device, a position at which the calibration pattern should be placed in the predetermined area as the three-dimensional space, superimposed on the predetermined image.

[0023] According to the above configuration, when capturing an image of the calibration pattern with a camera, the user can easily understand where in a predetermined area in three-dimensional space the calibration pattern should be placed.

[0024] In an eleventh aspect, the position at which the calibration pattern should be placed in the predetermined area is displayed on the display device so as to be superimposed on the predetermined image. With this configuration, a user can easily understand where the calibration pattern should be placed when photographing the calibration pattern with a camera.

[0025] In a twelfth aspect, the height of the calibration pattern in the plurality of captured images captured as images used for the calibration is superimposed on the predetermined image and displayed by the display device. With this configuration, the user can know at what height the captured images at which the calibration pattern has been arranged have already been captured. Therefore, it becomes easy for the user to place the calibration pattern at a height at which the calibration pattern has not yet been arranged and capture the captured images, and it is possible to assist the user in determining what kind of captured images should be captured.

[0026] In a thirteenth aspect, the height at which the calibration pattern should be disposed in the predetermined area is displayed on the display device so as to be superimposed on the predetermined image. With this configuration, a user can easily understand at what height the calibration pattern should be disposed when photographing the calibration pattern with a camera.

[0027] In a fourteenth aspect, the inclination of the calibration pattern in the plurality of captured images captured as images to be used for the calibration is superimposed on the predetermined image and displayed by the display device. With this configuration, the user can know at what inclination the captured images at which the calibration pattern has been arranged have already been captured. Therefore, it becomes easy for the user to place the calibration pattern at an inclination at which the calibration pattern has not yet been arranged and capture the captured images, and it is possible to assist the user in determining what kind of captured images should be captured.

[0028] In a fifteenth aspect, the display device displays at what angle the calibration pattern should be placed in the specified area, superimposed on the specified image. With this configuration, a user can easily understand at what angle the calibration pattern should be placed when photographing the calibration pattern with a camera.

[0029] In a sixteenth aspect, when the arrangement state of the calibration pattern in the predetermined area satisfies a predetermined condition, the image captured by the camera is imported as an image to be used for the calibration. With this configuration, the user does not need to perform an operation to import an image to be used for the calibration, thereby reducing the user's effort.

[0030] In a seventeenth aspect, when the number of the plurality of captured images captured as images to be used for the calibration exceeds a predetermined number, the processing device performs the calibration. With this configuration, the user does not need to perform an operation for the calibration, and the user's effort can be reduced. [Brief description of the drawings]

[0031] [Figure 1] System Schematic [Diagram 2] In the first embodiment, an image showing a calibration pattern being photographed and a position of the calibration pattern in a captured photographed image [Diagram 3] An image showing a calibration pattern and a predetermined straight line being photographed in the second embodiment [Figure 4] FIG. 4 is a schematic diagram showing feature points existing on a predetermined straight line in FIG. 3; [Diagram 5] Graph showing a comparative example of the distribution of the accuracy improvement effect on a specific straight line in FIG. [Figure 6] A graph showing the distribution of the accuracy improvement effect on a given line in FIG. [Figure 7] Flowchart showing the procedure for supporting calibration operations [Figure 8] Image showing the calibration pattern and scanning rectangle [Figure 9] Schematic diagram showing the distribution of accuracy improvement effects [Figure 10] FIG. 1 is a schematic diagram showing the positions of calibration patterns in a plurality of captured images; [Figure 11] Schematic diagram showing the distribution of accuracy improvement effect by using a calibration pattern on multiple captured images [Figure 12] A schematic diagram showing the distribution of the accuracy improvement effect in Figure 11 converted into a logarithmic representation. [Figure 13] FIG. 13 is a schematic diagram showing a mode of searching for a target arrangement position of a calibration pattern; [Figure 14] FIG. 1 is a schematic diagram showing a target arrangement position of a calibration pattern; [Figure 15] Schematic diagram showing the inclination of the calibration pattern [Figure 16] Schematic diagram showing the inclination distribution of the calibration pattern DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] (First embodiment) Hereinafter, a first embodiment embodied in a system used in a part assembly process (work process for a workpiece) by a robot (industrial machine) will be described with reference to the drawings.

[0033] As shown in FIG. 1, a system 10 includes a camera 11, a processing device 12, a display device 13, and the like.

[0034] Camera 11 (industrial camera) is capable of taking still and moving images, and transmits the data of the taken images to processing device 12. Camera 11 is fixed by stand 11a or the like so as to take images of a preset predetermined area 20. Predetermined area 20 is an area where parts are assembled (work on a workpiece) in a part assembly process by a robot, and has a predetermined width, depth, and height.

[0035] The display device 13 is configured with a liquid crystal screen or the like, and displays an image based on a signal input from the processing device 12 .

[0036] The processing device 12 is a computer equipped with a CPU, ROM, RAM, an operation unit, a storage device, an input / output interface, etc. The CPU executes various controls using programs and data stored in the ROM and RAM. The operation unit is composed of a keyboard, a mouse, etc., and inputs various instructions to the CPU by accepting user operations. The storage device stores various programs and data.

[0037] The processing device 12 causes the display device 13 to display a predetermined image 25 (captured image) captured by the camera 11. The predetermined image 25 is an image of the predetermined area 20 that is currently being captured (captured) by the camera 11. Fig. 1 shows a state in which a calibration pattern 30 is arranged within the predetermined area 20.

[0038] The calibration pattern 30 may have any known configuration, and may be formed, for example, in a plate shape with predetermined markers 30a marked in a predetermined arrangement. The calibration pattern 30 is used for processing to obtain camera parameters such as the lens center and focal length of the camera 11, i.e., for calibrating the camera 11.

[0039] The processing device 12 calibrates the camera 11 based on a plurality of captured images of the calibration pattern 30 captured by the camera 11. In detail, the camera 11 continuously captures the predetermined area 20, and the processing device 12 causes the display device 13 to continuously display the predetermined images 25 captured by the camera 11. However, at that stage, the predetermined images 25 are not taken into the processing device 12 as images to be used for calibration. Note that when the camera 11 starts capturing the predetermined area 20, the processing device 12 may determine whether calibration is possible. Whether calibration is possible can be determined based on, for example, the brightness of the image captured by the camera 11, the focus, the presence or absence of an obstruction, and the like. When the processing device 12 determines that calibration is possible, it may cause the display device 13 to display an instruction to execute calibration. When the processing device 12 determines that calibration is not possible, it may cause the display device 13 to display an instruction to correct the capturing conditions of the camera 11. In addition, if the processing device 12 has displayed an instruction to perform calibration on the display device 13 and then the calibration is not started within a specified period of time, the processing device 12 may cause the display device 13 to display the instruction to perform calibration again.

[0040] When the arrangement state of the calibration pattern 30 in the predetermined area 20 satisfies a predetermined condition, the processing device 12 imports the predetermined image 25 captured by the camera 11 as an image to be used for calibration. In other words, when importing the predetermined image 25 to be used for calibration into the processing device 12, the user does not need to perform any operation to import the predetermined image 25 into the processing device 12.

[0041] In detail, the processing device 12 specifies the position of the calibration pattern 30 based on the data of the image captured by the camera 11. A method for specifying the position of the calibration pattern 30 will be described later. The processing device 12 determines that the above-mentioned predetermined condition is satisfied when the specified position of the calibration pattern 30 is included in any of a plurality of position ranges that are desirable for performing calibration with high accuracy. The plurality of desirable position ranges are set, for example, near the four corners and near the center of the predetermined image 25.

[0042] Next, a description will be given of an assistance device that is applied to the system 10 and assists a user in performing an operation of calibrating the camera 11. In this embodiment, the assistance device (an industrial camera calibration operation assistance device) is configured with a processing device 12.

[0043] 2 is an image showing the calibration pattern 30 being photographed and the position of the calibration pattern 30 in the captured images. The processing device 12 causes the display device 13 to display the position of the calibration pattern 30 in the multiple captured images captured as images used for calibration, superimposed on the specified image 25.

[0044] In detail, the processing device 12 identifies the positions of the multiple markers 30a based on data of the captured image captured as an image used for calibration. For example, the processing device 12 identifies a contour line 31 (outline) of an area in which the identified multiple markers 30a exist as the position of the calibration pattern 30. Then, the processing device 12 displays the contour line 31 as a dashed line. The contour lines 32 and 33 represent the position of the calibration pattern 30 in a captured image that was captured with the calibration pattern 30 placed at a position different from the contour line 31.

[0045] As described above, contours 31-33 are displayed as the position of calibration pattern 30 that is already included in the captured captured image. This allows the user to easily determine the next position of calibration pattern 30 that is not yet included in the captured captured image. When calibration pattern 30 is placed in a desired position in predetermined area 20 by the user, processing device 12 determines that the above-mentioned predetermined condition is satisfied, and captures predetermined image 25 captured by camera 11 as an image to be used for calibration.

[0046] Then, the processing device 12 performs calibration each time the number of the captured images captured as images used for calibration exceeds a predetermined number. When the number of the captured images exceeds a predetermined number once, the processing device 12 performs calibration each time a new captured image is captured. This improves the accuracy of the calibration each time the calibration is performed. Since the method of performing calibration is well known, the description is omitted here. After that, the processing device 12 ends a series of processes for performing calibration when the camera parameters obtained as the calibration result are within the appropriate range. The processing device 12 may display the predetermined number of captured images required for calibration, the number of captured images currently captured, etc., on the display device 13. The processing device 12 may also display the degree of achievement or deficiency of the currently obtained camera parameters with respect to the values ​​in the appropriate range.

[0047] The present embodiment described above in detail has the following advantages.

[0048] The processing device 12 (calibration operation support device for industrial cameras) causes the display device 13 to display the positions of the calibration pattern 30 in the multiple captured images captured as images used for calibration, superimposed on the specified image 25. This allows the user to know in which position in the specified area 20 the captured images in which the calibration pattern 30 has been placed have already been captured. This makes it easy for the user to capture captured images by placing the calibration pattern 30 in a position where it has not yet been placed, thereby helping the user determine what kind of captured images should be captured.

[0049] When the arrangement of the calibration pattern 30 in the predetermined area 20 satisfies a predetermined condition, the processing device 12 imports the image captured by the camera 11 as an image to be used for calibration. With this configuration, the user does not need to perform any operation to import an image to be used for calibration, thereby reducing the user's effort.

[0050] When the number of the captured images used for calibration exceeds a predetermined number, the processing device 12 performs calibration. With this configuration, the user does not need to perform any operation for calibration, which reduces the user's effort.

[0051] The above embodiment can be modified as follows: The same parts as those in the above embodiment are denoted by the same reference numerals and the description thereof will be omitted.

[0052] The processing device 12 can also cause the display device 13 to display the position where the calibration pattern 30 should be placed in the predetermined area 20, superimposed on the predetermined image 25. Specifically, the processing device 12 causes the display device 13 to display any one of a plurality of position ranges desirable for performing calibration with high accuracy, in which a captured image in which the calibration pattern 30 is placed has not been captured, or a predetermined position within any one of the position ranges, superimposed on the predetermined image 25. With this configuration, the user can easily understand in which position the calibration pattern 30 should be placed when the calibration pattern 30 is photographed by the camera 11.

[0053] The processing device 12 can also display the heights of the calibration pattern 30 in the multiple captured images captured as images used for calibration on the display device 13 by superimposing them on the predetermined image 25. Specifically, the processing device 12 specifies the heights of the various parts of the calibration pattern 30 based on the data of the captured images captured as images used for calibration. The processing device 12 then displays the interiors of the contours 31 to 33 in colors corresponding to the heights of the various parts. With this configuration, the user can know at what height the captured images at which the calibration pattern 30 has been placed have already been captured. This makes it easy for the user to capture a captured image at a height at which the calibration pattern 30 has not yet been placed, thereby helping the user to determine what kind of captured image should be captured. It is also possible to calculate the average value of the heights of the various parts of the calibration pattern 30, and to display the interiors of the contours 31 to 33 in colors corresponding to the average value of the heights, or to display the average value of the heights numerically.

[0054] The processing device 12 can also display the height at which the calibration pattern 30 should be placed in the predetermined area 20 on the predetermined image 25 by the display device 13. Specifically, the processing device 12 displays on the display device 13 any of the height ranges, among a plurality of height ranges desirable for performing calibration with high accuracy, in which the captured image in which the calibration pattern 30 is placed has not been captured, or a predetermined position within any of the height ranges, by superimposing the height on the predetermined image 25. With this configuration, the user can easily understand at what height the calibration pattern 30 should be placed when the calibration pattern 30 is captured by the camera 11. Furthermore, when the user places the calibration pattern 30 at the desired height in the predetermined area 20, the processing device 12 may determine that the above-mentioned predetermined condition is satisfied and capture the predetermined image 25 captured by the camera 11 as an image to be used for calibration.

[0055] The processing device 12 can also display the inclination of the calibration pattern 30 in the multiple captured images captured as images used for calibration on the display device 13 by superimposing it on the predetermined image 25. Similarly to the above, the processing device 12 displays the inside of the contour lines 31 to 33 in a color according to the height of each part. The user can grasp the inclination of the calibration pattern 30 based on the degree of color change inside the contour lines 31 to 33. With this configuration, the user can grasp what inclination of the captured image at which the calibration pattern 30 has been arranged has already been captured. Therefore, it becomes easy for the user to place the calibration pattern 30 at an inclination at which it has not yet been arranged and capture the captured image, and therefore it is possible to assist the user in determining what kind of captured image should be captured. Note that the inclination of the calibration pattern 30 can be calculated based on the height of each part of the calibration pattern 30, and the calculated inclination can be displayed numerically.

[0056] The processing device 12 can also display, on the display device 13, the inclination at which the calibration pattern 30 should be arranged in the predetermined area 20, by superimposing it on the predetermined image 25. Specifically, the processing device 12 displays, on the display device 13, one of the inclination ranges, among a plurality of inclination ranges desirable for performing calibration with high accuracy, in which the captured image in which the calibration pattern 30 is arranged is not captured, or a predetermined inclination value within one of the inclination ranges. With this configuration, the user can easily understand the inclination at which the calibration pattern 30 should be arranged when the calibration pattern 30 is captured by the camera 11. Furthermore, when the user arranges the calibration pattern 30 in the predetermined area 20 with the desired inclination, the processing device 12 may determine that the above-mentioned predetermined condition is satisfied, and capture the predetermined image 25 captured by the camera 11 as an image to be used for calibration.

[0057] Second embodiment Hereinafter, a second embodiment will be described with reference to the drawings, in which the processing device 12 displays the distribution of the effect of improving the accuracy of calibration in a predetermined region 20 (hereinafter referred to as "accuracy improvement effect") of the calibration pattern 30 in a plurality of captured images captured as images to be used for calibration, superimposed on the predetermined image 25 and displayed by the display device 13. Note that the same parts as those in the first embodiment are denoted by the same reference numerals and description thereof will be omitted.

[0058] 3 is an image showing the calibration pattern 30 and the predetermined straight line L during shooting. The calibration pattern 30 is formed in a rectangular (rectangular) plate shape, and a plurality of feature points 30b are marked in an overall rectangular arrangement (predetermined arrangement). Here, the predetermined area 20 is regarded as a two-dimensional plane, and the position of each feature point 30b is regarded as a position on that two-dimensional plane. The predetermined straight line L is an arbitrary straight line drawn so as to overlap the calibration pattern 30 on that two-dimensional plane.

[0059] Fig. 4 is a schematic diagram showing the feature points 30b existing on the predetermined straight line L in Fig. 3. Here, four feature points 30b exist on the predetermined straight line L.

[0060] FIG. 5 is a graph showing a comparative example of the distribution of the accuracy improvement effect on the predetermined line L in FIG. 4. Each feature point 30b has the effect of improving the accuracy of calibration at each position. Therefore, the accuracy improvement effect is quantified by setting the accuracy improvement effect to 1.0 in an area (inside the contour line 31) where a plurality of feature points 30b exist in the calibration pattern 30, and setting the accuracy improvement effect to 0 in an area (outside the contour line 31) where no feature points 30b exist. In this case, the accuracy improvement effect is 0 if not inside the contour line 31, regardless of whether it is close to or far from the calibration pattern 30. Therefore, outside the contour line 31, it is not possible to show the difference in the accuracy improvement effect between a position close to the calibration pattern 30 and a position far from the calibration pattern 30.

[0061] FIG. 6 is a graph showing the distribution of the accuracy improvement effect on the predetermined straight line L in FIG. 4. Here, the accuracy improvement effect is quantified as a predetermined value that becomes smaller with the position of each feature point 30b as the apex and the farther away from the apex. More specifically, the predetermined value is a value based on each normal distribution 30c with the position of each feature point 30b as the apex. That is, the predetermined value is a value greater than 0 in the entire predetermined region 20. The distribution 30d of the accuracy improvement effect is a distribution of each total value obtained by totaling the above-mentioned predetermined values ​​at each position. In this case, outside the contour line 31, the closer to the calibration pattern 30, the greater the accuracy improvement effect, and the farther away from the calibration pattern 30, the smaller the accuracy improvement effect. Therefore, outside the contour line 31, the difference in the accuracy improvement effect can be shown between a position close to the calibration pattern 30 and a position far from the calibration pattern 30. Note that, although FIG. 6 shows the above-mentioned predetermined value on the predetermined straight line L as a two-dimensional graph, the above-mentioned predetermined value is a three-dimensional graph in the shape of a cone.

[0062] 7 is a flow chart showing the procedure for supporting the calibration operation. This series of processes is executed by the processing device 12.

[0063] First, the camera 11 captures an image of the predetermined area 20 (S10).

[0064] Next, the two-dimensional position of each of the feature points 30b is identified (S11). More specifically, in the predetermined image 25 (captured image) which is an image of the predetermined area 20 being captured by the camera 11, the predetermined area 20 is regarded as a two-dimensional plane, and the position of each of the feature points 30b is regarded as a position on the two-dimensional plane, and the position of each of the feature points 30b on the two-dimensional plane (hereinafter referred to as "two-dimensional position") is identified.

[0065] Next, it is determined whether or not a scanning rectangle has not been created (S12). As shown in FIG. 8, the scanning rectangle 40 is a rectangle (figure) corresponding to the contour line 31 (not shown) of an area in which a plurality of feature points 30b exist. In this determination, if it is determined that a scanning rectangle has not been created (S12: NO), the process proceeds to S14. On the other hand, in this determination, if it is determined that a scanning rectangle has not been created (S12: YES), the scanning rectangle 40 is created (S13). More specifically, a rectangle passing through each of the outermost feature points 30b is created based on the two-dimensional positions of each of the identified feature points 30b.

[0066] Next, it is determined whether or not the image capture condition (predetermined condition) is met (S14). More specifically, if the calibration pattern 30 is placed at a target arrangement position of the calibration pattern 30 described below or at a position near the target arrangement position, it is determined that the image capture condition is met, and otherwise it is determined that the image capture condition is not met. The position of the calibration pattern 30 is calculated based on the two-dimensional position of each of the identified feature points 30b. If it is determined that the image capture condition is not met (S14: NO), the process is executed again from S10.

[0067] On the other hand, if it is determined in the determination of S14 that the image capture condition is met (S14: YES), an image (predetermined image 25) of the predetermined area 20 being photographed by the camera 11 is captured (S15).

[0068] Next, it is determined whether calibration can be performed (S16). More specifically, if the number of captured images exceeds a predetermined number, it is determined that calibration can be performed. After the number of captured images exceeds the predetermined number once, it is determined that calibration can be performed every time a new captured image is captured. On the other hand, if the number of captured images does not exceed the predetermined number, it is determined that calibration cannot be performed. If it is determined that calibration cannot be performed (S16: NO), the process proceeds to S18. On the other hand, if it is determined that calibration can be performed (S16: YES), the process executes calibration (S17). Note that if the calibration of the camera 11 has not been performed, the processing device 12 cannot accurately calculate the position of each feature point 30b of the calibration pattern 30 in the three-dimensional space based on the captured images. For this reason, the processing device 12 cannot accurately grasp the distribution of the accuracy improvement effect in the predetermined area 20 as the three-dimensional space.

[0069] Next, a distribution of the accuracy improvement effect is created on a two-dimensional plane (S18). Fig. 9 is a schematic diagram showing the distribution of the accuracy improvement effect corresponding to the predetermined image 25 of Fig. 8. The closer to the center of the calibration pattern 30, the higher the value of the accuracy improvement effect contour line 51 becomes.

[0070] Next, the distribution of the accuracy improvement effect is displayed by superimposing it (S19). FIG. 10 is a schematic diagram showing the position of the calibration pattern 30 in a plurality of captured images. Here, the position of the calibration pattern 30 is shown by contour lines 31 and 32. FIG. 11 is a schematic diagram showing the distribution of the accuracy improvement effect corresponding to the predetermined image 25 in FIG. 10, and shows a plurality of accuracy improvement effects superimposed. The closer to the center of the two calibration patterns 30, the higher the accuracy improvement effect. At a position away from the center of the two calibration patterns 30, the difference in the accuracy improvement effect is difficult to see, but the further away from the center of the two calibration patterns 30, the lower the accuracy improvement effect. FIG. 12 is a schematic diagram showing the distribution of the accuracy improvement effect of FIG. 11 converted into a logarithmic display. In this figure, it is clearly shown that the accuracy improvement effect is lower as it is further away from the center of the two calibration patterns 30.

[0071] Next, the target arrangement position of the calibration pattern 30 on the two-dimensional plane is displayed (S20). FIG. 13 is a schematic diagram showing a mode of searching for the target arrangement position of the calibration pattern 30. Here, for convenience of explanation, the distribution of the accuracy improvement effect by one calibration pattern 30 is shown. While changing the position of the scanning rectangle 40 (scanning), the position where the sum of the accuracy improvement effect in the scanning rectangle 40 is the smallest is searched for. That is, the position of the calibration pattern 30 where the sum of the sums of the accuracy improvement effect at each position in the area corresponding to the calibration pattern 30 in the predetermined area 20 is the position where the calibration pattern 30 should be arranged in the predetermined area 20. Then, the position where the sum of the accuracy improvement effect in the scanning rectangle 40 is the smallest is displayed by the display device 13 while being superimposed on the predetermined image 25. FIG. 14 is a schematic diagram showing the target arrangement position 60 of the calibration pattern 30. The target arrangement position 60 is displayed by the same rectangle as the scanning rectangle 40.

[0072] Next, it is determined whether or not calibration has been performed (S21). If it is determined that calibration has not been performed (S21: NO), the process proceeds to S26. On the other hand, if it is determined that calibration has been performed (S21: YES), the three-dimensional position of each feature point 30b is identified (S22). In detail, the predetermined area 20 is regarded as a three-dimensional space using parameters related to the camera 11 acquired by the calibration, and the position of each feature point 30b is identified as a position in the three-dimensional space (three-dimensional position). After the calibration of the camera 11 is performed, the position of each feature point 30b of the calibration pattern 30 in the three-dimensional space can be accurately calculated based on the captured image using the parameters related to the camera 11 acquired by the calibration.

[0073] Next, a distribution of the accuracy improvement effect is created in a three-dimensional space (S23). In detail, the same process as in S18 and FIG. 9 is applied to the three-dimensional space and executed.

[0074] Next, the distribution of the accuracy improvement effect (height and gradient) is displayed in an overlapping manner (S24). In detail, the same process as that in S19 and Figs. 10 to 12 is applied to a three-dimensional space and executed.

[0075] Next, the target placement position of calibration pattern 30 in three-dimensional space is displayed (S25). More specifically, the same processing as in S20 and Figures 13 and 14 is applied to the three-dimensional space and executed, and the target placement position (height and tilt) of calibration pattern 30 is displayed as a three-dimensional position. Note that in addition to the position of calibration pattern 30 where the sum of the respective total values ​​of the accuracy improvement effect at each position in the region corresponding to calibration pattern 30 in predetermined region 20 is the smallest, the position of calibration pattern 30 where the sum of the respective total values ​​of the accuracy improvement effect is the second smallest may be displayed as the target placement position.

[0076] Next, it is determined whether or not calibration has been performed (S26). If it is determined that calibration has not been performed (S26: NO), the process proceeds to the process of S31. On the other hand, if it is determined that calibration has been performed (S26: YES), the inclination of the calibration pattern 30 is identified (S27). In detail, the three-dimensional position of each feature point 30b is identified, and the inclination of the plane through which each feature point 30b passes, that is, the inclination of the calibration pattern 30 is identified. FIG. 15 is a schematic diagram showing the inclination of the calibration pattern 30. Here, the inclination Rx and the inclination Ry are identified among the inclination Rx around the x-axis, the inclination Ry around the y-axis, and the inclination Rz around the z-axis. The inclination Rz around the z-axis is omitted because it has little effect on the accuracy improvement effect. For example, in the arrangement state P1, the inclination Rx=0° and the inclination Ry=0°. In the arrangement state P2, the inclination Rx=20° and the inclination Ry=20°.

[0077] Next, a gradient distribution of the calibration pattern 30 is created (S28). More specifically, a distribution in which the specified gradients Rx and Ry are at the apex and the values ​​become smaller the further away from the apex, is created based on a normal distribution.

[0078] Next, the tilt distributions are displayed in an overlapping manner (S29). Fig. 16 is a schematic diagram showing the tilt distribution of the calibration pattern 30. In this figure, the tilt distribution in the arrangement state P1 and the tilt distribution in the arrangement state P2 are displayed in an overlapping manner.

[0079] Next, the target arrangement angle of the calibration pattern 30 is displayed (S30). More specifically, the arrangement angle at which the distribution value is smallest within the range of the tilts Rx and Ry from -45° to 45° based on the tilt distribution in Fig. 16 is displayed as the target arrangement angle. Note that if the tilts Rx and Ry are out of the range of -45° to 45°, there is a risk that the calibration pattern 30 will not be recognized.

[0080] Next, it is determined whether or not the termination condition is satisfied (S31). In detail, if at least one of the following is satisfied, it is determined that the termination condition is satisfied, and if none of the following is satisfied, it is determined that the termination condition is not satisfied.

[0081] - The user performed an exit operation The number of multiple images imported has exceeded the end threshold. ·RMS (reprojection error) is below the threshold The difference between the previous and current RMS values ​​is below the threshold. The minimum value of the accuracy improvement effect in the predetermined region 20 is equal to or greater than the judgment value. The ratio of the area in which the minimum value of the accuracy improvement effect is equal to or greater than the judgment value in the predetermined area 20 is equal to or greater than a predetermined ratio. In the above judgment, if it is determined that the termination condition is not satisfied (S31: NO), the process is executed again from S10. On the other hand, in the above judgment, if it is determined that the termination condition is satisfied (S31: YES), this series of processes is terminated (END).

[0082] The present embodiment described above in detail has the following advantages. Here, only advantages different from the first embodiment will be described.

[0083] The processing device 12 of the camera 11 causes the display device 13 to display, superimposed on the predetermined image 25, the distribution of the effect of improving the calibration accuracy in the predetermined region 20 by the calibration pattern 30 in the multiple captured images captured as images used for calibration. This allows the user to understand to what extent the accuracy improvement effect of the calibration pattern 30 in the multiple captured images has been exerted on which parts of the predetermined region 20. This makes it easy for the user to place the calibration pattern 30 in a position where the accuracy improvement effect is insufficient and capture the captured image, thereby helping the user to determine what kind of captured image should be captured.

[0084] The calibration pattern 30 includes a plurality of feature points 30b, and the distribution of the accuracy improvement effect is a distribution of total values ​​at each position, where the position of each feature point 30b is the apex and the total value decreases the farther away from the apex. With this configuration, the distribution of the accuracy improvement effect in the specified region 20 can be easily expressed using a predetermined value that is largest at the position of each feature point 30b and decreases the farther away from each feature point 30b. Furthermore, since the distribution of the accuracy improvement effect is expressed as a distribution of total values ​​at each position, it is possible to prevent the process of obtaining the distribution of the accuracy improvement effect from becoming complicated even when the number of captured images captured is large.

[0085] The predetermined value is a value based on each normal distribution with the position of each feature point 30b as its apex. With this configuration, the accuracy improvement effect of each feature point 30b in the predetermined region 20 can be reflected up to positions far from each feature point 30b. Therefore, it becomes easy to quantitatively grasp to what part of the predetermined region 20 the accuracy improvement effect is reaching.

[0086] The predetermined value is a value greater than 0 throughout the entire predetermined region 20. This configuration makes it possible to prevent the occurrence of any part in the predetermined region 20 where the accuracy improvement effect is 0. This makes it easier to quantitatively grasp to what extent the accuracy improvement effect extends to which part of the predetermined region 20.

[0087] The position of calibration pattern 30 where the sum of the total values ​​obtained by adding up the predetermined values ​​at each position within the area corresponding to calibration pattern 30 is smallest, in other words, the position of calibration pattern 30 where the effect of improving accuracy by calibration pattern 30 in the already captured captured image is smallest, can be determined as the position at which calibration pattern 30 should be placed in predetermined area 20. Then, the position at which calibration pattern 30 should be placed in predetermined area 20 is displayed by display device 13, superimposed on predetermined image 25. Therefore, when photographing calibration pattern 30 with camera 11, the user can easily understand where calibration pattern 30 should be placed.

[0088] The predetermined area 20 is regarded as a two-dimensional plane, the position of each feature point 30b is regarded as a position on the two-dimensional plane, and the distribution of the accuracy improvement effect is superimposed on the predetermined image 25 and displayed by the display device 13. With this configuration, even if the calibration of the camera 11 has not been performed, the processing device 12 can grasp and display the distribution of the accuracy improvement effect in the predetermined area 20 as a two-dimensional plane. This makes it easy for the user to determine what kind of captured image should be captured when performing the calibration.

[0089] The positions at which the calibration pattern 30 should be placed in the predetermined area 20 as a two-dimensional plane are superimposed on the predetermined image 25 and displayed by the display device 13. With this configuration, even if the calibration of the camera 11 has not been executed, the processing device 12 can grasp the distribution of the accuracy improvement effect in the predetermined area 20 as a two-dimensional plane, and can thereby display the positions at which the calibration pattern 30 should be placed as superimposed on the predetermined image 25.

[0090] After calibration is performed, the parameters related to the camera 11 acquired by the calibration are used to regard the predetermined area 20 as a three-dimensional space, and the positions of the feature points 30b as positions within the three-dimensional space, and the distribution of the accuracy improvement effect is superimposed on the predetermined image 25 and displayed by the display device 13. With this configuration, the processing device 12 of the camera 11 can accurately grasp and display the distribution of the accuracy improvement effect in the predetermined area 20 as a three-dimensional space. Therefore, it becomes easy for the user to place the calibration pattern 30 at a position in the predetermined area 20 as a three-dimensional space where the accuracy improvement effect is insufficient, and capture a captured image.

[0091] The position at which calibration pattern 30 should be placed in predetermined area 20 as three-dimensional space is displayed by display device 13, superimposed on predetermined image 25. With this configuration, when photographing calibration pattern 30 with camera 11, the user can easily understand where in predetermined area 20 as three-dimensional space calibration pattern 30 should be placed.

[0092] The predetermined value is not limited to a value based on each normal distribution with the position of each feature point 30b as a vertex, but may also be a value based on an exponential distribution or triangular distribution with the position of each feature point 30b as a vertex. That is, the predetermined value may be a value that decreases the further away from the vertex, with the position of each feature point 30b as a vertex. Furthermore, the calibration pattern 30 is not limited to a configuration in which the feature points 30b are uniformly arranged, but may also be a configuration in which the density of the feature points 30b increases closer to the center of the calibration pattern 30. Furthermore, the predetermined value may be a value that decreases the further away from the center of the calibration pattern 30 or a value that decreases the further away from the contour line 31.

[0093] The first and second embodiments can be modified as follows: The same parts as those in the first and second embodiments are given the same reference numerals and the description thereof will be omitted.

[0094] The position of the calibration pattern 30 is not limited to the contours 31 to 33, and it is also possible to display only the corners of the contours 31 to 33, or to display the center of the calibration pattern 30.

[0095] The user may also be allowed to perform the operation for capturing the image used for calibration.

[0096] Calibration operations can also be performed by the user.

[0097] The camera 11 (industrial camera) may be fixed to the hand of a robot (industrial machine). In this case, the posture of the robot is controlled so that the camera 11 photographs a preset area 20, and then the camera 11 is calibrated. In this state, the posture of the robot can be controlled so that the position and angle at which the calibration pattern 30 is photographed are changed.

[0098] The camera 11 may have the function of the processing device 12, and the camera 11 may constitute an industrial camera calibration operation support device. Also, the display device 13 may have the function of the processing device 12, and the display device 13 may constitute a camera calibration operation support device.

[0099] The above-described embodiments and modifications may be combined as appropriate. [Explanation of symbols]

[0100] 10...system, 11...camera, 12...processing device (calibration operation support device for industrial cameras), 13...display device, 20...specified area, 25...specified image, 30...calibration pattern, 30b...feature points, 30d...distribution of accuracy improvement effect, 31...contour (position), 32...contour (position), 33...contour (position), 60...target placement position.

Claims

1. 1. A calibration operation support device for an industrial camera, which is applied to a system including a camera that captures an image of a preset predetermined area, a display device that displays a predetermined image that is an image of the predetermined area being captured by the camera, and a processing device that calibrates the camera based on a plurality of captured images of a calibration pattern captured by the camera, A calibration operation support device for an industrial camera, which displays, on the display device, the position of the calibration pattern in the plurality of captured images captured as images to be used for the calibration, superimposed on the specified image.

2. 2. The calibration operation support device for an industrial camera according to claim 1, wherein the position at which the calibration pattern is to be placed in the predetermined area is displayed by the display device so as to be superimposed on the predetermined image.

3. 3. The calibration operation support device for an industrial camera according to claim 1, wherein the height of the calibration pattern in the plurality of captured images captured as images used for the calibration is superimposed on the specified image and displayed by the display device.

4. 4. The calibration operation support device for an industrial camera according to claim 1, wherein the height at which the calibration pattern should be placed in the specified area is displayed by the display device in a manner superimposed on the specified image.

5. 5. The calibration operation support device for an industrial camera according to claim 1, wherein the inclination of the calibration pattern in the plurality of captured images captured as images to be used for the calibration is superimposed on the specified image and displayed by the display device.

6. 6. A calibration operation support device for an industrial camera according to claim 1, wherein the inclination at which the calibration pattern should be placed in the specified area is displayed on the display device by superimposing it on the specified image.

7. 7. The calibration operation support device for an industrial camera according to claim 1, wherein, when the arrangement state of the calibration pattern in the specified area satisfies a specified condition, the image captured by the camera is imported as an image to be used for the calibration.

8. 8. The calibration operation support device for an industrial camera according to claim 1, wherein the calibration is performed by the processing device when the number of the plurality of captured images captured as images to be used for the calibration exceeds a predetermined number.

Citation Information

Patent Citations

  • Information processing method and information processor

    JP2008135996A

  • Apparatus and method for processing information

    JP2008275340A

  • Information processing apparatus, information processing method, program, and system

    JP2016181753A

  • Computer-implemented method of calibrating camera

    JP2018007253A

  • Calibration method, calibration device, and calibration system having the device

    WO2009141998A1