Calibration method, calibration device, calibration system, and computer program

Abstract

This calibration method captures an image, using at least two cameras, of a member having a predetermined design in which a pattern changes depending on the position of a surface of the member and a marker arranged at a predetermined position (S11), and calibrates the at least two cameras using the image of the member captured by the at least two cameras (S12). According to this calibration method, displacement of a plurality of cameras can be reduced using a comparatively simple method.

Description

Technical Field

[0001] This disclosure relates to the technical field of calibration methods, calibration devices, calibration systems, and recording media for calibrating cameras.

Background Art

[0002] As a method for calibrating a camera, a method using a calibration member imaged by the camera is known. For example, Patent Document 1 discloses calibrating a camera by imaging a target provided with an Aruco marker. Patent Document 2 discloses estimating the position and orientation of a camera by imaging a large number of installed calibration boards and performing calibration. Patent Document 3 discloses calibrating a camera by imaging a calibration board with known geometric and optical characteristics. Patent Document 4 discloses performing calibration by imaging a square grid of a flat plate while shifting the position of a cart on which the camera is mounted.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Summary of the Invention

Problems to be Solved by the Invention

[0004] This disclosure aims to improve the related technologies described above.

Means for Solving the Problems

[0005] One aspect of the calibration method of this disclosure is to image a member having a predetermined pattern whose pattern changes according to the position of the member surface and a marker arranged at a predetermined position with at least two cameras, and use the images of the member captured by the at least two cameras to calibrate the at least two cameras.

[0006] One aspect of the calibration apparatus of this disclosure includes an acquisition unit that acquires an image of a member having a predetermined pattern whose pattern changes according to the position of the member surface and a marker arranged at a predetermined position, imaged with at least two cameras, and a calibration unit that performs the at least two calibrations using the images of the member captured by the at least two cameras.

[0007] One aspect of the calibration system of this disclosure includes a member having a predetermined pattern whose pattern changes according to the position of the member surface and a marker arranged at a predetermined position, a driving device that drives the member to change the position or angle of the member with respect to at least two cameras, and a calibration device that performs the at least two calibrations using the images of the member captured by the at least two cameras.

[0008] One aspect of the recording medium of this disclosure has recorded a computer program that operates a computer to calibrate at least two cameras by imaging a member having a predetermined pattern whose pattern changes according to the position of the member surface and a marker arranged at a predetermined position with at least two cameras and using the images of the member captured by the at least two cameras.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments of a calibration method, a calibration device, a calibration system, a computer program, and a recording medium will be described with reference to the drawings.

[0011] <First Embodiment> The calibration method according to the first embodiment will be described with reference to FIGS. 1 to 3.

[0012] (Configuration of Camera) First, the configuration of the camera that is the object of the calibration method according to the first embodiment will be described with reference to FIG. 1. FIG. 1 is a schematic configuration diagram showing a camera and a calibration member to be calibrated by the calibration method according to the first embodiment.

[0013] As shown in FIG. 1, in the calibration method according to the first embodiment, the first camera 110 and the second camera 120 are calibrated. The first camera 110 and the second camera 120 are arranged, for example, so as to be able to image the same subject from different angles. However, the first camera 110 and the second camera 120 may be arranged so as to be able to image the subject from the same angle. The first camera 110 and the second camera 120 may include, for example, solid-state imaging devices such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. Further, the first camera 110 and the second camera 120 may include an optical system that forms an image of the subject on the imaging surface of the solid-state imaging device, a signal processing circuit that processes the output of the solid-state imaging device to obtain the luminance value for each pixel, and the like.

[0014] In the calibration method according to the first embodiment, the first camera 110 and the second camera 120 image a common calibration member 200, respectively. The calibration member 200 is, for example, a plate-like member and is configured to be usable when held by a person. The calibration member 200 may be configured to be usable by being placed at a predetermined location or attached to a support member. When executing the calibration method according to the first embodiment, a user holding the calibration member 200 may move so that the calibration member 200 is within the imaging ranges of the first camera 110 and the second camera 120. Alternatively, the calibration member 200 may be used in a state fixed to a predetermined support member. In this case, the calibration member 200 may be arranged within the imaging ranges of the first camera 110 and the second camera 120 by the user moving the fixed support member. Alternatively, the calibration member 200 may be used in a state drivable by a predetermined driving device. In this case, the calibration member 200 may be arranged within the imaging ranges of the first camera 110 and the second camera 120 by being driven by the driving device (for example, the position and angle are changed). A more specific configuration of the calibration member 200 will be described in detail below.

[0015] (Configuration of the Calibration Member) Next, the configuration of the calibration member 200 used in the calibration method according to the first embodiment will be specifically described with reference to FIG. 2. FIG. 2 is a plan view showing the configuration of the calibration member used in the calibration method according to the first embodiment.

[0016] As shown in FIG. 2, the calibration member 200 used in the calibration method according to the first embodiment has a predetermined pattern. Here, the "predetermined pattern" is a pattern whose pattern changes depending on the position on the member surface. More specifically, the predetermined pattern may be any pattern that can determine which part of the calibration member 200 is captured in the captured image, or at what angle or in what orientation the calibration member 200 is captured (for example, when it is upside down). An example of the predetermined pattern is a camouflage pattern as shown in FIG. 2, but it is not limited thereto. When using the calibration member 200 with a camouflage pattern, it is desirable that the first camera 110 and the second camera 120 be configured as cameras capable of performing shape measurement (for example, 3D scanners, range finders, etc.). That is, it is desirable to perform 3D shape measurement with the first camera 110 and the second camera 120 and perform calibration according to the results. Such calibration will be described in detail in other embodiments described later.

[0017] The calibration member 200 further has a marker 205. The marker 205 is arranged at a predetermined position of the calibration member 200. The marker 205 may be arranged, for example, so as to overlap a predetermined pattern of the calibration member 200. A plurality of markers 205 may be arranged on the calibration member 200. In this case, the arrangement positions of the plurality of markers 205 may be a predetermined arrangement as shown in FIG. 2, for example.

[0018] Note that the arrangement of the plurality of markers 205 shown in FIG. 2 is an example, and the number and arrangement pattern of the plurality of markers 205 are not particularly limited. In the example shown in FIG. 2, the plurality of markers 205 are arranged so as to be concentrated near the center of the calibration member 200, but they may be arranged evenly throughout the calibration member 200. Alternatively, the plurality of markers 205 may be arranged only at specific positions (for example, the four corners of the calibration member 200, etc.) of the calibration member 200.

[0019] On the other hand, only one marker 205 may be arranged on the calibration member 200. In this case, it is preferable that the marker 205 can specify not only its position but also its orientation. That is, it is preferable that just by detecting one marker 205 from the captured image, it is possible to estimate which part of the calibration member is being imaged and in what orientation the calibration member is being imaged.

[0020] The calibration member 200 described above is typically configured as a planar member, but it may be a member having at least a partially curved surface. Also, when the shape of the subjects of the first camera 110 and the second camera 120 (that is, the objects to be imaged in the operation after calibration) is known, the calibration member 200 may have a shape corresponding to the shape of the subject. For example, when the subjects of the first camera 110 and the second camera 120 are "a human face", the calibration member 200 may be configured as a member having a shape close to that of a human face. Further, the calibration member 200 may be a member having unevenness. The unevenness in this case may be uniformly present on the calibration member 200, or may be unevenness present only at specific positions. For example, as described above, when the subjects of the first camera 110 and the second camera 120 are "a human face", unevenness corresponding to a human's eyes, nose, ears, mouth, etc. may be provided. Alternatively, unevenness corresponding to a predetermined pattern of the calibration member 200 may be provided.

[0021] The calibration member 200 may have a honeycomb structure in order to achieve weight reduction and enhance rigidity. For example, the calibration member 200 may be configured as an aluminum honeycomb board. However, the material constituting the calibration member 200 is not particularly limited.

[0022] (Flow of operations) Next, the flow of operations of the calibration method according to the first embodiment will be described with reference to FIG. 3. FIG. 3 is a flowchart showing the flow of operations of the calibration method according to the first embodiment.

[0023] As shown in FIG. 3, in the calibration method according to the first embodiment, first, the first camera 110 and the second camera 120 respectively capture images of the calibration member 200 (step S11). It is preferable that the imaging by the first camera 110 and the imaging by the second camera 120 be performed at as close a timing as possible (preferably simultaneously). Note that the images captured by the first camera 110 and the second camera 120 may include the entire calibration member 200, or may include only a part of the calibration member 200. When an image including only a part of the calibration member 200 is captured, the calibration member 200 may be arranged at a position such that a common part of the calibration member 200 is imaged by the first camera 110 and the second camera 120.

[0024] Subsequently, based on the images of the calibration member 200 captured by the first camera 110 and the second camera 120 (more specifically, the set of the image captured by the first camera 110 and the image captured by the second camera 120), camera calibration of the first camera 110 and the second camera 120 is performed (step S12). Specifically, calibration is performed using a predetermined pattern and the marker 205 included in the calibration member 200. Note that calibration using the predetermined pattern included in the calibration member 200 and calibration using the marker 205 included in the calibration member 200 will be described in detail in other embodiments described later.

[0025] Note that the calibration method is not particularly limited. For example, based on the "shift" estimated from the captured images of the first camera 110 and the second camera 120, the parameters of the first camera 110 and the second camera 120 may be changed. For example, the focus and angle of the first camera 110 and the second camera 120 may be controlled using software.

[0026] (Technical Effect) Next, the technical effect obtained by the calibration method according to the first embodiment will be described.

[0027] As described with reference to FIGS. 1 to 3, in the calibration method according to the first embodiment, by capturing an image of the calibration member 200 having a predetermined pattern and the marker 205, calibration of the first camera 110 and the second camera 120 (that is, at least two cameras) is performed. In this way, by using the predetermined pattern and the marker 205 included in the calibration member 200, it is possible to effectively reduce the "shift" occurring in a plurality of cameras in a relatively easy method.

[0028] <Second Embodiment> The calibration method according to the second embodiment will be described with reference to FIGS. 4 and 5. Note that the second embodiment only differs in part of the operations compared with the above-described first embodiment. For example, the configurations of the first camera 110, the second camera 120, and the calibration member 200 (see FIGS. 1 and 2) may be the same as those in the first embodiment. Therefore, hereinafter, the parts different from the first embodiment will be described in detail, and the description of other overlapping parts will be omitted as appropriate.

[0029] (Flow of operations) First, the flow of operations of the calibration method according to the second embodiment will be described with reference to FIG. 4. FIG. 4 is a flowchart showing the flow of operations of the calibration method according to the second embodiment. Note that in FIG. 4, the same reference numerals are given to the same processes as those shown in FIG. 3.

[0030] As shown in FIG. 4, in the calibration method according to the second embodiment, first, the first camera 110 and the second camera 120 respectively capture images of the calibration member 200 (step S11).

[0031] Subsequently, in the second embodiment, in particular, first calibration processing based on the marker 205 in the images captured by the first camera 110 and the second camera 120 is executed (step S21). In the first calibration processing, the marker 205 is detected from the captured image, and calibration is performed based on the detected marker. When a plurality of markers 205 are detected from the captured image, calibration may be performed using each of the plurality of markers 205 (that is, all the detected markers 205). Alternatively, calibration may be performed using only some of the plurality of detected markers 205. The first calibration processing may be processing for detecting a positional deviation occurring between the first camera 110 and the second camera 120 based on, for example, the position of the marker 205 in the image, and performing adjustment to reduce the deviation. Alternatively, the first calibration processing may be processing for detecting a directional deviation occurring between the first camera 110 and the second camera 120 based on, for example, the orientation of the marker 205 in the image, and performing adjustment to reduce the deviation. The first calibration processing may be a calibration processing with lower accuracy (in other words, rough) compared to the second calibration processing described later.

[0032] After the first calibration process, a second calibration process is executed based on a pattern of a predetermined pattern in the images captured by the first camera 110 and the second camera 120 (step S22). In the second calibration process, it is estimated which part of the calibration member 200 is imaged from the pattern in the captured image, and calibration is performed according to which part is imaged. The second calibration process may be a process of detecting a positional deviation occurring between the first camera 110 and the second camera 120 based on, for example, a pattern of a predetermined pattern in the image (specifically, the imaging position of the calibration member 200 estimated from the pattern), and performing adjustment to reduce the deviation. Alternatively, the second calibration process may be a process of detecting a directional deviation occurring between the first camera 110 and the second camera 120 based on, for example, a pattern of a predetermined pattern in the image (specifically, the imaging orientation of the calibration member 200 estimated from the pattern), and performing adjustment to reduce the deviation. The second calibration process may be a calibration process with higher accuracy (in other words, finer) compared to the first calibration process described above.

[0033] (Modification example) Next, the operation flow of a modification example of the calibration method according to the second embodiment will be described with reference to FIG. 5. FIG. 5 is a flowchart showing the operation flow of a modification example of the calibration method according to the second embodiment. In FIG. 5, the same reference numerals are assigned to the same processes as those shown in FIG. 4.

[0034] As shown in FIG. 5, in the calibration method according to the modification example of the second embodiment, first, the first camera 110 and the second camera 120 respectively image the calibration member 200 (step S11).

[0035] Subsequently, in the modification, in particular, second calibration processing is executed based on a pattern of a predetermined pattern in the images captured by the first camera 110 and the second camera 120 (step S22). In the second calibration processing, as in the case described with reference to FIG. 4, it is estimated which part of the calibration member 200 is imaged from the pattern in the captured image, and calibration is performed according to which part is imaged. The second calibration processing may be a process of detecting a positional deviation occurring between the first camera 110 and the second camera 120 based on a pattern of a predetermined pattern in the image (specifically, the imaging position of the calibration member 2 that is estimated from the pattern), and performing adjustment to reduce the deviation. Alternatively, the second calibration processing may be a process of detecting a directional deviation occurring between the first camera 110 and the second camera 120 based on, for example, a pattern of a predetermined pattern in the image (specifically, the imaging orientation of the calibration member 200 that is estimated from the pattern), and performing adjustment to reduce the deviation. The second calibration processing according to the modification may be a calibration processing with lower accuracy (in other words, rough) compared to the first calibration processing according to the modification described later.

[0036] After the first calibration process, a first calibration process based on the marker 205 in the images captured by the first camera 110 and the second camera 120 is executed (step S21). In the first calibration process, as in the case described with reference to FIG. 4, the marker 205 is detected from the captured images, and calibration is performed based on the detected marker. When a plurality of markers 205 are detected from the captured images, calibration may be performed using each of the plurality of markers 205 (that is, all the detected markers 205). Alternatively, calibration may be performed using only a part of the plurality of detected markers 205. The first calibration process may be a process of detecting a positional deviation occurring between the first camera 110 and the second camera 120 based on, for example, the position of the marker 205 in the image, and performing adjustment to reduce the deviation. Alternatively, the first calibration process may be a process of detecting a directional deviation occurring between the first camera 110 and the second camera 120 based on, for example, the orientation of the marker 205 in the image, and performing adjustment to reduce the deviation. The calibration process based on the position of the marker 205 and the calibration process based on the orientation of the marker 205 described above may be executed in combination with each other. That is, a calibration process based on both the position and the orientation of the marker 205 may be executed. The first calibration process according to the modification example may be a calibration process with high (in other words, fine) accuracy as compared with the second calibration process according to the modification example described above.

[0037] (Technical Effect) Next, the technical effect obtained by the calibration method according to the second embodiment will be described.

[0038] As described with reference to FIGS. 4 and 5, in the calibration method according to the second embodiment, the first calibration process based on the marker 205 included in the calibration member 200 and the second calibration process based on the pattern of a predetermined pattern included in the calibration member 200 are sequentially executed. In this way, it is possible to perform appropriate calibration by using each of the marker 205 and the predetermined pattern of the calibration member 200. As already described, if a rough adjustment is performed in the process performed first and a fine adjustment is performed in the process performed later among the first calibration process and the second calibration process, the deviation between the first camera 110 and the second camera 120 can be effectively reduced by stepwise calibration.

[0039] <Third Embodiment> The calibration method according to the third embodiment will be described with reference to FIG. 6. Note that the third embodiment is different only in part of the operation as compared with the first and second embodiments described above, and the other parts may be the same as those of the first and second embodiments. For this reason, hereinafter, parts different from the embodiments already described will be described in detail, and description of other overlapping parts will be omitted as appropriate.

[0040] (Flow of Operation) First, the flow of operation of the calibration method according to the third embodiment will be described with reference to FIG. 6. FIG. 6 is a flowchart showing the flow of operation of the calibration method according to the third embodiment. Note that in FIG. 6, the same reference numerals are given to the processes similar to those shown in FIG. 3.

[0041] As shown in FIG. 6, in the calibration method according to the third embodiment, first, the first camera 110 and the second camera 120 each capture an image of the calibration member 200 (step S11).

[0042] Subsequently, in the third embodiment, in particular, it is determined whether or not the number of images captured by the first camera 110 and the second camera 120 has reached a predetermined number (step S31). The "predetermined number" here is the number required for calibration using a plurality of images described later, and for example, an appropriate number may be determined by prior simulation or the like. When the number of images captured by the first camera 110 and the second camera 120 has not reached the predetermined number (step S31: NO), the process of step S11 is executed again. That is, the first camera 110 and the second camera 120 each capture an image of the calibration member 200. In this way, the imaging of the calibration member 200 by the first camera 110 and the second camera 120 is repeated until the number of images captured reaches the predetermined number.

[0043] On the other hand, when the number of images captured by the first camera 110 and the second camera 120 has reached the predetermined number (step S31: YES), calibration of the first camera 110 and the second camera 120 is performed based on a plurality of images of the calibration member 200 captured by the first camera 110 and the second camera 120 (step S32). More specifically, calibration is performed using a plurality of pairs of an image captured by the first camera 110 and an image captured by the second camera 120. Here, the calibration may be a process of performing calibration a plurality of times for the number of times the images are captured. Alternatively, it may be a process of integrating all or part of the images captured a plurality of times and performing calibration a smaller number of times than the number of imaging times. Alternatively, it may be a process of selecting only some of the images captured a plurality of times and performing calibration using only the selected images.

[0044] The calibration in step S32 described above may be executed as the first calibration process and the second calibration process as in the second embodiment (see FIGS. 4 and 5). Specifically, as the first calibration process, a process of detecting the marker 205 from images captured a plurality of times and performing calibration based on the detected marker may be executed. Further, as the second calibration process, a process of estimating which part of the calibration member 200 is imaged from the pattern of images captured a plurality of times and performing calibration according to which part is imaged may be executed. The order of executing the first calibration process and the second calibration process may be before and after each other.

[0045] (Technical effect) Next, the technical effect obtained by the calibration method according to the third embodiment will be described.

[0046] As described with reference to FIG. 6, in the calibration method according to the third embodiment, imaging of the calibration member 200 by the first camera 110 and the second camera 120 is performed a plurality of times until the number of captured images reaches a predetermined number. In this way, compared with the case where imaging is performed only once, it is possible to improve the calibration accuracy by the amount of increase in the images used for calibration. Further, even when an image inappropriate for calibration is captured, calibration can be performed using another image, so that it is possible to prevent inappropriate calibration from being executed.

[0047] <Fourth Embodiment> The calibration method according to the fourth embodiment will be described with reference to FIGS. 7 to 9. Note that the fourth embodiment is only different in part of the operation compared with the third embodiment described above, and the other parts may be the same as those of the third embodiment. Therefore, hereinafter, the parts different from the embodiments already described will be described in detail, and the description of other overlapping parts will be omitted as appropriate.

[0048] (Flow of operations) First, the flow of operations of the calibration method according to the fourth embodiment will be described with reference to FIG. 7. FIG. 7 is a flowchart showing the flow of operations of the calibration method according to the fourth embodiment. In FIG. 7, the same reference numerals are given to the same processes as those shown in FIG. 6.

[0049] As shown in FIG. 7, in the calibration method according to the fourth embodiment, first, the first camera 110 and the second camera 120 respectively capture images of the calibration member 200 (step S11).

[0050] Subsequently, it is determined whether or not the number of images captured by the first camera 110 and the second camera 120 has reached a predetermined number (step S31). If the number of captured images of the first camera 110 and the second camera 120 has not reached the predetermined number (step S31: NO), the process of step S11 is executed again in the same manner as in the third embodiment described above. However, in the fourth embodiment, in particular, after changing at least one of the position or angle of the calibration member 200 (step S41), the process of step S11 is executed. As a result, in the second and subsequent imaging operations, the calibration member 200 is imaged at a position or angle different from that until then. A specific example of the method of changing the position and angle of the calibration member 200 will be described in detail later.

[0051] When the imaging counts of the first camera 110 and the second camera 120 have reached a predetermined count (step S31: YES), camera calibration of the first camera 110 and the second camera 120 is performed based on a plurality of images of the calibration member 200 captured by the first camera 110 and the second camera 120 (step S32). More specifically, calibration is performed using a plurality of pairs of an image captured by the first camera 110 and an image captured by the second camera 120. Note that the calibration here may be a process of performing calibration a plurality of times for the number of times images are captured. Alternatively, it may be a process of integrating all or part of the images captured a plurality of times and performing calibration a smaller number of times than the number of imaging times. Alternatively, it may be a process of selecting only some of the images captured a plurality of times and performing calibration using only the selected images.

[0052] The calibration in step S32 described above may be executed as a first calibration process and a second calibration process as in the second embodiment (see FIGS. 4 and 5). Specifically, as the first calibration process, a process of detecting the marker 205 from the images captured a plurality of times and performing calibration based on the detected marker may be executed. Further, as the second calibration process, a process of estimating which part of the calibration member 200 is imaged from the pattern of the images captured a plurality of times and performing calibration according to which part is imaged may be executed. The order of executing the first calibration process and the second calibration process may be before and after each other.

[0053] (Change of position and angle) Next, a method for changing the position and angle of the calibration member will be specifically described with reference to FIGS. 8 and 9. FIG. 8 is a diagram showing an example of changing the position of the calibration member in the calibration method according to the fourth embodiment. FIG. 9 is a diagram showing an example of changing the angle of the calibration member in the calibration method according to the fourth embodiment.

[0054] As shown in FIG. 8, the calibration member 200 may have its position changed by being moved in the front-rear direction or the left-right direction. Further, the calibration member 200 may have its position changed by being moved in the up-down direction (i.e., the front side and the back side of the paper surface). Furthermore, the calibration member 200 may have its position changed by being moved in an oblique direction that combines the above-described front-rear direction, left-right direction, and up-down direction. The amount of movement of the calibration member 200 may be preset. When the calibration member 200 is moved multiple times, the amount of movement per time may be the same each time, or may be changed each time. For example, the amount of movement may be gradually increased, or the amount of movement may be gradually decreased.

[0055] As shown in FIG. 9, the calibration member 200 may have its angle changed by being rotated. In the example shown in FIG. 9, for the sake of convenience of explanation, an example of rotating clockwise about an axis near the center of the calibration member 200 is given, but it may be rotated in other modes. That is, the axis about which the calibration member 200 is rotated is not particularly limited, and any rotation axis may be used. Further, there may be two or more rotation axes about which the calibration member 200 is rotated. Also, the rotation direction of the calibration member 200 is not limited to one direction and may be rotated in various directions. The rotation axis and the rotation direction of the calibration member 200 may be preset. When the rotation of the calibration member 200 is executed multiple times, the amount of rotation per time may be the same each time, or may be changed each time. For example, the amount of rotation may be gradually increased, or the amount of rotation may be gradually decreased. Also, regarding the rotation direction, it may be the same each time, or may be changed each time.

[0056] The change in the position and angle of the calibration member 200 described above may be performed manually. When manually performing the calibration member 200, guidance information (i.e., information indicating the distance and direction to move the calibration member) may be presented to the user. The guidance information will be described in detail in other embodiments described later. Also, the change in the position and angle of the calibration member 200 may be automatically performed using a driving device or the like. The configuration including the driving device will be described in detail in other embodiments described later.

[0057] (Technical Effect) Next, the technical effect obtained by the calibration method according to the fourth embodiment will be described.

[0058] As described with reference to FIGS. 7 to 9, in the calibration method according to the fourth embodiment, a plurality of images are captured while changing the position or angle of the calibration member 200. In this way, since the calibration member 200 is imaged from different distances or angles, for example, compared with the case of imaging only at the same distance or angle, it is possible to improve the calibration accuracy by the amount of increase in the variation of the image of the calibration member 200. Also, even when an image is captured at a distance or angle inappropriate for calibration, calibration can be performed using another image, so that it is possible to prevent inappropriate calibration from being executed.

[0059] <Fifth Embodiment> The calibration method according to the fifth embodiment will be described with reference to FIGS. 10 to 13. Note that the fifth embodiment is only different in part of the operation compared with the first to fourth embodiments described above, and the other parts may be the same as those of the first to third embodiments. For this reason, in the following, the parts different from the embodiments already described will be described in detail, and the description of other overlapping parts will be omitted as appropriate.

[0060] (Operation Flow) First, the operation flow of the calibration method according to the fifth embodiment will be described with reference to FIG. 10. FIG. 10 is a flowchart showing the operation flow of the calibration method according to the fifth embodiment. In FIG. 10, the same reference numerals are given to the processes similar to those shown in FIG. 3.

[0061] As shown in FIG. 10, in the calibration method according to the fifth embodiment, first, the first camera 110 and the second camera 120 respectively capture images of the calibration member 200 (step S11).

[0062] Subsequently, in the fifth embodiment in particular, it is determined whether the position of the calibration member 200 is inappropriate (step S51). More specifically, it is determined whether the calibration member 200 is imaged by the first camera 110 and the second camera 120 at a position or angle suitable for performing calibration. Note that the determination method here is not particularly limited. For example, based on the captured image, it may be determined whether the calibration member 200 is within a predetermined range. The "predetermined range" here may be set in advance by prior simulation or the like.

[0063] When the position of the calibration member 200 is inappropriate (step S51: YES), information regarding the position or direction in which the calibration member is to be moved (hereinafter, appropriately referred to as "guidance information") is output (step S52). The guidance information may be information output to, for example, a user who holds the calibration member 200. In this case, information indicating how the calibration member 200 should be moved may be presented to the user. The user may move the calibration member 200 according to the guidance information. An example of the output of the guidance information to the user will be described in detail later. The guidance information may be information output to a driving device that drives the calibration member 200. In this case, information regarding the amount of movement or the direction of movement of the calibration member 200, or coordinate information of the movement target point of the calibration member 200, etc. may be output to the driving device. The driving device may drive the calibration member 200 according to the guidance information.

[0064] After the output of the guidance information, the process of step S11 is executed again. That is, the first camera 110 and the second camera 120 respectively capture images of the calibration member 200. Then, again, it is determined whether the position of the calibration member 200 is inappropriate (step S51). Thus, in the calibration method according to the fifth embodiment, the imaging of the images by the first camera 110 and the second camera 120 is repeated until the position of the calibration member 200 becomes appropriate.

[0065] On the other hand, when the position of the calibration member 200 is not inappropriate (step S51: NO), calibration of the first camera 110 and the second camera 120 is performed based on the images of the calibration member 200 captured by the first camera 110 and the second camera 120 (step S12). Specifically, calibration is performed using a predetermined pattern and marker 205 that the calibration member 200 has.

[0066] (Modification example) Next, with reference to FIG. 11, the operation flow of a modified example of the calibration method according to the fifth embodiment will be described. FIG. 11 is a flowchart showing the operation flow of a modified example of the calibration method according to the fifth embodiment. In FIG. 11, the same reference numerals are given to the same processes as those shown in FIG. 7.

[0067] As shown in FIG. 11, in a modified example of the calibration method according to the fifth embodiment, first, the first camera 110 and the second camera 120 respectively capture images of the calibration member 200 (step S11).

[0068] Subsequently, it is determined whether the position of the calibration member 200 is inappropriate (step S51). And when the position of the calibration member 200 is inappropriate (step S51: YES), guidance information indicating the position or direction to move the calibration member is output (step S52).

[0069] After the output of the guidance information, the process of step S11 is executed again. That is, the first camera 110 and the second camera 120 respectively capture images of the calibration member 200. And again, it is determined whether the position of the calibration member 200 is inappropriate (step S51). Thus, also in the modified example of the calibration method according to the fifth embodiment, the image capturing by the first camera 110 and the second camera 120 is repeated until the position of the calibration member 200 becomes appropriate.

[0070] On the other hand, when the position of the calibration member 200 is not inappropriate (step S51: NO), it is determined whether or not the number of images captured by the first camera 110 and the second camera 120 has reached a predetermined number (step S31). And when the number of captured images of the first camera 110 and the second camera 120 has not reached the predetermined number (step S31: NO), after changing at least one of the position or angle of the calibration member 200 (step S41), the process of step S11 is executed. In a modification of the fifth embodiment, in particular, it is also determined again whether or not the position of the calibration member 200 is inappropriate (step S51). And when the position of the calibration member 200 is inappropriate (step S51: YES), guidance information indicating the position or direction to move the calibration member is output (step S52).

[0071] After the output of the guidance information, the process of step S11 is executed again. That is, the first camera 110 and the second camera 120 respectively capture images of the calibration member 200. And again, it is determined whether or not the position of the calibration member 200 is inappropriate (step S51). Thus, in the modification of the calibration method according to the fifth embodiment, even after changing the position and angle of the calibration member, the imaging of images by the first camera 110 and the second camera 120 is repeated until the position of the calibration member 200 becomes appropriate.

[0072] On the other hand, when the number of captured images of the first camera 110 and the second camera 120 has reached the predetermined number (step S31: YES), calibration of the first camera 110 and the second camera 120 is performed based on a plurality of images of the calibration member 200 captured by the first camera 110 and the second camera 120 (step S3). More specifically, calibration is performed using a plurality of pairs of an image captured by the first camera 110 and an image captured by the second camera 120.

[0073] (Specific examples of guidance information) Next, the guidance information output by the calibration method according to the fifth embodiment will be specifically described with reference to FIGS. 12 and 13. FIG. 12 is a diagram (part 1) showing an example of the output mode of the guidance information by the calibration device according to the sixth embodiment. FIG. 13 is a diagram (part 2) showing an example of the output mode of the guidance information by the calibration device according to the sixth embodiment.

[0074] As shown in FIG. 12, the guidance information is presented to the user, for example, by a display device including a display. In the example shown in FIG. 12, an image indicating the current position of the calibration member 200 and information indicating the direction in which the calibration member 200 is to be moved (the text "Please move a little further to the right" and an arrow in the right direction) are presented to the user. Note that the guidance information may be displayed as information including the specific moving distance of the calibration member. For example, the text "Please move 30 cm further to the right" may be displayed. Also, when it is required to move the calibration member 200 in an oblique direction, the display may be made to directly guide in the oblique direction, or the display for guiding step by step obliquely may be made. For example, when guiding the calibration member 200 in the lower right direction, only the display for guiding in the right direction may be made first, and then only the display for guiding in the lower direction may be made.

[0075] As shown in FIG. 13, as guiding information, an image indicating the current position of the calibration member 200 and a frame indicating the movement target point of the calibration member 200 may be presented to the user. In this case, text prompting the user to move the calibration member 200 into the frame (for example, "Please move so as to fit within the frame") may be displayed. The frame that is the movement target point of the calibration member 200 may be the same size as the calibration member 200, or may be slightly larger than the calibration member. Further, in addition to or instead of the frame indicating the movement target point of the calibration member 200 as shown in FIG. 13, a frame indicating the movement target point of the marker 205 included in the calibration member 200 may be presented to the user. When the calibration member 200 has a plurality of markers 205, a plurality of frames corresponding to the plurality of markers 205 may be displayed.

[0076] Note that the above-described display mode of the guiding information is an example, and the guiding information may be output in other display modes. Also, when a plurality of types of display modes can be realized, one display mode may be selected and displayed from among the plurality of display modes. In this case, the user may be able to select the display mode. For example, the display mode may be switched according to the user's operation.

[0077] The guiding information may be output not only as visual display (i.e., image information), but also in other modes. Specifically, the guiding information may be output as audio information. The guiding information may be output including both display image information and audio information for audio notification. When the guiding information includes image information and audio information, both display by the image information and audio notification by the audio information may be performed simultaneously, or only one selected (i.e., only image display or only audio notification) may be performed.

[0078] (Technical Effect) Next, the technical effect obtained by the calibration method according to the fifth embodiment will be described.

[0079] As described with reference to FIGS. 10 to 13, in the calibration method according to the fifth embodiment, guiding information is output when the position of the calibration member 200 is inappropriate. In this way, even if the position of the calibration member is not suitable for calibration, it is possible to move it to an appropriate position based on the guiding information. As a result, even if the calibration member 200 cannot be initially placed at an appropriate position, it is finally possible to capture an image suitable for calibration.

[0080] <Sixth Embodiment> The calibration method according to the sixth embodiment will be described. Note that the sixth embodiment describes a specific example of the calibration member 200 used in the calibration method, and other parts may be the same as those in the first to fifth embodiments. Therefore, hereinafter, parts different from the already described embodiments will be described in detail, and descriptions of other overlapping parts will be omitted as appropriate.

[0081] The calibration member 200 used in the calibration method according to the sixth embodiment is configured such that at least one of the lightness and chroma of a predetermined pattern is higher than a predetermined value. The "predetermined value" here is a threshold value set for accurately detecting a predetermined pattern, and may be calculated, for example, by prior simulation or the like as a value capable of achieving a desired detection accuracy. The predetermined value may be set separately for each of the lightness and chroma. That is, the predetermined value for lightness and the predetermined value for chroma may be different values.

[0082] Note that it is preferable that both the lightness and chroma are equal to or higher than the predetermined value, but it is also acceptable if only one of them is equal to or higher than the predetermined value. However, since the lightness of the calibration member in the image is greatly affected by environmental parameters such as illumination, if only one of the lightness and chroma is set as the predetermined position, it is desirable that the chroma, which is less affected by the environmental parameters, is equal to or higher than the predetermined value.

[0083] In addition, the calibration member 200 is configured such that a predetermined pattern includes a plurality of hues. By including a plurality of hues in the predetermined pattern, it becomes possible to perform calibration not only using the shape of the pattern of the predetermined pattern but also using color information. If the predetermined pattern includes a plurality of hues, for example, alignment can be performed using "Colored Point Cloud Registration" which is an open CV. Specifically, alignment can be performed using a plurality of point clouds having color information. Note that the hues included in the predetermined pattern are not particularly limited, but appropriate hues (for example, hues with higher detection accuracy) may be selected according to the environment for imaging the image or the like.

[0084] (Technical effect) Next, the technical effect obtained by the calibration method according to the sixth embodiment will be described.

[0085] In the calibration method according to the sixth embodiment, the predetermined pattern of the calibration member 200 is set such that at least one of the lightness and the chroma is higher than a predetermined value and includes a plurality of hues. In this way, it becomes possible to perform calibration using the predetermined pattern with higher accuracy.

[0086] <Seventh Embodiment> The calibration method according to the seventh embodiment will be described. Note that the seventh embodiment describes a specific example of the calibration member 200 used in the calibration method, and other parts may be the same as those of the first to sixth embodiments. For this reason, in the following, parts different from the already described embodiments will be described in detail, and description of other overlapping parts will be omitted as appropriate.

[0087] The calibration member 200 used in the calibration method according to the seventh embodiment has a marker 205 composed of a plurality of two-dimensional codes. The two-dimensional code may be a stacked two-dimensional code or a matrix two-dimensional code. Examples of the stacked two-dimensional code include PDF417 and CODE49, but other stacked two-dimensional codes can also be applied as the marker 205 according to this embodiment. Examples of the matrix two-dimensional code include QR Code (registered trademark), DataMatrix, VeriCode, ArUko marker, etc., but other matrix two-dimensional codes can also be applied as the marker 205 according to this embodiment. Note that the calibration member 200 may include a plurality of types of two-dimensional codes as the marker 205. In this case, a stacked two-dimensional code and a matrix two-dimensional code may be combined and used.

[0088] Incidentally, according to the research of the inventor of the present application, it has been found that the ArUko marker, which is a matrix two-dimensional code, is suitable for the marker 205 included in the calibration member 200. Therefore, it is preferable that the calibration member 200 has an ArUko marker or a combination of an ArUko marker and another two-dimensional code as the marker 205. However, even when the marker 205 does not include an ArUko marker, the technical effects described below can be obtained accordingly.

[0089] (Technical effects) Next, the technical effects obtained by the calibration method according to the seventh embodiment will be described.

[0090] In the calibration method according to the seventh embodiment, the calibration member 200 It has a plurality of two-dimensional codes. By doing so, the detection accuracy of the marker 205 can be improved, so that calibration can be performed more appropriately. In addition, since the two-dimensional code itself can have information used for calibration (for example, information regarding position, etc.), calibration can be performed more easily. Furthermore, by arranging a plurality of two-dimensional codes, it becomes possible to detect position information more accurately compared to the case where only one two-dimensional code is arranged.

[0091] <Eighth Embodiment> The calibration device according to the eighth embodiment will be described with reference to FIGS. 14 to 16. Note that the calibration device according to the eighth embodiment may be configured as a device capable of realizing the calibration methods according to the first to seventh embodiments described above. Therefore, among the operations performed by the calibration device according to the eighth embodiment, the operations described in the first to seventh embodiments described above will be omitted as appropriate.

[0092] (Hardware Configuration) First, with reference to FIG. 14, the hardware configuration of the calibration device according to the eighth embodiment will be described. FIG. 14 is a block diagram showing the hardware configuration of the calibration device according to the eighth embodiment.

[0093] As shown in FIG. 14, the calibration device 300 according to the eighth embodiment includes a processor 11, a RAM (Random Access Memory) 12, a ROM (Read Only Memory) 13, and a storage device 14. The calibration device 300 may further include an input device 15 and an output device 16. The processor 11, the RAM 12, the ROM 13, the storage device 14, the input device 15, and the output device 16 are connected via a data bus 17.

[0094] Processor 11 reads a computer program. For example, processor 11 is configured to read a computer program stored in at least one of RAM 12, ROM 13, and storage device 14. Alternatively, processor 11 may read a computer program stored in a computer-readable recording medium using a recording medium reader (not shown). Processor 11 may obtain (i.e., read) a computer program from a device (not shown) disposed outside calibration device 300 via a network interface. By executing the read computer program, processor 11 controls RAM 12, storage device 14, input device 15, and output device 16. In particular, in this embodiment, when the computer program read by processor 11 is executed, various functional blocks for executing various processes related to calibration are realized in processor 11. Examples of processor 11 include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (field-programmable gate array), a DSP (Demand-Side Platform), and an ASIC (Application Specific Integrated Circuit). Processor 11 may use one of the above examples or may use a plurality in parallel.

[0095] RAM 12 temporarily stores the computer program executed by processor 11. RAM 12 temporarily stores data temporarily used by processor 11 when processor 11 is executing a computer program. RAM 12 may be, for example, D-RAM (Dynamic RAM).

[0096] ROM 13 stores the computer program executed by processor 11. ROM 13 may also store fixed data. ROM 13 may be, for example, P-ROM (Programmable ROM).

[0097] The storage device 14 stores data that the calibration device 300 stores in the long term. The storage device 14 may operate as a temporary storage device of the processor 11. The storage device 14 may include, for example, at least one of a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device.

[0098] The input device 15 is a device that receives an input instruction from a user of the calibration device 300. The input device 15 may include, for example, at least one of a keyboard, a mouse, and a touch panel. The input device 15 may be a dedicated controller (operation terminal). Further, the input device 15 may include a terminal (for example, a smartphone or a tablet terminal) owned by the user. The input device 15 may be a device capable of voice input including, for example, a microphone.

[0099] The output device 16 is a device that outputs information regarding the calibration device 300 to the outside. For example, the output device 16 may be a display device (for example, a display) capable of displaying information regarding the calibration device 300. The display device here may be a TV monitor, a personal computer monitor, a smartphone monitor, a tablet terminal monitor, or a monitor of other portable terminals. Further, the display device may be a large monitor or digital signage installed in various facilities such as stores. Further, the output device 16 may be a device that outputs information in a form other than an image. For example, the output device 16 may be a speaker that outputs information regarding the calibration device 300 by voice.

[0100] (Functional Configuration) Next, with reference to FIG. 15, the functional configuration of the calibration device 300 according to the eighth embodiment will be described. FIG. 15 is a block diagram showing the functional configuration of the calibration device according to the eighth embodiment.

[0101] As shown in FIG. 15, the calibration device 300 according to the eighth embodiment is connected to the first camera 110 and the second camera 120 that are the calibration targets, respectively. The calibration device 300 includes an image acquisition unit 310 and a calibration unit 320 as processing blocks for realizing its functions. Note that each of the image acquisition unit 310 and the calibration unit 320 may be realized by the above-described processor 11 (see FIG. 1).

[0102] The image acquisition unit 310 is configured to be able to acquire an image of the calibration member 200 captured by the first camera 110 and an image of the calibration member 200 captured by the second camera 120. The image acquisition unit 310 may include storage means (memory) for storing the acquired images. The image acquisition unit 310 may store, for example, the image of the first camera 110 and the image of the second camera 120 captured at the same timing as a set of two images. The images acquired by the image acquisition unit 310 are configured to be output to the calibration unit 320.

[0103] The calibration unit 320 is configured to be able to perform calibration of the first camera 110 and the second camera 120 based on the image of the first camera 110 and the image of the second camera 120 acquired by the image acquisition unit 310. The calibration unit 320 is configured to be able to control the parameters of the first camera 110 and the second camera 120 so that calibration can be performed. Note that for the specific calibration method, since the methods of the first to seventh embodiments described above can be appropriately adopted, detailed description thereof is omitted here.

[0104] (Flow of operation) Next, with reference to FIG. 16, the flow of operation of the calibration device 300 according to the eighth embodiment will be described. FIG. 16 is a flowchart showing the flow of operation of the calibration device according to the eighth embodiment.

[0105] As shown in FIG. 3, when the operation of the calibration apparatus 300 according to the first embodiment is started, first, the image acquisition unit 310 acquires an image of the calibration member 200 captured by the first camera 110 and an image of the calibration member 200 captured by the second camera 120 (step S81). When adopting a configuration in which images are repeatedly captured until a predetermined number of images is reached as in the third and fourth embodiments described above, the image acquisition unit 310 may acquire the captured image of the first camera 110 and the captured image of the second camera 120 each time imaging is performed by the first camera 110 and the second camera 120. Further, the image acquisition unit 310 may function as a determination unit that determines whether or not the captured images of the first camera 110 and the second camera 120 have reached a predetermined number.

[0106] Subsequently, the calibration unit 320 performs calibration of the first camera 110 and the second camera 120 based on the image of the first camera 110 acquired by the image acquisition unit 310 and the image of the second camera 120 (step S82). When adopting a configuration in which the first calibration process and the second calibration process are performed as in the second embodiment described above, the calibration unit 320 may be configured to include a first calibration unit that performs the first calibration process and a second calibration unit that performs the second calibration process. Further, when adopting a configuration in which guidance information is output as in the fifth embodiment, the calibration unit 320 may be configured to include a guidance information output unit that outputs guidance information.

[0107] (Technical Effect) Next, the technical effect obtained by the calibration apparatus 300 according to the eighth embodiment will be described.

[0108] As described with reference to FIGS. 14 to 16, in the calibration apparatus 300 according to the eighth embodiment, calibration of the first camera 110 and the second camera 120 (that is, at least two cameras) is performed by imaging an image of the calibration member 200 having a predetermined pattern and the marker 205. By doing so, it is possible to effectively reduce the "deviation" occurring in a plurality of cameras in a relatively easy manner by using the predetermined pattern and the marker 205 included in the calibration member 200.

[0109] <Ninth Embodiment> The calibration system according to the ninth embodiment will be described with reference to FIGS. 17 and 18. Note that the calibration system according to the ninth embodiment may be configured as a system capable of implementing the calibration methods according to the first to seventh embodiments described above. Therefore, among the operations performed by the calibration apparatus according to the ninth embodiment, the operations described in the first to seventh embodiments described above will be omitted as appropriate. Further, the calibration system according to the ninth embodiment may have the same hardware configuration (FIG. 14) as the calibration apparatus 300 according to the eighth embodiment described above. Therefore, the description of the portions overlapping with the eighth embodiment already described will be omitted as appropriate.

[0110] (Functional Configuration) First, the functional configuration of the calibration system according to the ninth embodiment will be described with reference to FIG. 17. FIG. 17 is a block diagram showing the functional configuration of the calibration apparatus according to the ninth embodiment.

[0111] As shown in FIG. 17, the calibration system according to the ninth embodiment includes a first camera 110, a second camera 120, a calibration member 200, a calibration apparatus 300, and a driving apparatus 400. Note that when adopting a configuration that outputs the guidance information of the fifth embodiment, the calibration system may include a display device having a display, a speaker, and the like.

[0112] The driving device 400 is configured to be able to drive the calibration member 200. Specifically, it is configured as a device capable of changing the position and angle of the calibration member 200 with respect to the first camera 110 and the second camera 120. The driving device 400 drives the calibration member 200 based on information regarding driving (hereinafter, appropriately referred to as "driving information") output from the calibration device 300. That is, the operation of the driving device 400 may be controlled by the calibration device 300. The driving device 400 may be configured to include, for example, various actuators and the like, but its configuration is not particularly limited. When a specific support member is arranged near the subjects of the first camera 110 and the second camera 120, the driving device 400 may be configured integrally with the support member. For example, when the subject is a person sitting on a chair, the driving device 400 may be configured integrally with the chair. In this case, the calibration member 200 may be supported so as to be drivable, for example, at the headrest portion of the chair.

[0113] (Operation of the driving device) Next, with reference to FIG. 18, the operation of the driving device 400 included in the calibration system according to the ninth embodiment will be described in detail. FIG. 18 is a flowchart showing the flow of the operation of the driving device of the calibration system according to the ninth embodiment.

[0114] As shown in FIG. 18, the drive device 400 according to the ninth embodiment first acquires drive information from the calibration device 300 (step S91). The drive information may be, for example, the information output in the process of step S41 of the calibration method according to the fourth embodiment (see FIG. 7). That is, the drive information may be the information output as the information for changing the position and angle of the calibration member 200 in a series of processes of imaging images a plurality of times. Further, the drive information may be the information output in the process of step S52 of the calibration method according to the fifth embodiment (see FIGS. 10 and 11). That is, the drive information may be the guiding information indicating the position and direction in which the calibration member 200 is moved.

[0115] Subsequently, the drive device 400 drives the calibration member 200 based on the acquired drive information (step S92). In the case where the calibration member 200 is driven a plurality of times, the processes of steps S91 and S92 described above may be repeatedly executed.

[0116] In addition to or instead of the driving based on the drive information described above, the drive device 400 may be configured to perform a pre-programmed operation. For example, the drive device 400 may be set to drive the calibration member 200 at a predetermined timing so as to have a position and an angle determined according to the timing.

[0117] (Technical Effect) Next, the technical effect obtained by the calibration system according to the ninth embodiment will be described.

[0118] As described with reference to FIG. 17, in the calibration system according to the ninth embodiment, the calibration member is automatically driven by the drive device 400. In this way, the labor of manually moving the calibration member 200 can be saved. Further, it is possible to realize a more precise movement as compared with the case of manually moving the calibration member 200.

[0119] <Specific application examples> Specific application examples of the calibration method of the first to seventh embodiments, the calibration device of the eighth embodiment, and the calibration system of the ninth embodiment described above will be described.

[0120] (Three-dimensional face shape measurement device) Each of the above-described embodiments is applicable to a three-dimensional face shape measurement device that measures the three-dimensional shape of a face. The three-dimensional face shape measurement device can measure the three-dimensional shape of the face of a person as the subject by imaging the face of the person with two left and right cameras (i.e., corresponding to the first camera 110 and the second camera 120) and synthesizing those images. More specifically, the right camera images the right side of the face, and the left camera images the left side of the face. Then, by synthesizing the right side shape of the face created from the right side image of the face and the left side shape of the face created from the left side image of the face, a three-dimensional shape of the entire face of the person (e.g., up to the ears) is created. The three-dimensional face shape measurement device may, for example, image an image while irradiating a sine wave pattern on the subject and perform measurement using the sine wave grating shift method.

[0121] In the three-dimensional face shape measurement device, as described above, since the process of synthesizing the images captured by the two cameras is performed, if there is a deviation between the two cameras, the three-dimensional shape of the person's face cannot be appropriately measured. However, if each of the above-described embodiments is applied, the calibration of the two cameras can be appropriately executed, so that it becomes possible to appropriately measure the three-dimensional shape of the person's face.

[0122] In an apparatus capable of capturing a three-dimensional image, such as the three-dimensional facial shape measurement apparatus described above, as another embodiment of the calibration method, calibration using a three-dimensional image may be performed. That is, when the first camera 110 and the second camera 120 are configured as cameras capable of capturing three-dimensional images (for example, 3D scanners, range finders, etc.), calibration of the first camera 110 and the second camera 120 may be performed using the three-dimensional image of the calibration member 200. Hereinafter, such calibration will be described in detail.

[0123] (Calibration Using a Three-Dimensional Image) With reference to FIG. 19, the flow of the operation of calibration using a three-dimensional image will be described. FIG. 19 is a flowchart showing the flow of the operation of calibration using a three-dimensional image.

[0124] As shown in FIG. 19, in the calibration method using a three-dimensional image, first, the first camera 110 and the second camera 120 each capture a three-dimensional image of the calibration member 200 (step S101). The three-dimensional image of the calibration member 200 is captured as having a predetermined pattern and marker 205. In addition, when having a configuration capable of projecting a predetermined light pattern such as a sine wave pattern (for example, a projector, etc.), the calibration member 200 may be imaged in a state where the predetermined light pattern is projected. In this case, for example, a reflective member may be attached to the calibration member 200 as the marker 205, and the position and orientation of the reflective member as the marker 205 may be specified using the reflected light when the pattern is irradiated (that is, the light reflected by the reflective member).

[0125] Subsequently, based on the three-dimensional image of the calibration member 200, camera calibration of the first camera 110 and the second camera 120 is performed (step S102). That is, calibration is performed using a predetermined pattern and the marker 205 in the three-dimensional image. More specifically, the positions of the first camera 110 and the second camera 120 may be adjusted so that the captured three-dimensional images match between the first camera 110 and the second camera 120.

[0126] As described above, according to the calibration method using the three-dimensional image, calibration of the camera capable of capturing the three-dimensional image can be appropriately executed based on the predetermined pattern and the marker 205 in the three-dimensional image.

[0127] A program for operating the configuration of the embodiment so as to realize the functions of each of the above-described embodiments is recorded on a recording medium, the program recorded on the recording medium is read out as code, and a processing method executed by a computer is also included in the scope of each embodiment. That is, a computer-readable recording medium is also included in the scope of each embodiment. Further, not only the recording medium on which the above-described program is recorded, but also the program itself is included in each embodiment.

[0128] As the recording medium, for example, a floppy (registered trademark) disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a magnetic tape, a non-volatile memory card, or a ROM can be used. Also, not limited to those that execute processing with the program recorded on the recording medium alone, those that operate on an OS and execute processing in cooperation with the functions of other software and expansion boards are also included in the scope of each embodiment.

[0129] This disclosure is not limited to the above-described embodiments. This disclosure can be appropriately modified within a range not contrary to the gist or idea of the invention that can be read from the claims and the entire specification, and a calibration method, a calibration device, a calibration system, a computer program, and a recording medium accompanied by such a modification are also included in the technical idea of this disclosure.

[0130] <Supplementary Note> Regarding the embodiments described above, it can be further described as follows in the supplementary note below, but is not limited thereto.

[0131] (Supplementary Note 1) The calibration method described in Supplementary Note 1 is a calibration method characterized by imaging a member having a predetermined pattern whose pattern changes according to the position on the surface of the member and a marker arranged at a predetermined position with at least two cameras, and performing calibration of the at least two cameras using the images of the member captured by the at least two cameras.

[0132] (Supplementary Note 2) The calibration method described in Supplementary Note 2 is the calibration method described in Supplementary Note 1, characterized by performing, as the calibration, a first calibration based on the marker in the image of the member and a second calibration based on the pattern of the predetermined pattern in the image of the member.

[0133] (Supplementary Note 3) The calibration method described in Supplementary Note 3 is the calibration method described in Supplementary Note 1 or 2, characterized by imaging the member a plurality of times with the at least two cameras and performing calibration of the at least two cameras using the images of the plurality of members.

[0134] (Supplementary Note 4) The calibration method described in Supplementary Note 4 is the calibration method described in Supplementary Note 3, characterized by imaging the member a plurality of times at different positions or angles.

[0135] (Supplementary Note 5) The calibration method described in Supplementary Note 5 is the calibration method according to any one of Supplementary Notes 1 to 4, characterized by outputting information indicating a position or direction for moving the member so that the member is in a position suitable for imaging an image of the member.

[0136] (Appendix 6) The calibration method described in Appendix 6 is a calibration method according to any one of Appendices 1 to 5, wherein the predetermined pattern has at least one of lightness and chroma higher than a predetermined value and includes a plurality of hues, and the marker is a plurality of two-dimensional codes.

[0137] (Appendix 7) The calibration method described in Appendix 7 is a calibration method according to any one of Claims 1 to 6, characterized in that a three-dimensional image of the member is captured by the at least two cameras, and calibration of the at least two cameras is performed using the three-dimensional image.

[0138] (Appendix 8) The calibration device described in Appendix 8 includes an acquisition means for acquiring an image obtained by imaging a member having a predetermined pattern whose pattern changes according to the position on the member surface and a marker arranged at a predetermined position with at least two cameras, and a calibration means for performing the at least two calibrations using the images of the member imaged by the at least two cameras.

[0139] (Appendix 9) The calibration system described in Appendix 9 includes a member having a predetermined pattern whose pattern changes according to the position on the member surface and a marker arranged at a predetermined position, a driving device for driving the member to change the position or angle of the member with respect to at least two cameras, and a calibration device for performing the at least two calibrations using the images of the member imaged by the at least two cameras.

[0140] (Appendix 10) The computer program described in Supplementary Note 10 causes a computer to operate so as to image a member having a predetermined pattern whose pattern changes according to the position on the surface of the member and a marker arranged at a predetermined position with at least two cameras, and to calibrate the at least two cameras using the images of the member imaged by the at least two cameras.

[0141] (Supplementary Note 11) The recording medium described in Supplementary Note 11 is a recording medium characterized in that the computer program described in Supplementary Note 10 is recorded thereon.

[0142] To the extent permitted by law, this application claims the priority based on Japanese Patent Application No. 2020-198243 filed on November 30, 2020, and incorporates all of its disclosures herein. Also, to the extent permitted by law, all published gazettes and papers described in this specification are incorporated herein.

Explanation of Reference Signs

[0143] 110 First camera 120 Second camera 200 Calibration member 205 Marker 300 Calibration device 310 Image acquisition unit 320 Calibration unit 400 Driving device

Claims

1. By at least one computer, a member having a predetermined pattern whose pattern changes according to the position on the member surface and a marker arranged at a predetermined position is imaged a plurality of times by at least two cameras while rotating the member and changing the rotation amount until the next imaging, calibrating the at least two cameras using the images of the plurality of members imaged by the at least two cameras, wherein the calibration includes a first calibration based on the marker in the image of the member and a second calibration having a different accuracy from the first calibration based on the pattern of the predetermined pattern in the image of the member A calibration method characterized by the above.

2. The calibration method according to claim 1, characterized in that information indicating a position or direction for moving the member is output so that the member is in a position suitable for imaging an image of the member.

3. The predetermined pattern has at least one of lightness and chroma higher than a predetermined value and includes a plurality of hues, The marker is a plurality of two-dimensional codes The calibration method according to claim 1 or 2, characterized by the above.

4. Imaging a three-dimensional image of the member with the at least two cameras, Calibrating the at least two cameras using the three-dimensional image The calibration method according to any one of claims 1 to 3, characterized by the above.

5. An acquisition means for acquiring images obtained by imaging a member having a predetermined pattern whose pattern changes according to the position on the member surface and a marker arranged at a predetermined position a plurality of times by at least two cameras while rotating the member and changing the rotation amount until the next imaging, A calibration means for calibrating the at least two cameras using the images of the plurality of members imaged by the at least two cameras Comprising wherein the calibration includes a first calibration based on the marker in the image of the member and a second calibration having a different accuracy from the first calibration based on the pattern of the predetermined pattern in the image of the member A calibration device characterized by the above.

6. A member having a predetermined pattern whose pattern changes according to the position on the surface of the member and a marker disposed at a predetermined position, A driving device that drives the member and rotates it so as to change the angle of the member with respect to at least two cameras, A calibration device that performs calibration of the at least two cameras using images of the member taken a plurality of times while rotating the member with at least two cameras and changing the amount of rotation until the next imaging, Comprising, The calibration includes a first calibration based on the marker in the image of the member and a second calibration having a different accuracy from the first calibration based on the pattern of the predetermined pattern in the image of the member. A calibration system characterized by this.

7. A member having a predetermined pattern whose pattern changes according to the position on the surface of the member and a marker disposed at a predetermined position is imaged a plurality of times with at least two cameras while rotating the member and changing the amount of rotation until the next imaging, Calibration of the at least two cameras is performed using images of the member taken by the at least two cameras, The calibration includes a first calibration based on the marker in the image of the member and a second calibration having a different accuracy from the first calibration based on the pattern of the predetermined pattern in the image of the member. A computer program characterized by operating a computer in such a manner.

8. By at least one computer, A member having a predetermined pattern whose pattern changes according to the position on the surface of the member and a marker disposed at a predetermined position is imaged with at least two cameras, It is determined whether the position of the member is inappropriate when the member is imaged with the at least two cameras, When it is determined that the position of the member is inappropriate, information indicating a position or direction to move the member so that the member is in a position suitable for imaging an image of the member is output, Calibration of the at least two cameras is performed using the image of the member taken by the at least two cameras, The calibration includes a first calibration based on the marker in the image of the member and a second calibration with a different accuracy from the first calibration based on the pattern of the predetermined pattern in the image of the member. A calibration method characterized by the above.

9. Acquisition means for acquiring images of a member having a predetermined pattern whose pattern changes according to the position on the member surface and a marker arranged at a predetermined position, taken by at least two cameras; Determination means for determining whether the position of the member is inappropriate when the member is imaged by the at least two cameras; Output means for outputting information indicating a position or direction to move the member so that the member is in a position suitable for imaging the image of the member when it is determined that the position of the member is inappropriate; Calibration means for calibrating the at least two cameras using the images of the member taken by the at least two cameras Comprising The calibration includes a first calibration based on the marker in the image of the member and a second calibration with a different accuracy from the first calibration based on the pattern of the predetermined pattern in the image of the member. A calibration device characterized by the above.

10. A member having a predetermined pattern whose pattern changes according to the position on the member surface and a marker arranged at a predetermined position, A driving device for driving the member to change the position or angle of the member with respect to at least two cameras; Determination means for determining whether the position of the member is inappropriate when the member is imaged by the at least two cameras; Output means for outputting information indicating a position or direction to move the member so that the member is in a position suitable for imaging the image of the member when it is determined that the position of the member is inappropriate; A calibration device for calibrating at least two cameras using the images of the member taken by the at least two cameras Comprising The calibration includes a first calibration based on the marker in the image of the member and a second calibration with a different accuracy from the first calibration based on the pattern of the predetermined pattern in the image of the member. A calibration system characterized by the above.

11. Imaging a member having a predetermined pattern whose pattern changes according to the position of the member surface and a marker arranged at a predetermined position with at least two cameras, Determining whether the position of the member is inappropriate when imaging the member with the at least two cameras, When it is determined that the position of the member is inappropriate, outputting information indicating the position or direction to move the member so that the member is in a position suitable for imaging an image of the member, Calibrating the at least two cameras using the images of the member captured by the at least two cameras, The calibration includes a first calibration based on the marker in the image of the member and a second calibration having a different accuracy from the first calibration based on the pattern of the predetermined pattern in the image of the member A computer program characterized by operating a computer as described above.

Images