Calibration system and calibration method

Abstract

The present disclosure provides a calibration system and a calibration method that are capable of calibrating a camera without using a calibration jig. [Solution] The calibration system disclosed herein is a calibration system 1 that includes a calculation unit 302 that calculates camera parameters of the fixed camera 10 based on fixed images of a subject S taken by a fixed camera 10 fixedly placed at a predetermined location and moving images of the subject S taken by a movable mobile camera 20. The calculation unit 302 calculates a first optical flow of the subject using a trained model and a second optical flow of the subject using geometric calculations based on the fixed image and moving image that have the same timestamp, and calculates the camera parameters of the fixed camera 10 based on the first optical flow and the second optical flow.

Description

[Technical Field]

[0001] The present disclosure relates to a calibration system and a calibration method. [Background technology]

[0002] Methods for calibrating camera parameters (e.g., camera position, orientation, focal length, etc.) have been developed. For example, Patent Document 1 discloses a method for calibrating the camera parameters of a fixed camera that is fixedly placed in a space using a mobile camera that can move freely in the space and a calibration jig. This calibration method calibrates the camera parameters of the fixed camera based on the camera parameters of the mobile camera and images of the calibration jig captured by the mobile camera and the fixed camera, respectively. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2024-019855 Summary of the Invention [Problem to be solved by the invention]

[0004] The inventors have found the following problem with the calibration method disclosed in Patent Document 1. A method that requires a calibration jig when calibrating a camera requires that the calibration jig be installed in the shooting environment. Therefore, this calibration method is difficult to adopt in places where it is difficult to install a calibration jig, such as roads with heavy traffic or inside factories, and therefore has low versatility.

[0005] The present disclosure has been made to solve such problems, and provides a calibration system and a calibration method that are capable of calibrating a camera without using a calibration jig. [Means for solving the problem]

[0006] A calibration system according to the present disclosure includes a calculation unit that calculates camera parameters of the fixed camera based on fixed images of a subject captured by a fixed camera fixedly disposed at a predetermined location and moving images of the subject captured by a movable camera, wherein the calculation unit calculates a first optical flow of the subject by a trained model and a second optical flow of the subject by geometric calculation based on the fixed image and the moving image having the same timestamp, and calculates the camera parameters of the fixed camera based on the first optical flow and the second optical flow. This allows the fixed camera to be calibrated without using a calibration jig.

[0007] The calculation unit may calculate camera parameters of the fixed camera that reduce a difference between the first optical flow and the second optical flow.

[0008] The calculation unit may calculate the first optical flow using a trained model that has been machine-learned using training data that inputs the fixed image and the moving image and outputs the first optical flow.

[0009] The calculation unit may calculate camera parameters and depth estimates of the moving camera based on a plurality of frame images captured by the moving camera, and may calculate the second optical flow, which includes the camera parameters of the fixed camera as variables, based on the fixed image, the moving image, and further the camera parameters and depth estimates of the moving camera.

[0010] The calibration method according to the present disclosure is a calibration method for calculating camera parameters of a fixed camera based on fixed images of a subject taken by a fixed camera fixedly placed at a predetermined location and moving images of the subject taken by a movable camera, the calibration method including the steps of: calculating a first optical flow of the subject using a trained model based on the fixed image and the moving image having the same timestamp; calculating a second optical flow of the subject using geometric calculation based on the fixed image and the moving image having the same timestamp; and calculating the camera parameters of the fixed camera based on the first optical flow and the second optical flow. [Effects of the Invention]

[0011] The present disclosure provides a calibration system and a calibration method that are capable of calibrating a camera without using a calibration jig. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram showing a calibration system according to the first embodiment. [Figure 2] FIG. 2 is a block diagram showing the calibration device according to the first embodiment. [Figure 3] FIG. 3 is a diagram for explaining a method for transferring an object on a moving image onto a fixed image. [Figure 4] FIG. 4 is a flowchart of the calibration method according to the first embodiment. [Figure 5] FIG. 5 shows an application example of the calibration system according to the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, specific embodiments of the present disclosure will be described in detail with reference to the drawings. However, the present disclosure is not limited to the following embodiments. In addition, the following description and drawings have been simplified appropriately for clarity of explanation.

[0014] (Embodiment 1) <Proofreading system> First, a calibration system 1 according to the first embodiment will be described with reference to Fig. 1 and Fig. 2. Fig. 1 is a block diagram showing the calibration system according to the first embodiment. Fig. 2 is a block diagram showing the calibration device according to the first embodiment.

[0015] As shown in FIG. 1, the calibration system 1 includes a fixed camera 10, a mobile camera 20, and a calibration device 30. There may be multiple fixed cameras 10 and multiple mobile cameras 20. The fixed cameras 10, the mobile cameras 20, and the calibration device 30 are interconnected via a network NW in a state in which information can be transmitted. The network NW includes communication lines such as the Internet, a LAN (Local Area Network), and a WAN (Wide Area Network), as well as communication networks such as base stations. The network NW may be wired or wireless, as long as it is connected in a state in which information can be transmitted.

[0016] The fixed camera 10 is a camera fixedly placed at a predetermined location. The fixed camera 10 transmits the captured image (hereinafter, referred to as a fixed image) to the calibration device 30. The mobile camera 20 is a camera that can move freely. The mobile camera 20 may be provided on a moving body such as a car or a robot. The mobile camera 20 transmits the captured video (hereinafter referred to as "moving video") to the calibration device 30.

[0017] The calibration device 30 is a device that calibrates the camera parameters of the fixed camera 10 and the mobile camera 20. Camera parameters include two types: internal parameters and external parameters. Internal parameters are parameters specific to the optical system, such as focal length, lens distortion, and optical center coordinates. External parameters are parameters such as the camera position and shooting direction. A camera whose internal parameters and external parameters are known is called a calibrated camera. On the other hand, a camera whose internal parameters and external parameters are unknown is called an uncalibrated camera. As shown in FIG. 2, the calibration device 30 includes an image acquisition unit 301, a calculation unit 302, and a storage unit 303.

[0018] The video acquisition unit 301 acquires fixed video from the fixed camera 10 and moving video from the mobile camera 20. The fixed camera 10 and the mobile camera 20 are time-synchronized. Therefore, the fixed video and the moving video are provided with time-synchronized time stamps.

[0019] The calculation unit 302 calculates the camera parameters of the fixed camera 10 based on the fixed video and moving video acquired by the video acquisition unit 301. The calculation unit 302 first calculates the optical flow of the subject S using two methods based on the fixed video and moving video. The calculation unit 302 then calculates the camera parameters of the fixed camera 10 based on the two calculated optical flows. The method of calculating the camera parameters of the fixed camera 10 will be described in detail later.

[0020] The storage unit 303 stores a trained model used by the calculation unit 302. The trained model is machine-learned in advance in the calculation unit 302 or an external device such as an edge or server (cloud). The trained model will be described in detail later.

[0021] <Method of calculating camera parameters of fixed camera 10> A method for calculating the camera parameters of fixed camera 10 will be described in detail below. First, calculation unit 302 extracts frame images from the fixed video and moving video acquired by video acquisition unit 301 based on predefined conditions. Here, a frame image extracted from the fixed video is referred to as a fixed image, and a frame image extracted from the moving video is referred to as a fixed image. Here, the fixed image and the moving image are images of the same subject S captured at the same timestamp. Note that the timestamps of the fixed image and the moving image do not need to match exactly.

[0022] Then, the calculation unit 302 calculates the optical flow of the subject S between the fixed image and the moving image (hereinafter referred to as the first optical flow) using the trained model stored in the storage unit 303. The first optical flow is calculated using a trained model that has been machine-learned using training data, with the fixed image and the moving image as input and the first optical flow as output. First optical flow u1 ms The calculation formula is shown in the following formula (1). m is the moving image, I s is a fixed image, Φ θ is the trained model.

[0023]

number

[0024] The machine learning method is not particularly limited, and may be supervised learning, unsupervised learning, reinforcement learning, deep learning, etc. Furthermore, in a trained model, input images during learning are not limited to fixed images and moving images, and may be images captured by a separate camera.

[0025] Next, calculation unit 302 calculates the optical flow of subject S between the fixed image and the moving image (hereinafter referred to as second optical flow) through geometric calculation. More specifically, calculation unit 302 transfers subject S on the moving image onto the fixed image, and calculates the second optical flow based on the pixel coordinates of subject S on the fixed image and the pixel coordinates of subject S' transferred onto the fixed image.

[0026] 3 is a diagram for explaining a method for transferring a subject on a moving image onto a fixed image. As shown in FIG. 3, calculation unit 302 first reprojects subject S on moving image 21 onto three-dimensional coordinates. Then, calculation unit 302 projects subject S reprojected onto three-dimensional coordinates onto fixed image 11. In this way, subject S on moving image 21 is transferred onto fixed image 11.

[0027] Pixel coordinates p of the object S' transferred onto the fixed image 11 * is calculated by the following formula (2) and formula (3): m ∈SE(3) and K m are the camera parameters of the moving camera 20, and X m ∈SE(3) is an external parameter, K m is an internal parameter. X s and K. s are the camera parameters of the fixed camera 10, and X s is an external parameter, K s is an intrinsic parameter. D is a depth estimate of the moving camera 20. Note that here, the camera parameter X s and K. s In other words, the calculation unit 302 calculates the camera parameter X of the fixed camera 10, which is a variable. s and K. sを By calculating this, the fixed camera 10 is calibrated.

[0028]

number

[0029]

number

[0030] Here, the camera parameters and depth estimates of the moving camera 20 are calibration results that can be obtained by calibrating the moving camera 20 based on a plurality of frame images included in the moving video. In the first embodiment, the calibration of the moving camera 20 is performed by the calculation unit 302, but the calibration may also be performed by the moving camera 20 itself. The camera parameters X of the moving camera 20 m ∈SE(3) and K m is obtained by estimating the self-position from the origin for a plurality of frame images constituting the moving video. The depth estimate D of the moving camera 20 is obtained by performing monocular depth estimation for a plurality of frame images constituting the moving video.

[0031] Note that the calculation unit 302 assigns a consistent scale between the external parameters and depth estimates of the moving camera 20 in calibrating the moving camera 20. Therefore, the calculation unit 302 can calculate the optical flow without acquiring scale information from a three-dimensional measurement sensor such as a radar (RADAR) or a light detection and ranging (LiDAR). Therefore, calibration between the fixed camera 10, the moving camera 20, and the three-dimensional measurement sensor is not required.

[0032] The calculation unit 302 may calculate the camera parameters of the fixed camera 10 with higher accuracy by acquiring correct scale information. For example, the calibration system 1 may be equipped with a three-dimensional measurement sensor to acquire correct scale information. Alternatively, the calculation unit 302 may use a deep learning model (for example, a metric-aware monocular depth estimator) that has acquired the correct scale when calibrating the mobile camera 20.

[0033] In addition, the calibration system 1 may perform stereoscopic vision between multiple fixed cameras 10 or between a fixed camera 10 and a mobile camera 20 to obtain more accurate depth estimates.

[0034] The calculation unit 302 calculates the pixel coordinates of the object S′ transferred onto the fixed image 11 based on the formulas (2) and (3), and then calculates the second optical flow. ms The calculation formula for is shown in the following formula (4), where p0 is the pixel coordinate of the subject S on the fixed image 11. In this way, the second optical flow including the camera parameters of the fixed camera 10 as variables is calculated.

[0035]

number

[0036] As described above, in calculating the first optical flow and the second optical flow, the calculation unit 302 calculates the optical flow for the entire image region without performing image processing on the fixed image and the moving image in advance. Therefore, the calculation unit 302 does not need to predefine some regions, such as a region of interest or an observation target, in the fixed image and the moving image. Therefore, the calibration system 1 according to the first embodiment can be applied to any subject S.

[0037] Finally, the calculation unit 302 calculates the camera parameters of the fixed camera 10 based on the first optical flow and the second optical flow. More specifically, the calculation unit 302 calculates the camera parameters of the fixed camera 10 so that the difference between the first optical flow and the second optical flow becomes smaller. The calculation formula for the camera parameters of the fixed camera 10 is shown in the following formula (5).

[0038]

number

[0039] w ms is the confidence level for the first optical flow. ms is output from the machine-learned model along with the first optical flow. ms The w imposes a penalty on optical flow that is expected to be inaccurate due to, for example, the effects of occlusions or reflections on the fixed camera 10 or the moving camera 20. ms is obtained by, for example, performing masking due to the influence of occlusion or by using a confidence estimation model. Regarding masking due to the influence of occlusion, if there is an unobservable area in a moving image or a fixed image due to the influence of parallax, w ms = 0. For the confidence estimation model, fine tuning may be applied to a trained model, for example. ms is not required.

[0040] As described above, the calculation unit 302 calculates the first optical flow by machine learning and the second optical flow by geometric calculation based on the fixed video and the moving video. Then, the calculation unit 302 calculates the camera parameters of the fixed camera 10 that minimize the difference between the first optical flow and the second optical flow.

[0041] <Calibration method> Next, a method for calibrating the fixed camera 10 using the calibration system 1 according to the first embodiment will be described. FIG. 4 is a flowchart of the calibration method according to the first embodiment. Note that the order of the steps is not limited to this and may be changed as appropriate. The flowchart shown in FIG. 4 is started, for example, when the calibration device 30 receives a command signal or when a predetermined time has elapsed.

[0042] First, the video acquisition unit 301 acquires fixed video and moving video (step S101). The video acquisition unit 301 acquires fixed video from the fixed camera 10 and moving video from the moving camera 20. The fixed video and moving video acquired by the video acquisition unit 301 are assigned timestamps.

[0043] Next, the calculation unit 302 calibrates the moving camera 20 (step S102). The calculation unit 302 calibrates the moving camera 20 based on a plurality of frame images included in the moving video acquired by the video acquisition unit 301. By calibrating the moving camera 20, the calculation unit 302 acquires the camera parameters and depth estimates of the moving camera 20.

[0044] Next, calculation unit 302 extracts fixed images from the fixed video and moving images from the moving video (step S103). Calculation unit 302 extracts frame images from the fixed video and moving video acquired by video acquisition unit 301 based on predefined conditions. Here, frame images extracted from the fixed video are called fixed images, and frame images extracted from the moving video are called fixed images. Here, fixed images and moving images are images of the same subject S captured at the same timestamp. Note that the timestamps of the fixed images and moving images do not need to match exactly.

[0045] Next, the calculation unit 302 calculates a first optical flow of the subject S using the trained model (step S104). The calculation unit 302 calculates the first optical flow using a trained model stored in the storage unit 303, which receives a fixed image and a moving image as input and outputs the first optical flow.

[0046] Next, the calculation unit 302 calculates a second optical flow of the subject S by geometric calculation (step S105). The calculation unit 302 calculates the second optical flow, which includes the camera parameters of the fixed camera 10 as variables, based on the fixed image, the moving image, and the camera parameters and depth estimates of the moving camera 20 calculated by calibrating the moving camera 20 (step S102).

[0047] Finally, the calculation unit 302 calculates the camera parameters of the fixed camera 10 based on the first optical flow and the second optical flow (step S106). More specifically, the calculation unit 302 calculates the camera parameters of the fixed camera 10 that reduce the difference between the first optical flow and the second optical flow.

[0048] As described above, the calibration system 1 according to the first embodiment calculates the first optical flow of the subject S using a trained model and the second optical flow of the subject S using geometric calculations based on a fixed image and a moving image having the same timestamp. Then, the calibration system 1 calculates the camera parameters of the fixed camera 10 based on the first optical flow and the second optical flow. This allows the calibration system 1 to calibrate the fixed camera 10 without using a calibration jig.

[0049] Therefore, the calibration system 1 can be used in places where it is difficult to install a calibration jig, such as roads with heavy traffic or inside factories. Fig. 5 shows an application example of the calibration system according to the first embodiment. For example, as shown in Fig. 5, the calibration system 1 can be used on roads with heavy traffic, with a surveillance camera as the fixed camera 10 and a camera mounted on an automobile as the mobile camera 20.

[0050] Furthermore, in the present disclosure, part or all of the processing of the calibration system 1 can be realized by causing a CPU (Central Processing Unit) to execute a computer program.

[0051] The above-described program includes a set of instructions (or software code) that, when loaded into a computer, causes the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drives (SSD) or other memory technologies, CD-ROMs, digital versatile discs (DVDs), Blu-ray discs or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices. The program may also be transmitted on a transitory computer-readable medium or a communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0052] The present invention is not limited to the above-described embodiment, and can be modified as appropriate within the scope of the invention. [Explanation of symbols]

[0053] 1. Calibration System 10 Fixed Cameras 11 Fixed Images 20 Moving Camera 21 Moving Images 30 Calibration device 301 Video Acquisition Unit 302 Calculation Unit 303 Storage section NW Network S Subject

Claims

1. 1. A calibration system comprising: a calculation unit that calculates camera parameters of a fixed camera based on a fixed image, which is an image of a subject taken by a fixed camera fixedly disposed at a predetermined location, and a moving image, which is an image of the subject taken by a movable camera, The calculation unit Calculating a first optical flow of the object using a trained model and a second optical flow of the object using geometric calculations based on the fixed image and the moving image having the same timestamp; calculating camera parameters of the fixed camera based on the first optical flow and the second optical flow; Calibration system.

2. The calculation unit calculating camera parameters of the fixed camera that reduce a difference between the first optical flow and the second optical flow; The calibration system of claim 1 .

3. The calculation unit Calculating the first optical flow using a trained model that has been machine-learned using training data in which the fixed image and the moving image are input and the first optical flow is output.

3. A calibration system according to claim 1 or 2.

4. The calculation unit Calculating camera parameters and a depth estimate of the moving camera based on a plurality of frame images captured by the moving camera; calculating the second optical flow based on the fixed image, the moving image, and further the camera parameters and depth estimates of the moving camera, the second optical flow including the camera parameters of the fixed camera as variables; The calibration system of claim 3 .

5. A calibration method for calculating camera parameters of a fixed camera based on fixed images of a subject taken by a fixed camera fixedly disposed at a predetermined location and moving images of the subject taken by a movable camera, the method comprising: Calculating a first optical flow of the object using a trained model based on the fixed image and the moving image having the same timestamp; calculating a second optical flow of the object by geometric calculation based on the fixed image and the moving image having the same timestamp; and calculating camera parameters of the fixed camera based on the first optical flow and the second optical flow. Calibration method.

Images