Correction system and correction method

Abstract

According to the invention, the correction system and the correction method which can correct the camera without using a correction jig can be provided. A correction system according to the present disclosure is provided with a calculation unit (302) that calculates camera parameters of a fixed camera (10) on the basis of a fixed image obtained by imaging a subject (S) by the fixed camera (10) fixedly disposed at a predetermined location, and a moving image obtained by imaging the subject (S) by the fixed camera (10) fixedly disposed at the predetermined location. A calculation unit (302) calculates a first optical flow of a subject (S) by learning a model on the basis of a fixed image and a moving image having the same timestamp, calculates a second optical flow of the subject by geometric calculation, and calculates a second optical flow of the subject by geometric calculation on the basis of the first optical flow and the second optical flow. Based on the first optical flow and the second optical flow, camera parameters of the fixed camera (10) are calculated.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a correction system and a correction method. BACKGROUND

[0002] A method of correcting camera parameters (for example, a position, an orientation, a focal distance, and the like of a camera) has been developed. For example, in Patent Literature 1, a method of correcting camera parameters of a fixed camera fixedly arranged in a space by a mobile camera capable of freely moving in the space and a correction jig is disclosed. The correction method corrects the camera parameters of the fixed camera based on camera parameters of the mobile camera, and captured images of the correction jig captured by the mobile camera and the fixed camera, respectively. PRIOR ART DOCUMENTS PATENT LITERATURE

[0003] Patent Literature 1: Japanese Patent Application Publication No. 2024-019855 SUMMARY PROBLEMS TO BE SOLVED BY THE INVENTION

[0004] With regard to the correction method disclosed in Patent Literature 1, the inventors have found the following problem. The method in which the correction jig must be used when correcting the camera requires the correction jig to be arranged in a captured environment. Therefore, the correction method is difficult to adopt in places where it is difficult to arrange the correction jig, such as a road with heavy traffic, a factory, and the like, and has low versatility.

[0005] The present disclosure has been made to solve such a problem, and provides a correction system and a correction method in which correction of a camera can be performed without using a correction jig. TECHNICAL SOLUTION TO THE PROBLEM

[0006] The correction system according to the present disclosure includes a calculation unit that calculates camera parameters of a fixed camera based on a fixed image and a mobile image, the fixed image being an image obtained by the fixed camera fixedly arranged at a predetermined place capturing a subject, the mobile image being an image obtained by a mobile camera capable of moving capturing the subject, wherein the calculation unit calculates a first optical flow of the subject by a learned model based on the fixed image and the mobile image having the same time stamp, calculates a second optical flow of the subject by a geometric calculation, and calculates the camera parameters of the fixed camera based on the first optical flow and the second optical flow. Thus, the fixed camera can be corrected without using a correction jig.

[0007] The calculation unit can calculate the camera parameters of the fixed camera that make a difference between the first optical flow and the second optical flow smaller.

[0008] It can also be that the calculation section calculates the first optical flow using a learned model that is trained using training data, which takes the fixed image and the moving image as input and outputs the first optical flow.

[0009] It can also be that the calculation section calculates camera parameters of the moving camera and a depth estimation value based on a plurality of frame images captured by the moving camera, and calculates the second optical flow that includes the camera parameters of the fixed camera as a variable based on the fixed image, the moving image, and the camera parameters of the moving camera and the depth estimation value.

[0010] The correction method according to the present disclosure calculates camera parameters of a fixed camera based on a fixed image and a moving image, the fixed image being an image obtained by the fixed camera fixedly arranged at a predetermined place capturing a subject, the moving image being an image obtained by a moving camera capturing the subject, the correction method including: a step of calculating a first optical flow of the subject by a learned model based on the fixed image and the moving image having the same time stamp; a step of calculating a second optical flow of the subject by geometry calculation based on the fixed image and the moving image having the same time stamp; and a step of calculating the camera parameters of the fixed camera based on the first optical flow and the second optical flow. Effects of Invention

[0011] According to the present disclosure, it is possible to provide a correction system and a correction method that enable correction of a camera without using a correction jig. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 is a block diagram showing a correction system according to Embodiment 1. Figure 2 is a block diagram showing a correction device according to Embodiment 1. Figure 3 is a diagram for explaining a method of copying a subject on a moving image to a fixed image. Figure 4 is a flowchart of a correction method according to Embodiment 1. Figure 5 is an application example of the correction system according to Embodiment 1. DETAILED DESCRIPTION

[0013] A specific embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings. However, the present disclosure is not limited to the following embodiment. Furthermore, the following description and drawings are appropriately simplified in order to make the description clear.

[0014] (Embodiment 1) <Correction system> First, referring to Figure 1 and Figure 2 a correction system 1 according to Embodiment 1 will be described. Figure 1 is a block diagram showing the correction system according to Embodiment 1. Figure 2 is a block diagram showing a correction device according to Embodiment 1.

[0015] As shown in Figure 1 , the correction system 1 is provided with a fixed camera 10, a mobile camera 20, and a correction device 30. In addition, the fixed camera 10 and the mobile camera 20 can each be plural. The fixed camera 10, the mobile camera 20, and the correction device 30 are connected to each other in a state capable of transmitting information via a network NW. The network NW includes a communication line and a communication network such as the Internet, a LAN (Local Area Network), a WAN (Wide Area Network), and a base station. The network NW can be wired or wireless as long as it is connected in a state capable of transmitting information.

[0016] The fixed camera 10 is a camera fixedly arranged at a prescribed place. The fixed camera 10 transmits an image (hereinafter referred to as a fixed image) captured to the correction device 30. The mobile camera 20 is a camera capable of freely moving. The mobile camera 20 can be provided, for example, to a mobile body such as a vehicle or a robot. The mobile camera 20 transmits an image (hereinafter referred to as a mobile image) captured to the correction device 30.

[0017] The correction device 30 is a device that corrects camera parameters of the fixed camera 10 and the mobile camera 20. The camera parameters include two kinds of internal parameters and external parameters. The internal parameters are, for example, parameters inherent to an optical system such as a focal distance, a lens distortion, and an optical center coordinate. The external parameters are, for example, parameters such as a position of the camera and a direction of capturing. A camera whose internal parameters and external parameters are known is referred to as a corrected camera. On the other hand, a camera whose internal parameters and external parameters are unknown is referred to as an uncorrected camera. As shown in Figure 2 , the correction device 30 is provided with an image acquisition unit 301, a calculation unit 302, and a storage unit 303.

[0018] The image acquisition unit 301 acquires the fixed image from the fixed camera 10 and acquires the mobile image from the mobile camera 20. In addition, the fixed camera 10 and the mobile camera 20 are synchronized in time. Therefore, the fixed image and the mobile image are given time stamps synchronized in time.

[0019] The calculation section 302 calculates the camera parameter of the fixed camera 10 on the basis of the fixed image and the moving image acquired by the image acquisition section 301. The calculation section 302 first calculates the optical flow of the subject S by two methods on the basis of the fixed image and the moving image. Then, the calculation section 302 calculates the camera parameter of the fixed camera 10 on the basis of the two calculated optical flows. The method of calculating the camera parameter of the fixed camera 10 is described later in detail.

[0020] The storage section 303 stores a learned model used by the calculation section 302. The learned model is subjected to machine learning in advance in the calculation section 302 or an external device such as an edge or a server (cloud). The learned model is described later in detail.

[0021] <Method of calculating camera parameter of fixed camera 10> The method of calculating the camera parameter of the fixed camera 10 is described in detail below. First, the calculation section 302 extracts frame images from the fixed image and the moving image acquired by the image acquisition section 301 on the basis of a condition defined in advance. Here, the frame image extracted from the fixed image is set as a fixed image, and the frame image extracted from the moving image is set as a moving image. Here, the fixed image and the moving image are images obtained by photographing the same subject S at the same time stamp. Note that the time stamps of the fixed image and the moving image need not be completely identical.

[0022] Then, the calculation section 302 calculates the optical flow of the subject S between the fixed image and the moving image (hereinafter referred to as first optical flow) using the learned model stored in the storage section 303. The first optical flow is calculated using the learned model subjected to machine learning using training data, which takes the fixed image and the moving image as input and outputs the first optical flow. The first optical flow u1 ms is calculated by the following mathematical expression (1). m is the moving image, I s is the fixed image, Φ θ is the learned model.

[0023] [mathematical expression 1] u1 ms = Φ θ (I m , I s ) … (1)

[0024] Note that the method of machine learning is not particularly limited, and can be unsupervised learning, reinforcement learning, deep learning, or the like in addition to supervised learning. Furthermore, in the learned model, the input image at the time of learning is not limited to the fixed image and the moving image, and can be an image photographed by another camera.

[0025] Next, the calculation section 302 calculates the optical flow of the object S between the fixed image and the moving image (hereinafter referred to as second optical flow) by geometric calculation. More specifically, the calculation section 302 copies the object S on the moving image onto the fixed image, and calculates the second optical flow based on the pixel coordinates of the object S on the fixed image and the pixel coordinates of the object S' copied onto the fixed image.

[0026] Figure 3 is a diagram for explaining a method of copying the object on the moving image onto the fixed image. As shown in Figure 3 , the calculation section 302 first reprojects the object S on the moving image 21 to three-dimensional coordinates. Then, the calculation section 302 projects the object S reprojected to the three-dimensional coordinates onto the fixed image 11. According to the above, the object S on the moving image 21 is copied onto the fixed image 11.

[0027] The pixel coordinates p ＊ are calculated by the following mathematical expression (2) and mathematical expression (3). X m ∈ SE(3) and K m is the camera parameter of the moving camera 20, X m ∈ SE(3) is the extrinsic parameter, K m is the intrinsic parameter. X s and K s is the camera parameter of the fixed camera 10, X s is the extrinsic parameter, K s is the intrinsic parameter. D is the depth estimation value of the moving camera 20. Here, the camera parameter X s and K s are variables. In other words, the calculation section 302 performs the correction of the fixed camera 10 by calculating the camera parameter X s and K s of the fixed camera 10 as variables.

[0028] [Mathematical Expression 2] p * = Π -1 (T m→s Π(D, p m | K m ) | K s )... (2)

[0029] [Mathematical Expression 3] where

[0030] Here, the camera parameters of the mobile camera 20 and the depth estimation value are correction results that can be obtained by correcting the mobile camera 20 based on the plurality of frame images included in the mobile image. In addition, the correction of the mobile camera 20 is performed by the calculation portion 302 in the present embodiment 1, but the correction can be performed by the mobile camera 20 itself. The camera parameters X m ∈ SE(3) and K m are obtained, for example, by performing self-position estimation with respect to the origin on the plurality of frame images that constitute the mobile image. The depth estimation value D of the mobile camera 20 is obtained, for example, by performing monocular depth estimation on the plurality of frame images that constitute the mobile image.

[0031] In addition, in the correction of the mobile camera 20, the calculation portion 302 assigns a uniform scale between the extrinsic parameters of the mobile camera 20 and the depth estimation value. Therefore, the calculation portion 302 can calculate the optical flow without having to obtain scale information from a three-dimensional measurement sensor such as a radar (RADAR) or a laser radar (LiDAR: Light Detection And Ranging) or the like. Therefore, it is not necessary to fix the correction between the fixed camera 10 and the mobile camera 20 and the three-dimensional measurement sensor.

[0032] In addition, the calculation portion 302 can calculate the camera parameters of the fixed camera 10 with higher accuracy by obtaining correct scale information. For example, in order to obtain correct scale information, the correction system 1 can be provided with a three-dimensional measurement sensor. Furthermore, in the correction of the mobile camera 20, the calculation portion 302 can use a depth learning model (for example, Metric-aware monocular depth estimator or the like) that has obtained correct scale.

[0033] In addition, in order to obtain a depth estimation value with higher accuracy, the correction system 1 can perform stereo vision between a plurality of fixed cameras 10 or between the fixed camera 10 and the mobile camera 20.

[0034] After calculating the pixel coordinates of the subject S' copied onto the fixed image 11 based on mathematical expression (2) and mathematical expression (3), the calculation portion 302 calculates the second optical flow. The second optical flow u2 ms The calculation formula of the second optical flow is shown in mathematical expression (4) below. p0 is the pixel coordinates of the subject S on the fixed image 11. According to the above, the second optical flow including the camera parameters of the fixed camera 10 as a variable is calculated.

[0035] [mathematical expression 4] u2 ms = p * -p0…(4)

[0036] In the above, in the calculation of the first optical flow and the second optical flow, the calculation section 302 does not perform image processing on the fixed image and the moving image in advance, but performs the calculation of the optical flow on the entire region of the image. Therefore, the calculation section 302 does not have to define a partial region such as a region of interest or an observation object in advance in the fixed image and the moving image. Therefore, the correction system 1 according to the present embodiment 1 can be applied to any object S.

[0037] Finally, the calculation section 302 calculates the camera parameter of the fixed camera 10 on the basis of the first optical flow and the second optical flow. More specifically, the camera parameter of the fixed camera 10 is calculated in such a manner that the difference between the first optical flow and the second optical flow becomes smaller. The calculation formula of the camera parameter of the fixed camera 10 is shown in the following mathematical formula (5).

[0038] [mathematical formula 5]

[0039] w ms is the reliability with respect to the first optical flow. w ms is output from the learned model subjected to machine learning together with the first optical flow. ms A penalty is given to the optical flow for which good accuracy is not expected, for example, due to the influence of the fixed camera 10 or the moving camera 20 being blocked or reflected. ms is obtained, for example, by performing masking due to the influence of the blocking, or by using a reliability estimation model, or the like. With respect to the masking due to the influence of the blocking, for example, if there is an unobservable region in the moving image or the fixed image due to the influence of the parallax, w ms = 0. With respect to the reliability estimation model, for example, fine tunning can be applied to the learned model. In addition, w ms is not necessary for the calculation of the camera parameter of the fixed camera 10.

[0040] As explained above, the calculation section 302 calculates the first optical flow by machine learning and the second optical flow by geometric calculation on the basis of the fixed image and the moving image. Then, the calculation section 302 calculates the camera parameter of the fixed camera 10 that makes the difference between the first optical flow and the second optical flow the smallest.

[0041] <Correction method> Next, a method of correcting the fixed camera 10 using the correction system 1 according to the present embodiment 1 will be explained. Figure 4 is a flowchart of the correction method according to the present embodiment 1. In addition, the order of the steps is not limited to this, and can be appropriately replaced. Figure 4The flowchart shown starts, for example, when the correction device 30 receives a certain command signal or when a prescribed time elapses as a trigger.

[0042] First, the image acquisition unit 301 acquires the fixed image and the moving image (step S101). The image acquisition unit 301 acquires the fixed image from the fixed camera 10 and acquires the moving image from the moving camera 20. In addition, the fixed image and the moving image acquired by the image acquisition unit 301 are given time stamps.

[0043] Next, the calculation unit 302 performs correction of the moving camera 20 (step S102). The calculation unit 302 performs correction of the moving camera 20 based on a plurality of frame images included in the moving image acquired by the image acquisition unit 301. By the correction of the moving camera 20, the calculation unit 302 acquires the camera parameter and the depth estimation value of the moving camera 20.

[0044] Next, the calculation unit 302 extracts the fixed image from the fixed image and extracts the moving image from the moving image (step S103). The calculation unit 302 extracts frame images from the fixed image and the moving image acquired by the image acquisition unit 301 based on a condition defined in advance. Here, the frame image extracted from the fixed image is set as the fixed image, and the frame image extracted from the moving image is set as the moving image. Here, the fixed image and the moving image are images obtained by photographing the same subject S at the same time stamp. In addition, the time stamps of the fixed image and the moving image do not need to be completely identical.

[0045] Next, the calculation unit 302 calculates the first optical flow of the subject S by the learned model (step S104). The calculation unit 302 calculates the first optical flow using the learned model stored in the storage unit 303, which takes the fixed image and the moving image as input and outputs the first optical flow.

[0046] Next, the calculation unit 302 calculates the second optical flow of the subject S by geometric calculation (step S105). The calculation unit 302 calculates the second optical flow including the camera parameter of the fixed camera 10 as a variable based on the fixed image, the moving image, and the camera parameter and the depth estimation value of the moving camera 20 calculated by the correction of the moving camera 20 (step S102).

[0047] Finally, the calculation unit 302 calculates the camera parameter of the fixed camera 10 based on the first optical flow and the second optical flow (step S106). More specifically, the calculation unit 302 calculates the camera parameter of the fixed camera 10 that makes the difference between the first optical flow and the second optical flow smaller.

[0048] As explained above, the correction system 1 according to Embodiment 1 calculates the first optical flow of the subject S by learning the completed model based on the fixed image and the moving image having the same time stamp, and calculates the second optical flow of the subject S by geometric calculation. Then, based on the first optical flow and the second optical flow, the camera parameter of the fixed camera 10 is calculated. Thus, the correction system 1 can perform the correction of the fixed camera 10 without using a correction jig.

[0049] Therefore, the correction system 1 can be used even in a place where it is difficult to set a correction jig, such as a road with heavy traffic, a factory, and the like. Figure 5 is an application example of the correction system according to Embodiment 1. The correction system 1 can be adopted on a road with heavy traffic, for example, as shown in Figure 5

[0050] Further, the present disclosure can realize a part or all of the processing of the correction system 1 by causing a CPU (Central Processing Unit) to execute a computer program.

[0051] The above-described program contains the following command group (or software code) for causing a computer to execute one or more functions described in the embodiments when the program is read into the computer. The program can also be stored in a non-transitory computer-readable medium or a tangible storage medium. As examples but not limitation, the computer-readable medium or the tangible storage medium includes RAM (Random-Access Memory), ROM (Read-Only Memory), flash memory, SSD (Solid-State Drive), or other memory technology, CD-ROM, DVD (Digital Versatile Disc), Blu-ray (registered trademark) disc, or other optical disc storage, magnetic cassettes, magnetic tapes, magnetic disk storage, or other magnetic storage devices. The program can also be transmitted on a transitory computer-readable medium or a communication medium. As examples but not limitation, the transitory computer-readable medium or the communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0052] In addition, the present disclosure is not limited to the above-described embodiments, and can be appropriately changed within the scope of the gist. Explanation of Reference Numerals

[0053] 1 correction system 10 fixed camera 11 fixed image 20 moving camera 21 moving image ​30 correction device 301 image acquisition unit 302 calculation unit 303 storage unit NW network S subject

Claims

1. A correction system comprising a calculation unit configured to calculate camera parameters of a fixed camera based on a fixed image and a moving image, the fixed image being an image obtained by the fixed camera fixedly arranged at a predetermined place capturing a subject, the moving image being an image obtained by a movable moving camera capturing the subject, the calculation unit configured to calculate a first optical flow of the subject by a learned model based on the fixed image and the moving image having a same time stamp, and calculate a second optical flow of the subject by a geometric calculation, the calculation unit configured to calculate the camera parameters of the fixed camera based on the first optical flow and the second optical flow.

2. The correction system according to claim 1, wherein the calculation unit calculates the camera parameters of the fixed camera that make a difference between the first optical flow and the second optical flow smaller.

3. The correction system according to claim 1 or 2, wherein the calculation unit calculates the first optical flow using a learned model that is machine-learned using training data, with the fixed image and the moving image as inputs and the first optical flow as an output.

4. The correction system according to claim 3, wherein the calculation unit calculates camera parameters of the moving camera and a depth estimation value based on a plurality of frame images captured by the moving camera, the calculation unit calculates the second optical flow including the camera parameters of the fixed camera as a variable based on the fixed image, the moving image, and the camera parameters of the moving camera and the depth estimation value.

5. A correction method of calculating camera parameters of a fixed camera based on a fixed image and a moving image, the fixed image being an image obtained by the fixed camera fixedly arranged at a predetermined place capturing a subject, the moving image being an image obtained by a movable moving camera capturing the subject, the correction method comprising: a step of calculating a first optical flow of the subject by a learned model based on the fixed image and the moving image having a same time stamp; a step of calculating a second optical flow of the subject by a geometric calculation based on the fixed image and the moving image having a same time stamp; and a step of calculating the camera parameters of the fixed camera based on the first optical flow and the second optical flow. ​

Images