Camera parameter arithmetic apparatus, method, program and recording medium, and river water flow measuring apparatus, method, program and recording medium

Abstract

To provide a camera parameter arithmetic apparatus, a method, a program and a recording medium thereof capable of reducing restriction of an imaging object, and a river water flow measuring apparatus, a method, a program and a recording medium thereof using the same.SOLUTION: The camera parameter arithmetic apparatus, on the basis of first and second image regions, in which subjects overlap, of a first image captured by a first camera whose first height from a reference surface, a first optical axis direction, and a first focal distance are known and a second image captured by a second camera so as to overlap at least partially with the subject of the first image, calculates a parallax between the first and second cameras, calculates the second optical axis direction of the second camera based on the first optical axis direction and the parallax, sets at least two sets of first and second line segments corresponding to each other in the first and second image regions, and calculates the camera parameters of the second camera based on the first height, the first optical axis direction, the first focal distance, the second optical axis direction and at least two sets of the first and second line segments.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a camera parameter calculation device for obtaining camera parameters, a camera parameter calculation method, a camera parameter calculation program and a recording medium thereof, and a river flow measurement device, a river flow measurement method, a river flow measurement program and a recording medium thereof using the same.

Background Art

[0002] When obtaining predetermined quantities related to an object imaged in an image based on the image captured by a camera, so-called camera parameters are required. Generally, these camera parameters include, for example, internal parameters such as the focal length and aberration in the imaging optical system (imaging optical system) of the camera, and external parameters such as the optical axis direction of the imaging optical system and the position of the camera (three-dimensional position of the projection center). Such camera parameters are generally calculated based on six or more members representing known calibration points with three-dimensional coordinate values arranged so as to be imaged in the image and six or more calibration points imaged in the image. In a method using such calibration points, it takes time and effort to install the calibration points, and in some cases, it may be difficult to install the calibration points depending on the imaging target (subject). Therefore, it is desired to obtain camera parameters without using calibration points. As such a technique, for example, there is a camera calibration device disclosed in Patent Document 1.

[0003] The camera calibration device disclosed in this Patent Document 1, in the image plane of an image in which the world coordinate space is photographed by a camera, the first world coordinate space normal vector and the second world coordinate space normal vector, each of which is a normal vector to a reference plane in the world coordinate space and has the same length as each other, and a first image plane normal vector and a second image plane normal vector corresponding thereto, respectively, an acquisition unit that acquires the first image plane normal vector and the second image plane normal vector; a projective depth calculation unit that calculates a projective depth vector having, as vector elements, four projective depths corresponding to the first start point and the first end point of the first image plane normal vector and the second start point and the second end point of the second image plane normal vector; and a camera parameter calculation unit that calculates the internal parameters and external parameters of the camera based on the calculated projective depth vector, the first image plane start point vector and the first image plane end point vector corresponding to the first start point and the first end point, respectively, and the second image plane start point vector and the second image plane end point vector corresponding to the second start point and the second end point, respectively. In this camera calibration device, according to the paragraph

[0016] , for the first and second world coordinate space normal vectors, sides of a building or a shelf that extend perpendicularly to the horizontal plane are used.

Prior Art Document

Patent Document

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] By the way, in order to obtain camera parameters without using calibration points, as described above, in the image, it is necessary that an object such as a building or a shelf that extends perpendicularly to the horizontal plane is captured, and there are restrictions on the imaging target.

[0006] The present invention is an invention made in view of the above circumstances, and its object is to provide a camera parameter calculation device, a camera parameter calculation method, a camera parameter calculation program and a recording medium thereof that can further reduce the constraints on an imaging target, as well as a river flow measurement device, a river flow measurement method, a river flow measurement program and a recording medium thereof using the same.

Means for Solving the Problems

[0007] As a result of various studies, the inventor has found that the above object is achieved by the following present invention. That is, a camera parameter calculation device according to one aspect of the present invention includes a first image captured by a first camera that knows a first height from a predetermined reference plane, a first direction of a first optical axis, and a first focal length of a first imaging optical system, and a second camera different from the first camera. A parallax between the first and second cameras is obtained based on first and second image regions in which the subject overlaps in the second image captured so as to overlap at least a part of the subject of the first image, and a second direction of a second optical axis in the second camera is obtained based on the first direction of the first optical axis and the obtained parallax. A direction processing unit, a corresponding line segment processing unit that sets at least two pairs of corresponding first and second line segments from the first and second images respectively in the first and second image regions, the first height, the first direction of the first optical axis, the first focal length of the first imaging optical system, the second direction of the second optical axis obtained by the direction processing unit, and a camera parameter processing unit that obtains camera parameters of the second camera based on the at least two pairs of first and second line segments set by the corresponding line segment processing unit. Preferably, in the above camera parameter calculation device, the first and second directions of the first and second optical axes are represented by first and second u-axis rotation angles with respect to the u-axis and first and second v-axis rotation angles with respect to the v-axis when a state in which the w-direction in a camera coordinate system (local coordinate system) that is a uvw orthogonal coordinate system with the projection center as the coordinate origin coincides with the Z-direction in a ground coordinate system (world coordinate system) that is an XYZ orthogonal coordinate system is set as the initial state, and the camera parameters of the second camera obtained by the camera parameter processing unit are the second height from the reference plane and the second focal length of the second imaging optical system in the second camera when the XY plane in the ground coordinate system is used as the reference plane. Preferably, the above camera parameter calculation device further includes the first and second cameras and a direction measurement unit that measures the first direction of the first optical axis in the first camera. Preferably, in the above camera parameter calculation device, the direction measurement unit is a three-axis acceleration sensor provided in the first camera that measures the inclination of the first camera.

[0008] When not considering the deviation of the principal point position and aberration in the imaging optical system of the camera, the position of the actual point corresponding to the point on the image can be obtained based on the direction of the optical axis, height, and focal length from the position of the point on the image. The camera parameter calculation device can obtain the second direction of the second optical axis in the second camera by correcting the first direction of the first optical axis in the first camera based on the parallax between the first and second images. The camera parameter calculation device can obtain the actual length of each of the at least two sets of second line segments corresponding thereto by obtaining the actual length of each of the at least two sets of first line segments from the positions of each end point of each of the at least two sets of first line segments, and can obtain the second height and the second focal length in the second camera by performing an inverse operation of the calculation method for obtaining the actual length from the positions of each end point. Therefore, when obtaining camera parameters, the camera parameter calculation device does not need to use, for example, calibration points, buildings, shelves, etc., and can further reduce the constraints on the imaging target.

[0009] In another aspect, in the above-described camera parameter calculation device, the direction processing unit includes, in the first and second image regions, a corresponding point search unit that searches for corresponding points corresponding to each other between the first and second images, a parallax calculation unit that obtains the parallax based on each corresponding point searched by the corresponding point search unit, and a direction calculation unit that obtains the second direction of the second optical axis in the second camera based on the first direction of the first optical axis and the parallax obtained by the parallax calculation unit, or includes, in the first and second image regions, an input unit that receives the input of corresponding points corresponding to each other in the first and second images, a parallax calculation unit that obtains the parallax based on each corresponding point received by the input unit, and a direction calculation unit that obtains the second direction of the second optical axis in the second camera based on the first direction of the first optical axis and the parallax obtained by the parallax calculation unit.

[0010] When such a camera parameter calculation device includes a corresponding point search unit, it can automatically obtain each corresponding point that corresponds to each other in the corresponding point search. When the camera parameter calculation device includes an input unit, even when it is difficult to obtain corresponding points by corresponding point search for some images, the input unit receives the input of each corresponding point that corresponds to each other, so that the parallax can be surely obtained and the second direction of the second optical axis can be obtained.

[0011] In another aspect, in these above-mentioned camera parameter calculation devices, the corresponding line segment processing unit includes a display unit that displays at least one of the first and second images, an input unit that receives the input of an input line segment as one of the first and second line segments in one of the first and second images, and an extraction unit that extracts a corresponding line segment corresponding to the input line segment received by the input unit in the other of the first and second images, or includes a display unit that displays the first and second images and an input unit that receives the input of the first and second line segments.

[0012] When such a camera parameter calculation device includes an extraction unit, if an input line segment is input as one of the first and second line segments, the extraction unit extracts a corresponding line segment as the other of the first and second line segments, so that the labor of inputting the first and second line segments can be saved. When the camera parameter calculation device does not include an extraction unit, since the input unit receives the input of the first and second line segments, the first and second line segments can be surely set.

[0013] In another aspect, in these above-described camera parameter calculation devices, the at least two sets of first and second line segments are three or more sets of first and second line segments, and the camera parameters of the second camera obtained by the camera parameter processing unit are the second height from the reference plane and the second focal length of the second imaging optical system in the second camera. The camera parameter processing unit uses an arithmetic expression for obtaining the length of the line segment based on the second height and the second focal length to obtain the camera parameters of the second camera such that the error with respect to the length of each of the at least two sets of first and second line segments is minimized. Preferably, in the above-described camera parameter calculation device, the camera parameter processing unit obtains the camera parameters of the second camera by solving the arithmetic expression by the least squares method.

[0014] Since the camera parameters of the second camera obtained by the camera parameter processing unit are two, namely the second height and the second focal length, the camera parameters of the second camera can be obtained from two sets of first and second line segments. Since the above-described camera parameter calculation device obtains the camera parameters of the second camera based on three or more sets of first and second line segments having redundancy with respect to two unknowns, the camera parameters can be obtained more appropriately.

[0015] Another aspect of the present invention relates to a camera parameter calculation method. The method includes: capturing a first image with a first camera from which a first height from a predetermined reference plane, a first direction of a first optical axis, and a first focal length of a first imaging optical system are known; capturing a second image with a second camera different from the first camera such that the subject of the first image is at least partially overlapped; obtaining a parallax between the first and second cameras based on the first and second image regions where the subject overlaps; a direction processing step of obtaining a second direction of a second optical axis in the second camera based on the first direction of the first optical axis and the obtained parallax; a corresponding line segment processing step of setting at least two pairs of corresponding first and second line segments from the first and second images respectively in the first and second image regions; and a camera parameter processing step of obtaining camera parameters of the second camera based on the first height, the first direction of the first optical axis, the first focal length of the first imaging optical system, the second direction of the second optical axis obtained in the direction processing step, and the at least two pairs of first and second line segments set in the corresponding line segment processing step. Another aspect of the present invention relates to a camera parameter calculation program which causes a computer to function as any of the above-described camera parameter calculation devices.

[0016] Such a camera parameter calculation method and camera parameter calculation program do not require, for example, the use of calibration points, buildings, shelves, etc. when obtaining camera parameters, and can further reduce the constraints on the imaging target.

[0017] Another aspect of the present invention relates to a recording medium which is a computer-readable recording medium storing the above-described camera parameter calculation program.

[0018] According to this, when obtaining camera parameters, for example, it is not necessary to use calibration points, buildings, shelves, etc., and a camera parameter calculation program that can further reduce the constraints on the imaging target can be provided.

[0019] A river water flow measurement device according to another aspect of the present invention includes any one of the above-described camera parameter calculation devices, and a plurality of surface images consecutive in time series obtained by imaging the surface of a river with the second camera are orthogonally corrected based on the camera parameters of the second camera obtained by the camera parameter calculation device to generate a plurality of corrected surface images, and a flow velocity processing unit that obtains a surface flow velocity on the surface of the river based on the plurality of corrected surface images generated by the ortho-correction unit.

[0020] Such a river water flow measurement device can obtain camera parameters required for ortho-correction without using, for example, calibration points, buildings, shelves, etc., so that restrictions on the imaging target can be further reduced, and when attempting to measure, the surface flow velocity of the river can be measured more quickly.

[0021] In another aspect, in the above-described river water flow measurement device, the at least two sets of first and second line segments include a set of first A and second A line segments set on the near bank of the river with respect to the second camera, a set of first B and second B line segments set on the far bank of the river with respect to the second camera, and a set of first C and second C line segments set along the width direction of the river, and are three or more sets of first and second line segments.

[0022] Such a river water flow measurement device sets at least two sets of first and second line segments in three sets: the near bank, the far bank, and the river width, so that the surface flow velocity of the river can be measured with good balance accuracy from the near side to the far side of the river along the river width direction.

[0023] In another aspect, in the above-described river water flow measurement device, a flow rate processing unit that obtains the flow rate of the river based on the surface flow velocity obtained by the flow velocity processing unit, the shape of the cross section of the river, and the water level of the river is further provided. Such a river water flow measurement device can measure the flow rate of the river because it is provided with a flow rate processing unit.

[0024] In another aspect, in these above-described river water flow measurement devices, the first and second cameras, and a direction measurement unit for measuring the first direction of the first optical axis in the first camera are further provided. Such a river water flow measurement device includes the first and second cameras and the direction measurement unit, so it can be actually measured at the measurement site only by this river water flow measurement device.

[0025] A river water flow measurement method according to another aspect of the present invention includes the above-described camera parameter calculation method, and an ortho-correction step of ortho-correcting each of a plurality of surface images continuous in time series when the surface of a river is imaged by the second camera based on the camera parameters of the second camera obtained by the camera parameter calculation method to generate a plurality of corrected surface images, and a flow velocity processing step of obtaining a surface flow velocity on the surface of the river based on the plurality of corrected surface images generated in the ortho-correction step. A river water flow measurement program according to another aspect of the present invention is a river water flow measurement program for causing a computer to function as the above-described river water flow measurement device. Such a river water flow measurement method and river water flow measurement program can further reduce the constraints on the imaging target and can measure the surface flow velocity of a river more quickly when attempting to measure.

[0026] A recording medium according to another aspect of the present invention is a computer-readable recording medium on which the above-described river water flow measurement program is recorded. According to this, a river water flow measurement program that can further reduce the constraints on the imaging target and can measure the surface flow velocity of a river more quickly when attempting to measure can be provided.

Effects of the Invention

[0027] The camera parameter calculation device, camera parameter calculation method, camera parameter calculation program, and recording medium thereof according to the present invention can further reduce the constraints on the imaging target. According to the present invention, a river water flow measurement device, river water flow measurement method, river water flow measurement program, and recording medium thereof using this can be provided.

Brief Description of the Drawings

[0028]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Embodiments for Carrying Out the Invention

[0029] Hereinafter, one or more embodiments of the present invention will be described with reference to the drawings. However, the scope of the invention is not limited to the disclosed embodiments. In each figure, components with the same reference numerals are shown to be the same components, and the description thereof will be omitted as appropriate. In this specification, when referring to components generically, they are indicated by reference numerals without subscripts, and when referring to individual components, they are indicated by reference numerals with subscripts.

[0030] The camera parameter calculation device in the embodiment captures a first image captured by a first camera from which a first height from a predetermined reference plane, a first direction of a first optical axis, and a first focal length of a first imaging optical system are known, and a second image captured by a second camera different from the first camera so as to at least partially overlap with the subject of the first image. A direction processing unit that obtains a parallax between the first and second cameras based on the first and second image regions where the subjects overlap, and obtains a second direction of a second optical axis in the second camera based on the first direction of the first optical axis and the obtained parallax; A corresponding line segment processing unit that sets at least two pairs of first and second line segments corresponding to each other from the first and second images in the first and second image regions; the first height, the first direction of the first optical axis, the first focal length of the first imaging optical system, the second direction of the second optical axis obtained by the direction processing unit, and at least two sets of first and second line segments set by the corresponding line segment processing unit And a camera parameter processing unit that obtains camera parameters of the second camera based on the above.

[0031] First, the calculation principle of the camera parameters of the second camera in such a camera parameter calculation device will be described.

[0032] FIG. 1 is a diagram for explaining the collinearity condition in camera imaging. In FIG. 1, the XYZ orthogonal coordinate system is a ground coordinate system (world coordinate system) set on the ground, and the uvw orthogonal coordinate system is a camera coordinate system (local coordinate system (first local coordinate system)) set in the camera with the projection center o of the camera (image) as the coordinate origin. The xy orthogonal coordinate system is a pixel coordinate system (local coordinate system (second local coordinate system)) set in the image PC when the image PC is arranged so as to be parallel to the uv plane of the camera coordinate system uvw, with the intersection of the normal line of the image PC passing through the projection center o and the image PC as the coordinate origin. The pixel coordinate system xy is also a two-dimensional orthogonal coordinate system set in the two-dimensional image sensor IM of the camera, with the intersection of the normal line of the two-dimensional image sensor IM passing through the projection center o of the camera and the imaging surface (imaging plane of the imaging optical system) of the two-dimensional image sensor IM as the coordinate origin.

[0033] With the state where the w - direction of the camera coordinate system uvw coincides with the Z - direction of the ground coordinate system (the state where the w - axis of the camera coordinate system uvw and the Z - axis of the ground coordinate system are parallel to each other) as the initial state, when the camera rotates by an angle κ around the w - axis, then rotates by an angle φ around the v - axis, and finally rotates by an angle ω around the u - axis, the collinearity condition that the position (X, Y, Z) of an object on the ground, the projection center o(X0, Y0, Z0), and the position (x, y) of the object on the image are on a straight line is represented by the following equations (1 - 1) and (1 - 2). Here, assuming the focal length of the imaging optical system in the camera is c, the coefficient a regarding the rotation of the camera ij is represented by the following equations (2 - 1) to (2 - 9).

[0034]

Number

[0035]

Number

[0036] From these equations (1 - 1), (1 - 2) and (2 - 1) to (2 - 9), assuming there is no deviation in the principal point position and no distortion, generally, the camera parameters X0, Y0, Z0, ω, φ, κ, c can be obtained from six or more calibration points with known positions in the ground coordinate system and the positions of these calibration points on the image (positions in the pixel coordinate system).

[0037] On the other hand, the above equations (1 - 1) and (1 - 2) can be transformed as follows (3 - 1) and (3 - 2).

[0038]

Number

[0039] Here, the distance D 12 between two points D1(X1, Y1) and D2(X2, Y2) on a plane at an arbitrary altitude Z is represented by the following equation (4). Here, h = Z - Z0.

[0040]

Number

[0041] Also, assume that the rotation angle of the camera about the w-axis changes from angle κ to angle κ', and as a result, two points D1(X1, Y1) and D2(X2, Y2) move to two points D1'(X1', Y1') and D2'(X2', Y2') respectively. Then, the following equations 5-1 and 5-2 hold. Here, the coefficient a ij ' is the coefficient at angle κ'.

[0042]

Equation

[0043] Substituting the coefficients a ij , a ij ' into Equation 5-2, it can be transformed as shown in the following Equation 6, and it can be seen that the variable A is 0 regardless of the angle κ.

[0044]

Equation

[0045] From this, it can be seen that even when the angle κ changes, the distance between two points on the XY plane at the same elevation does not change. That is, the result of geometric correction (ortho-correction) is the same regardless of the angle κ. Therefore, when obtaining the coefficients a ij , a ij ', the angle κ can be any value. For example, 0° or 90° at which cos and sin are integer values are preferred.

[0046] Therefore, from Equations 4 and 6, the first height h1 (h1 = Z - Z 01 , the projection center o1(X 01 , Y 01 , Z 01) If there are a first image captured by a first camera that can determine the first direction of the first optical axis and the first focal length of the first imaging optical system, and a second image captured by a second camera different from the first camera so as to at least partially overlap the subject of the first image, the parallax between the first and second images is obtained based on the first and second images, and the first direction of the first optical axis is corrected based on the obtained parallax, whereby the second direction of the second optical axis in the second camera can be obtained. At least two or more line segments corresponding to each other are set in each of the first and second images, the length of each of the set at least two or more line segments is obtained for the first image by using Equation 4, and the length of each of the obtained at least two or more line segments and the obtained second direction of the second optical axis are substituted into Equation 4 for the second image, and by solving Equation 4 for the substituted second image, the camera parameters h2 and c2 of the second camera can be obtained (h2 = Z - Z 02 , the projection center o2 (X 02 , Y 02 , Z 02 ) of the second camera. The second image can be orthorectified based on the obtained camera parameters h2 and c2 of the second camera.

[0047] Next, taking a river flow measurement device equipped with a camera parameter calculation device as an example, the camera parameter calculation device in the embodiment, the camera parameter calculation method and camera parameter calculation program implemented thereon, and the river flow measurement device, the river flow measurement method and river flow measurement program implemented thereon will be described more specifically together with each recording medium on which each program is recorded.

[0048] FIG. 2 is a block diagram showing the configuration of a river water flow measuring device including a camera parameter calculation device according to an embodiment. FIG. 3 is a diagram for explaining a direction processing unit in a first aspect in the river water flow measuring device (camera parameter calculation device). FIG. 4 is a diagram for explaining a direction processing unit in a second aspect in the river water flow measuring device (camera parameter calculation device). FIG. 5 is a first diagram for explaining a corresponding line segment processing unit in the first and second aspects in the river water flow measuring device (camera parameter calculation device). FIG. 6 is a second diagram for explaining a corresponding line segment processing unit in the first and second aspects in the river water flow measuring device (camera parameter calculation device). FIG. 7 is a diagram for explaining an inspection line set in an image of a river as an example. FIG. 8 is a diagram showing an orthorectified image of the image shown in FIG. 7 as an example. FIG. 9 is a diagram showing calculation results of surface flow velocity and water volume as an example.

[0049] The river water flow measuring device 1000 including a camera parameter calculation device according to an embodiment includes, for example, as shown in FIG. 2, an image acquisition unit 1, a control processing unit 2, a corresponding line segment processing unit 3, an interface unit (IF unit) 4, and a storage unit 5.

[0050] The image acquisition unit 1 is connected to the control processing unit 2 and, in accordance with the control of the control processing unit 2, captures a first image captured by a first camera in which a first height from a predetermined reference plane, a first direction of a first optical axis, and a first focal length of a first imaging optical system are known, and a second image captured by a second camera different from the first camera so as to at least partially overlap the subject of the first image.

[0051] In this embodiment, for example, the image acquisition unit 1 includes a first camera 11-1, a second camera 11-2, and a direction measurement unit 12. The first and second cameras 11-1 and 11-2 are devices that generate images of the subject (imaging target) by imaging the subject, respectively. For example, an imaging optical system (imaging optical system) that forms an optical image of the subject on a predetermined imaging surface, an image sensor that is arranged with a light receiving surface coinciding with the imaging surface and converts the optical image of the subject into an electrical signal, and an image processing unit that generates image data representing an image of the subject by performing image processing on the output of the image sensor, etc. It is a digital camera. Each digital camera of such first and second cameras 11-1 and 11-2 may be a monochrome camera, may be a color camera, may be a visible light camera, or may be an infrared camera. The direction measurement unit 12 is a device that measures the first direction of the first optical axis in the first camera 11-1, and is configured to include, for example, a three-axis acceleration sensor that measures the inclination of the first camera 11-1. The first direction of the first optical axis can be obtained from the inclination of the first camera 11-1. In this embodiment, for the sake of simplicity, a so-called tablet (tablet computer) equipped with a three-axis velocity sensor and a camera, a so-called smartphone (portable communication terminal device with information processing function), etc. are used for the first camera 11-1 and the direction measurement unit 12. The first camera 11-1 and the second camera 11-2 may be of the same type of product or different types of products.

[0052] Note that the image acquisition unit 1 may be, for example, an interface circuit that inputs and outputs data to and from an external device. The external device is a storage medium such as a USB (Universal Serial Bus) memory and an SD card (registered trademark) that stores the first and second images. Alternatively, the external device is a drive device that reads data from a recording medium such as a CD-ROM (Compact Disc Read Only Memory), CD-R (Compact Disc Recordable), DVD-ROM (Digital Versatile Disc Read Only Memory), and DVD-R (Digital Versatile Disc Recordable) that records the first and second images. The interface circuit as the image acquisition unit 1 may be connected to the external device by wire or wirelessly. Alternatively, the image acquisition unit 1 may be, for example, a communication interface circuit that transmits and receives communication signals to and from an external device. The external device is connected to the communication interface circuit via a network (WAN (Wide Area Network, including a public communication network)) or LAN (Local Area Network), and is a server device that manages the first and second images. In any of these cases where the image acquisition unit 1 is involved, the first height, the first direction, and the first focal length in the first camera are associated with the first image, and these are also acquired together when the first image is acquired. When the image acquisition unit 1 is an interface circuit or a communication interface circuit, the image acquisition unit 1 may be used in combination with the IF unit 4 (that is, the IF unit 4 may be used as the image acquisition unit 1). Alternatively, the image acquisition unit 1 may be an input unit that receives the input of the first image, the first height, the first direction, and the first focal length in the first camera, and the second image. In this case, the image acquisition unit 1 may be used in combination with the input unit 31 described later in the corresponding line segment processing unit 3.

[0053] In this embodiment, the second camera 11-2 of the image acquisition unit 1 also functions as a device that generates a plurality of consecutive surface images in time series by imaging the surface of the river. Such a plurality of surface images may be stored in a storage medium, a recording medium, or a server device, and acquired by the river flow measurement device 1000 via the IF unit 4, or may be acquired via the input unit 31, similar to the above-described first and second images.

[0054] The corresponding line segment processing unit 3 sets at least two pairs of first and second line segments corresponding to each other from the first and second images, respectively, in the first and second image regions where the subject overlaps in the first and second images.

[0055] In this embodiment, for example, the corresponding line segment processing unit 3 includes an input unit 31, a display unit 32, and an extraction unit 33(23) (the corresponding line segment processing unit of the first aspect). The display unit 32 is a device that is connected to the control processing unit 2 and displays at least one of the first and second images according to the control of the control processing unit 2. The input unit 31 is a device that is connected to the control processing unit 2 and receives the input of an input line segment as one of the first and second line segments in one of the first and second images. For example, the input line segment is input when both end points of the input line segment are input from the input unit 31. Each of these input unit 31 and display unit 32 also functions as each of the input unit and display unit in the river water flow measurement device (camera parameter calculation device). In the input unit 31, various commands such as a command for instructing the start of measurement of water flow (start of calculation of camera parameters), and various data necessary for the operation of the river water flow measurement device (camera parameter calculation device) 1000, such as the name of the river in the plurality of surface images, are input. To the display unit 32, the commands and data input from the input unit 31, and the flow velocity and flow rate obtained by the river water flow measurement device 1000 are output. The input unit 31 is, for example, a plurality of input switches, a keyboard, a mouse, etc. to which a predetermined function is assigned. The display unit 32 is, for example, a display device such as a CRT display, an LCD (liquid crystal display device), and an organic EL display. The extraction unit 33(23) extracts a corresponding line segment corresponding to the input line segment received by the input unit 31 in the other of the first and second images, and in this embodiment, is functionally configured in the control processing unit 2 by the execution of a control processing program described later. The extraction unit 33(23) will be further described later.

[0056] Note that the input unit 31 and the display unit 32 may be configured by a touch panel. When configuring this touch panel, the input unit 3 is a position input device that detects and inputs an operation position, such as a resistive film method or a capacitance method. In this touch panel, the position input device is provided on the display surface of the display unit 32, and one or more input content candidates that can be input to the display unit 32 are displayed. When the user touches the display unit 32 on which the input content to be input is displayed, the position is detected by the position input device, and the display content displayed at the detected position is input to the river water flow measurement device 1000 as the user's operation input content. In such a touch panel, since the user can intuitively understand the input operation, a river water flow measurement device 1000 that is easy for the user to handle is provided.

[0057] The IF unit 4 is connected to the control processing unit 2 and is a circuit that inputs and outputs data to and from, for example, external devices according to the control of the control processing unit 2. For example, it is an interface circuit of RS-232C using a serial communication method, an interface circuit using the Bluetooth (registered trademark) standard, and an interface circuit using the USB standard. Further, the IF unit 4 may be a communication interface circuit that transmits and receives communication signals to and from external devices, such as a data communication card or a communication interface circuit according to the IEEE802.11 standard.

[0058] The memory unit 5 is a circuit connected to the control processing unit 2 and stores various predetermined programs and various predetermined data according to the control of the control processing unit 2. The various predetermined programs include, for example, a control processing program, and the control processing program includes a control program, a direction processing program, an extraction program, a camera parameter processing program, an ortho-correction program, a flow velocity processing program, a flow rate processing program, and the like. The control program is a program for controlling each part 1, 3 to 5 of the river water flow measuring device (camera parameter calculation device) 1000. The direction processing program is for the first image captured by the first camera from which the first height from a predetermined reference plane, the first direction of the first optical axis, and the first focal length of the first imaging optical system are known, and the second camera different from the first camera. Based on the first and second image regions where the subject overlaps in the second image captured so as to at least partially overlap the subject of the first image, the parallax between the first and second cameras is obtained, and based on the first direction of the first optical axis and the obtained parallax, the second direction of the second optical axis in the second camera is obtained. The extraction program is a program for extracting a corresponding line segment corresponding to the input line segment received by the input unit in the other of the first and second images. The camera parameter processing program is based on the first height, the first direction of the first optical axis, the first focal length of the first imaging optical system, the second direction of the second optical axis obtained by the direction processing program, and the at least two sets of first and second line segments set by the corresponding line segment processing unit 3. A program for obtaining the camera parameters of the second camera. The ortho-correction program is a program for ortho-correcting each of a plurality of surface images consecutive in time series captured by the second camera of the river surface based on the camera parameters of the second camera obtained by the camera parameter calculation program to generate a plurality of corrected surface images. The flow velocity processing program is a program for obtaining the surface flow velocity on the surface of the river based on the plurality of corrected surface images generated by the ortho-correction program. The flow rate processing program is a program for obtaining the flow rate of the river based on the surface flow velocity obtained by the flow velocity processing program, the shape of the cross section of the river, and the water level of the river.Each such program is provided, for example, by a recording medium such as a CD-ROM or a DVD-ROM that is readable by a computer storing each such program, and is stored in the storage unit 5. Alternatively, for example, each program is downloaded from a server device that manages each such program and is stored in the storage unit 5. The various predetermined data includes the first and second images, the first height, the first direction, and the first focal length in the first camera that generated the first image, the second direction obtained by the direction processing program in the second camera that generated the second image, and the second height and the second focal length obtained by the camera parameter calculation program, the plurality of surface images, the surface flow velocity obtained by the flow velocity processing program, the flow rate obtained by the flow rate processing program, and river information representing predetermined data related to the river, etc., and data necessary for executing each of these programs is included. Such a storage unit 5 includes, for example, a ROM (Read Only Memory) which is a non-volatile memory element, an EEPROM (Electrically Erasable Programmable Read Only Memory) which is a rewritable non-volatile memory element, and the like. And the storage unit 5 includes a RAM (Random Access Memory) or the like which serves as a working memory of the so-called control processing unit 2 that stores data and the like generated during the execution of the predetermined program. The storage unit 5 may be configured to include a hard disk device or a solid state drive (SSD) having a relatively large storage capacity.

[0059] The first focal length of the first camera 11-1 may be the focal length stored in the first camera 11-1 itself or the focal length described in the instruction manual. However, since there may be cases where the size of the image sensor is different from the effective size actually used for image generation in the image sensor, in order to obtain the actual focal length, for example, by the method of Z. Zhang using a calibration board, etc., the first focal length of the first camera 11-1 is measured in advance and stored in the storage unit 5 in advance.

[0060] The memory unit 5 functionally includes a river information storage unit 51 that stores the river information. In this embodiment, the river information is information representing data necessary for obtaining the flow rate of the river, such as cross-sectional shape information representing the shape of the cross-section of the river, and water level information representing the water level of the river. The river information is measured in advance by known conventional means, and is input from the input unit 31, for example, and stored in the river information storage unit 51. Alternatively, for example, when the plurality of surface images are acquired by the IF unit 4 from the storage medium, the recording medium, and the server device, the river information is stored in the storage medium, the recording medium, and the server device along with the plurality of surface images, acquired by the IF unit 4, and stored in the river information storage unit 51.

[0061] The control processing unit 2 controls each part 1, 3 to 5 of the river water flow measuring device (camera parameter calculation device) 1000 according to the functions of the respective parts, obtains camera parameters, and is a circuit for measuring the water flow of the river, such as the flow velocity and the flow rate. The control processing unit 2 is configured to include, for example, a CPU (Central Processing Unit) and its peripheral circuits. When the control processing program is executed, the control processing unit 2 functionally includes a control unit 21, a direction processing unit 22, an extraction unit 23(33), a camera parameter processing unit 24, an ortho correction unit 25, a flow velocity processing unit 26, and a flow rate processing unit 27.

[0062] The control unit 21 controls each part 1, 3 to 5 of the river water flow measuring device (camera parameter calculation device) 1000 according to the functions of the respective parts, and is in charge of the overall control of the river water flow measuring device (camera parameter calculation device) 1000.

[0063] The direction processing unit 22 captures a first image captured by a first camera from which a first height from a predetermined reference plane, a first direction of a first optical axis, and a first focal length of a first imaging optical system can be known, and a second camera different from the first camera. Based on the first and second image regions of the subject that overlap at least partially in the second image of the first image, the parallax between the first and second cameras is obtained, and based on the first direction of the first optical axis and the obtained parallax, the second direction of the second optical axis in the second camera is obtained.

[0064] In the present embodiment, the direction processing unit 22 includes a corresponding point search unit 221, a parallax calculation unit 222, and a direction calculation unit 223 (the direction processing unit 22 of the first aspect). These corresponding point search unit 221, parallax calculation unit 222, and direction calculation unit 223 are functionally configured in the control processing unit 2 by executing the direction processing program.

[0065] The corresponding point search unit 221 searches for corresponding points corresponding to each other between the first and second images in the first and second image regions.

[0066] More specifically, the corresponding point search unit 221 extracts a plurality of feature points from the first image region of the first image and extracts a plurality of feature points from the second image region of the second image. The feature points are locations in the image that can be distinguished from others, for example, the corners or edges of an object captured in the image. There are various methods for extracting these feature points, and for example, A-KAZE or the like is used. Then, the corresponding point search unit 221 searches for feature points extracted from the other of the first and second image regions corresponding to the feature points (one corresponding point) extracted from one of the first and second image regions by corresponding point search and selects them as corresponding points (the other corresponding point). There are various methods for this corresponding point search, such as kNN, and for example, brute-force matching is used.

[0067] Alternatively, for example, the corresponding point search unit 221 extracts feature points from one of the first and second image regions. For example, feature points are extracted from the first image region of the first image. Then, the corresponding point search unit 221 searches for corresponding points (the other corresponding points) corresponding to the feature points (one corresponding point) extracted from one of the first and second image regions from the other of the first and second image regions. In the above example, corresponding points are searched from the second image region of the second image. For this corresponding point search, for example, the Lucas-kanade method or the like is used.

[0068] For example, FIG. 3 shows the result of corresponding point search. FIG. 3B shows the first image PC11, FIG. 3A shows the second image PC21, in FIG. 3B, a plurality of the first corresponding points extracted from the first image PC11 are indicated by ● as the plurality of the one corresponding points, in FIG. 3A, a plurality of the second corresponding points in the second image PC21 corresponding to each of these plurality of first corresponding points ● are indicated by ● as the plurality of the other corresponding points, and the correspondence relationship between the first corresponding point ● and the second corresponding point ● is represented by connecting the mutually corresponding first corresponding point ● and second corresponding point ● with a line segment.

[0069] The disparity calculation unit 222 obtains the disparity based on each corresponding point searched by the corresponding point search unit 221. The disparity (the distance and angular difference between the first and second cameras) is obtained by known conventional means, for example, the eight-point method (eight-point algorithm) for estimating the fundamental matrix (translation matrix and rotation matrix) of the stereo camera from eight sets of each corresponding point, or a method of estimating the fundamental matrix by minimizing the error of the fundamental matrix by iterative calculation using eight or more sets of each corresponding point.

[0070] The direction calculation unit 223 obtains the second direction of the second optical axis in the second camera based on the first direction of the first optical axis and the disparity obtained by the disparity calculation unit 222. More specifically, the direction calculation unit 223 obtains the second direction of the second optical axis by correcting the first direction of the first optical axis with the angular difference in the disparity obtained by the disparity calculation unit 222.

[0071] In the above description, the direction processing unit 22 is configured to include a corresponding point search unit 221, a disparity calculation unit 222, and a direction calculation unit 223. However, the direction processing unit 22 may include an input unit that receives inputs of corresponding points corresponding to each other in the first and second images in the first and second image regions, a disparity calculation unit that obtains the disparity based on each corresponding point received by the input unit, and a direction calculation unit that obtains the second direction of the second optical axis in the second camera based on the first direction of the first optical axis and the disparity obtained by the disparity calculation unit (the direction processing unit 22 of the second aspect). The input unit 31 of the corresponding line segment processing unit 3 is also used as the input unit of this direction processing unit 22. For example, FIG. 4 shows the state of input of each corresponding point. FIG. 4B shows the first image PC12, FIG. 4A shows the second image PC22, FIG. 4B shows five first corresponding points MK11 to MK15 input by the input unit 31 with respect to the first image PC12 as circles, and FIG. 4A shows five second corresponding points MK21 to MK25 in the second image PC22 corresponding to each of these five first corresponding points MK11 to MK15, which are marked with two blank triangles △ facing each other in the left-right direction and two filled triangles ▲ facing each other in the up-down direction within a rectangle. Depending on the image, it may be difficult to extract feature points or search for corresponding points. In such cases, by manually inputting each corresponding point for the first and second images, the disparity between the first and second images can be surely obtained, and thus, the second direction of the second optical axis can be obtained.

[0072] The extraction unit 23(33) extracts corresponding line segments corresponding to the input line segment received by the input unit in the other of the first and second images. For example, FIG. 5 shows the first image PC13, and FIG. 6 shows the second image PC23. The display unit 32 displays at least one of these first and second images PC13, PC23, for example, the first image PC13. Note that the second image PC23 may be displayed on the display unit 32 on a separate screen or the same screen. For such a first image PC13, as shown in FIG. 5, the user (operator) inputs, using the input unit 31, both endpoints PT11 and PT12 of the first A line segment set on the near bank of the river as the first input line segment, and both endpoints PT13 and PT14 of the first B line segment set on the far bank of the river as the second input line segment. The line segment connecting point PT12 and point PT14 becomes the first C line segment set along the width direction of the river as the third input line segment. Then, as shown in FIG. 6, the extraction unit 23(33) searches for four corresponding points PT21 to PT24 corresponding to these four points PT11 to PT14 respectively by corresponding point search, and as the first to third corresponding line segments corresponding to the first to third input line segments respectively, the first A line segment D PT11PT12 and the second A line segment D that forms a pair PT21PT2 , the first B line segment D PT13PT14 and the second B line segment D that forms a pair PT23PT24 , and the first C line segment D PT12PT14 and the second C line segment D that forms a pair PT22PT24 are each extracted.

[0073] In the camera parameter calculation device, a predetermined reference plane may be arbitrarily and appropriately set, but in the river flow measurement device 1000, the predetermined reference plane is set to the water surface of the river. Therefore, as shown in FIG. 5, the first A line segment D PT11PT12 , the first B line segment D PT13PT14 and the first C line segment D PT12PT14 are set to be on a plane substantially coinciding with the water surface of the river, and the second A line segment D, the second B line segment D, and the second C line segment D extracted by corresponding point search PT21PT22 , the second B line segment D PT23PT24 and the second C line segment D PT22PT24It is on a plane that substantially coincides with the water surface of the river, and the second height of the second camera 11-2 is the height from the water surface of the river.

[0074] In the above description, as the first aspect, the corresponding line segment processing unit 3 is configured to include the input unit 31, the display unit 32, and the extraction unit 33 (23). However, the corresponding line segment processing unit 3 may include a display unit that displays the first and second images, and an input unit that receives the input of the first and second line segments (the corresponding line segment processing unit 3 of the second aspect). For example, the display unit 32 displays the first image PC13 shown in FIG. 5 and the second image PC23 shown in FIG. 6 on the same screen. The user inputs four first to fourth points PT11 to PT14 with the input unit 31 for the first image PC13, and inputs four first to fourth points PT21 to PT24 with the input unit 31 for the second image PC23. As a result, the input of three sets of first A and second A line segments D PT11PT12 、D PT21PT2 、first B and second B line segments D PT13PT14 、D PT23PT24 、and first C and second C line segments D PT12PT14 、D PT22PT24 is received. According to this, since the input of the first and second line segments is received by the input unit 31, the first and second line segments can be surely set.

[0075] The camera parameter processing unit 24 obtains the camera parameters of the second camera based on the first height, the first direction of the first optical axis, the first focal length of the first imaging optical system, the second direction of the second optical axis obtained by the direction processing unit, and the at least two sets of first and second line segments set by the corresponding line segment processing unit. In the present embodiment, the at least two sets of first and second line segments are three or more sets of first and second line segments, and the camera parameters of the second camera obtained by the camera parameter processing unit 24 are the second height from the reference plane and the second focal length of the second imaging optical system in the second camera. The camera parameter processing unit 24 uses the arithmetic expression of Equation 4 for obtaining the length of the line segment based on the second height and the second focal length to obtain the camera parameters of the second camera such that the error with respect to the length of each of the at least two sets of first and second line segments is minimized. For example, the camera parameter processing unit 24 obtains the camera parameters of the second camera by solving the arithmetic expression of Equation 4 by the least squares method. More specifically, the camera parameter processing unit 24 uses the first height h1, the first directions κ1, φ1, ω1, and the first focal length c1 in the first camera 11-1 in Equation 4 to obtain the length of the first A line segment D PT11PT12 , the length of the first B line segment D PT13PT14 , and the length of the first C line segment D PT12PT14 . Thereby, the lengths of the second A line segment D PT21PT22 , the second B line segment D PT23PT24 , and the second C line segment D PT22PT24 are obtained. For the angle κ1, for example, 0° is used as described above, and the same applies to the angle κ2. Subsequently, the camera parameter processing unit 24 uses the second directions κ2, φ2, ω2 obtained by the direction processing unit 22 and the lengths of the obtained second A line segment D PT21PT22 , the second B line segment D PT23PT24 , and the second C line segment D PT22PT24 in Equation 4 to generate an arithmetic expression for the camera parameters. Then, the camera parameter processing unit 24 obtains the second height h2 and the second focal length c2 in the camera parameters of the second camera by solving this arithmetic expression for the camera parameters by the least squares method.

[0076] The orthorectification unit 25 orthorectifies each of a plurality of surface images consecutive in time series of the surface of the river imaged by the second camera based on the camera parameters of the second camera obtained by the camera parameter processing unit to generate a plurality of rectified surface images. Note that any one of the plurality of surface images may be used as the second image.

[0077] The surface velocity processing unit 26 obtains the surface velocity on the surface of the river based on the plurality of rectified surface images generated by the orthorectification unit. Various methods for obtaining the surface velocity of a river based on a plurality of surface images consecutive in time series are known. For example, there are PIV (Particle Image Velocimetry) disclosed in Japanese Patent No. 4025161, STIV (Space-Time Image Velocimetry) disclosed in Japanese Patent No. 6910506, and the like. In this STIV, based on the plurality of rectified surface images generated by the orthorectification unit, an STI regarding an inspection line set as a measurement target in the surface of the river is generated. Based on the generated STI, the angle of the diagonal pattern is obtained, and based on the obtained angle of the diagonal pattern, the imaging time interval in the plurality of surface images consecutive in time series, and the actual length of the inspection line, the surface velocity at the inspection line is obtained. The STI is generated by extracting each image of the inspection line from each of the plurality of rectified surface images and juxtaposing these extracted images in time series order with one end aligned. Feature points on the image due to an object or the like flowing according to the flow of the river are diagonally connected by the flow of the river in the STI to form the diagonal pattern. Since the angle (tilt angle) of the diagonal pattern depends on the surface velocity, the surface velocity of the river can be obtained based on the angle of the diagonal pattern.

[0078] The flow rate processing unit 27 determines the flow rate of the river based on the surface flow velocity obtained by the flow velocity processing unit, the shape of the cross-section of the river, and the water level of the river. The flow rate may be determined from the surface flow velocity at a single inspection line, and in order to determine it more accurately, it may be determined from the multiple surface flow velocities at multiple inspection lines. For example, the flow rate processing unit 27 first divides the cross-section of the river up to the water level of the river into a plurality of sections, and estimates the average flow velocity of each of the plurality of sections based on the multiple surface flow velocities at each of the plurality of inspection lines. Generally, since the flow velocity distribution in the vertical water depth can be represented by a logarithmic distribution or an exponential distribution, the flow velocity at each water depth can be estimated from the surface flow velocity. Subsequently, the flow rate processing unit 27 determines the flow rate of each section by multiplying the area of the section by the average flow velocity of the section. Then, the flow rate processing unit 27 determines the flow rate of the river by summing up the flow rates of the plurality of sections. If the size of the section is a minute size, the sum becomes an integral.

[0079] For example, as shown in FIG. 7, for a river image PCR obtained by imaging the river obliquely, a plurality of, in this example, 17 inspection lines DL1 to DL17 along the flow are set in the input unit 31 so as to be juxtaposed along the width direction. When the river image PCR shown in FIG. 7 is ortho-corrected by the ortho-correction unit 25, a corrected river image PCC shown in FIG. 8 is generated. The surface flow velocities and the flow rates at each of the inspection lines DL1 to DL17 obtained by the flow velocity processing unit 26 and the flow rate processing unit 27 respectively are shown in the upper row of FIG. 9. The horizontal axis represents the positions of each of the inspection lines DL1 to DL17, the left vertical axis represents the surface flow velocity, and the right vertical axis represents the flow rate. The cross-sectional shape of the river is shown in the lower row of FIG. 9. The horizontal axis represents the positions of each of the inspection lines DL1 to DL17, the vertical axis represents the depth, and the water surface of the river is set at a depth of 0 (reference level). In this example, from the second inspection line DL2 to the twelfth inspection line DL12 are the water surface of the river.

[0080] Then, the control processing unit 2 displays on the display unit 32 the camera parameters of the second camera obtained by the camera parameter processing unit 24, the flow velocities at each inspection line obtained by the flow velocity processing unit 26, and the flow rate of the river obtained by the flow rate processing unit 27.

[0081] These control processing unit 2, input unit 31, display unit 32, IF unit 4, and storage unit 5 can be configured by, for example, a computer such as a desktop type, notebook type, or tablet type. Also, for example, the river water flow measurement device (camera parameter calculation device) 1000 can be configured by a smartphone equipped with a camera and a three-axis acceleration sensor as the first camera 11-1 and the direction measurement unit 12, and a tablet equipped with a camera as the second camera 11-2, control processing unit 2, input unit 31, display unit 32, IF unit 4, and storage unit 5.

[0082] Next, the operation of this embodiment will be described. FIG. 10 is a flowchart showing the operation in the river water flow measurement device (camera parameter calculation device).

[0083] The user installs, for example, a smartphone equipped with a three-axis acceleration sensor and a camera, and a tablet equipped with a camera as the river water flow measurement device 1000 on the river to be measured, and activates each of the smartphone and the tablet. After activation, each of the smartphone and the tablet in the river water flow measurement device 1000 executes initialization of each necessary part and starts its operation. In the tablet of the river water flow measurement device 1000, by executing its control processing program, the control processing unit 2 is functionally configured with a control unit 21, a direction processing unit 22, an extraction unit 23(33), a camera parameter processing unit 24, an ortho correction unit 25, a flow velocity processing unit 26, and a flow rate processing unit 27, and the direction processing unit 22 is functionally configured with a corresponding point search unit 221, a parallax calculation unit 222, and a direction calculation unit 223.

[0084] In FIG. 10, the river water flow measuring device 1000 measures the first direction of the first optical axis by the smartphone, acquires a first image, acquires a second image by the tablet, and stores the second image in the storage unit 5 (S1).

[0085] The user (operator) inputs the first height of the smartphone into the tablet. The river water flow measuring device 1000 receives the input of the first height by the tablet and stores the input in the storage unit 5 (S2). As a result, the tablet acquires and stores the first height. The first height is actually measured by the user. For example, the river water flow measuring device 1000 is installed on a bridge over a river, and the distance from the smartphone (first camera 11-1) to the water surface of the river is actually measured as the first height. In the present embodiment, since it may be difficult to actually measure the first height from the water surface of the river, for example, the distance D from the smartphone (first camera 11-1) to the river (waterside) is actually measured by a laser rangefinder or the like. R is actually measured and obtained by the following formula 7. For example, the distance D R is input into the tablet, and the first height is obtained by the arithmetic processing of formula 7 by the tablet.

[0086]

Equation

[0087] In addition, the distance D between two points substantially at the same height as the water surface of the river 12 is actually measured, and the first height may be obtained by the above formula 4.

[0088] Subsequently, the river water flow measuring device 1000 searches for a plurality of corresponding points for the first and second images by the direction processing unit 22 of the tablet by searching for corresponding points, obtains the parallax between the first and second images based on these plurality of corresponding points, and obtains the second direction of the second optical axis based on the obtained parallax and the first direction of the first optical axis (S3, direction processing step).

[0089] Subsequently, the river flow measurement device 1000 displays the first and second images on the display unit 32 of the tablet (S4, display step of the corresponding line segment processing step). The user inputs, as each input line segment, the two end points of the first A line segment set on the near bank of the river and the two end points of the first B line segment set on the far bank of the river to the tablet as the first A line segment, the first B line segment, and the first C line segment respectively. The river flow measurement device 1000 receives the input of the two end points of the first A line segment and the two end points of the first B line segment by the tablet and stores them in the storage unit 5 (S5, input step of the corresponding line segment processing step). As a result, the tablet acquires and stores the first A line segment, the first B line segment, and the first C line segment as each input line segment.

[0090] Subsequently, the river flow measurement device 1000 extracts, by the extraction unit 23(33) of the tablet, the second A line segment, the second B line segment, and the second C line segment as each corresponding line segment corresponding to the first A line segment, the first B line segment, and the first C line segment as each input line segment received by the input unit 31, and stores them in the storage unit 5 (S6, extraction step of the corresponding line segment processing step).

[0091] Subsequently, the river flow measurement device 1000 obtains the lengths of the second A line segment, the second B line segment, and the second C line segment by obtaining the lengths of the first A line segment, the first B line segment, and the first C line segment by using the first height, the first direction, and the first focal length in Equation 4, and generates an arithmetic expression for the camera parameters by using the second direction obtained by the direction processing unit 22 in the process S3 and the lengths of the second A line segment, the second B line segment, and the second C line segment obtained above in Equation 4, and obtains the second height and the second focal length by solving the generated arithmetic expression for the camera parameters by the least squares method and stores them in the storage unit 5 (S7, camera parameter processing step).

[0092] Subsequently, the river flow measurement device 1000 acquires a plurality of surface images continuous in time series by imaging the surface of the river with the second camera 11-2 of the tablet and stores them in the storage unit 5 (S8, surface image acquisition step).

[0093] Subsequently, the river water flow measurement device 1000 orthogonally corrects each of the plurality of surface images continuous in the time series based on the camera parameters of the second camera obtained by the camera parameter processing unit 24 by the ortho-correction unit 25 of the tablet to generate a plurality of corrected surface images, and stores them in the storage unit 5 (S9, ortho-correction step).

[0094] Subsequently, the river water flow measurement device 1000 obtains the surface flow velocity at each inspection line on the surface of the river based on the plurality of corrected surface images generated by the ortho-correction unit 25 by the flow velocity processing unit 26 of the tablet, and stores it in the storage unit 5 (S10, flow velocity processing step).

[0095] Subsequently, the river water flow measurement device 1000 obtains the flow rate of the river based on the surface flow velocity at each inspection line obtained by the flow velocity processing unit 26, the shape of the cross-section of the river, and the water level of the river by the flow rate processing unit 27 of the tablet, and stores it in the storage unit 5 (S11, flow rate processing step).

[0096] Then, the river water flow measurement device 1000 displays on the display unit 32 of the tablet the camera parameters of the second camera obtained by the camera parameter processing unit 24, the surface flow velocity at each inspection line obtained by the flow velocity processing unit 26, and the flow rate of the river obtained by the flow rate processing unit 27, and ends this process (S12). Note that the river water flow measurement device 1000 may output these values to an external device via the IF unit 4 as necessary.

[0097] When the deviation of the principal point position and aberration in the imaging optical system of the camera are not considered, the position of the actual point corresponding to the point on the image can be obtained based on the direction of the optical axis, height, and focal length from the position of the point on the image. The camera parameter calculation device provided in the river flow measurement device 1000, as well as the camera parameter calculation method and camera parameter calculation program implemented thereon, can obtain the second direction of the second optical axis in the second camera 11-2 by correcting the first direction of the first optical axis in the first camera 11-1 based on the parallax between the first and second images. The above camera parameter calculation device, camera parameter calculation method, and camera parameter calculation program can obtain the actual length of each of the at least two sets of second line segments corresponding thereto by obtaining the actual length of each of the at least two sets of first line segments from the positions of each end point of each of the at least two sets of first line segments, and can obtain the height and focal length in the second camera 11-2 by performing an inverse calculation of the calculation method for obtaining the actual length from the positions of each end point. Therefore, when obtaining camera parameters, the above camera parameter calculation device, camera parameter calculation method, and camera parameter calculation program do not need to use, for example, calibration points, buildings, shelves, etc., and can further reduce the constraints on the imaging target. Although the second direction of the second optical axis in the camera parameters can also be calculated together with the second height and the second focal length, the above camera parameter calculation device, camera parameter calculation method, and camera parameter calculation program obtain the second height and the second focal length in the camera parameters after obtaining the second direction of the second optical axis by correcting the first direction of the first optical axis, so that it is possible to reduce the occurrence of local solution outliers, and the second height and the second focal length can be stably obtained as realistic values.

[0098] Since the above camera parameter calculation device, camera parameter calculation method, and camera parameter calculation program include a corresponding point search unit 221, each corresponding point corresponding to each other in the corresponding point search can be automatically obtained.

[0099] Since the above camera parameter calculation device, camera parameter calculation method, and camera parameter calculation program include the extraction unit 23(33), if an input line segment is input as one of the first and second line segments, the extraction unit 23(33) extracts a corresponding line segment as the other of the first and second line segments. Therefore, the labor of inputting the first and second line segments can be saved.

[0100] The camera parameters of the second camera obtained by the camera parameter processing unit 24 are two, namely the second height in the second camera 11-2 and the second focal length of the second imaging optical system. Therefore, it is possible to obtain the camera parameters of the second camera with two sets of first and second line segments. Since the above camera parameter calculation device, camera parameter calculation method, and camera parameter calculation program obtain the camera parameters of the second camera based on three or more sets of first and second line segments having redundancy for two unknowns, the camera parameters of the second camera can be obtained more appropriately.

[0101] According to the embodiment, a recording medium recording a camera parameter calculation program can be provided. According to the embodiment, a river water flow measurement device 1000, a river water flow measurement method, and a river water flow measurement program each including the camera parameter calculation device, the camera parameter calculation method, and the camera parameter calculation program can be provided, and a recording medium recording the river water flow measurement program can be provided.

[0102] The above river water flow measurement device 1000, river water flow measurement method, and river water flow measurement program can obtain camera parameters required for ortho-correction without using, for example, calibration points, buildings, shelves, etc. Therefore, the constraints on the imaging target can be further reduced, and when attempting to measure, the surface velocity of the river can be measured more quickly. In particular, the surface velocity of the river can be measured even from a location on the river where six or more calibration points cannot be arranged.

[0103] The above river water flow measurement device 1000, river water flow measurement method, and river water flow measurement program set at least two sets of first and second line segments into three sets of the near bank, the far bank, and the river width. Therefore, the surface velocity of the river can be measured with good balance accuracy from the near side to the far side of the river along the river width direction.

[0104] Since the above river water flow measurement device 1000, river water flow measurement method, and river water flow measurement program perform flow rate processing, the flow rate of the river can be measured.

[0105] Since the above river water flow measurement device 1000 includes the first and second cameras 11-1 and 11-2 and the direction measurement unit 12, it can be actually measured at the measurement site only by the river water flow measurement device 1000.

[0106] To express the present invention, the embodiments have been appropriately and fully described above with reference to the drawings. However, it should be recognized that those skilled in the art can easily make changes and / or improvements to the above embodiments. Therefore, as long as the changes or improvements made by those skilled in the art do not depart from the scope of the claims described in the claims, such changes or improvements are construed to be included within the scope of the claims of the claims.

Explanation of Signs

[0107] 1000 River flow measurement device equipped with a camera parameter calculation device, 1 Image acquisition unit, 2 Control processing unit, 3 Corresponding line segment processing unit, 4 IF unit, 5 Storage unit, 11-1 First camera, 11-2 Second camera, 12 Direction measurement unit, 21 Control unit, 22 Direction processing unit, 23(33), Extraction unit, 24 Camera parameter processing unit, 25 Orthogonal correction unit, 26 Flow velocity processing unit, 27 Flow rate processing unit, 31 Input unit, 32 Display unit, 51 River information storage unit

Claims

1. A first image captured by a first camera that can determine a first height from a predetermined reference plane, a first direction of a first optical axis, and a first focal length of a first imaging optical system, and a second camera different from the first camera. A parallax between the first and second cameras is obtained based on first and second image regions in which the subject overlaps in the first and second images captured so as to overlap at least partially with the subject of the first image, and the first direction of the first optical axis and the obtained parallax. A direction processing unit that obtains a second direction of a second optical axis in the second camera; In the first and second image regions, a corresponding line segment processing unit that sets at least two pairs of first and second line segments corresponding to each other from the first and second images respectively; A camera parameter processing unit that obtains camera parameters of the second camera based on the first height, the first direction of the first optical axis, the first focal length of the first imaging optical system, the second direction of the second optical axis obtained by the direction processing unit, and the at least two pairs of first and second line segments set by the corresponding line segment processing unit; A camera parameter calculation device.

2. The direction processing unit In the first and second image regions, a corresponding point search unit that searches for corresponding points corresponding to each other between the first and second images; A parallax calculation unit that obtains the parallax based on each corresponding point searched by the corresponding point search unit; A direction calculation unit that obtains a second direction of the second optical axis in the second camera based on the first direction of the first optical axis and the parallax obtained by the parallax calculation unit, or Or In the first and second image regions, an input unit that receives inputs of corresponding points corresponding to each other in the first and second images respectively; A parallax calculation unit that obtains the parallax based on each corresponding point received by the input unit; A direction calculation unit that obtains a second direction of the second optical axis in the second camera based on the first direction of the first optical axis and the parallax obtained by the parallax calculation unit, The camera parameter calculation device according to claim 1.

3. The corresponding line segment processing unit A display unit that displays at least one of the first and second images; In one of the first and second images, an input unit that receives an input of an input line segment as one of the first and second line segments; In the other of the first and second images, an extraction unit that extracts a corresponding line segment corresponding to the input line segment received by the input unit, or Or ​ A display unit that displays the first and second images, and an input unit that receives input of the first and second line segments. The camera parameter calculation device according to claim 1.

4. The at least two sets of first and second line segments are three or more sets of first and second line segments, The camera parameters of the second camera obtained by the camera parameter processing unit are the second height from the reference plane and the second focal length of the second imaging optical system in the second camera, The camera parameter processing unit uses an arithmetic expression for obtaining the length of the line segment based on the second height and the second focal length, and determines the camera parameters of the second camera so that the error with respect to the length of each of the at least two sets of first and second line segments is minimized. The camera parameter calculation device according to claim 1.

5. A first image captured by a first camera in which the first height from a predetermined reference plane, the first direction of the first optical axis, and the first focal length of the first imaging optical system are known, and a second camera different from the first camera. A direction processing step of obtaining a parallax between the first and second cameras based on the first and second image regions in which the subject of the first image overlaps at least partially with the subject of the second image, and obtaining a second direction of the second optical axis in the second camera based on the first direction of the first optical axis and the obtained parallax; A corresponding line segment processing step of setting at least two sets of first and second line segments corresponding to each other from the first and second images in the first and second image regions; A camera parameter processing step of obtaining camera parameters of the second camera based on the first height, the first direction of the first optical axis, the first focal length of the first imaging optical system, the second direction of the second optical axis obtained in the direction processing step, and the at least two sets of first and second line segments set in the corresponding line segment processing step. A camera parameter calculation method.

6. A camera parameter calculation program for causing a computer to function as the camera parameter calculation device according to any one of claims 1 to 4.

7. A computer-readable recording medium on which the camera parameter calculation program according to claim 6 is recorded.

8. The camera parameter calculation device according to any one of claims 1 to 4, An orthorectification unit that orthorectifies each of a plurality of surface images consecutive in time series obtained by imaging the surface of the river with the second camera based on the camera parameters of the second camera obtained by the camera parameter calculation device to generate a plurality of corrected surface images; A flow velocity processing unit that obtains a surface flow velocity on the surface of the river based on the plurality of corrected surface images generated by the orthorectification unit. A river water flow measurement device.

9. The at least two sets of first and second line segments include three or more sets of first and second line segments including a set of first A and second A line segments set on the near bank of the river with respect to the second camera, a set of first B and second B line segments set on the far bank of the river with respect to the second camera, and a set of first C and second C line segments set along the width direction of the river. The river water flow measurement device according to claim 8.

10. The river water flow measurement device further includes a flow rate processing unit that obtains the flow rate of the river based on the surface flow velocity obtained by the flow velocity processing unit, the shape of the cross section of the river, and the water level of the river. The river water flow measurement device according to claim 8.

11. The first and second cameras; The river water flow measurement device according to claim 8, further comprising a direction measurement unit that measures the first direction of the first optical axis in the first camera. The river water flow measurement device according to claim 8.

12. The camera parameter calculation method according to claim 5; An orthorectification step of orthorectifying each of a plurality of surface images consecutive in time series obtained by imaging the surface of the river with the second camera based on the camera parameters of the second camera obtained by the camera parameter calculation method to generate a plurality of corrected surface images; A flow velocity processing step of obtaining a surface flow velocity on the surface of the river based on the plurality of corrected surface images generated by the orthorectification step. A river water flow measurement method.

13. A river water flow measurement program for causing a computer to function as the river water flow measurement device according to claim 8.

14. A computer-readable recording medium on which the river water flow measurement program according to claim 13 is recorded.

Images