Camera parameter calculation device, said method, said program, and said recording medium, and river water flow measurement device, said method, said program, and said recording medium

Abstract

The present invention provides a camera parameter calculation device, method, program and recording medium thereof that can reduce restrictions on an image capture subject, as well as a river water flow measuring device, method, program and recording medium thereof that use the same. [Solution] The camera parameter calculation device of the present invention calculates the parallax between the first and second cameras based on first and second image regions in which the subjects of a first image captured by a first camera in which a first height from a reference plane, a first optical axis direction, and a first focal length are known, and a second image captured by a second camera so as to overlap at least partially with the subject of the first image, calculates the second optical axis direction of the second camera based on the first optical axis direction and the parallax, sets at least two sets of first and second line segments that correspond to each other in the first and second image regions, and calculates the camera parameters of the second camera based on the first height, the first optical axis direction, the first focal length, the second optical axis direction, and the at least two sets of the first and second line segments.

Description

[Technical field]

[0001] The present invention relates to a camera parameter calculation device, a camera parameter calculation method, a camera parameter calculation program and a recording medium thereof for determining camera parameters, as well as a river water flow measurement device, a river water flow measurement method, a river water flow measurement program and a recording medium thereof using the same. [Background technology]

[0002] When determining a predetermined amount of an object captured in an image based on an image captured by a camera, so-called camera parameters are required. These camera parameters generally include internal parameters such as focal length and aberration in the imaging optical system (imaging optical system) of the camera, and external parameters such as the optical axis direction of the imaging optical system and the position of the camera (three-dimensional position of the projection center). Such camera parameters are generally calculated based on six or more orientation points captured in an image by arranging components representing known orientation points with three-dimensional coordinate values ​​in six or more locations. In such a method using orientation points, it takes time and effort to set the orientation points, and depending on the imaging target (subject), it may be difficult to set the orientation points, so it is desirable to determine the camera parameters without using the orientation points. An example of such a technology is a camera calibration device disclosed in Patent Document 1.

[0003] The camera calibration device disclosed in Patent Document 1 includes an acquisition unit that acquires a first image plane normal vector and a second image plane normal vector, which correspond to a first world coordinate space normal vector and a second world coordinate space normal vector, respectively, in an image plane of an image captured by a camera in a world coordinate space, and which are normal vectors with respect to a reference plane in the world coordinate space and have the same length, and an acquisition unit that acquires a first start point and a first end point of the first image plane normal vector and a second start point and a second end point of the second image plane normal vector. a camera parameter calculation unit that calculates an internal parameter and an external parameter of the camera based on the calculated projective depth vector, a first image plane start point vector and an end point vector corresponding to the first start point and the first end point, respectively, and a second image plane start point vector and an end point vector corresponding to the second start point and the second end point, respectively. In this camera calibration device, according to paragraph

[0016] , the first and second world coordinate space normal vectors use sides of a building or shelf or the like extending perpendicular to a horizontal plane. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7151879 Summary of the Invention [Problem to be solved by the invention]

[0005] Incidentally, the camera calibration device disclosed in Patent Document 1 determines camera parameters without using orientation points, so as described above, the image must include objects that extend perpendicular to the horizontal plane, such as buildings and shelves, and there are limitations on the objects that can be captured.

[0006] The present invention has been made in consideration of the above-mentioned circumstances, and its object is to provide a camera parameter calculation device, a camera parameter calculation method, a camera parameter calculation program and a recording medium thereof that can further reduce the constraints on the subject of imaging, as well as a river water flow measurement device, a river water flow measurement method, a river water flow measurement program and a recording medium thereof using the same. [Means for solving the problem]

[0007] As a result of various investigations, the present inventors have found that the above object can be achieved by the present invention described below. That is, a camera parameter calculation device according to one aspect of the present invention includes a direction processing unit that calculates a parallax between a first image captured by a first camera in which a first height from a predetermined reference plane, a first direction of a first optical axis, and a first focal length of a first imaging optical system are known, and a second image captured by a second camera different from the first camera so that the first image overlaps at least partially with a subject of the first image, and calculates a second direction of the second optical axis in the second camera based on the first direction of the first optical axis and the calculated parallax; a corresponding line segment processing unit that sets at least two pairs of first and second line segments that correspond to each other from the first and second images, respectively, in the first and second image regions; and a camera parameter processing unit that calculates camera parameters of the second camera based on the first height, the first direction of the first optical axis, the first focal length of the first imaging optical system, the second direction of the second optical axis calculated by the direction processing unit, and the at least two pairs of first and second line segments set by the corresponding line segment processing unit. Preferably, in the above-mentioned camera parameter calculation device, the first and second directions of the first and second optical axes are expressed by first and second u-axis rotation angles with respect to the u-axis and first and second v-axis rotation angles with respect to the v-axis, respectively, when the initial state is a state in which the w direction in a camera coordinate system (local coordinate system) that is a uvw orthogonal coordinate system with the projection center as the coordinate origin coincides with the Z direction in a ground coordinate system (world coordinate system) that is an XYZ orthogonal coordinate system, and the camera parameters of the second camera calculated by the camera parameter processing unit are a second height from the reference plane of the second camera and a second focal length of the second imaging optical system when the XY plane in the ground coordinate system is the reference plane. Preferably, in the above-mentioned camera parameter calculation device, the camera parameter calculation device further includes the first and second cameras and a direction measurement unit that measures the first direction of the first optical axis of the first camera. Preferably, in the above-mentioned camera parameter calculation device, the direction measurement unit is a three-axis acceleration sensor provided in the first camera that measures the tilt of the first camera.

[0008] The position of an actual point corresponding to a point on an image can be obtained from the position of the point on the image based on the direction, height, and focal length of the optical axis when the deviation of the principal point position and aberration in the imaging optical system of the camera are not taken into consideration. The camera parameter calculation device can obtain the second direction of the second optical axis in the second camera by correcting the first direction of the first optical axis in the first camera based on the parallax between the first and second images. The camera parameter calculation device can obtain the actual lengths of the first line segments in at least two pairs of first line segments from the positions of their respective endpoints, thereby obtaining the actual lengths of the second line segments in the at least two pairs of first line segments corresponding to the first line segments, and can obtain the second height and the second focal length in the second camera by inverse calculation of the calculation method for obtaining the actual lengths from the positions of the endpoints. Therefore, the camera parameter calculation device does not need to use, for example, a control point, a building, a shelf, etc., when obtaining the camera parameters, and can further reduce the constraints on the imaging target.

[0009] In another aspect, in the above-mentioned camera parameter calculation device, the direction processing unit includes a corresponding point searcher that searches for corresponding points that correspond to each other between the first and second images in the first and second image regions, a parallax calculation unit that calculates the parallax based on each corresponding point searched for by the corresponding point searcher, and a direction calculation unit that calculates a second direction of the second optical axis in the second camera based on a first direction of the first optical axis and the parallax calculated by the parallax calculation unit, or includes an input unit that accepts input of corresponding points that correspond to each other in the first and second images in the first and second image regions, a parallax calculation unit that calculates the parallax based on each corresponding point accepted by the input unit, and a direction calculation unit that calculates the second direction of the second optical axis in the second camera based on the first direction of the first optical axis and the parallax calculated by the parallax calculation unit.

[0010] When such a camera parameter calculation device includes a corresponding point search unit, it can automatically find corresponding points by searching for corresponding points. When the camera parameter calculation device includes an input unit, even if it is difficult to find corresponding points by searching for corresponding points depending on the image, the input unit receives input of corresponding points, so that it is possible to reliably find the parallax and the second direction of the second optical axis.

[0011] In another aspect, in the above-mentioned camera parameter calculation device, the corresponding line segment processing unit includes a display unit that displays at least one of the first and second images, an input unit that accepts input of an input line segment as one of the first and second line segments in one of the first and second images, and an extraction unit that extracts a corresponding line segment in the other of the first and second images that corresponds to the input line segment accepted by the input unit, or includes a display unit that displays the first and second images and an input unit that accepts input of the first and second line segments.

[0012] When such a camera parameter calculation device includes an extraction unit, if an input line segment is input as one of the first and second line segments, the extraction unit extracts a corresponding line segment as the other of the first and second line segments, thereby saving the effort of inputting the first and second line segments.When the camera parameter calculation device does not include an extraction unit, the input of the first and second line segments is accepted by the input unit, so that the first and second line segments can be reliably set.

[0013] In another aspect, in the above-mentioned camera parameter calculation device, the at least two sets of first and second line segments are three or more sets of first and second line segments, the camera parameters of the second camera calculated by the camera parameter processing unit are a second height from the reference plane in the second camera and a second focal length of a second imaging optical system, and the camera parameter processing unit calculates the camera parameters of the second camera using an arithmetic expression for calculating the length of a line segment based on the second height and the second focal length so that an error in each length of the at least two sets of first and second line segments is minimized. Preferably, in the above-mentioned camera parameter calculation device, the camera parameter processing unit calculates the camera parameters of the second camera by solving the arithmetic expression by the least squares method.

[0014] The camera parameters of the second camera calculated by the camera parameter processing unit are two, the second height and the second focal length, and therefore the camera parameters of the second camera can be calculated using two pairs of the first and second line segments. The camera parameter calculation device calculates the camera parameters of the second camera based on three or more pairs of the first and second line segments having redundancy for two unknowns, and therefore can calculate the camera parameters more appropriately.

[0015] A camera parameter calculation method according to another aspect of the present invention includes a direction processing step of calculating a parallax between a first image captured by a first camera in which a first height from a predetermined reference plane, a first direction of a first optical axis, and a first focal length of a first imaging optical system are known, and a second image captured by a second camera different from the first camera so as to overlap at least a portion of a subject of the first image, based on a first and second image area in which the subject overlaps, and calculating a second direction of the second optical axis in the second camera based on the first direction of the first optical axis and the calculated parallax; a corresponding line segment processing step of setting at least two sets of first and second line segments that correspond to each other from the first and second images, respectively, in the first and second image areas; and a camera parameter processing step of calculating camera parameters of the second camera based on the first height, the first direction of the first optical axis, the first focal length of the first imaging optical system, the second direction of the second optical axis calculated in the direction processing step, and the at least two sets of first and second line segments set in the corresponding line segment processing step. A camera parameter calculation program according to another aspect of the present invention is a camera parameter calculation program for causing a computer to function as any one of the camera parameter calculation devices described above.

[0016] Such a camera parameter calculation method and camera parameter calculation program eliminates the need to use, for example, orientation points, buildings, shelves, etc., when determining the camera parameters, and can further reduce restrictions on the subject to be imaged.

[0017] A recording medium according to another aspect of the present invention is a computer-readable recording medium having the above-mentioned camera parameter calculation program recorded thereon.

[0018] According to this, when obtaining the camera parameters, it is not necessary to use, for example, orientation points, buildings, shelves, etc., and a camera parameter calculation program can be provided that can further reduce the restrictions on the subject to be imaged.

[0019] A river water flow measuring device according to another aspect of the present invention comprises any one of the camera parameter calculation devices described above, an orthorectification unit that orthorectifies each of a plurality of consecutive surface images of the river surface captured by the second camera in a time series based on the camera parameters of the second camera calculated by the camera parameter calculation device to generate a plurality of corrected surface images, and a flow velocity processing unit that calculates the surface flow velocity on the surface of the river based on the plurality of corrected surface images generated by the orthorectification unit.

[0020] Such a river water flow measuring device can obtain the camera parameters required for ortho-correction without the need to use, for example, control points, buildings, shelves, etc., thereby reducing restrictions on the object to be imaged and making it possible to measure the surface flow velocity of the river more quickly when measurements are required.

[0021] In another aspect, in the above-mentioned river water flow measuring device, the at least two sets of first and second line segments are three or more sets of first and second line segments, including a set of 1A and 2A line segments set on the near bank of the river relative to the second camera, a set of 1B and 2B line segments set on the far bank of the river relative to the second camera, and a set of 1C and 2C line segments set along the width direction of the river.

[0022] Such a river water flow measuring device sets at least two sets of first and second line segments into three sets: the near bank, the far bank, and the river width, so that the surface flow velocity of the river can be measured with balanced accuracy from the front to the back of the river along the river width direction.

[0023] In another aspect, the above-mentioned river water flow measuring device further includes a flow rate processing unit that calculates the flow rate of the river based on the surface flow velocity calculated by the flow rate processing unit, the cross-sectional shape of the river, and the water level of the river. Since such a river water flow measuring device includes a flow rate processing unit, it is possible to measure the flow rate of the river.

[0024] In another aspect, the above-mentioned river water flow measuring device further includes the first and second cameras and a direction measuring unit that measures the first direction of the first optical axis of the first camera. Since such a river water flow measuring device includes the first and second cameras and the direction measuring unit, actual measurement can be performed at the measurement site using only the river water flow measuring device.

[0025] A river water flow measurement method according to another aspect of the present invention includes the above-mentioned camera parameter calculation method, an orthorectification step of orthorectifying each of a plurality of time-series consecutive surface images of the surface of the river captured by the second camera based on the camera parameters of the second camera calculated by the camera parameter calculation method to generate a plurality of corrected surface images, and a flow velocity processing step of calculating a surface flow velocity on the surface of the river based on the plurality of corrected surface images generated in the orthorectification step. A river water flow measurement program according to another aspect of the present invention is a river water flow measurement program for causing a computer to function as the above-mentioned river water flow measurement device. Such a river water flow measurement method and river water flow measurement program can further reduce restrictions on the image capture target, and can measure the surface flow velocity of the river more quickly when measurement is attempted.

[0026] According to another aspect of the present invention, a computer-readable recording medium is provided that stores the above-mentioned river water flow measurement program. This can reduce restrictions on the imaging target, and can provide a river water flow measurement program that can measure the surface flow velocity of a river more quickly when measurement is attempted. Effect of the Invention

[0027] The camera parameter calculation device, the camera parameter calculation method, the camera parameter calculation program, and the recording medium thereof according to the present invention can reduce the restrictions on the imaging target. According to the present invention, it is possible to provide a river water flow measurement device, a river water flow measurement method, a river water flow measurement program, and the recording medium thereof using the camera parameter calculation device, the camera parameter calculation method, the camera parameter calculation program, and the recording medium thereof. [Brief description of the drawings]

[0028] [Figure 1] FIG. 13 is a diagram for explaining a collinearity condition in imaging by a camera. [Diagram 2] 1 is a block diagram showing a configuration of a river water flow measuring device equipped with a camera parameter calculation device in an embodiment. [Diagram 3] 4 is a diagram for explaining a direction processing unit of a first embodiment in the river water flow measuring device (camera parameter calculation device). FIG. [Figure 4] 13 is a diagram for explaining a direction processing unit of a second aspect in the river water flow measuring device (camera parameter calculation device). FIG. [Diagram 5] FIG. 2 is a first diagram for explaining a corresponding line segment processing unit of a first and second embodiment in the river water flow measuring device (camera parameter calculation device). [Figure 6] FIG. 2 is a second diagram for explaining the corresponding line segment processing unit of the first and second aspects in the river water flow measuring device (camera parameter calculation device). [Figure 7] FIG. 13 is a diagram for explaining an example of an inspection line set on an image of a river. [Figure 8] FIG. 8 is a diagram showing an example of an image obtained by orthorectifying the image shown in FIG. 7. [Figure 9] FIG. 11 is a diagram showing, as an example, the results of calculation of surface flow velocity and water volume. [Figure 10] 5 is a flowchart showing the operation of the river water flow measuring device (camera parameter calculation device). DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] Hereinafter, one or more embodiments of the present invention will be described with reference to the drawings. However, the scope of the invention is not limited to the disclosed embodiments. In addition, configurations with the same reference numerals in each drawing indicate that they are the same configurations, and their description will be omitted as appropriate. In this specification, when referring to a general term, a reference numeral without a subscript is used, and when referring to an individual configuration, a reference numeral with a subscript is used.

[0030] A camera parameter calculation device in an embodiment includes: a direction processing unit that calculates a parallax between a first image captured by a first camera in which a first height from a predetermined reference plane, a first direction of a first optical axis, and a first focal length of a first imaging optical system are known, and a second image captured by a second camera different from the first camera so that the first image overlaps at least partially with a subject of the first image, and calculates a second direction of the second optical axis in the second camera based on the first direction of the first optical axis and the calculated parallax; a corresponding line segment processing unit that sets at least two pairs of first and second line segments that correspond to each other from the first and second images, respectively, in the first and second image regions; and a camera parameter processing unit that calculates camera parameters of the second camera based on the first height, the first direction of the first optical axis, the first focal length of the first imaging optical system, the second direction of the second optical axis calculated by the direction processing unit, and the at least two pairs of first and second line segments set by the corresponding line segment processing unit.

[0031] First, the principle of calculation of the camera parameters of the second camera in such a camera parameter calculation device will be described.

[0032] FIG. 1 is a diagram for explaining the collinearity conditions in the imaging of a camera. In FIG. 1, the XYZ Cartesian coordinate system is a ground coordinate system (world coordinate system) set on the ground, the uvw Cartesian coordinate system is a camera coordinate system (local coordinate system (first local coordinate system)) set on the camera with the projection center o of the camera (image) as the coordinate origin, and the xy Cartesian coordinate system is a pixel coordinate system (local coordinate system (second local coordinate system)) set on the image PC with the intersection point of the normal of the image PC passing through the projection center o and the image PC as the coordinate origin when the image PC is arranged so as to be parallel to the uv plane of the camera coordinate system uvw. The pixel coordinate system xy is also a two-dimensional Cartesian coordinate system set on the two-dimensional image sensor IM of the camera with the intersection point of the normal of the two-dimensional image sensor IM passing through the projection center o of the camera and the imaging surface (imaging surface of the imaging optical system) of the two-dimensional image sensor IM as the coordinate origin.

[0033] The initial state is when the w direction of the camera coordinate system uvw and the Z direction of the ground coordinate system are the same (the w axis of the camera coordinate system uvw and the Z axis of the ground coordinate system are parallel to each other). When the camera rotates around the w axis by an angle κ, then around the v axis by an angle φ, and finally around the u axis by an angle ω, the position (X, Y, Z) of the object on the ground and the projection center o(X 0 , Y 0 , Z 0 The collinear condition that the position (x, y) of the object on the image is on a straight line is expressed by the following equations 1-1 and 1-2. Here, if the focal length of the imaging optical system in the camera is c, the coefficient a related to the rotation of the camera is ij are expressed by the following formulas 2-1 to 2-9.

[0034]

number

[0035]

number

[0036] From these formulas 1-1, 1-2, and 2-1 to 2-9, if there is no deviation of the principal point position or distortion, the camera parameter X 0 , Y 0 , Z 0 , ω, φ, κ, and c can be obtained from six or more control points whose positions in the ground coordinate system are known and the positions of each of these control points on the image (positions in the pixel coordinate system).

[0037] On the other hand, the above formulas 1-1 and 1-2 can be transformed into the following formulas 3-1 and 3-2.

[0038]

number

[0039] Here, two points D on the plane at any altitude Z 1 (X 1 , Y 1), D 2 (X 2 , Y 2 ) distance D 12 is expressed by the following equation 4, where h=ZZ 0 It is.

[0040]

number

[0041] In addition, the rotation angle of the camera around the w axis changes from angle κ to angle κ'. As a result, the two points D 1 (X 1 , Y 1 ), D 2 (X 2 , Y 2 ) each has 2 points D 1 '(X 1 ', Y 1 '), D 2 '(X 2 ', Y 2 If we move to each of them, the following equations 5-1 and 5-2 hold. Here, the coefficient a ij ' is the coefficient when the angle is κ'.

[0042]

number

[0043] Coefficient a in Equation 5-2 ij , a ij By substituting κ′, the equation can be transformed into the following equation 6, and it can be seen that the variable A is 0 regardless of the angle κ.

[0044]

number

[0045] As a result, even if the angle κ changes, the distance between two points on the XY plane at the same altitude does not change. In other words, the result of geometric correction (orthorectification) is the same regardless of the angle κ. For this reason, the coefficient a ij , a ijWhen calculating ', the angle κ may be any value, and is preferably, for example, 0° or 90°, whose cosine and sinue are integer values.

[0046] Therefore, from Equation 4 and Equation 6, the first height h from a predetermined reference plane (the plane of height Z described above) is 1 (h 1 =ZZ 01 , the projection center of the first camera o 1 (X 01 , Y 01 , Z 01 ), if there is a first image captured by a first camera in which the first direction of the first optical axis and the first focal length of the first imaging optical system are known, and a second image captured by a second camera different from the first camera so as to overlap at least a part of the subject of the first image, the parallax between the first and second images is calculated based on the first and second images, and the first direction of the first optical axis is corrected based on the calculated parallax, thereby obtaining the second direction of the second optical axis in the second camera. At least two or more line segments corresponding to each other are set in the first and second images, respectively, and the lengths of the at least two or more line segments set are calculated for the first image by using Equation 4, and the lengths of the at least two or more line segments calculated and the calculated second direction of the second optical axis are substituted into Equation 4 for the second image, and the camera parameters h 2 , c 2 (h 2 =ZZ 02 , the projection center of the second camera o 2 (X 02 , Y 02 , Z 02 ) The camera parameters of the second camera h 2 , c 2 The second image can be orthorectified based on

[0047] Next, using a river water flow measuring device equipped with a camera parameter calculation device as an example, the camera parameter calculation device in the embodiment, the camera parameter calculation method and camera parameter calculation program implemented therein, as well as the river water flow measuring device, the river water flow measurement method and river water flow measurement program implemented therein will be explained in more detail, along with each recording medium on which each program is recorded.

[0048] FIG. 2 is a block diagram showing the configuration of a river water flow measuring device including a camera parameter calculation device in an embodiment. FIG. 3 is a diagram for explaining a direction processing unit of a first aspect in the river water flow measuring device (camera parameter calculation device). FIG. 4 is a diagram for explaining a direction processing unit of a second aspect in the river water flow measuring device (camera parameter calculation device). FIG. 5 is a first diagram for explaining a corresponding line segment processing unit of a first and second aspect in the river water flow measuring device (camera parameter calculation device). FIG. 6 is a second diagram for explaining a corresponding line segment processing unit of a first and second aspect in the river water flow measuring device (camera parameter calculation device). FIG. 7 is a diagram for explaining an inspection line set in an image of a river as an example. FIG. 8 is a diagram showing an image obtained by orthorectifying the image shown in FIG. 7 as an example. FIG. 9 is a diagram showing a calculation result of a surface flow velocity and a water volume as an example.

[0049] A river water flow measuring device 1000 equipped with a camera parameter calculation device in an embodiment includes, for example, an image acquisition unit 1, a control processing unit 2, a corresponding line segment processing unit 3, an interface unit (IF unit) 4, and a memory unit 5, as shown in Figure 2.

[0050] The image acquisition unit 1 is connected to the control processing unit 2 and is a device that acquires, under the control of the control processing unit 2, a first image captured by a first camera, the first height from a predetermined reference plane, the first direction of a first optical axis, and the first focal length of a first imaging optical system being known, and a second image captured by a second camera different from the first camera so as to overlap at least partially with a subject of the first image.

[0051] In this embodiment, for example, the image acquisition unit 1 includes a first camera 11-1, a second camera 11-2, and a direction measurement unit 12. The first and second cameras 11-1 and 11-2 are devices that capture an image of a subject (image target) to generate an image of the subject, and are, for example, digital cameras that include an imaging optical system (imaging optical system) that captures an optical image of the subject on a predetermined imaging plane, an image sensor that is arranged with its light receiving surface aligned with the imaging plane and converts the optical image of the subject into an electrical signal, and an image processing unit that generates image data that is data representing an image of the subject by image processing the output of the image sensor. Each of the first and second cameras 11-1 and 11-2 digital cameras may be a monochrome camera, a color camera, a visible light camera, or an infrared camera. The direction measurement unit 12 is a device that measures the first direction of the first optical axis in the first camera 11-1, and is, for example, configured to include a three-axis acceleration sensor that measures the inclination of the first camera 11-1. A first direction of the first optical axis can be obtained from the inclination of the first camera 11-1. In this embodiment, for the first camera 11-1 and the direction measurement unit 12, a so-called tablet (tablet computer) or a so-called smartphone (portable communication terminal device with information processing function) equipped with a three-axis speed sensor and a camera is used for convenience. The first camera 11-1 and the second camera 11-2 may be the same type of product or different types of products.

[0052] The image acquisition unit 1 may be, for example, an interface circuit that inputs and outputs data to and from an external device. The external device may be, for example, a storage medium such as a USB (Universal Serial Bus) memory and an SD card (registered trademark) that stores the first and second images. Alternatively, the external device may be a drive device that reads data from a storage medium such as a CD-ROM (Compact Disc Read Only Memory), a CD-R (Compact Disc Recordable), a DVD-ROM (Digital Versatile Disc Read Only Memory), and a DVD-R (Digital Versatile Disc Recordable) that records the first and second images. The interface circuit as the image acquisition unit 1 may be connected to the external device by wire or wirelessly. Alternatively, the image acquisition unit 1 may be, for example, a communication interface circuit that transmits and receives communication signals to and from an external device, and the external device is a server device that is connected to the communication interface circuit via a network (WAN (Wide Area Network, including a public communication network)) or LAN (Local Area Network) and manages the first and second images. When the image acquisition unit 1 is any of these, the first image is associated with a first height, a first direction, and a first focal length in the first camera, and these are also acquired when the first image is acquired. When the image acquisition unit 1 is an interface circuit or a communication interface circuit, the image acquisition unit 1 may also be used as the IF unit 4 (i.e., the IF unit 4 may be used as the image acquisition unit 1). Alternatively, the image acquisition unit 1 may be an input unit that accepts input of the first image, the first height, the first direction, and the first focal length in the first camera, and the second image. In this case, the image acquisition unit 1 may also be used as an input unit 31 in the corresponding line segment processing unit 3, which will be described later.

[0053] In this embodiment, the second camera 11-2 of the image acquisition unit 1 also functions as a device that captures the surface of the river to generate multiple surface images that are continuous in time series. Note that, like the above-mentioned first and second images, such multiple surface images may be stored in a storage medium, a recording medium, or a server device and acquired by the river water flow measuring device 1000 via the IF unit 4, or may be acquired via the input unit 31.

[0054] The corresponding line segment processing unit 3 sets at least two pairs of corresponding first and second line segments from each of the first and second images in a first and second image area where subjects overlap in the first and second images.

[0055] In this embodiment, for example, the corresponding line segment processing unit 3 includes an input unit 31, a display unit 32, and an extraction unit 33 (23) (corresponding line segment processing unit of the first aspect). The display unit 32 is connected to the control processing unit 2, and is a device that displays at least one of the first and second images according to the control of the control processing unit 2. The input unit 31 is connected to the control processing unit 2, and is a device that accepts input of an input line segment as one of the first and second line segments in one of the first and second images. For example, the input line segment is input by inputting both end points of the input line segment from the input unit 31. The input unit 31 and the display unit 32 also function as the input unit and the display unit in the river water flow measuring device (camera parameter calculation device), and the input unit 31 receives various commands, such as a command to start measuring the water flow (start calculating the camera parameters), and various data required for operating the river water flow measuring device (camera parameter calculation device) 1000, such as the name of the river in the multiple surface images, and the display unit 32 outputs the commands and data input from the input unit 31, and the flow speed and flow rate calculated by the river water flow measuring device 1000. The input unit 31 is, for example, multiple input switches to which predetermined functions are assigned, a keyboard, a mouse, etc. The display unit 32 is, for example, a display device such as a CRT display, an LCD (liquid crystal display device), an organic EL display, etc. The extraction unit 33 (23) extracts a corresponding line segment in the other of the first and second images, which corresponds to the input line segment received by the input unit 31, and in this embodiment, is functionally configured in the control processing unit 2 by executing a control processing program described later. The extraction unit 33 (23) will be described further below.

[0056] The input unit 31 and the display unit 32 may be configured as a touch panel. In the case of configuring this touch panel, the input unit 3 is a position input device that detects and inputs an operation position, for example, a resistive film type or a capacitive type. In this touch panel, a position input device is provided on the display surface of the display unit 32, and one or more input contents that can be input are displayed on the display unit 32. When a user touches the display unit 32 displaying the input contents that the user wants to input, the position is detected by the position input device, and the display contents displayed at the detected position are input to the river water flow measuring device 1000 as the user's operation input contents. In such a touch panel, the user can easily intuitively understand the input operation, and therefore, a river water flow measuring device 1000 that is easy for the user to handle is provided.

[0057] The IF unit 4 is connected to the control processing unit 2 and is a circuit that inputs and outputs data to and from, for example, an external device under the control of the control processing unit 2, such as an interface circuit for RS-232C, which is a serial communication method, an interface circuit using the Bluetooth (registered trademark) standard, and an interface circuit using the USB standard. The IF unit 4 may also be a communication interface circuit that transmits and receives communication signals to and from an external device, such as a data communication card or a communication interface circuit conforming to the IEEE802.11 standard.

[0058] The storage unit 5 is a circuit connected to the control processing unit 2 and stores various predetermined programs and various predetermined data under the control of the control processing unit 2. The various predetermined programs include, for example, a control processing program, and the control processing program includes a control program, a direction processing program, an extraction program, a camera parameter processing program, an orthorectification program, a flow rate processing program, and a flow rate processing program. The control program is a program for controlling each unit 1, 3 to 5 of the river water flow measuring device (camera parameter calculation device) 1000. The direction processing program is a program for calculating the parallax between the first and second cameras based on a first image captured by a first camera in which a first height from a predetermined reference plane, a first direction of a first optical axis, and a first focal length of a first imaging optical system are known, and a second image captured by a second camera different from the first camera so as to overlap at least a part of the subject of the first image, based on a first and second image area in which the subject overlaps, and for calculating the second direction of the second optical axis in the second camera based on the first direction of the first optical axis and the calculated parallax. The extraction program is a program for extracting a corresponding line segment corresponding to the input line segment received by the input unit in the other of the first and second images. The camera parameter processing program is a program for determining camera parameters of the second camera based on the first height, the first direction of the first optical axis, the first focal length of the first imaging optical system, the second direction of the second optical axis determined by the direction processing program, and the at least two pairs of first and second line segments set by the corresponding line segment processing unit 3. The orthorectification program is a program for orthorectifying each of a plurality of time-series consecutive surface images of the surface of the river captured by the second camera based on the camera parameters of the second camera determined by the camera parameter calculation program to generate a plurality of corrected surface images. The flow velocity processing program is a program for determining a surface flow velocity on the surface of the river based on a plurality of corrected surface images generated by the orthorectification program. The flow rate processing program is a program for determining the flow rate of the river based on the surface flow velocity determined by the flow velocity processing program, the cross-sectional shape of the river, and the water level of the river.Each of these programs is provided by, for example, a computer-readable recording medium, such as a CD-ROM or a DVD-ROM, on which each of the programs is recorded, and is stored in the storage unit 5. Alternatively, each of the programs is downloaded from a server device that manages each of the programs, and is stored in the storage unit 5. The various types of predetermined data include data necessary for executing each of these programs, such as the first and second images, the first height, the first direction, and the first focal length of the first camera that generated the first image, the second direction of the second camera that generated the second image, which was determined by the direction processing program, and the second height and the second focal length determined by the camera parameter calculation program, the multiple surface images, the surface flow velocity determined by the flow velocity processing program, the flow rate determined by the flow rate processing program, and river information representing predetermined data related to the river. Such a storage unit 5 includes, for example, a ROM (Read Only Memory), which is a nonvolatile storage element, an EEPROM (Electrically Erasable Programmable Read Only Memory), which is a rewritable nonvolatile storage element, or the like. The storage unit 5 includes a RAM (Random Access Memory) that stores data generated during execution of the predetermined program and serves as a so-called working memory of the control processing unit 2. The storage unit 5 may be configured to include a hard disk device or a solid state drive (SSD) with a relatively large storage capacity.

[0059] The first focal length of first camera 11-1 may be a focal length stored in first camera 11-1 itself or a focal length described in the instruction manual. However, since there are cases where the size of the image sensor differs from the effective size of the image sensor actually used for image generation, in order to obtain the actual focal length, the first focal length of first camera 11-1 is actually measured in advance, for example, by Z. Zhang's method using a calibration board, and is stored in advance in memory unit 5.

[0060] The storage unit 5 functionally includes a river information storage unit 51 that stores the river information. In this embodiment, the river information is, for example, information representing data required to calculate the flow rate of the river, such as cross-sectional shape information representing the cross-sectional shape of the river and water level information representing the water level of the river. The river information is measured in advance by a known conventional method, for example, input from the input unit 31, and stored in the river information storage unit 51. Alternatively, for example, when the multiple surface images are acquired by the IF unit 4 from the storage medium, the recording medium, and the server device, the river information is stored in the storage medium, the recording medium, and the server device in association with the multiple surface images, acquired by the IF unit 4, and stored in the river information storage unit 51.

[0061] The control processing unit 2 is a circuit for controlling each of the units 1, 3 to 5 of the river water flow measuring device (camera parameter calculation device) 1000 according to the function of each unit, obtaining camera parameters, and measuring the water flow of the river, such as flow velocity and flow rate. The control processing unit 2 is configured with, for example, a CPU (Central Processing Unit) and its peripheral circuits. The control processing unit 2 functionally includes a control unit 21, a direction processing unit 22, an extraction unit 23 (33), a camera parameter processing unit 24, an ortho-correction unit 25, a flow velocity processing unit 26, and a flow rate processing unit 27 by executing a control processing program.

[0062] The control unit 21 controls each of the units 1, 3 to 5 of the river water flow measuring device (camera parameter calculation device) 1000 according to the function of each unit, and is in charge of the overall control of the river water flow measuring device (camera parameter calculation device) 1000.

[0063] The direction processing unit 22 calculates the parallax between the first and second cameras based on a first and second image areas in which a subject overlaps between a first image captured by a first camera, in which a first height from a predetermined reference plane, a first direction of a first optical axis, and a first focal length of a first imaging optical system are known, and a second image captured by a second camera different from the first camera so as to overlap at least partially with the subject of the first image, and calculates a second direction of the second optical axis of the second camera based on the first direction of the first optical axis and the calculated parallax.

[0064] In this embodiment, the direction processing unit 22 includes a corresponding point search unit 221, a disparity calculation unit 222, and a direction calculation unit 223 (first aspect direction processing unit 22). The corresponding point search unit 221, the disparity calculation unit 222, and the direction calculation unit 223 are functionally configured in the control processing unit 2 by execution of the direction processing program.

[0065] The corresponding point search section 221 searches for corresponding points that correspond to each other between the first and second images in the first and second image regions.

[0066] More specifically, the corresponding point search unit 221 extracts a plurality of feature points from the first image region of the first image, and extracts a plurality of feature points from the second image region of the second image. The feature points are parts in an image that can be distinguished from others, such as corners and edges of an object captured in the image. There are various methods for extracting the feature points, and for example, A-KAZE is used. Then, the corresponding point search unit 221 searches for a feature point extracted from one of the first and second image regions (one corresponding point) from the other of the first and second image regions by a corresponding point search, and selects it as a corresponding point (the other corresponding point). There are various methods for this corresponding point search, such as kNN, and for example, Brute-Force Matcher is used.

[0067] Alternatively, for example, the corresponding point search unit 221 extracts feature points from one of the first and second image regions. For example, feature points are extracted from the first image region of the first image. Then, the corresponding point search unit 221 searches for a corresponding point (the other corresponding point) that corresponds to the feature point (the one corresponding point) extracted from one of the first and second image regions, from the other of the first and second image regions. In the above example, the corresponding point is searched for from the second image region of the second image. For example, the Lucas-Kanade method or the like is used for this corresponding point search.

[0068] For example, the results of the corresponding point search are shown in Fig. 3. Fig. 3B shows the first image PC11, Fig. 3A shows the second image PC21, Fig. 3B shows a plurality of first corresponding points extracted from the first image PC11 as the plurality of one corresponding points, Fig. 3A shows a plurality of second corresponding points in the second image PC21 corresponding to each of the plurality of first corresponding points, and the correspondence relationship between the first corresponding points and the second corresponding points is represented by connecting the corresponding first corresponding points and the second corresponding points with a line segment.

[0069] The disparity calculation unit 222 calculates the disparity based on each corresponding point searched for by the corresponding point search unit 221. The disparity (the interval and angle difference between the first and second cameras) is calculated by a known conventional method, such as an 8-point method (8-point algorithm) that estimates a fundamental matrix (translation matrix and rotation matrix) of a stereo camera from each of eight pairs of corresponding points, or a method that estimates the fundamental matrix by minimizing an error in the fundamental matrix through iterative calculation using eight or more pairs of corresponding points.

[0070] The direction calculation unit 223 calculates a second direction of the second optical axis in the second camera based on the first direction of the first optical axis and the parallax calculated by the parallax calculation unit 222. More specifically, the direction calculation unit 223 calculates the second direction of the second optical axis by correcting the first direction of the first optical axis by the angle difference in the parallax calculated by the parallax calculation unit 222.

[0071] In the above description, the direction processing unit 22 is configured to include the corresponding point search unit 221, the parallax calculation unit 222, and the direction calculation unit 223. However, the direction processing unit 22 may include an input unit that receives input of corresponding points that correspond to each other in the first and second images in the first and second image regions, a parallax calculation unit that calculates the parallax based on the corresponding points received by the input unit, and a direction calculation unit that calculates the second direction of the second optical axis in the second camera based on the first direction of the first optical axis and the parallax calculated by the parallax calculation unit (direction processing unit 22 of the second embodiment). The input unit of this direction processing unit 22 also serves as the input unit 31 of the corresponding line segment processing unit 3. For example, FIG. 4 shows how the corresponding points are input. Fig. 4B shows the first image PC12, Fig. 4A shows the second image PC22, Fig. 4B shows five first corresponding points MK11-MK15 inputted by the input unit 31 for the first image PC12 with circles, Fig. 4A shows five second corresponding points MK21-MK25 in the second image PC22 corresponding to the five first corresponding points MK11-MK15 with marks including two blank triangles △ facing each other in the left-right direction and two filled triangles ▲ facing each other in the up-down direction within a rectangle. Depending on the image, it may be difficult to extract feature points or to search for corresponding points. In such cases, by manually inputting each corresponding point for the first and second images, it is possible to reliably obtain the parallax between the first and second images, and therefore the second direction of the second optical axis can be obtained.

[0072] The extraction unit 23 (33) extracts a corresponding line segment corresponding to the input line segment received by the input unit in the other of the first and second images. For example, FIG. 5 shows a first image PC13, and FIG. 6 shows a second image PC23. The display unit 32 displays at least one of the first and second images PC13 and PC23, for example, the first image PC13. The second image PC23 may be displayed on the display unit 32 on a different screen or the same screen. For such a first image PC13, the user (operator) inputs, as shown in FIG. 5, both end points PT11 and PT12 of a first A line segment set on the near bank of the river as a first input line segment, and inputs both end points PT13 and PT14 of a first B line segment set on the far bank of the river as a second input line segment, using the input unit 31. The line segment connecting point PT12 and point PT14 becomes a first C line segment set along the width direction of the river as a third input line segment. Then, as shown in FIG. 6, the extraction unit 23 (33) searches for four corresponding points PT21 to PT24 corresponding to these four points PT11 to PT14, respectively, by performing a corresponding point search, and extracts the first A line segment D PT11PT12 The second line segment D that pairs with PT21PT2 , 1B line segment D PT13PT14 The second line segment D that pairs with PT23PT24 , and the first C line segment D PT12PT14 The second line segment D that pairs with PT22PT24 Extract each one.

[0073] In the camera parameter calculation device, the predetermined reference plane may be set arbitrarily and appropriately, but in the river water flow measurement device 1000, the predetermined reference plane is set to the water surface of the river. Therefore, as shown in FIG. 5, the first A line segment D PT11PT12 , 1B line segment D PT13PT14 and 1C line segment D PT12PT14 is set to be on a plane that approximately coincides with the water surface of the river, and the second A line segment D is extracted by the corresponding point search. PT21PT22 , 2nd B line segment D PT23PT24 and the second C line segment D PT22PT24is on a plane that approximately coincides with the water surface of the river, and the second height of the second camera 11-2 is the height from the water surface of the river.

[0074] In the above description, the corresponding line segment processing unit 3 is configured to include the input unit 31, the display unit 32, and the extraction unit 33 (23) in the first embodiment, but the corresponding line segment processing unit 3 may include a display unit that displays the first and second images, and an input unit that accepts input of the first and second line segments (corresponding line segment processing unit 3 in the second embodiment). For example, the display unit 32 displays the first image PC13 shown in FIG. 5 and the second image PC23 shown in FIG. 6 on the same screen, and the user inputs four first to fourth points PT11 to PT14 for the first image PC13 through the input unit 31, and inputs four first to fourth points PT21 to PT24 for the second image PC23 through the input unit 31. As a result, three pairs of firstA and secondA line segments D PT11PT12 , D PT21PT2 , 1B and 2B line segments D PT13PT14 , D PT23PT24 , and 1C and 2C line segments D PT12PT14 , D PT22PT24 According to this, since the input of the first and second line segments is received by the input unit 31, the first and second line segments can be reliably set.

[0075] The camera parameter processing unit 24 determines the camera parameters of the second camera based on the first height, the first direction of the first optical axis, the first focal length of the first imaging optical system, the second direction of the second optical axis determined by the direction processing unit, and the at least two pairs of first and second line segments set by the corresponding line segment processing unit. In this embodiment, the at least two pairs of first and second line segments are three or more pairs of first and second line segments, and the camera parameters of the second camera determined by the camera parameter processing unit 24 are the second height from the reference plane in the second camera and the second focal length of the second imaging optical system, and the camera parameter processing unit 24 determines the camera parameters of the second camera so that the error for each length of the at least two pairs of first and second line segments is minimized using the calculation formula of Equation 4 that determines the length of a line segment based on the second height and the second focal length. For example, the camera parameter processing unit 24 determines the camera parameters of the second camera by solving the calculation formula of Equation 4 by the least squares method. More specifically, the camera parameter processing unit 24 calculates the first height h 1 , first direction κ 1 , φ 1 , ω 1 and the first focal length c 1 By using this in Equation 4, the first A line segment D PT11PT12 , 1B line segment D PT13PT14 and 1C line segment D PT12PT14 Calculate the length of each of the following: PT21PT22 , 2nd B line segment D PT23PT24 and the second C line segment D PT22PT24 The length of each of the angles κ is calculated. 1 For example, 0° is used for the angle κ 2 Next, the camera parameter processing unit 24 calculates the second direction κ 2 , φ 2 , ω 2 and the second line segment D PT21PT22 , 2nd B line segment D PT23PT24 and the second C line segment D PT22PT24The camera parameter processing unit 24 then uses the lengths of the first and second camera 1 and 2 in Equation 4 to generate a calculation formula for the camera parameters. The camera parameter processing unit 24 then solves the calculation formula for the camera parameters by the least squares method to obtain the second height h 2 and the second focal length c 2 Request.

[0076] The orthorectification unit 25 orthorectifies each of a plurality of time-series consecutive surface images of the river surface captured by the second camera based on the camera parameters of the second camera calculated by the camera parameter processing unit to generate a plurality of corrected surface images. Any of the plurality of surface images may be used as the second image.

[0077] The flow velocity processing unit 26 calculates the surface flow velocity on the surface of the river based on a plurality of corrected surface images generated by the orthorectification unit. There are various known methods for calculating the surface flow velocity of a river based on a plurality of time-series consecutive surface images, such as PIV (Particle Image Velocimetry) disclosed in Japanese Patent No. 4025161 and STIV (Space-Time Image Velocimetry) disclosed in Japanese Patent No. 6910506. In this STIV, an STI is generated for an inspection line set as a measurement target on the surface of the river based on a plurality of corrected surface images generated by the orthorectification unit, an angle of a diagonal pattern is calculated based on the generated STI, and a surface flow velocity on the inspection line is calculated based on the calculated angle of the diagonal pattern, the imaging time interval in the plurality of time-series consecutive surface images, and the actual length of the inspection line. The STI is generated by extracting each image of the inspection line from each of the plurality of corrected surface images, and arranging the extracted images in chronological order with one end aligned. In STI, feature points on an image due to objects flowing with the river flow are connected diagonally by the river flow to form the diagonal pattern. The angle (slope angle) of the diagonal pattern depends on the surface flow velocity, so the surface flow velocity of the river can be calculated based on the angle of the diagonal pattern.

[0078] The flow rate processing unit 27 calculates the flow rate of the river based on the surface flow rate calculated by the flow rate processing unit, the shape of the cross section of the river, and the water level of the river. The flow rate may be calculated from the surface flow rate at one inspection line, or may be calculated from multiple surface flow rates at multiple inspection lines to obtain a more accurate flow rate. For example, the flow rate processing unit 27 first divides the cross section of the river up to the water level of the river into multiple sections, and estimates the average flow rate of each of the multiple sections based on the multiple surface flow rates at each of the multiple inspection lines. In general, the flow rate distribution of the vertical water depth can be expressed as a logarithmic distribution or an exponential distribution, so that the flow rate of each water depth can be estimated from the surface flow rate. Next, the flow rate processing unit 27 calculates the flow rate of each of the multiple sections by multiplying the area of ​​the section by the average flow rate of the section. Then, the flow rate processing unit 27 calculates the flow rate of the river by summing up the flow rates of the multiple sections. If the size of the section is infinitesimal, the sum becomes an integral.

[0079] For example, as shown in FIG. 7, a river image PCR obtained by capturing an image of a river from an oblique angle is set by the input unit 31 so that a plurality of inspection lines DL1 to DL17 along the flow, 17 in this example, are arranged side by side along the width direction. When the river image PCR shown in FIG. 7 is orthorectified by the orthorectification unit 25, a corrected river image PCC shown in FIG. 8 is generated. The surface flow velocities and flow rates at the inspection lines DL1 to DL17 calculated by the flow velocity processing unit 26 and the flow rate processing unit 27 are shown in the upper part of FIG. 9. The horizontal axis is the position of each of the inspection lines DL1 to DL17, the left vertical axis is the surface flow velocity, and the right vertical axis is the flow rate. The lower part of FIG. 9 shows the cross-sectional shape of the river. The horizontal axis is the position of each of the inspection lines DL1 to DL17, the vertical axis is the depth, and the water surface of the river is set to a depth of 0 (reference level). In this example, the water surface of the river is from the second inspection line DL2 to the twelfth inspection line DL12.

[0080] Then, the control processing unit 2 displays on the display unit 32 the camera parameters of the second camera calculated by the camera parameter processing unit 24, the flow velocities at each inspection line calculated by the flow velocity processing unit 26, and the flow rate of the river calculated by the flow rate processing unit 27.

[0081] The control processing unit 2, the input unit 31, the display unit 32, the IF unit 4, and the storage unit 5 can be configured, for example, by a computer of a desktop type, a notebook type, a tablet type, etc. Also, for example, the river water flow measuring device (camera parameter calculation device) 1000 can be configured by a smartphone equipped with a camera and a three-axis acceleration sensor as the first camera 11-1 and the direction measuring unit 12, and a tablet equipped with a camera as the second camera 11-2, the control processing unit 2, the input unit 31, the display unit 32, the IF unit 4, and the storage unit 5.

[0082] Next, the operation of this embodiment will be described with reference to Fig. 10, which is a flow chart showing the operation of the river water flow measuring device (camera parameter calculation device).

[0083] A user places, for example, a smartphone equipped with a third axis acceleration sensor and a camera, and a tablet equipped with a camera as the river water flow measuring device 1000 in the river to be measured, and starts up the smartphone and the tablet. After starting up, the smartphone and the tablet in the river water flow measuring device 1000 execute initialization of the necessary parts and start their operation. In the tablet of the river water flow measuring device 1000, by executing the control processing program, the control processing unit 2 is functionally configured with a control unit 21, a direction processing unit 22, an extraction unit 23 (33), a camera parameter processing unit 24, an ortho-correction unit 25, a flow velocity processing unit 26, and a flow rate processing unit 27, and the direction processing unit 22 is functionally configured with a corresponding point search unit 221, a parallax calculation unit 222, and a direction calculation unit 223.

[0084] In FIG. 10, the river water flow measuring device 1000 measures a first direction of a first optical axis with the smartphone, obtains a first image, obtains a second image with the tablet, and stores them in the memory unit 5 (S1).

[0085] A user (operator) inputs the first height of the smartphone into the tablet. The river water flow measuring device 1000 accepts the input of the first height by the tablet and stores it in the memory unit 5 (S2). As a result, the tablet acquires and stores the first height. The first height is actually measured by the user. For example, the river water flow measuring device 1000 is installed on a bridge over a river, and the distance from the smartphone (first camera 11-1) to the water surface of the river is actually measured as the first height. In this embodiment, since it may be difficult to actually measure the first height from the water surface of the river, the distance D from the smartphone (first camera 11-1) to the river (water's edge) is measured using, for example, a laser range finder or the like. R is measured and calculated by the following formula 7. For example, the distance D R is input to the tablet, and the first height is determined by the calculation of Equation 7 using the tablet.

[0086]

number

[0087] The distance D between the two points at approximately the same height as the water surface of the river 12 may be actually measured, and the first height may be calculated by the above-mentioned formula 4.

[0088] Next, the river water flow measuring device 1000 uses the direction processing unit 22 of the tablet to search for multiple corresponding points in the first and second images, calculates the disparity between the first and second images based on the multiple corresponding points, and calculates the second direction of the second optical axis based on the calculated disparity and the first direction of the first optical axis (S3, direction processing step).

[0089] Next, the river water flow measuring device 1000 displays the first and second images on the display unit 32 of the tablet (S4, display step of the corresponding line segment processing step). The user inputs both end points of the 1A line segment set on the near bank of the river and both end points of the 1B line segment set on the far bank of the river, as the 1A line segment, the 1B line segment, and the 1C line segment, as the input line segments, into the tablet. The river water flow measuring device 1000 accepts the input of both end points of the 1A line segment and the 1B line segment through the tablet and stores them in the storage unit 5 (S5, input step of the corresponding line segment processing step). As a result, the tablet acquires and stores the 1A line segment, the 1B line segment, and the 1C line segment, as the input line segments.

[0090] Next, the river water flow measuring device 1000 extracts, by the extraction unit 23 (33) of the tablet, the 2A line segment, the 2B line segment, and the 2C line segment as corresponding line segments corresponding to the 1A line segment, the 1B line segment, and the 1C line segment as the input line segments received by the input unit 31, and stores them in the memory unit 5 (S6, extraction process of the corresponding line segment processing process).

[0091] Next, the river water flow measuring device 1000 uses the first height, first direction and first focal length in equation 4 to determine the lengths of line segment 1A, line segment 1B and line segment 1C, thereby determining the lengths of line segment 2A, line segment 2B and line segment 2C, and generates an equation for the camera parameters using the second direction determined by the direction processing unit 22 in process S3 and the lengths of line segment 2A, line segment 2B and line segment 2C determined in equation 4, and determines the second height and second focal length by solving the generated equation for the camera parameters using the least squares method, and stores them in the memory unit 5 (S7, camera parameter processing step).

[0092] Next, the river water flow measuring device 1000 captures a plurality of time-series consecutive surface images by capturing images of the surface of the river with the second camera 11-2 of the tablet, and stores them in the memory unit 5 (S8, surface image capturing step).

[0093] Next, the river water flow measuring device 1000 orthorectifies each of the multiple chronologically consecutive surface images using the orthorectification unit 25 of the tablet based on the camera parameters of the second camera determined by the camera parameter processing unit 24 to generate multiple corrected surface images, and stores them in the memory unit 5 (S9, orthorectification process).

[0094] Next, the river water flow measuring device 1000 calculates the surface flow velocity at each inspection line on the surface of the river based on the multiple corrected surface images generated by the ortho-correction unit 25 using the flow velocity processing unit 26 of the tablet, and stores it in the memory unit 5 (S10, flow velocity processing process).

[0095] Next, the river water flow measuring device 1000 calculates the flow rate of the river using the flow rate processing unit 27 of the tablet based on the surface flow velocity at each inspection line calculated by the flow velocity processing unit 26, the cross-sectional shape of the river, and the water level of the river, and stores the flow rate in the memory unit 5 (S11, flow rate processing process).

[0096] Then, the river water flow measuring device 1000 displays on the display unit 32 of the tablet the camera parameters of the second camera calculated by the camera parameter processing unit 24, the surface flow velocities at each inspection line calculated by the flow velocity processing unit 26, and the flow rate of the river calculated by the flow rate processing unit 27, and ends this process (S12). Note that the river water flow measuring device 1000 may output these values ​​to an external device via the IF unit 4 as necessary.

[0097] The position of an actual point corresponding to a point on an image can be obtained from the position of the point on the image based on the direction of the optical axis, the height, and the focal length, when the deviation of the principal point position and the aberration in the imaging optical system of the camera are not taken into consideration. The camera parameter calculation device provided in the river water flow measuring device 1000, and the camera parameter calculation method and the camera parameter calculation program implemented therein can obtain the second direction of the second optical axis in the second camera 1-2 by correcting the first direction of the first optical axis in the first camera 11-1 based on the parallax between the first and second images. The camera parameter calculation device, the camera parameter calculation method, and the camera parameter calculation program can obtain the actual lengths of the first line segments from the positions of the respective end points of the at least two sets of first line segments, thereby obtaining the actual lengths of the second line segments in the at least two sets corresponding to the first line segments, and can obtain the height and the focal length in the second camera 11-2 by inverse calculation of the calculation method for obtaining the actual lengths from the positions of the respective end points. Therefore, the camera parameter calculation device, camera parameter calculation method, and camera parameter calculation program can reduce restrictions on the imaging target without using, for example, a ground point, a building, or a shelf when calculating the camera parameters. The second direction of the second optical axis in the camera parameters can also be calculated together with the second height and the second focal length, but the camera parameter calculation device, camera parameter calculation method, and camera parameter calculation program calculate the second direction of the second optical axis by correcting the first direction of the first optical axis, and then calculate the second height and the second focal length in the camera parameters. This reduces local solution outliers, and allows the second height and the second focal length to be stably calculated as realistic values.

[0098] The camera parameter calculation device, the camera parameter calculation method, and the camera parameter calculation program include the corresponding points search section 221, so that corresponding points can be automatically found by the corresponding points search.

[0099] The above camera parameter calculation device, camera parameter calculation method, and camera parameter calculation program are equipped with an extraction unit 23 (33), so that if an input line segment is input as one of the first and second line segments, the extraction unit 23 (33) extracts a corresponding line segment as the other of the first and second line segments, thereby eliminating the need to input the first and second line segments.

[0100] The camera parameters of the second camera calculated by the camera parameter processing unit 24 are two, namely, the second height of the second camera 11-2 and the second focal length of the second imaging optical system, and therefore it is possible to calculate the camera parameters of the second camera using two pairs of the first and second line segments. The camera parameter calculation device, camera parameter calculation method, and camera parameter calculation program calculate the camera parameters of the second camera based on three or more pairs of the first and second line segments having redundancy for two unknowns, and therefore it is possible to more appropriately calculate the camera parameters of the second camera.

[0101] According to the embodiment, a recording medium having a camera parameter calculation program recorded thereon can be provided. According to the embodiment, a river water flow measuring device 1000 having a camera parameter calculation device, a camera parameter calculation method, and a camera parameter calculation program, respectively, a river water flow measuring method, and a river water flow measuring program, respectively, can be provided, and a recording medium having a river water flow measuring program recorded thereon can be provided.

[0102] The above-mentioned river water flow measuring device 1000, river water flow measuring method, and river water flow measuring program can obtain the camera parameters required for orthorectification without using, for example, a control point, a building, a shelf, etc., so that the restrictions on the imaged object can be further reduced, and the surface flow velocity of the river can be measured more quickly when measurement is attempted. In particular, the surface flow velocity of the river can be measured even from a place on the river where six or more control points cannot be arranged.

[0103] The above-mentioned river water flow measuring device 1000, river water flow measuring method and river water flow measuring program set at least two sets of first and second line segments into three sets: the near bank, the far bank and the river width, so that the surface flow velocity of the river can be measured with balanced accuracy from the front to the back of the river along the river width direction.

[0104] The above-mentioned river water flow measuring device 1000, river water flow measuring method and river water flow measuring program perform flow rate processing, and therefore can measure the flow rate of a river.

[0105] The river water flow measuring device 1000 includes the first and second cameras 11-1, 11-2 and the direction measuring unit 12, so that actual measurements can be made at the measurement site using only the river water flow measuring device 1000.

[0106] In order to express the present invention, the present invention has been described adequately and sufficiently through the embodiments with reference to the drawings in the above description, but it should be recognized that those skilled in the art can easily change and / or improve the above-mentioned embodiments. Therefore, unless the changes or improvements made by those skilled in the art are at a level that departs from the scope of the claims described in the claims, the changes or improvements are interpreted as being included in the scope of the claims. [Explanation of symbols]

[0107] 1000 River flow measurement device equipped with a camera parameter calculation device, 1 Image acquisition unit, 2 Control processing unit, 3 Corresponding line processing unit, 4 IF unit, 5 Memory unit, 11-1 First camera, 11-2 Second camera, 12 Direction measurement unit, 21 Control unit, 22 Direction processing unit, 23 (33), Extraction unit, 24 Camera parameter processing unit, 25 Ortho-correction unit, 26 Flow velocity processing unit, 27 Flow rate processing unit, 31 Input unit, 32 Display unit, 51 River information storage unit

Claims

1. a direction processing unit that calculates a parallax between a first image captured by a first camera, the first height from a predetermined reference plane, a first direction of a first optical axis, and a first focal length of a first imaging optical system, and a second image captured by a second camera different from the first camera so as to overlap at least a part of the subject of the first image, based on a first and second image area in which the subject overlaps, and calculates a second direction of the second optical axis of the second camera based on the first direction of the first optical axis and the calculated parallax; a corresponding line segment processing unit that sets at least two pairs of first and second line segments that correspond to each other from the first and second images in the first and second image regions, respectively; a camera parameter processing unit that calculates camera parameters of the second camera based on the first height, a first direction of the first optical axis, a first focal length of the first imaging optical system, a second direction of the second optical axis calculated by the direction processing unit, and the at least two pairs of first and second line segments set by the corresponding line segment processing unit. Camera parameter calculation device.

2. The direction processing unit includes: a corresponding point search unit that searches for corresponding points between the first and second images in the first and second image regions; a disparity calculation unit that calculates the disparity based on each corresponding point searched for by the corresponding point search unit; a direction calculation unit that calculates a second direction of a second optical axis in the second camera based on a first direction of the first optical axis and the parallax calculated by the parallax calculation unit, or an input unit that receives an input of corresponding points in the first and second images, the corresponding points being in the first and second image regions; a disparity calculation unit that calculates the disparity based on each corresponding point received by the input unit; a direction calculation unit that calculates a second direction of a second optical axis in the second camera based on a first direction of the first optical axis and the parallax calculated by the parallax calculation unit, 2. The camera parameter calculation device according to claim 1.

3. The corresponding line segment processing unit a display unit that displays at least one of the first and second images; an input unit that receives an input of an input line segment as one of the first and second line segments in one of the first and second images; an extraction unit that extracts, from the other of the first and second images, a corresponding line segment that corresponds to the input line segment received by the input unit; or a display unit that displays the first and second images; an input unit that receives an input of the first and second line segments; 2. The camera parameter calculation device according to claim 1.

4. the at least two pairs of first and second line segments are three or more pairs of first and second line segments; the camera parameters of the second camera calculated by the camera parameter processing unit are a second height of the second camera from the reference plane and a second focal length of a second imaging optical system; the camera parameter processing unit determines camera parameters of the second camera by using an arithmetic expression for determining a length of a line segment based on the second height and the second focal length so that an error in each length of the at least two pairs of the first and second line segments is minimized; 2. The camera parameter calculation device according to claim 1.

5. a direction processing step of calculating a parallax between a first image captured by a first camera, which knows a first height from a predetermined reference plane, a first direction of a first optical axis, and a first focal length of a first imaging optical system, and a second image captured by a second camera different from the first camera, such that the second image overlaps with a subject in the first image in at least a portion of the first image, based on a first and second image region in which the subject overlaps, and calculating a second direction of a second optical axis in the second camera based on the first direction of the first optical axis and the calculated parallax; a corresponding line segment processing step of setting at least two pairs of first and second line segments corresponding to each other from the first and second images, respectively, in the first and second image regions; a camera parameter processing step of determining camera parameters of the second camera based on the first height, a first direction of the first optical axis, a first focal length of the first imaging optical system, a second direction of the second optical axis determined in the direction processing step, and the at least two pairs of first and second line segments set in the corresponding line segment processing step. Camera parameter calculation method.

6. A camera parameter calculation program for causing a computer to function as the camera parameter calculation device according to any one of claims 1 to 4.

7. 7. A computer-readable recording medium on which the camera parameter calculation program according to claim 6 is recorded.

8. A camera parameter calculation device according to any one of claims 1 to 4, an orthorectification unit that orthorectifies each of a plurality of time-series consecutive surface images of the surface of the river captured by the second camera based on the camera parameters of the second camera calculated by the camera parameter calculation device to generate a plurality of corrected surface images; A flow velocity processing unit that calculates a surface flow velocity on the surface of the river based on a plurality of corrected surface images generated by the orthorectification unit. River water flow measuring device.

9. The at least two sets of first and second line segments are three or more sets of first and second line segments, including a set of first A and second A line segments set on the near bank of the river relative to the second camera, a set of first B and second B line segments set on the far bank of the river relative to the second camera, and a set of first C and second C line segments set along the width direction of the river. The river water flow measuring device according to claim 8.

10. a flow rate processing unit that calculates a flow rate of the river based on the surface flow rate calculated by the flow rate processing unit, a cross-sectional shape of the river, and a water level of the river; The river water flow measuring device according to claim 8.

11. the first and second cameras; a direction measurement unit that measures the first direction of the first optical axis of the first camera, The river water flow measuring device according to claim 8.

12. A camera parameter calculation method according to claim 5; an orthorectification step of orthorectifying each of a plurality of time-series consecutive surface images of the surface of the river captured by the second camera based on the camera parameters of the second camera calculated by the camera parameter calculation method to generate a plurality of corrected surface images; and a flow velocity processing step of calculating a surface flow velocity on the surface of the river based on a plurality of corrected surface images generated in the orthorectification step. River water flow measurement method.

13. A river water flow measuring program for causing a computer to function as the river water flow measuring device according to claim 8.

14. A computer-readable recording medium having recorded thereon the river water flow measurement program according to claim 13.

Images