Cloud measurement system and cloud height measurement method

Abstract

To measure a cloud height with high accuracy for a long time.SOLUTION: A cloud measurement device includes: a stereo camera 30 for photographing a stereo image by a plurality of cameras that can change a pan angle and a tilt angle; a camera control unit 210 for controlling the pan angle and the tilt angle of each camera to control photographing; an image reception unit 204 for receiving the stereo image from the stereo camera; a matching coordinate acquisition unit for acquiring feature points of a subject captured in the stereo image, and acquiring a combination of reference coordinates at which the feature points should be located on the image and coordinates of the acquired feature points as matching coordinates; a direction difference detection unit 209 for acquiring a displacement amount of the feature points on the basis of the matching coordinates, and acquiring a difference amount of pan directions and tilt directions of the plurality of cameras to generate correction information; a correction processing unit 201 for correcting the stereo image on the basis of the correction information; and a cloud base height estimation unit 203 for estimating a cloud base height on the basis of the corrected stereo image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a cloud measurement system for measuring the state of clouds in the sky and a cloud height measurement method. [Background technology]

[0002] When controlling aircraft takeoffs and landings at airports, it is necessary to have constantly changing weather information around the airport. METAR (Regular Airport Weather Report) is a type of weather reporting system for reporting aviation weather information, and is used to understand the weather conditions at airports, air bases, etc.

[0003] METAR includes information on various observation items such as wind direction, wind speed, visibility, weather, and cloud height. Of these observation items, cloud height measurement is based on the observer's experience, and automation has not progressed much. Conventional cloud measurement systems include, for example, laser-based ceilometers. Ceilometers can measure cloud height directly above, but their horizontal measurement range is narrow, making it impossible to measure cloud height over the entire sky.

[0004] As a technology related to such cloud height measurement, for example, Patent Document 1 discloses measuring cloud height over a wide range using optical images taken by a stereo camera with a pair of wide-angle cameras facing the zenith. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2019-60754 Summary of the Invention [Problem to be solved by the invention]

[0006] Patent Document 1 describes a method for correcting camera lens distortion by calibrating a camera using stars visible in an image captured at night on a clear day. However, the technology described in Patent Document 1 uses stars for correction, so correction can only be performed at night on a clear day. Furthermore, in Patent Document 1, the camera is fixed facing the zenith, and Patent Document 1 does not take into consideration deviations in the camera orientation that occur due to aging or the like when using a camera whose shooting direction can be changed using a movable platform.

[0007] The present invention has been made in view of the above points, and has as its object to make it possible to measure cloud height with high accuracy over a long period of time. [Means for solving the problem]

[0008] In order to achieve the above object, in a preferred embodiment, the cloud measurement system of the present invention includes a stereo camera having multiple cameras, each of which has adjustable pan and tilt angles, that captures stereo images including multiple images with parallax between them; a camera control unit that controls the capture of the stereo images by controlling the pan and tilt angles of the multiple cameras; a receiving unit that receives the stereo images from the stereo camera; a matching coordinate acquisition unit that acquires feature points of the subject captured in the stereo images and acquires, as matching coordinates, combinations of reference coordinates where the feature points should be located on the stereo images and the coordinates of the feature points captured on the stereo images; a difference detection unit that acquires the amount of displacement of the feature points on the stereo images based on the matching coordinates and, based on the amount of displacement, acquires the amount of difference in orientation in the pan and tilt directions for at least one of the multiple cameras to generate correction information; a correction processing unit that corrects the stereo images based on the correction information; and an altitude estimation unit that estimates the height of the cloud base of the clouds captured in the stereo images based on the stereo images corrected by the correction processing unit. [Effects of the Invention]

[0009] According to the present invention, it is possible to measure cloud height over the entire sky with high accuracy. Other novel features of the present invention and the technical problems solved thereby will become apparent from the description and drawings of this specification. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a schematic diagram showing the configuration of a cloud measurement system according to an embodiment of the present invention. [Figure 2] FIG. 1 is a block diagram showing a simplified configuration of a cloud height measurement device. [Figure 3] FIG. 2 is a conceptual diagram showing weather information stored in a weather information storage unit. [Figure 4] 10 is a flowchart illustrating an example of cloud height measurement processing. [Figure 5] FIG. 4 is a data configuration diagram showing an example of a shooting order table. [Figure 6] FIG. 4 is a data configuration diagram showing an example of a correction table. [Figure 7] 10 is a flowchart showing the flow of a photographing process. [Figure 8] 10 is a flowchart showing the flow of a subject detection process. [Figure 9] FIG. 10 is a schematic diagram for explaining an example of a fixed object table. [Figure 10] FIG. 10 is a schematic diagram for explaining an outline of a matching process. [Figure 11] 10 is a flowchart showing the flow of cloud shape recognition processing. [Figure 12] 10 is a flowchart showing the flow of an orientation deviation calculation process. [Figure 13] 10 is a flowchart showing the flow of a correction process. [Figure 14] 10 is a flowchart showing the flow of cloud height estimation processing. [Figure 15] This is a conceptual diagram showing an example of a METAR, which is a type of weather report format. DETAILED DESCRIPTION OF THE INVENTION

[0011] An embodiment of the present invention will be described below with reference to the drawings. Please note that the embodiment described below is an example for explaining the present invention, and that some details have been omitted or simplified as appropriate for clarity of explanation.

[0012] FIG. 1 is a schematic diagram showing the configuration of a cloud measurement system 10 according to one embodiment of the present invention.

[0013] The cloud measurement system 10 includes a cloud height measurement device 20, a stereo camera 30, a visible light camera 31, a ceilometer 32, and an external device 40. The cloud height measurement device 20 is connected to the stereo camera 30, the visible light camera 31, the ceilometer 32, and the external device 40 via a network 50. The network 50 is, for example, a two-way communication network such as the Internet. The stereo camera 30, the visible light camera 31, and the ceilometer 32 are placed at a measurement point such as an airport. Note that the cloud height measurement device 20 and the stereo camera 30, the visible light camera 31, and the ceilometer 32 may be directly connected using, for example, a dedicated cable without using the network 50.

[0014] The cloud height measurement device 20 calculates the height of clouds in the sky (cloud height) based on two images (stereo images) taken of the sky by the stereo camera 30 at the same time, and generates a weather report formula such as METAR that includes the cloud height.

[0015] The stereo camera 30 is configured with two cameras positioned with a predetermined baseline length and captures two images (stereo images) with parallax. The stereo camera 30 has, for example, a pan-tilt head and a zoom lens that allows optical zooming. The stereo camera 30 may use a visible light camera or a near-infrared camera. Using a near-infrared camera enables imaging of the sky even at night. Using a visible light camera enables capturing high-resolution color and brightness images. The stereo camera 30 adds the shooting time as attribute information to the stereo images of the sky and transmits them to the cloud height measurement device 20 via the network 50. The stereo images are used to calculate the distance to the subject, i.e., the cloud, based on the parallax when estimating the cloud base height. The stereo camera 30 may also be configured by distributing three or more cameras and appropriately combining two of them. In this case, multiple stereo images with different baseline lengths can be obtained.

[0016] The visible light camera 31 captures the same imaging range as the stereo camera 30, adds the shooting time as attribute information to the obtained visible light image, and transmits it to the cloud height measurement device 20 via the network 50. Note that if a visible light camera is used for the stereo camera 30, the visible light camera 31 may be omitted and one of the stereo images may be used as the visible light image.

[0017] The ceilometer 32 measures the altitude (cloud base altitude) of clouds directly above, adds the measurement time as attribute information, and transmits the information to the cloud height measurement device 20 via the network 50. When multiple layers of clouds exist directly above, the ceilometer 32 can measure the cloud base altitude of each layer.

[0018] The external device 40 includes, for example, an anemometer, a visibility measuring device, a meteorological satellite, etc., and outputs wind direction / wind speed information and visibility information.

[0019] The cloud height measurement device 20 includes a correction processing unit 201, a stereo vision ranging unit 202, a cloud base altitude estimation unit 203, an image receiving unit 204, a communication unit 205, an object recognition unit 207, a cloud shape recognition unit 208, a direction difference detection unit 209, a camera control unit 210, a weather information storage unit 211, and a correction information storage unit 212.

[0020] The stereo vision distance measurement unit 202 detects parallax in the stereo images using a method such as block matching or feature matching, and calculates the distance to the cloud.

[0021] The cloud base height estimation unit 203 calculates the cloud height based on the distance to the cloud obtained by the stereo visual ranging unit 202. The calculated cloud height is corrected based on the cloud base height directly above the ceilometer 32, measured by the ceilometer 32. The cloud base height estimation unit 203 also acquires information such as wind direction, wind speed, and visibility provided by the external device 40 via the network 50, and stores the information in the weather information storage unit 211. Furthermore, the cloud base height estimation unit 203 generates a weather report format such as METAR based on the corrected cloud height and the wind direction, wind speed, visibility, and the like acquired from the external device 40. Note that in addition to METAR, other weather report formats may be generated, such as SPECI (Special Airport Actual Weather Report), TAF (Operational Airport Forecast Weather Report), TREND (Landing Airport Forecast Weather Report), VOLMET (Operational Airport Forecast Weather Report for Volmet Broadcasting), SCAN (Aviation Weather Observatory Actual Weather Report), and the like.

[0022] The image receiving unit 204 acquires stereo images from the stereo camera 30 and visible light images from the visible light camera 31 via the communication unit 205 , and stores these acquired images in the weather information holding unit 211 .

[0023] The communication unit 205 connects to the stereo camera 30 and the visible light camera 31 via the network 50, transmits control information for controlling them, and receives image information from these cameras. The communication unit 205 also communicates with the ceilometer 32 and the external device 40 via the network 50, and receives information provided by these devices.

[0024] When the image contains a reference subject, the subject recognition unit 207 performs a feature point matching process between the photographic data of the reference subject stored in the subject information storage unit 213 and the subject in the received image, and obtains the amount of displacement of the subject on the image in pixel units.

[0025] The cloud shape recognition unit 208 performs cloud shape recognition processing to identify high-level clouds captured in the stereo image, and performs feature point matching processing on the area where the high-level clouds exist to obtain the matched coordinates.

[0026] In this embodiment, the correction amount is acquired based on the feature point coordinates obtained by the matching process performed in either the object recognition unit 207 or the cloud shape recognition unit 208. Therefore, these can be collectively considered as a matching coordinate acquisition unit.

[0027] The orientation difference detection unit 209 performs orientation difference detection processing to obtain the amount of displacement of a fixed object, celestial body, or high clouds, and if the value differs from the value in the correction table stored in the correction information storage unit 212, updates the value and updates the correction table and the image used to calculate the amount of displacement stored in the correction information storage unit 212. If the amount of displacement of the fixed object, celestial body, or high clouds is greater than a predetermined threshold, the orientation difference detection unit 209 updates the pan-tilt angle in the shooting order table stored in the correction information storage unit 212 to a pan-tilt angle that takes into account the amount of correction calculated from the amount of displacement, stores the updated shooting order table in the correction information storage unit 212, and outputs the result of whether or not correction is necessary as "correction required."

[0028] The camera control unit 210 periodically controls the stereo camera 30 by instructing it on the pan / tilt angle and zoom magnification via the communication unit 205 in accordance with the information in the shooting order table, and performs shooting. Similarly, the camera control unit 210 also performs shooting with the stereo camera 30 when the output of the direction difference detection unit 209 is "correction required." The weather information storage unit 211 stores the stereo images and visible light images received by the image receiving unit 204, the information acquired from the external device 40 by the cloud base height estimation unit 203, the results of cloud height estimation by the cloud base height estimation unit 203, and the generated weather report formula.

[0029] The correction information storage unit 212 stores a shooting order table in which the order of shooting is set, a correction table in which correction information is set, and images used when updating the correction table.

[0030] The object information storage unit 213 stores information about fixed objects and celestial bodies that is used to update the correction table.

[0031] The UI unit 214 is an interface unit that presents various types of information to the user and accepts various operations by the user.

[0032] The internal clock 215 connects to an NTP (Network Time Protocol) server (not shown) via the communication unit 205 and the network 50, synchronizes its own time information, and supplies the time information to each part of the cloud height measurement device 20, the stereo camera 30, the visible light camera 31, the ceilometer 32, etc. Note that instead of using an NTP server to synchronize the time information, a satellite positioning system such as a GPS (Global Positioning System) may be used.

[0033] FIG. 2 is a block diagram showing a simplified configuration of the cloud height measurement device 20.

[0034] The ceiling height measuring device 20 includes a processor 101, a memory 102, a storage 103, an input device 104, an output device 105, and a communication module 106.

[0035] The processor 101 is configured with an arithmetic device such as a CPU (Central Processing Unit) and / or a GPU (Graphics Processing Unit). The functions of the above-mentioned parts of the cloud height measurement device 20 are realized by the processor 101 executing a program stored in the memory 102 while using storage resources (memory 102 and storage 103), a communication module 106, etc.

[0036] The memory 102 is composed of storage elements such as DRAM (Dynamic Random Access Memory) and is used to store programs that realize the various functions of the cloud height measurement device 20, as well as various data used by the processor 101 when executing the programs.

[0037] The storage 103 is a non-volatile storage device such as a hard disk drive (HDD) or a solid state drive (SSD), and serves as a weather information holding unit 211 , a correction information holding unit 212 , and a subject information holding unit 213 .

[0038] The input device 104 is a device such as a keyboard, mouse, or touch panel through which a user inputs information and performs various operations. The output device 105 is a device such as a display or printer for presenting various types of information to a user. The input device 104 and the output device 105 function as a UI unit 214.

[0039] The communication module 106 is an interface such as a network interface card (NIC) for communicating with each device connected to the network 50 via the network 50. The communication module 106 serves as a communication unit 205.

[0040] A general computer such as a personal computer or a server computer can be used as the cloud height measurement device 20, but it may also be configured as a dedicated device or as a system having multiple computers or devices. Furthermore, to realize some or all of the functions of each of the above-mentioned parts, it may also be configured to include dedicated circuits such as an FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit), or CPLD (Complex Programmable Logic Device).

[0041] The program may be installed in storage 103 from a program source and read into memory 102 for execution by processor 101. The program source may be, for example, a storage medium readable by a program distribution server or cloud height measurement device 20. When the program source is a program distribution server, the program distribution server may include a processor and storage resources for storing the program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. Furthermore, in this embodiment, all of the functions of each of the above-mentioned units may be realized as a single program, or each function of each unit may be realized as one or more programs.

[0042] FIG. 3 is a conceptual diagram showing the weather information stored in the weather information storage unit 211. As shown in FIG.

[0043] The image information 2111 includes stereo images with parallax captured by the stereo camera 30 at the same time, and visible light images captured by the visible light camera 31 at the same time as the stereo camera 30. The wind direction and speed information 2112 is information indicating the wind direction and wind speed at the measurement point, obtained from the external device 40. The visibility information 2113 is information indicating the visibility at the measurement point, obtained from the external device 40. The cloud base height information 2115 is information indicating the cloud base height (the lowest height of the cloud) directly above it, measured by the ceilometer 32. The weather report formula 2116 is a METAR generated by the cloud base height estimation unit 203.

[0044] 4 is a flowchart showing an example of cloud height measurement processing in this embodiment. Note that, as a premise of the cloud height measurement processing, it is assumed that the cloud base altitude estimation unit 203 periodically acquires wind direction and speed information 2112, visibility information 2113, cloud top altitude information 224, and cloud base altitude information 2115 from the ceilometer 32 and external device 40 and stores them in the weather information storage unit 211. Furthermore, this processing and each of the processing described below are specifically realized by the processor 101 executing a program stored in the memory 102.

[0045] The cloud height measurement process is started, for example, in response to a predetermined start operation from the user. When the process is started, the cloud height measurement device 20 causes the camera control unit 210 to perform initial settings of the stereo camera 30 and the visible light camera 31 via the network 50. During the initial settings, the camera control unit 210 sends time information indicated by the internal clock 215 to the stereo camera 30 and the visible light camera 31, and synchronizes the shooting time added to the captured image with the time information of the internal clock 215. The camera control unit 210 also performs calibration between the two cameras that make up the stereo camera 30. In addition, in preparation for subsequent shooting, the shooting order table is read out from the correction information storage unit 212 (step S401).

[0046] 5 is a data structure diagram showing an example of the structure of a shooting order table. In this embodiment, data with a table structure is used, but shooting order information may be stored in other data formats such as a list structure. This also applies to the correction table described below.

[0047] The shooting order table 500 includes a first table 501 used to control one of the cameras constituting the stereo camera 30 and a second table 502 used to control the other camera. In this embodiment, imaging is performed multiple times with different combinations of pan angle, tilt angle, and zoom magnification. In the shooting order table 500, the shooting order is registered in each table for each combination of pan angle, tilt angle, and zoom magnification used to perform imaging for each camera constituting the stereo camera 30.

[0048] The "Order" column 5001 contains ascending serial numbers indicating the order in which the images were taken. The "Pan" column 5002 contains information about the pan angle, which indicates the rotation angle around the vertical axis indicating the orientation of the pan head of the stereo camera 30, and is set as an angle between "0" and "359" degrees. A pan angle of 0 degrees is set to the same orientation for the multiple cameras that make up the stereo camera 30. For example, the orientation can be set based on the orientation of an axis perpendicular to the baseline and the horizontal direction, or the azimuth angle. Negative values ​​and values ​​greater than 360 degrees are converted to a value between 0 and 359 by adding or subtracting 360. The "Tilt" column 5003 contains information about the tilt angle, which indicates the angle in the elevation direction of the pan head of the stereo camera 30, and is listed as a zenith angle between "0" and "90" degrees. A zenith angle of 90 degrees means that the optical axis of the camera, i.e., the axis perpendicular to the sensor surface, is perpendicular to the vertical axis, and a zenith angle of 0 degrees means that it is parallel to the vertical axis and points straight up. In this embodiment, it is assumed that the pan angle and tilt angle have axes that are in the same direction among the multiple cameras that make up the stereo camera 30, and the information set in the shooting order table is also used as common information among the multiple cameras. The "Zoom" column 5004 sets the zoom magnification of the stereo camera 30. "1" indicates that the zoom magnification is 1x, i.e., shooting at a magnification that is the standard on the wide-angle side, and "2" indicates shooting at a zoom magnification that is twice that.

[0049] In this embodiment, the camera used as stereo camera 30 is capable of optical zooming, and at a zoom magnification of 1x, the viewing angle is 60 degrees in both the horizontal direction (pan direction) and the vertical direction (tilt direction), and at a zoom magnification of 2x, it is 30 degrees. In order to capture images using such a camera so that it can cover the entire sky without gaps, in this embodiment, the reference pan angle and tilt angle are changed by 60 degrees each. Specifically, after capturing an image of one viewing angle at a zoom magnification of 1x, the zoom magnification is increased to 2x, and the same field of view is divided into four and captured four times, with 1 + 4 = 5 images being one set.

[0050] When capturing an entire sky, with a zoom magnification of 1x, 6 directions (360 degrees divided by 60 degrees) are required for the pan direction, and 2 directions (90 degrees divided by 60 degrees) are required for the tilt direction. Therefore, combining these, 6 x 2 = 12 directions are required for capturing an entire sky. When capturing the entire field of view captured at the standard magnification with a zoom magnification of 2x, as mentioned above, four shots are required in each direction (field of view), resulting in 12 x 4 = 48 directions. However, in this case, the zenith angle exceeds 90°, which means that clouds will not be captured at a zoom magnification of 2x. Specifically, the combination of pan angle, tilt angle, and zoom magnification results in a zoom magnification of "2" and a tilt angle of "90 + 15" degrees. There are 12 such pairs in the shooting order table 500, and if the shooting of these pairs is excluded from the shooting order table 500 or the shooting is skipped, it becomes possible to cover the entire sky at both zoom magnifications of 1x and 2x with 60-12=48 shots. This shortens processing time and is also effective in saving storage capacity.

[0051] Here, for simplicity's sake, the zoom magnifications are 1x and 2x, but if, for example, a camera capable of optically zooming 10x with the same field of view is used and images are taken at both 1x and 10x zoom magnifications, then to cover the entire sky, 12 images would be required at 1x zoom, and 900 images at 10x zoom, for a total of 912 images. Note that the closer the camera is to the zenith, the greater the overlap in field of view angles, so by taking images at pan and tilt angles that minimize the overlap, the number of images taken can be reduced, shortening the time required for taking images and estimating cloud height.

[0052] Returning to Fig. 4, once the initial settings are complete, the cloud height measurement device 20 controls the stereo camera 30 and the visible light camera 31 by the camera control unit 210 to capture images of the entire sky in accordance with the shooting order table 500. The cloud height measurement device 20 acquires the stereo images and visible light images obtained as a result of the shooting by the image receiving unit 204, and stores them in the weather information storage unit 211 as image information 2111. The stereo images obtained as a result of the shooting are passed in order to the object recognition unit 207, cloud shape recognition unit 208, and direction difference detection unit 209, which transmit the stereo images, update the correction table stored in the correction information storage unit 212, and store the images used for correction (step S402).

[0053] FIG. 6 is a data configuration diagram showing an example of the configuration of the correction table.

[0054] Similar to the shooting order table 500, the correction table 600 also includes a first table 601 corresponding to one of the two cameras that make up the stereo camera 30 and a second table 602 corresponding to the other.

[0055] The "#" column 6010 contains ascending serial numbers indicating the order in which shooting will be performed. The "PTZ setting" column 6020 includes a "pan" column 6021, a "tilt" column 6022, and a "zoom" column 6023, which correspond to the "pan" column 5002, the "tilt" column 5003, and the "zoom" column 5004 in the shooting order table 500, and each column contains the pan angle, tilt angle, and zoom magnification.

[0056] The "Correction Amount" column 6030 includes a "Pan" column 6031, a "Tilt" column 6032, a "Rotation" column 6033, and a "Scaling Ratio" column 6034, and sets the amount of correction for an image captured with the combination of pan angle, tilt angle, and zoom magnification set in the corresponding "PTZ Setting" column 6020. The "Pan" column 6031 and "Tilt" column 6032 in the "Correction Amount" column 6030 set the correction angle in the X-axis direction and the correction angle in the Y-axis direction on the image, respectively. Similarly, the "Rotation" column 6033 sets the amount of correction indicating the correction angle in the rotation direction around the center of the image. Furthermore, the "Scaling Ratio" column 6034 sets the scaling ratio corresponding to the amount of deviation in zoom magnification due to, for example, deterioration over time of the zoom function of the stereo camera 30.

[0057] The "Fixed Object" column 6041 of the "Reference" column 6040 registers the reference fixed object used to calculate the correction amount set in the "Correction Angle" column 6030, and the "Infinity" column 6042 registers celestial bodies and high clouds. Also, the "Update History" column 6005 registers the date and time when the information in the row was updated.

[0058] In FIG. 6, an upward arrow in the table indicates that the value is the same as the value in the row above.

[0059] 4, the cloud height measurement device 20 uses the correction processing unit 201 to read the correction table 600 from the correction information storage unit 212 and the stereo images from the weather information storage unit 211, and performs correction processing according to the correction table 600. After the correction processing, the stereo images are stored in the weather information storage unit 211 and are also sent to the stereo vision distance measurement unit 202 (step S403).

[0060] The ceiling height measurement device 20 determines whether or not photographing has been completed for all the orders set in the photographing order table 500, and if there are still orders remaining in the photographing order table 500, photographing for the next order is carried out (step S404).

[0061] After completing the above-mentioned processing for all orders set in the shooting order table 500, the cloud height measurement device 20 performs the cloud height estimation processing described below using the stereo images taken in step S402 and stored in the weather information storage unit 211, or, if there are stereo images corrected in step S403, the corrected images, and generates a weather report formula 116 (step S405).

[0062] The cloud height measurement device 20 determines whether an unrecoverable abnormality has occurred in the processing up to step S404, which may cause a problem such as being unable to estimate cloud height in step S405 or not completing cloud height estimation within a predetermined time (step S406). If an abnormality has occurred, there is a possibility that a problem such as a misalignment in the imaging directions of the two cameras constituting the stereo camera 30 has occurred, and the cloud height measurement device 20 notifies the user of the abnormality via the UI unit 214 and terminates the cloud height measurement processing (step S407). If no such abnormality has occurred, the cloud height measurement device 20 waits until the next image capture timing, and at the next image capture timing, returns to the image capture processing of step S402 and repeats the above processing at a predetermined interval, for example, every 10 minutes (step S408).

[0063] The weather report formula 2116 generated by the above processing and stored in the weather information storage unit is provided to airports and the like as appropriate.

[0064] FIG. 7 is a flowchart showing the flow of the photographing process in step S402.

[0065] The camera control unit 210 controls the orientation and zoom magnification of the stereo camera 30 in accordance with the information set in the "pan" column 5002, the "tilt" column 5003, and the "zoom" column 5004 of the shooting order table 500 (step S701). Once the orientation and zoom magnification of the stereo camera 30 are set, the camera control unit 210 releases the shutters of the stereo camera 30 to capture stereo images. The captured stereo images are acquired by the image receiving unit 204 and the communication unit 205 via the network 50 and stored in the weather information storage unit 211 (step S702).

[0066] The cloud height measuring device 20 reads out the stereo images stored in the weather information storage unit 211 by the object recognition unit 207, and performs object detection processing described below to acquire coordinates of feature points of fixed objects and celestial bodies in the images (step S703). The cloud height measuring device 20 determines whether or not a fixed object or celestial body is included in the stereo images, i.e., whether or not a fixed object or celestial body has been detected in step S703 and the coordinates of the feature points have been acquired (step S704). If neither a fixed object nor a celestial body has been detected in step S703, the cloud height measuring device 20 performs cloud shape recognition processing described below by the cloud shape recognition unit 208 to detect high-level clouds and acquire the coordinates of their feature points (step S705). Thereafter, the cloud height measuring device 20 determines whether or not high-level clouds have been detected in step S705, and if high-level clouds have not been detected, ends the photographing process for this photographing order (step S706).

[0067] If the presence of a fixed object or celestial body is confirmed in step S704, or the presence of high clouds is confirmed in step S706, the cloud height measurement device 20 performs an orientation shift calculation process (described later) using the orientation difference detection unit 209, and calculates the amount of displacement based on the acquired feature point coordinates of the fixed object, celestial body, or high cloud. The amount of displacement calculated here includes the amount of translation, which is the amount of displacement in each of the pan and tilt directions, the amount of rotation, which is the amount of displacement in the rotation direction around the image center as an axis, and the amount of displacement such as the image enlargement / reduction ratio due to changes in zoom magnification, etc. (step S707).

[0068] The orientation difference detection unit 209 compares the translational amount of the obtained change amount with a predetermined threshold, and if the translational amount is equal to or less than the predetermined threshold, sets a correction amount to the value of the corresponding column in the "correction amount" column 6030 of the correction table 600 based on the obtained displacement amount (step S709). On the other hand, if the translational amount exceeds the threshold, the orientation difference detection unit 209 corrects the values ​​of the "pan" column 5002 and "tilt" column 5003 of the shooting sequence table 500, and the "pan" column 6021 and "tilt" column 6021 in the "PTZ setting" column 6020 of the correction table 600 by adding or subtracting the amount of translational movement obtained. If the displacement amount includes displacement of rotation and scaling ratio, the correction amount is set in the "Rotation" column 6033 and the "Scaling ratio" column 6034 of the "Correction amount" column 6030, and the values ​​of the "Pan" column 6031 and the "Tilt" column 6032 in the "Correction amount" column 6030 are reset to zero and updated. After this, the cloud height measurement device 20 returns to the processing of step S701, and the camera control unit 210 takes an image again using the new pan angle and tilt angle (S710). FIG. 8 is a flowchart showing the procedure of the subject detection process in step S703.

[0069] In the object detection process, the object recognition unit 207 acquires the pan angle, tilt angle, and zoom magnification set by the pan-tilt-zoom control in step S701 (steps S801 and S802) from information attached to the image read from the weather information storage unit 211. The object recognition unit 207 also reads out a fixed object table that stores information about fixed objects that serve as references from the object information storage unit 213 (step S803).

[0070] Fig. 9 is a schematic diagram for explaining an example of a fixed object table. Like the shooting order table 500 and the correction table 600, a fixed object table is provided for each camera constituting the stereo camera 30, but Fig. 9 shows only one of them.

[0071] Each row in the fixed object table 900 stores information about a reference fixed object. A “PTZ” column 901 in the fixed object table 900 stores the pan angle and tilt angle, which indicate the camera orientation when the fixed object was photographed, and the zoom magnification used for the photograph. A “Fixed Object” column 902 stores the type of fixed object photographed. A “Image” column 903 stores a path indicating the save destination of the image file of the photographed fixed object. The image file identified by this path is used when performing feature point matching with stereo images. Here, a PNG file is used as an example of an image file. A “Feature Point Coordinates” column 904 stores a path to a file that records the coordinates of the feature points of the fixed object captured in the image file identified by the “Image” column 903. The coordinates of the feature points registered in the file identified by this path are used as reference coordinates when obtaining the displacement of the feature points. Here, a CSV file is used as an example of a file that records feature point coordinates. A “Update History” column 905 stores the date and time when the information for the row was registered.

[0072] Images 910, 920, and 930 each represent an image recorded in an image file identified in each row of the "Image" column 903. Tables 940, 950, and 960 each represent feature point information recorded in a CSV file identified in each row of the "Feature Point Coordinates" column 904 in tabular form.

[0073] Image 910 depicts a lightning rod 911 as a fixed object, with three feature points indicated by arrows 912, 913, and 914. The coordinates of these three feature points are represented by X and Y coordinates on the image and are recorded in the "X" and "Y" columns of each row of table 940. Image 920 depicts a terrain (mountain) 921 as a fixed object, with the X and Y coordinate values ​​of one feature point 922 recorded in a CSV file as X and Y coordinate information as shown in table 950. Image 930 depicts a control tower 931 as a fixed object, with the X and Y coordinates of three feature points similarly recorded in a CSV file as shown in each row of table 960. These files are stored in the subject information storage unit 213 along with the fixed object table 900.

[0074] To improve the accuracy of the translation amount, rotation angle, and enlargement / reduction ratio obtained in the direction deviation calculation described below, it is desirable to obtain multiple feature points for one combination of pan, tilt, and zoom.

[0075] In addition to the information about the fixed objects described above, the object information storage unit 213 also stores information that can identify celestial objects that have a standard brightness rating. The position of a celestial object changes depending on the time and date. This can be addressed by preparing tables similar to the fixed object table 900 for each time period, or by changing the celestial object information used depending on the season. The coordinates of the celestial objects stored here can be obtained, for example, from information provided by an external database or by astronomical calculations.

[0076] 8, the object recognition unit 207 refers to the "PTZ" column 901 of the fixed object table 900 and determines whether a fixed object serving as a reference corresponding to the pan angle, tilt angle, and zoom magnification obtained in steps S801 and S802 is registered in the fixed object table 900 (step S804). If a fixed object is registered, the object recognition unit 207 performs feature point matching between the stereo image read from the weather information storage unit 211 and the image information stored in the object information storage unit 213.

[0077] FIG. 10 is a schematic diagram for explaining an overview of the matching process. Image 910 and table 940 are a table showing the image of lightning rod 911 as a reference fixed object registered in the first row of the fixed object table described in FIG. 9, and the contents of a file in which its feature points are recorded. Image 1000 is a stereo image (one of two images constituting a stereo image) of lightning rod 1001 read from meteorological information storage unit 211. Arrows 1002, 1003, and 1004 are feature points corresponding to the feature points indicated by arrows 912, 913, and 914 on image 910, respectively, detected by feature point matching. Table 1010 shows information on the X and Y coordinates of the feature points indicated by arrows 1002, 1003, and 1004. Comparing the coordinates registered in table 940 with the coordinates in table 1010 and calculating the difference reveals that the feature points of image 1000, i.e., the stereo image read from weather information storage unit 211, are translated 100 pixels in the positive direction of the X axis and 150 pixels in the negative direction of the Y axis from the feature points of image 910, i.e., the image stored in object information storage unit 213. Object recognition unit 207 acquires, as matching coordinates of the feature points, pairs of coordinates of the feature points obtained by the matching process and corresponding coordinates in the file indicated in "feature point coordinates" column 904 of fixed object table 900 (step S805). Then, object recognition unit 207 obtains a determination result that "fixed object included" (step S806).

[0078] On the other hand, if it is determined in step S804 that the stereo images do not include any registered fixed objects, the range of the angle of view captured in the stereo images is calculated in terms of declination and right ascension (step S807). Next, the object recognition unit 207 checks whether a celestial object with a reference brightness level exists within the calculated range of declination and right ascension during the time period indicated by the timestamp attached to the stereo images or the reception time of the stereo images (step S808). If such a celestial object exists, the object recognition unit 207 acquires information about the celestial object and proceeds to the processing of step S805, where it performs feature point matching in the same manner as in the case of fixed objects, and acquires matching coordinates, which are a combination of the coordinates of the position where the celestial object should originally be captured and the coordinates of the position (feature point) of the celestial object captured in the image. In this case, the determination result in step S806 is "celestial object included."

[0079] If it is determined in step S808 that no suitable celestial object exists, the object recognition unit 207 determines that there are no matching coordinates of the feature points (step S809), and obtains a determination result that "no celestial object is included" (step S810).

[0080] Finally, the object recognition unit 207 determines whether or not a fixed object or a celestial object is included, obtained through the above processing, and if the determination result indicates that a fixed object or a celestial object is included, outputs the matching coordinates of the feature points (step S811).

[0081] FIG. 11 is a flowchart showing the flow of the cloud shape recognition process in step S705.

[0082] If it is determined in step S704 that no fixed object or celestial body serving as a reference is stepped on in the stereo images, the cloud shape recognition unit 208 receives the stereo images from the object recognition unit 207 (step S1101) and performs cloud shape recognition on the received stereo images. For example, image recognition processing using a learning model that has been machine-learned in advance can be applied to the cloud shape recognition (step S1102).

[0083] The cloud shape recognition unit 208 determines whether high-level clouds (cirrus, cirrostratus, cirrocumulus) are included based on the results of the cloud shape recognition (step S1103). If it is determined that high-level clouds are included, the cloud shape recognition unit 208 performs feature point matching between the two images constituting the stereo image, using an image captured by one of the cameras constituting the stereo camera 30 as a reference, in a segmentation area on the image containing the high-level clouds. The feature point matching coordinates obtained here are acquired as a matching image for the image captured by the other camera (step S1304). Then, the cloud shape recognition unit 208 obtains a determination result that "high-level clouds are included" (step S1305).

[0084] If it is determined in step S1103 that high-level clouds are not included, the cloud shape recognition unit 208 determines that there are no matching coordinates of the feature points (step S1306) and obtains a determination result of "high-level clouds are not included" (step S1307). Finally, the cloud shape recognition unit 208 outputs the determination result of whether there are matching coordinates of the feature points, and if matching coordinates are obtained, outputs the coordinates (a set of coordinates of the feature points matched between the image images), and ends the process (step S1308).

[0085] 12 is a flowchart showing the flow of the orientation shift calculation process in step S707. The orientation difference detection unit 209 receives a determination result indicating whether a fixed object, a celestial body, or a high-level cloud is included from the object recognition unit 207 or the cloud shape recognition unit 208 (step S1201). The orientation difference detection unit 209 determines whether the received determination result indicates that a fixed object, a celestial body, or a high-level cloud is included (step S1202). If the determination result is positive, the orientation difference detection unit 209 uses, for example, ICP (Interactive Closest Point) or the least squares method to determine the translation amount, rotation amount, and scaling factor that will minimize the distance between the coordinates of the matched feature points from the matching coordinates of the feature points received along with the determination result. An affine transformation matrix representing the obtained translation amount, rotation amount, and scaling factor is then generated (step S1203), and converted into the X and Y translation, rotation angle, and scaling factor used in step S709 or S710 (step S1204). On the other hand, if it is determined in step S1202 that no fixed objects, celestial bodies, or high clouds are included, the orientation difference detection unit 209 sets the X and Y translations to 0, the rotation angle to 0 degrees, and the scaling factor to 1 (no scaling) (step S1205).

[0086] If the received determination result in step S1202 indicates that high clouds are included, then in steps S1203 and S1204, processing is performed on the image captured by the other camera in the processing of step S705 described in FIG. 11, and the X and Y translations of the image captured by one of the cameras are set to 0, the rotation angle to 0 degrees, and the scaling factor to 1 (no scaling).

[0087] FIG. 13 is a flowchart showing the flow of the correction process in step S403.

[0088] Correction processing unit 201 reads out the stored stereo images from weather information storage unit 211 (step S1301) and, for each of the two images constituting the stereo images, obtains the correction information in "correction amount" column 6030 corresponding to the shooting order of the stereo images, read out from correction table 600 in correction information storage unit 212 (step S1302). Correction processing unit 201 then generates an affine transformation matrix for each image constituting the stereo images based on the correction information obtained from correction table 600 (step S1303). Correction processing unit 201 transforms each image constituting the stereo images using the generated affine transformation matrix (step S1304) and stores the transformed images in weather information storage unit 211 (step S1305). Correction processing unit 201 then determines whether processing has been completed for all captured stereo images. If there are any unprocessed stereo images, the correction processing returns to step S1301 and continues the correction processing until processing has been completed for all images (step S1306).

[0089] FIG. 14 is a flowchart showing the flow of the cloud height estimation process in step S405.

[0090] In the cloud height estimation process, the stereo vision ranging unit 202 reads stereo images (or, if any, images corrected in the correction process of step S403) from the weather information storage unit 211 (step S1401). The stereo vision ranging unit 202 performs semantic segmentation on the read stereo images using a cloud shape recognition process based on ten cloud shapes to detect clouds. The cloud shape recognition process can be realized, for example, by applying image recognition processing using a learning model that has been machine-learned in advance (step S1402). Then, for each detected cloud shape region, the stereo vision ranging unit 202 performs disparity extraction processing using a block matching method or a feature matching method to extract disparity (step S1403), and obtains the distance to the cloud based on the extracted disparity (step S1404). The stereo vision ranging unit 202 determines whether this process has been performed on all stereo images. If there are unprocessed stereo images, the process returns to step S1401 and performs processing on the unprocessed stereo images (step S1405).

[0091] After the stereo vision ranging unit 202 completes processing of the stereo images, the cloud base altitude estimation unit 203 calculates the cloud height of the cloud from the acquired distance to the cloud and the orientation (tilt angle) of the stereo camera. The cloud base altitude estimation unit 203 further reads cloud base altitude information 2115 from the weather information storage unit 211 and corrects the calculated cloud height using the cloud height provided by the ceilometer. For example, for cloud A whose cloud height is measured by a ceilometer, if the cloud height acquired by the cloud height measurement device 20 differs from the cloud height obtained by the ceilometer, the cloud base altitude estimation unit 203 corrects the cloud height of cloud A to the cloud height obtained by the ceilometer. The cloud base altitude estimation unit 203 also corrects the cloud heights of clouds other than cloud A using the same ratio as the correction ratio for cloud A (step S1406). Finally, the cloud base height estimation unit 203 generates a weather report formula from the corrected cloud height and information such as wind direction and speed information 2112 and visibility information 2113 stored in the weather information storage unit 211 (step S1407).

[0092] If no clouds are detected in the processing of steps S1402 to S1404, the processing of steps S1403 to S1406 is omitted, and in step S1407, a weather report formula 2116 indicating that no clouds are present is generated.

[0093] Furthermore, if a cloud shape is recognized in step S1402 but the parallax cannot be extracted from the stereo image in step S1403, there is a possibility that there is a misalignment in the imaging directions of the two cameras that make up the stereo camera 30. In this case, the UI unit 214 notifies the user of the occurrence of an abnormality (step S408).

[0094] Figure 15 is a conceptual diagram showing an example of a METAR, a type of weather report format. As shown in the figure, when information 1500 about each item of weather described in normal sentences is converted into a weather report format 2116, the amount of text can be significantly reduced without reducing the amount of information.

[0095] According to the embodiment described above, by correcting to eliminate displacement of the feature point matching coordinates of fixed objects, celestial bodies, and high-level clouds, it is possible to correct the loss of stereo parallelism caused by secular changes in the multiple cameras that make up the stereo camera and the platform that determines their orientation, etc., and to maintain high-precision cloud height measurements. Furthermore, by performing correction using fixed objects and high-level clouds during the day and using fixed objects such as illuminated buildings and mountain shadows and celestial bodies at night, it is possible to perform correction regardless of the time of day or night.

[0096] In the embodiment described above, the shooting order table, correction table, fixed object information table, etc. are provided corresponding to each of the multiple cameras that make up the stereo camera, but the information corresponding to the multiple cameras may also be collected into a single table.

[0097] Furthermore, the correction amount is obtained by matching feature points using the coordinates of known fixed objects or celestial bodies for each camera constituting the stereo camera. However, feature point matching can also be performed by comparing images captured by the stereo cameras. In this case, in step S402, the cloud height measurement device uses an image captured by one of the cameras constituting the stereo camera as a reference and obtains the displacement amount of feature points on the image captured by the other camera from the displacement amount of parallax, thereby obtaining a correction amount for the image captured by the other camera. Then, in step S403, the cloud height measurement device generates a corrected image by performing affine transformation on the image captured by the other camera using the obtained correction amount. Furthermore, instead of updating the shooting order table when the translation amount exceeds a threshold, the correction amount for the pan angle and tilt angle of the other camera can be stored as correction information, and the orientation of one camera can be controlled by adding or subtracting this correction amount to the pan angle and tilt angle indicated in the shooting order table during shooting.

[0098] The above-described embodiments show typical aspects of the present invention, and the present invention is not limited thereto. Various modifications of the present invention are possible without departing from the spirit of the present invention. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those having all of the described configurations. [Explanation of symbols]

[0099] 10 Cloud measurement system, 20 Cloud height measurement device, 30 Stereo camera, 31 Visible light camera, 32 Ceilometer, 201 Correction processing unit, 202 Stereo visual ranging unit, 203 Cloud base altitude estimation unit, 204 Image receiving unit, 205 Communication unit, 207 Object recognition unit, 208 Cloud shape recognition unit, 209 Direction difference detection unit, 210 Camera control unit, 211 Weather information storage unit, 212 Correction information storage unit, 213 Object information storage unit, 214 UI unit, 215 Internal clock

Claims

1. a stereo camera having a plurality of cameras each capable of changing the angle of a pan direction and an angle of a tilt direction, and capturing a stereo image including a plurality of images having parallax with respect to each other; a camera control unit that controls the pan angle and tilt angle of the plurality of cameras to control the stereo camera from capturing the stereo image; a receiving unit that receives the stereo images from the stereo camera; a matching coordinate acquisition unit that acquires feature points of a subject captured in the stereo images, and acquires, as matching coordinates, a combination of reference coordinates where the feature points should be located on the stereo images and the coordinates of the feature points of the subject acquired on the stereo images; a difference detection unit that acquires a displacement amount of the feature point on the stereo image based on the matching coordinates, and acquires an orientation difference amount between the pan direction and the tilt direction for at least one of the plurality of cameras based on the displacement amount, thereby generating correction information; a correction processing unit that corrects the stereo images based on the correction information; and an altitude estimation unit that estimates the height of clouds captured in the stereo images based on the stereo images corrected by the correction processing unit.

2. The cloud measurement system according to claim 1, characterized in that the zoom magnification of each of the multiple cameras can be changed, the difference detection unit acquires a magnification ratio of the stereo image based on the matching coordinates, and the correction information includes the magnification ratio.

3. The cloud measurement system according to claim 1 further comprises: a subject information storage unit that stores subject information relating to a predetermined reference subject having the reference coordinates as a feature point; The cloud measurement system according to claim 1, characterized in that the matching coordinate acquisition unit has a subject detection unit that, when the reference subject is captured in the stereo images, acquires as the matching coordinates a combination of the reference coordinates obtained from the subject information and the coordinates of a feature point on the stereo images that corresponds to the reference coordinates.

4. The cloud measurement system of claim 3, wherein the matching coordinate acquisition unit further includes a cloud shape recognition unit that identifies high-level clouds captured in the stereo images, matches a first image captured by a first camera among the plurality of cameras as a reference image with a second image captured by another camera, and acquires a combination of corresponding feature points between the first image and the second image as the matching coordinates.

5. 5. The cloud measurement system according to claim 4, wherein the cloud shape recognition unit identifies the high-level clouds and acquires the matching coordinates when the reference subject cannot be detected by the subject detection unit.

6. The cloud measurement system of claim 3, characterized in that the information regarding the reference subject includes reference image information that serves as a reference for photographing the subject corresponding to each of the multiple cameras, and the subject detection unit performs a matching process between each of the multiple images included in the stereo image and the reference image information corresponding to each of the multiple images to obtain the matching coordinates.

7. The cloud measurement system according to claim 3, characterized in that the subject detection unit identifies a celestial object captured in the stereo image, acquires information about the identified celestial object, acquires the coordinates of the position where the celestial object should be captured in the stereo image as the reference coordinates, and acquires a combination of the reference coordinates and the coordinates of the point where the celestial object is captured in the stereo image as the matching coordinates.

8. When the orientation difference amount exceeds a predetermined threshold value, the difference detection unit corrects control amounts in the pan direction and tilt direction for a camera that captured the stereo image from which the orientation difference amount exceeding the threshold value was acquired, 2. The cloud measurement system according to claim 1, wherein the camera control unit controls the pan angle and tilt angle of the plurality of cameras in accordance with the corrected control amount.

9. The difference detection unit resets the correction information for a camera that captured a stereo image in which the orientation difference amount exceeding the threshold value is acquired when the control amount in the pan direction and the tilt direction is corrected for the camera. The cloud measurement system described in claim 8.

10. A cloud height measurement method in a cloud height measurement system, which measures cloud height using stereo images captured by a stereo camera having a lens whose zoom magnification can be optically changed and a plurality of cameras whose pan angle and tilt angle can be changed, and which captures stereo images including a plurality of images with parallax between them, comprising: controlling the pan angle, tilt angle, and zoom magnification of the plurality of cameras to acquire the stereo images captured by the stereo cameras; acquiring feature points of the subject captured in the stereo images, and acquiring, as matching coordinates, a combination of reference coordinates at which the feature points should be located on the stereo images and the coordinates of the feature points of the subject acquired on the stereo images; acquiring a displacement amount of the feature point on the stereo image based on the matching coordinates, acquiring a direction difference amount between the pan direction and the tilt direction for at least one of the plurality of cameras based on the displacement amount, and generating correction information; A cloud height measurement method that corrects the stereo images based on the correction information, and estimates the height of clouds captured in the stereo images based on the corrected stereo images.

11. The cloud height measurement method described in claim 10, characterized in that if, in generating the correction information, the orientation difference amount in either the pan direction or the tilt direction exceeds a predetermined threshold value, the pan direction angle and the tilt direction angle of the multiple cameras are corrected based on the orientation difference amount to obtain new stereo images, and the process of obtaining the stereo images and subsequent processes are repeated using the new stereo images.

12. The cloud height measuring method according to claim 10 , wherein the process of generating the correction information includes a process of obtaining a magnification ratio of the stereo image based on the matching coordinates, and the correction information includes the obtained magnification ratio.

13. The process of acquiring the matching coordinates includes matching the feature points between a reference image of the subject captured by each of the plurality of cameras and each of the plurality of images included in the stereo image, and acquiring the matching coordinates for each of the plurality of images; The cloud height measuring method according to claim 12 , wherein the process of generating the correction information generates the correction information for each of the plurality of cameras based on the matching coordinates acquired for each of the plurality of images.

14. The cloud height measuring method according to claim 13, wherein the subject includes at least one of a fixed object and a celestial body.

15. The cloud height measurement method according to claim 12, wherein the process of acquiring the matching coordinates includes identifying high clouds captured in the stereo images, matching a first image captured by a first camera among the plurality of cameras as a reference image with a second image captured by another camera, and acquiring a combination of corresponding feature points between the first image and the second image as the matching coordinates.

Images