Cloud measurement system and cloud height measurement method

Abstract

To highly accurately measure cloud heights of clouds in the whole sky.SOLUTION: A cloud measurement system comprises: a stereo camera 30 that can change zoom magnification in an imaging direction and optically; a picture reception unit 204 for acquiring the imaged picture; a cloud edge detection unit 207 that recognizes a cloud displayed on the picture, detects a boundary of the cloud, and acquires an edge region including the boundary from a plurality of sub regions in the picture; a cloud imaging order setting unit 209 that on the basis of the edge region, sets target sub regions to be imaged while changing the zoom magnification of the camera and their imaging order; a camera control unit 210 that controls the stereo camera 30 so as to perform enlargement imaging on the target sub regions according to the set imaging order to acquire a stereo picture for each target sub region, a stereo vision distance measurement unit 202 for acquiring a distance to a cloud displayed on the stereo picture using a parallax, and a cloud bottom height estimation unit 203 for acquiring a cloud bottom height of the cloud on the basis of the acquired distance.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a cloud measurement system and a cloud height measurement method. [Background technology]

[0002] When controlling aircraft takeoffs and landings at an airport, it is necessary to have constantly changing weather information around the airport. METAR (Regular Airport Weather Report) is a type of weather reporting system for reporting aviation weather information, and is used to understand the weather conditions at airports, air bases, etc.

[0003] METAR includes information on various observation items such as wind direction, wind speed, visibility, weather, and cloud height. Of these observation items, cloud height measurement is based on the observer's experience, and automation has not progressed much. Conventional cloud measurement systems include, for example, laser-based ceilometers. Ceilometers can measure cloud height directly above, but their horizontal measurement range is narrow, making it impossible to measure cloud height over the entire sky.

[0004] As a technology related to such cloud height measurement, for example, Patent Document 1 discloses measuring cloud height over a wide range using optical images taken by a stereo camera with a pair of wide-angle cameras facing the zenith. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2019-60754 Summary of the Invention [Problem to be solved by the invention]

[0006] The technology described in Patent Document 1 uses a stereo camera consisting of a wide-angle camera fixed toward the zenith, which can capture images over a wide viewing angle. However, the number of pixels in the image sensor used in the camera is limited. For this reason, if a wide area is captured at once using a wide-angle camera, the resolution of the resulting image will be low, making it difficult to expect high measurement accuracy.

[0007] The present invention has been made in view of the above points, and has as its object to provide a cloud height measuring device that can measure the cloud height of clouds throughout the entire sky with high accuracy. [Means for solving the problem]

[0008] In a preferred embodiment, the cloud measurement device of the present invention includes a plurality of cameras capable of changing the shooting direction and optically zooming magnification, an image receiving unit that acquires images captured by the plurality of cameras via a communication line, a cloud edge detection unit that recognizes clouds captured in the images acquired by the receiving unit, detects the boundaries of the clouds, and acquires sub-areas including boundaries as edge areas from among a plurality of sub-areas obtained by dividing the image into a plurality of areas that can be enlarged and photographed by changing the zoom magnification of the camera, a cloud shooting order setting unit that determines a sub-area to be photographed by changing the zoom magnification of the camera from among the plurality of sub-areas based on the edge area and sets the order in which the sub-areas to be photographed are photographed, a camera control unit that controls the plurality of cameras to photograph the shooting areas corresponding to the sub-areas to be photographed by changing the zoom magnification according to the shooting order set by the cloud shooting order setting unit, and acquires stereo images for each sub-area to be photographed by changing the zoom magnification, a stereo visual ranging unit that acquires the distance to the clouds captured in the stereo images based on the stereo images, and a cloud base altitude estimation unit that acquires the cloud base altitude of the clouds captured in the stereo images based on the acquired distance. [Effects of the Invention]

[0009] According to the present invention, it is possible to measure the cloud height of clouds throughout the sky with higher accuracy. Other novel features of the present invention and the technical problems solved thereby will become apparent from the description and drawings of this specification. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a schematic diagram showing the configuration of a cloud measurement system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram showing a simplified hardware configuration of the cloud height measurement device. [Figure 3] FIG. 2 is a conceptual diagram showing weather information stored in a weather information storage unit. [Figure 4] 10 is a flowchart illustrating an example of cloud height measurement processing. [Figure 5] FIG. 10 is a data configuration diagram showing an example of the configuration of a shooting order table. [Figure 6] 10 is a flowchart showing the details of wide-angle shooting processing. [Figure 7] 10 is a flowchart showing details of telephoto shooting processing. [Figure 8] 10 is a flowchart showing details of a cloud edge detection process. [Figure 9] FIG. 10 is a schematic diagram showing the state of cloud boundaries recognized by cloud shape recognition processing. [Figure 10] FIG. 10 is a schematic diagram showing the relationship between a cloud image and sub-regions. [Figure 11] FIG. 10 is a schematic diagram showing the relationship between cloud boundaries and sub-regions. [Figure 12] 10 is a flowchart showing details of a photographing order setting process. [Figure 13] FIG. 10 is a schematic diagram showing how a movement vector is obtained. [Figure 14] FIG. 10 is a schematic diagram for explaining pan-tilt operation time. [Figure 15] 10 is a flowchart showing details of a cloud height estimation process. [Figure 16] This is a conceptual diagram showing an example of a METAR, which is a type of weather report format. [Figure 17] 10 is a flowchart of a cloud edge detection process according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] An embodiment of the present invention will be described below with reference to the drawings. Please note that the embodiment described below is an example for explaining the present invention, and that some details have been omitted or simplified as appropriate for clarity of explanation.

[0012] FIG. 1 is a schematic diagram showing the configuration of a cloud measurement system 10 according to one embodiment of the present invention.

[0013] The cloud measurement system 10 includes a cloud height measurement device 20, a stereo camera 30, a visible light camera 31, a ceilometer 32, and an external device 40. The cloud height measurement device 20 is connected to the stereo camera 30, the visible light camera 31, the ceilometer 32, and the external device 40 via a network 50. The network 50 is, for example, a two-way communication network such as the Internet. The stereo camera 30, the visible light camera 31, and the ceilometer 32 are placed at a measurement point such as an airport. Note that the cloud height measurement device 20 and the stereo camera 30, the visible light camera 31, and the ceilometer 32 may be directly connected using, for example, a dedicated cable without using the network 50.

[0014] The cloud height measurement device 20 calculates the height of clouds in the sky (cloud height) based on two images (stereo images) taken of the sky by the stereo camera 30 at the same time, and generates a weather report formula such as METAR that includes the cloud height.

[0015] The stereo camera 30 is configured with two cameras arranged with a predetermined baseline length and captures two images (stereo images) with parallax. The stereo camera 30 has, for example, a pan-tilt head and a zoom lens that allows optical zooming. The stereo camera 30 may use a visible light camera or a near-infrared camera. Using a near-infrared camera enables imaging of the sky even at night. Using a visible light camera enables capturing high-resolution color and brightness images. The stereo camera 30 adds the shooting time as attribute information to the stereo images of the sky and transmits them to the cloud height measurement device 20 via the network 50. The stereo images are used to calculate the distance to the subject based on the parallax, i.e., to calculate the cloud height. The stereo camera 30 may also be configured by distributing three or more cameras and appropriately combining two of them. In this case, multiple stereo images with different baseline lengths can be obtained.

[0016] The visible light camera 31 captures the same imaging range as the stereo camera 30, adds the shooting time as attribute information to the obtained visible light image, and transmits it to the cloud height measurement device 20 via the network 50. Note that if a visible light camera is used for the stereo camera 30, the visible light camera 31 may be omitted and one of the stereo images may be used as the visible light image.

[0017] The ceilometer 32 measures the altitude (cloud base altitude) of clouds directly above, adds the measurement time as attribute information, and transmits the information to the cloud height measurement device 20 via the network 50. When multiple layers of clouds exist directly above, the ceilometer 32 can measure the cloud base altitude of each layer.

[0018] The external device 40 includes, for example, an anemometer, a visibility measuring device, a meteorological satellite, etc., and outputs wind direction / wind speed information and visibility information.

[0019] The cloud height measurement device 20 includes a stereo vision ranging unit 202, a cloud base altitude estimation unit 203, an image receiving unit 204, a communication unit 205, a cloud edge detection unit 207, a cloud movement estimation unit 208, a cloud photographing order setting unit 209, a camera control unit 210, a weather information storage unit 211, a cloud photographing order information storage unit 212, a UI unit 214, and an internal clock 215.

[0020] The stereo vision distance measurement unit 202 detects parallax in the stereo images using a method such as block matching or feature matching, and calculates the distance to the cloud.

[0021] The cloud base height estimation unit 203 calculates the cloud height based on the distance to the cloud obtained by the stereo visual ranging unit 202. The calculated cloud height is corrected based on the cloud base height directly above the ceilometer 32, measured by the ceilometer 32. The cloud base height estimation unit 203 also acquires information such as wind direction, wind speed, and visibility provided by the external device 40 via the network 50, and stores the information in the weather information storage unit 211. Furthermore, the cloud base height estimation unit 203 generates a weather report format such as METAR based on the corrected cloud height and the wind direction, wind speed, visibility, and other information acquired from the external device 40. Note that in addition to METAR, other weather report formats may be generated, such as SPECI (Special Airport Actual Weather Report), TAF (Operational Airport Forecast Weather Report), TREND (Landing Airport Forecast Weather Report), VOLMET (Operational Airport Forecast Weather Report for Volmet Broadcasting), SCAN (Aviation Weather Observatory Actual Weather Report), and the like.

[0022] The image receiving unit 204 acquires stereo images from the stereo camera 30 and visible light images from the visible light camera 31 via the communication unit 205 , and stores these acquired images in the weather information holding unit 211 .

[0023] The communication unit 205 connects to the stereo camera 30 and the visible light camera 31 via the network 50, transmits control information for controlling them, and receives image information from these cameras. The communication unit 205 also communicates with the ceilometer 32 and the external device 40 via the network 50, and receives information provided by these devices.

[0024] The cloud edge detection unit 207 receives the stereo images from the image receiving unit 204, performs cloud shape recognition processing to recognize the cloud shapes, and acquires areas (called edge areas) that include the edge portions of the clouds in the received images.

[0025] The cloud movement estimation unit 208 uses the time-series stereo images stored in the weather information storage unit 211 to extract characteristic points of the edge of the cloud recognized by the cloud edge detection unit 207 and obtains its movement vector.

[0026] The cloud photographing order setting unit 209 predicts the transition destination of the edge area acquired by the cloud edge detection unit 207 based on the movement vector acquired by the cloud movement estimation unit 208, and sets the photographing order of the predicted transition destination.

[0027] The camera control unit 210 controls the stereo camera 30 via the communication unit 205 and the network 50, and captures stereo images according to the shooting order table stored in the cloud shooting order information storage unit 212 and the shooting order set by the cloud shooting order setting unit 209. The meteorological information storage unit 211 stores the stereo images and visible light images received by the image receiving unit 204, information acquired from the external device 40 by the cloud base height estimation unit 203, the results of cloud height estimation by the cloud height estimation unit, and the generated weather report formula.

[0028] The cloud photographing order information storage unit 212 stores a photographing order table in which the photographing order of areas is set by the cloud photographing order setting unit 209 .

[0029] The UI unit 214 is an interface unit that presents various types of information to the user and accepts various operations by the user.

[0030] The internal clock 215 connects to an NTP (Network Time Protocol) server (not shown) via the communication unit 205 and the network 50, synchronizes its own time information, and supplies the time information to each part of the cloud height measurement device 20, the stereo camera 30, the visible light camera 31, the ceilometer 32, etc. Note that instead of using an NTP server to synchronize the time information, a satellite positioning system such as a GPS (Global Positioning System) may be used.

[0031] FIG. 2 is a block diagram showing a simplified hardware configuration of the cloud height measurement device 20. As shown in FIG.

[0032] The cloud height measurement device 20 includes a processor 101, a memory 102, a storage 103, an input device 104, an output device 105, and a communication module 106.

[0033] The processor 101 is configured with an arithmetic device such as a CPU (Central Processing Unit) and / or a GPU (Graphics Processing Unit). The functions of the above-mentioned parts of the cloud height measurement device 20 are realized by the processor 101 executing a program stored in the memory 102 while using storage resources (memory 102 and storage 103), a communication module 106, etc.

[0034] The memory 102 is composed of storage elements such as DRAM (Dynamic Random Access Memory) and is used to store programs that realize the various functions of the cloud height measurement device 20, as well as various data used by the processor 101 when executing the programs.

[0035] The storage 103 is a non-volatile storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and serves as a meteorological information storage unit 211 and a cloud photographing order information storage unit 212 .

[0036] The input device 104 is a device such as a keyboard, mouse, or touch panel through which a user inputs information and performs various operations. The output device 105 is a device such as a display or printer for presenting various types of information to a user. The input device 104 and the output device 105 function as a UI unit 214.

[0037] The communication module 106 is an interface such as a network interface card (NIC) for communicating with each device connected to the network 50 via the network 50. The communication module 106 serves as a communication unit 205.

[0038] A general computer such as a personal computer or a server computer can be used as the cloud height measurement device 20, but it may also be configured as a dedicated device or as a system having multiple computers or devices. Furthermore, to realize some or all of the functions of each of the above-mentioned parts, it may also be configured to include dedicated circuits such as an FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit), or CPLD (Complex Programmable Logic Device).

[0039] The program may be installed in storage 103 from a program source and read into memory 102 for execution by processor 101. The program source may be, for example, a storage medium readable by a program distribution server or cloud height measurement device 20. When the program source is a program distribution server, the program distribution server may include a processor and storage resources for storing the program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. Furthermore, in this embodiment, all of the functions of each of the above-mentioned units may be realized as a single program, or each function of each unit may be realized as one or more programs.

[0040] FIG. 3 is a conceptual diagram showing the weather information stored in the weather information storage unit 211. As shown in FIG.

[0041] The image information 2111 includes stereo images with parallax captured by the stereo camera 30 at the same time, and visible light images captured by the visible light camera 31 at the same time as the stereo camera 30. The wind direction and speed information 2112 is information indicating the wind direction and wind speed at the measurement point, obtained from the external device 40. The visibility information 2113 is information indicating the visibility at the measurement point, obtained from the external device 40. The cloud base height information 2115 is information indicating the cloud base height (the lowest height of the cloud) directly above it, measured by the ceilometer 32. The weather report formula 2116 is a METAR generated by the cloud base height estimation unit 203.

[0042] 4 is a flowchart illustrating an example of cloud height measurement processing in this embodiment. Note that, as a premise for the cloud height measurement processing, it is assumed that the cloud base altitude estimation unit 203 periodically acquires wind direction and speed information 2112, visibility information 2113, cloud top altitude information 224, and cloud base altitude information 2115 from the ceilometer 32 and external device 40 and stores them in the weather information storage unit 211. Furthermore, this processing and each of the processing described below are specifically realized by the processor 101 executing a program stored in the memory 102.

[0043] The cloud height measurement process is started, for example, in response to a predetermined start operation from the user. When the process is started, the cloud height measurement device 20 causes the camera control unit 210 to perform initial settings of the stereo camera 30 and the visible light camera 31 via the network 50. During the initial settings, the camera control unit 210 sends time information indicated by the internal clock 215 to the stereo camera 30 and the visible light camera 31, and synchronizes the shooting time added to the captured image with the time information of the internal clock 215. The camera control unit 210 also performs calibration between the two cameras that make up the stereo camera 30. In addition, in preparation for subsequent shooting, a shooting order table is read out from the cloud shooting order information storage unit 212 (step S401).

[0044] 5 is a data structure diagram showing an example of the structure of a shooting order table. Note that although table-structured data is used here, shooting order information may be stored in other data formats, such as a list structure that can indicate the shooting order.

[0045] The shooting order table 500 holds setting information in a “pan” column 5002 , a “tilt” column 5003 , and a “zoom” column 5004 corresponding to information in an “order” column 5001 .

[0046] The "Order" column 5001 contains ascending serial numbers indicating the order in which images are taken. The "Pan" column 5002 contains information indicating the pan angle, which is the rotation angle around the vertical axis indicating the orientation of the pan head of the stereo camera 30, and is set as an angle between "0" and "359" degrees. A pan angle of 0 degrees is set to the same orientation for the multiple cameras that make up the stereo camera 30. For example, the orientation can be set based on the orientation of an axis perpendicular to the baseline and the horizontal direction, or the azimuth angle. Negative values ​​and values ​​greater than 360 degrees are converted to a value between 0 and 359 by adding or subtracting 360. The "Tilt" column 5003 contains information indicating the pan angle, which is the angle in the elevation direction of the pan head of the stereo camera 30, and is listed as a zenith angle between "0" and "90" degrees. A zenith angle of 90 degrees means that the optical axis of the camera, i.e., the axis perpendicular to the sensor surface, is perpendicular to the vertical axis, and a zenith angle of 0 degrees means that it is parallel to the vertical axis and points straight up. The "Zoom" column 5004 sets the zoom magnification of the stereo camera 30. "1" indicates that the zoom magnification is 1x, that is, shooting at a magnification that is the standard on the wide-angle side.

[0047] In this embodiment, it is assumed that the pan angle and tilt angle have axes in a common direction among the multiple cameras that make up the stereo camera 30, and the information set in the shooting order table is also used as common information among the multiple cameras.

[0048] 4, the cloud height measurement device 20 controls the stereo camera 30 in accordance with the shooting order table 500 read out in step S401 by the camera control unit 210, and performs wide-angle shooting processing (step S402) and telephoto shooting processing (step S403), which will be described later, to capture stereo images. At this time, visible light images are also captured by the visible light camera. The captured stereo images and visible light images are acquired by the image receiving unit 204 and stored in the weather information holding unit 211 as image information 2111.

[0049] The camera control unit 210 determines whether or not shooting has been performed based on all the setting information set in the shooting order table 500, i.e., whether or not shooting of the entire sky has been completed. If shooting of the entire sky has not been completed, the process returns to step S402, and shooting is performed according to the next shooting order setting (step S404).

[0050] If the entire sky has been photographed, the cloud height estimation process described later is then performed, and a weather report format is generated (step S405).

[0051] The cloud height measurement device 20 determines whether an irrecoverable abnormality has occurred in the processing up to step S404, such as the inability to extract characteristic points from the image, the inability to estimate cloud height, or the inability to complete cloud height estimation within a predetermined time (step S406). If an irrecoverable abnormality has occurred, there is a possibility that a problem has occurred, such as a misalignment in the imaging directions of the two cameras constituting the stereo camera 30, and the cloud height measurement device 20 notifies the abnormality via the UI unit 214 (step S407). If no abnormality has occurred, the cloud height measurement device 20 waits until the next timing for capturing an image, and at that point returns to the capturing processing of step S402 and repeats the above processing at a predetermined interval, for example, every 10 minutes (step S408).

[0052] The weather report formula 2116 generated by the above process and stored in the weather information storage unit 211 is provided to airports and the like as appropriate.

[0053] FIG. 6 is a flowchart showing the details of the wide-angle shooting process in step S402.

[0054] The camera control unit 210 controls the platform on which the stereo camera 30 is mounted, in accordance with the information set in the "pan" column 5002 and the "tilt" column 5003 of the shooting order table, to control the orientation of the stereo camera 30. The camera control unit 210 also controls the zoom of the stereo camera 30 in accordance with the zoom magnification set in the "zoom" column 5004 of the shooting order table (step S501).

[0055] Once the orientation and zoom settings of the stereo camera 30 are complete, the camera control unit 210 releases the shutter of the stereo camera 30 to expose it and capture stereo images. At this time, the camera control unit 210 releases the shutter multiple times at different times to capture images and obtain time-series stereo images so that the movement of clouds (displacement of cloud positions over time) can be obtained. The captured stereo images are obtained from the stereo camera 30 by the image receiving unit 204 and stored in the weather information holding unit 211 (step S502).

[0056] Next, the cloud edge detection unit 207 receives the stereo images from the image receiving unit 204 and acquires sub-regions including the boundaries of clouds in the images by cloud edge detection processing described later (step S503). The cloud height measurement device 20 determines whether or not sub-regions have been acquired by the cloud edge detection unit 207 (step S604). If sub-regions have been acquired, the cloud height measurement device 20 sets the shooting order for those sub-regions by cloud shooting order setting processing described later (step S605) and outputs it as the shortest shooting order (step S606). If the cloud edge detection unit 207 has not acquired any sub-regions, the cloud height measurement device 20 simply ends the wide-angle shooting processing.

[0057] FIG. 7 is a flowchart showing the details of the telephoto shooting process in step S403.

[0058] The camera control unit 210 determines whether the shortest shooting order has been set by the cloud shooting order setting unit 209. If the shortest shooting order has not been set, there is no sub-area to be photographed with telephotography, and so the camera control unit 210 ends the telephotography processing (step S701).

[0059] If a photographing order has been set by the cloud photographing order setting unit 209, the camera control unit 210 controls the direction (pan angle and / or tilt angle) and zoom of the stereo camera 30 according to the set shortest photographing order (step S702). Once the direction and zoom of the stereo camera 30 have been set, the camera control unit 210 exposes the stereo camera 30 to photograph stereo images. The photographed stereo images are acquired from the stereo camera 30 by the image receiving unit 204 and stored in the weather information holding unit 211 (step S703).

[0060] The camera control unit 210 determines whether all shooting in the shortest shooting order set by the cloud shooting order setting unit 209 is complete, that is, whether acquisition of stereo images for the sub-areas specified in the shortest shooting order is complete, and if not, repeats the process from step S701 until completion. If completed, the camera control unit 210 ends the telephoto shooting process (step S704).

[0061] FIG. 8 is a flowchart showing the details of the cloud edge detection process in step S603.

[0062] In the cloud edge detection process, first, the cloud edge detection unit 207 receives the stereo images received by the image receiving unit 204 from the image receiving unit 204. Note that the transfer of images between processing units does not need to be performed directly between the processing units, and may be performed via the weather information storage unit 211 (step S801).

[0063] The cloud edge detection unit 207 performs cloud shape recognition processing to acquire cloud shapes. Fig. 9 is a schematic diagram showing the state of cloud boundaries recognized by the cloud shape recognition processing. As a result of performing the cloud shape recognition processing on a cloud image 901, a cloud image 902 is obtained in which a cloud boundary 903 is acquired as the cloud shape. For example, the cloud on the lower left side of the cloud image 902, surrounded by the cloud boundary 903, is a cumulus cloud, and shows a state in which it is separated from the other area (stratocumulus cloud) (step S802).

[0064] Returning to FIG. 8, the cloud edge detection unit 207 then detects edge regions, which are sub-regions that include the boundaries of the clouds.

[0065] Fig. 10 is a schematic diagram showing the relationship between a captured cloud image and sub-regions. In Fig. 10, cloud image 901 is the same cloud image as cloud image 901 shown in Fig. 9, and cloud image 1000 is obtained by dividing this cloud image into nine sub-regions 1001 to 1009 (in the figure, reference numbers for referencing each region are assigned in the upper right corner of each region framed by dashed lines in cloud image 1000. The same applies to other drawings below.) For example, if sub-region 1005 is captured using a telephoto lens with a zoom magnification increased to 3x, the image of sub-region 1005 will fill the viewing angle, and as shown in zoomed image 1010, the resolution will be three times higher than before zooming.

[0066] 11 is a schematic diagram showing the relationship between the recognized cloud boundaries and sub-regions. Cloud image 1100 is an image obtained by superimposing cloud image 902 shown in FIG. 9 on cloud image 1000, which has dashed lines indicating sub-regions. It can be seen that cloud boundary 903 is included in six of the nine sub-regions shown in cloud image 1100: sub-regions 1001, 1002, 1004, 1005, 1006, and 1007. In this example, cloud edge detection unit 207 detects these six sub-regions 1001, 1002, 1004, 1005, 1006, and 1007 as edge regions (step S803).

[0067] 8, the cloud edge detection unit 207 determines whether the number of sub-regions acquired in step S803 exceeds the predetermined sub-region coverage ratio. If the number of acquired sub-regions exceeds the predetermined sub-region coverage ratio, the cloud edge detection process ends (step S804).

[0068] In step S804, if the number of acquired sub-regions does not exceed the predetermined sub-region coverage ratio, the cloud edge detection unit 207 detects and adds an internal region, which is a region surrounded by the boundary of the cloud.

[0069] For example, in the cloud image 1100 shown in FIG. 11, there are nine subregions in the cloud image 1100, but six edge regions (subregions including cloud boundaries) are detected, resulting in a coverage rate of 6÷9=66.7%. If this coverage rate of 66.7% is below a predetermined minimum coverage rate threshold, the cloud edge detection unit 207 determines that cloud height cannot be estimated with sufficient accuracy and adds regions inside the cloud shape other than the cloud boundary subregions to the edge region, thereby increasing the coverage rate and improving the cloud height estimation accuracy. Taking the cloud image 1100 as an example, subregions 1008 and 1009 are newly added to the edge region. As a result, the number of subregions detected as edge regions becomes eight, resulting in a coverage rate of 8÷9=88.9% (step S805).

[0070] If the number of detected edge areas increases as a result of adding the internal area, the processing time increases, and there is a risk that the cloud height estimation process will not be completed within the preset time constraints. Therefore, the cloud edge detection unit 207 determines whether the coverage rate after adding the internal area to the edge area exceeds a predetermined maximum coverage rate. If the coverage rate is within the upper limit, the cloud edge detection unit 207 ends the cloud edge detection process (step S806).

[0071] On the other hand, if the coverage rate exceeds the maximum coverage rate in step S806, the cloud edge detection unit excludes part of the internal region added as an edge region from the edge region to bring the coverage rate within the threshold, and then terminates the process. For example, in FIG. 11, if subregion 1008 is excluded from subregions 1008 and 1009 added as edge regions, seven subregions will be detected as edge regions, and the coverage rate will be 7 / 9=77.8%. If this coverage rate is below the predetermined maximum coverage rate threshold, it can be said that there is no problem in terms of processing time constraints (step S807).

[0072] FIG. 12 is a flowchart showing the details of the photographing order setting process in step S605.

[0073] The cloud movement estimation unit 208 receives cloud images (stereo images) and edge region information from the cloud edge detection unit 207 (step S1201). The cloud movement estimation unit 208 acquires feature points from the received images. The feature points can be acquired by acquiring image feature amounts such as AKAZE features or HOG features using an image processing library such as OpenCV, an open source software library (step S1202). The cloud movement estimation unit 208 further performs a feature point matching process on the time-series cloud images from which the feature amounts have been acquired (step S1203).

[0074] The cloud movement estimation unit 208 performs a trajectory detection process, such as optical flow, on the result of the matching process to obtain movement vectors of the feature points.

[0075] 13 is a schematic diagram showing how a movement vector is obtained. Cloud image 1310 is a cloud image taken at a certain time, with a dashed line indicating a sub-region and a cloud boundary 1304 recognized in the cloud edge detection process (step S603) superimposed on it. Cloud image 1320 is a cloud image taken at a different time (e.g., t seconds later), with a dashed line indicating a sub-region and a cloud boundary 1306 recognized in the cloud edge detection process (step S603) superimposed on it.

[0076] Arrows 1331, 1332, 1333, 1334, and 1335 indicate the correspondence between some of the feature points acquired by the feature point acquisition process (step S1202) for cloud image 1310 and the feature points in cloud image 1320 that were matched by the matching process (step 1203). For example, arrow 1331 indicates that the cloud boundary (feature point) that was in edge region 1311 in cloud image 1310 has transitioned to edge region 1321 in cloud image 1320. It is possible to acquire a movement vector for each edge region from the displacement of the feature points associated with the arrows on the image (the difference between the coordinate position in cloud image 1310 and the coordinate position in cloud image 1320).

[0077] If there is a cloud image captured at a time later than the time cloud image 1320 was captured, for example, 2t seconds after the time cloud image 1310 was captured, when comparing this cloud image with cloud image 1310, the amount of movement of the edge region is expected to be even greater than the amount of movement of the edge region obtained by comparing with cloud image 1320 (step S1204).

[0078] After acquiring the movement vector, the cloud photographing order setting unit 209 acquires the pan-tilt operation time.

[0079] 14 is a schematic diagram illustrating pan-tilt operation time. Image 1410 shows an example of the operation pattern of the stereo camera 30 when the zoom of the stereo camera 30 is zoomed in three times from a wide-angle state to capture a sub-region within a cloud image captured at a wide angle. Image 1420 shows an example of the operation pattern of the stereo camera 30 when, after capturing an image of a sub-region at a wide angle, another sub-region is captured.

[0080] First, consider an image 1410 in which the stereo camera 30 is zoomed in 3x from a wide-angle state to capture one of the cloud regions captured at a wide angle. In this case, the stereo camera 30 is initially directed toward the center of the image, and from this state, there are four patterns: a pattern in which the stereo camera 30 moves by one sub-region in the tilt direction (arrow 1411), a pattern in which the stereo camera 30 moves by one sub-region in the pan direction (arrow 1412), a pattern in which the stereo camera 30 moves by one sub-region each in the tilt direction and the pan direction (arrow 1413), and a pattern in which only zooming is performed without movement in either the tilt or pan direction (arrow 1414). Even if movement in the pan direction and tilt direction cannot be performed simultaneously and the movement times in the pan direction and tilt direction are different, it is sufficient to consider these four patterns.

[0081] Next, consider the case where, with reference to image 1420, one of the nine sub-regions is photographed with 3x zoom, and then another sub-region is photographed. In this case, there are eight movement patterns indicated by arrows 1421 to 1428. Therefore, there are a maximum of eight movement patterns to consider.

[0082] From the above, a total of 12 patterns of operation time can be obtained: four patterns involving zoom operation and eight patterns without zoom operation. If the camera head can perform tilt and pan operations independently and in parallel, it can be considered that the operation in the direction with the longer operation time is the rate-limiting one. For example, assuming that the operation speed in the tilt direction and the operation speed in the pan direction are the same, the operation indicated by arrow 1421, which moves two sub-areas in the tilt direction, and the operation indicated by arrow 1423, which moves two sub-areas in the tilt direction and one sub-area in the pan direction, require the same amount of time because the amount of operation in the tilt direction, which has a longer operation distance, is the same. Therefore, in this case, the number of patterns of operation time required is reduced (step S805).

[0083] The cloud photographing order setting unit 209 predicts the edge area where the cloud boundary transitions based on the movement vector acquired in step S1204 and the operation time acquired in step S1205, and sets the photographing order for zooming. For example, the time required to photograph the four edge areas 1321, 1322, 1323, and 1324 shown in cloud image 1320 in Fig. 13 can take multiple combinations, such as photographing in the order 1324 → 1323 → 1321 → 1322, or photographing in the order 1323 → 1324 → 1321 → 1322. In such cases, the number of combinations can be reduced by imposing a constraint such as prioritizing photographing sub-areas adjacent in one direction, thereby making it possible to determine the photographing order with a small amount of calculation (step S806).

[0084] FIG. 15 is a flowchart showing the details of the cloud height estimation process in step S405. The stereo vision ranging unit 202 reads the stereo images acquired in steps S402 and S403 from the weather information storage unit 211 (step S1501). The stereo vision ranging unit performs semantic segmentation using ten cloud shapes in a cloud shape recognition process to detect clouds and their shapes in the stereo images. This detection can be performed, for example, by image recognition processing using a learning model that has been machine-learned in advance (step S1502). The stereo vision ranging unit 202 performs disparity extraction processing using a block matching method or a feature matching method for each cloud shape region (step S1503), and obtains the distance to the detected cloud based on the extracted disparity (step S1504). The stereo vision ranging unit 202 determines whether the above processing has been performed for all images. If there are any remaining stereo images, the above processing is performed on those images (step S1505).

[0085] When the above processing has been completed for all captured stereo images, the cloud height measurement device 20 uses the cloud base height estimation unit 203 to calculate the cloud height from the distance to the cloud and the camera direction (tilt angle) acquired by the stereo vision ranging unit 202. The cloud base height estimation unit 203 further reads cloud base height information 2115 from the weather information storage unit 211 and corrects the calculated cloud height using the cloud height at the same time provided by the ceilometer. For example, for cloud A whose cloud height is measured by a ceilometer, if the cloud height acquired by the cloud height measurement device 20 differs from the cloud height acquired by the ceilometer, the cloud base height estimation unit 203 corrects the cloud height of cloud A to the cloud height acquired by the ceilometer. The cloud base height estimation unit 203 also corrects the cloud heights of clouds other than cloud A using the same ratio as the correction ratio for cloud A (step S1506). Finally, the cloud base height estimation unit 203 reads out the wind direction and speed information 2112 and visibility information 2113, and generates a weather report formula 2116 by combining this with the acquired information such as cloud shape and cloud height (step S907).

[0086] If no clouds are detected in steps S1502 to S1504, the cloud base altitude estimation unit 203 omits the processing of steps S1503 to S1506, and in step S1507 generates a weather report formula 2116 indicating that no clouds are present.

[0087] Figure 16 is a conceptual diagram showing an example of a METAR, a type of weather report format. As shown in the figure, when information 1800 about each item of weather described in normal sentences is converted into a weather report format 2116, the amount of text can be significantly reduced without reducing the amount of information.

[0088] According to this embodiment, cloud height measurement is performed using high-resolution stereo images captured with a telephoto lens, allowing for more accurate cloud height measurement than when using low-resolution reference images captured with a wide angle. Telephoto photography is performed by selecting an area where clouds exist that has been identified based on images captured with a wide angle lens, allowing for more efficient cloud height measurement in a shorter time than when the entire sky is captured with a telephoto lens. Furthermore, telephoto photography is performed in a shooting order determined taking into account the time required for camera setup, allowing for more efficient cloud height measurement in a shorter time.

[0089] In the above-described embodiment, the stereo camera captures images at two zoom magnifications: 1x and 3x. However, depending on the type of cloud, capturing images at a higher magnification may not always be appropriate. In the second embodiment described below, cloud height measurement is performed by using different zoom magnifications depending on the type of cloud.

[0090] The cloud height measurement system of the second embodiment differs from the first embodiment in the cloud edge detection process shown in Fig. 8. The configuration of the cloud height measurement system and the processes other than the cloud edge area detection process are not significantly different from those of the first embodiment, so a description of these will be omitted here.

[0091] Fig. 17 is a flowchart of cloud edge detection processing in the second embodiment. In Fig. 17, the same reference numerals as in Fig. 8 are used to denote processing that is similar to the processing described in Fig. 8, and descriptions thereof will be omitted.

[0092] After cloud shape recognition in step S802, the cloud edge detection unit 207 determines whether cumulus clouds are included in the stereo images captured from the acquired cloud shapes (step S1701). If cumulus clouds are included in the stereo images, the cloud edge detection unit 207 determines the number of divisions into sub-regions to be 9, i.e., divides the region captured at the standard zoom magnification into 9 sub-regions (step S1702). If it is determined in step S1701 that cumulus clouds are not included, the cloud edge detection unit 207 determines whether stratus clouds are included in the image, and if stratus clouds are not included, ends the cloud edge detection process (S1703). If stratus clouds are included, the cloud edge detection unit 207 sets the number of divisions into sub-regions to 4, which is fewer than the number for cumulus clouds. Stratus clouds have low spatial frequency in the image and small brightness changes, so edge detection accuracy may be reduced in images captured at high resolution, i.e., at a high zoom magnification. In this embodiment, taking this into consideration, in the case of stratus clouds, the number of divisions into sub-regions is reduced compared to cumulus clouds, and telephoto shooting processing is performed using a low zoom magnification (step S1704).

[0093] After the division number is determined, the cloud edge detection unit 207 divides the captured stereo image into the determined division number of sub-regions and detects edge regions from the divided sub-regions (step S803). Next, the cloud edge detection unit 207 determines whether a sufficient number of sub-regions have been detected as edge regions, and if a sufficient number of sub-regions have been detected as edge regions, the cloud edge detection unit 207 ends the cloud edge detection process (step S804).

[0094] If a sufficient number of sub-regions are not detected as edge regions in step S804, the cloud edge detection unit 207 acquires the brightness change rate within each sub-region for those sub-regions not detected as edge regions (step S1705).The cloud edge detection unit 207 then selects sub-regions whose brightness change rate is greater than a predetermined brightness change rate, adds them as edge regions, and terminates the cloud edge detection process (step S1706).

[0095] Thereafter, for the sub-regions detected as edge regions, the order of telephotography is determined in the same manner as in the first embodiment, and telephotography is performed.

[0096] In this way, by switching the zoom magnification depending on the type of cloud and taking photographs, it is possible to perform highly accurate cloud height measurements regardless of the type of cloud being measured.

[0097] According to the embodiment described above, high-resolution telephoto zoom photography is performed while changing the shooting direction of the multiple cameras that make up the stereo camera, making it possible to cover the entire sky and measure cloud height with high accuracy. Furthermore, telephoto zoom photography is performed by recognizing the cloud shape and selecting the cloud boundary, making it possible to measure cloud height within a limited time constraint, such as a 10-minute cycle. Furthermore, the order of telephoto zoom photography is set taking into account the operating time required to control the setting of the camera (platform) to the shooting direction, making it possible to measure cloud height more efficiently.

[0098] The above-described embodiment illustrates a typical aspect of the present invention, and is not intended to limit the scope of the present invention. Various modifications to the present invention are possible without departing from the spirit and scope of the present invention. For example, the zoom magnification and the number of divisions into sub-regions used in the above-described embodiment are merely examples for easily understanding the present invention, and can be appropriately determined depending on the angle of view of the camera lens used and the resolution of the captured stereo image. Furthermore, in the above-described embodiment, all processing is performed using stereo images captured by a stereo camera. However, for example, images captured by a visible light camera may be used to detect edge regions, and the edge regions thus obtained may be captured by a telephoto stereo camera. [Explanation of symbols]

[0099] 10. Cloud measurement system, 20. Cloud height measurement device, 30. Stereo camera, 31. Visible light camera, 32. Ceilometer, 50. Network, 202. Stereo visual ranging unit, 203. Cloud base height estimation unit, 204. Image receiving unit, 205. Communication unit, 207. Cloud edge detection unit, 208. Cloud movement estimation unit, 209. Cloud shooting order setting unit, 210. Camera control unit, 211. Weather information storage unit, 212. Cloud shooting order information storage unit, 214. UI unit, 215. Internal clock

Claims

1. A plurality of cameras whose shooting direction and optical zoom magnification can be changed; an image receiving unit that receives images captured by the plurality of cameras via a communication line; a cloud edge detection unit that recognizes clouds captured in the image acquired by the image receiving unit, detects the boundaries of the clouds, and acquires, as an edge region, a sub-region that includes the boundaries from among a plurality of sub-regions obtained by dividing the image into a plurality of regions that can be enlarged and photographed by changing the zoom magnification of the plurality of cameras; A cloud photographing order setting unit determines a sub-area to be photographed by changing the zoom magnification of the plurality of cameras from among the plurality of sub-areas based on the edge area, and sets the photographing order of the sub-areas to be photographed; A camera control unit that controls the plurality of cameras to photograph a photographing area corresponding to a sub-area photographed by changing the zoom magnification according to the photographing order set by the cloud photographing order setting unit, and acquires a stereo image for each sub-area photographed by changing the zoom magnification; a stereo vision distance measuring unit that acquires distances to clouds captured in the stereo images based on the stereo images; A cloud measurement system having a cloud base altitude estimation unit that acquires the cloud base altitude of the clouds captured in the stereo image based on the distance.

2. The cloud measurement system according to claim 1 further comprises a cloud movement estimation unit that acquires a movement vector of the boundary in the edge region, The cloud photographing order setting unit determines the sub-area to be photographed by changing the zoom magnification from among the multiple sub-areas based on the movement vector and the operating time required to control the photographing direction of the multiple cameras.

3. The cloud measurement system according to claim 2, characterized in that the cloud photographing order setting unit sets the photographing order of the sub-areas to be photographed by changing the zoom magnification based on the operating time required to control the photographing direction of the multiple cameras.

4. The cloud measurement system described in claim 2, characterized in that the cloud movement estimation unit acquires corresponding feature points from each of multiple images of the same area taken at different times by at least one of the multiple cameras, and acquires the movement vector based on the displacement of the feature points between each of the multiple images.

5. The cloud measurement system described in claim 1, characterized in that when the number of sub-areas acquired as the edge area is less than a predetermined number, the cloud edge detection unit adds sub-areas that depict the interior surrounded by the boundary of the cloud to the edge area.

6. The cloud measurement system described in claim 5, characterized in that when the number of sub-areas acquired as the edge area by adding a sub-area that captures the interior surrounded by the cloud boundary to the edge area exceeds a predetermined upper limit, the cloud edge detection unit excludes from the edge area a portion of the sub-area that captures the interior surrounded by the cloud boundary so that the number of edge areas is within the upper limit.

7. The cloud measurement system according to claim 1, wherein the cloud edge detection unit identifies the shape of a cloud captured in the image and divides the image into a different number of sub-regions according to the identified shape of the cloud.

8. The cloud measurement system according to claim 7, characterized in that, when the number of the sub-areas acquired as the edge area is less than a predetermined number, the cloud edge detection unit selects sub-areas from the sub-areas not considered to be the edge area whose brightness change rate is greater than a predetermined value and adds them to the edge area.

9. 8. The cloud measurement system according to claim 7, wherein the camera control unit sets zoom magnifications of the plurality of cameras in accordance with the number of sub-regions divided by the cloud edge detection unit.

10. A cloud height measurement method in a cloud measurement system that controls a stereo camera composed of multiple cameras with variable zoom magnifications and measures cloud height based on stereo images captured by the stereo camera, Recognizing clouds captured in images acquired from at least one of the plurality of cameras and detecting boundaries of the clouds; A sub-area including the boundary of the detected cloud is acquired as an edge area from among a plurality of sub-areas obtained by dividing the image into a plurality of areas that can be enlarged by changing the zoom magnification; determining, as a target area for imaging, a sub-area to be enlarged and imaged by changing the zoom magnification from among the plurality of sub-areas based on the edge area, and setting an imaging order for the target areas for imaging; Controlling the plurality of cameras in accordance with the set shooting order, changing the zoom magnification to take an enlarged image of the shooting target area, and acquiring a stereo image of the shooting target area; A cloud height measurement method for acquiring the cloud base altitude of the cloud based on the parallax of the cloud captured in the acquired stereo image.

11. The cloud height measurement method of claim 10, wherein the process of setting the shooting order acquires a movement vector of the boundary in the edge area, and selects a sub-area to be photographed by changing the zoom magnification based on the movement vector and the time required to control the shooting direction of the multiple cameras, and determines it as the target area to be photographed.

12. The cloud height measuring method according to claim 11 , wherein the process of setting the photographing order sets the photographing order of the photographing target area based on the operation time required to control the photographing directions of the plurality of cameras.

13. The process of acquiring the edge region includes recognizing a cloud shape of the cloud captured in the image, and determining the number of divisions into the edge region according to the recognized cloud shape; The cloud height measuring method according to claim 10 , wherein the process of acquiring the stereo images sets zoom magnifications of the plurality of cameras in accordance with the number of divisions into the edge regions.

Images