Information processing device, control method, program

Abstract

To provide a mechanism that can estimate the height of a water surface from an image obtained by photographing a river.SOLUTION: The present invention comprises: selection means that selects a plurality of portions from the boundary between an image area of a water surface and an image area other than the water surface, which are extracted from an image obtained by photographing a predetermined range including the water surface; and water level estimation means that estimates the height of the water surface on the basis of the heights corresponding to the selected plurality of portions specified from positions in the image relative to the plurality of portions.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This invention relates to a water level measuring device that measures the height of the water surface from images taken of a river. [Background technology]

[0002] Early detection of a disaster is crucial to mitigating its impact. This is especially true for river flooding, where rapid evacuation is essential, making early detection even more critical. Therefore, monitoring river water level fluctuations is extremely important.

[0003] Common methods for measuring water levels include mechanical methods using floats, ultrasound, and water pressure. However, these mechanical methods have problems such as a lack of observation points, equipment failure, and maintenance costs. To solve these problems, Patent Document 1 describes a water level observation system that uses image processing. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2008-057994 [Disclosure of the Invention] [Problems that the invention aims to solve]

[0005] However, the water level observation system using the image processing described above has challenges in terms of robustness. Patent Document 1 determines the water surface boundary by a rapid decrease in brightness, but the distribution of brightness changes greatly depending on the situation, such as sunlight reflection on the water surface and shadows cast on the water surface from embankments, so there is a risk of overlooking or falsely detecting the water surface boundary.

[0006] To accommodate these diverse situations, increasing the number of brightness change patterns indicating the water surface boundary or lowering the detection threshold could lead to increased system complexity. Furthermore, such measures could result in the system forcibly detecting the water surface even under adverse conditions where measurement should be impossible, potentially reducing the reliability of water level measurement results.

[0007] To solve the above problems, this invention provides a mechanism that can estimate the height of the water surface from images taken of a river. [Means for solving the problem]

[0008] The present invention is characterized by comprising: a selection means for selecting a plurality of locations from the boundary between an image region of the water surface and an image region other than the water surface, extracted from an image of a predetermined range including the water surface; and a water level estimation means for estimating the altitude of the water surface based on the altitude corresponding to the plurality of locations identified from the in-image positions of the selected plurality of locations. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide a mechanism that can estimate the height of the water surface from images taken of a river. [Brief explanation of the drawing]

[0010] [Figure 1] Block diagram showing an example of an embodiment of the present invention. [Figure 2] Block diagram showing an example of the configuration of the water region estimation image correction unit. [Figure 3] A flowchart showing an example of the processing performed by the water level calculation unit. [Figure 4] A flowchart illustrating an example of image correction processing in aquatic regions. [Figure 5] A flowchart showing an example of positional misalignment correction processing. [Figure 6] A diagram showing an example of an input image and a correction image. [Figure 7] Figure showing an example of input image and water region estimation image. [Figure 8]Figure showing an example of a water area estimation image [Figure 9] Figure showing an example of measurement area data [Figure 10] Figure showing an example of correction image data [Figure 11] Figure showing an example of elevation information data [Figure 12] Figure showing an example of measurement result data [Figure 13] Figure showing an example of a measurement result list screen [Figure 14] Figure showing an example of an image analysis result screen [Figure 15] Figure showing an example of a water level estimation result screen [Figure 16] Figure showing an example of the hardware configuration of a water level measuring device

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0012] FIG. 1 is a block diagram showing an example of an embodiment of the present invention.

[0013] The imaging device 11 is a surveillance camera installed so as to capture an image of a river where the water level is to be measured and the levee adjacent thereto. This surveillance camera may have a fixed viewing angle or may be equipped with a PTZ function. The image captured by the imaging device 11 is transmitted to the water level measuring device 12.

[0014] The water level measuring device 12 includes a water area estimation unit 121, a water area estimation image correction unit 122, an elevation information storage unit 123, and a water level calculation unit 124. The water area estimation unit 121 divides each pixel of the input image into either a water area or a non-water area. The water area estimation image correction unit 122 performs processes such as correcting the deviation of the imaging viewing angle. The elevation information storage unit 123 stores the elevation information to be referred to during water level measurement. The water level calculation unit 124 calculates the water level at the time of image capture based on the water area and the elevation information. The measurement result storage unit 125 stores the input image used for measurement, the water area estimation image, the water level measurement result, and the like.

[0015] In the embodiment of the present invention, the water region estimation unit 121 performs semantic segmentation on the input image using a trained neural network model. The data for training includes images of the river and the surrounding land area, including images taken by the imaging device 11. These training images are collected under multiple conditions, such as sunny, rainy, foggy, and at night. The images may be full-color images taken in visible light or grayscale images taken in infrared light. This allows the system to learn the general characteristics of water regions in diverse situations and to stably recognize water regions.

[0016] Figure 2 is a block diagram showing the configuration of the water region estimation image correction unit. The measurement region storage unit 211 stores information about the range of the shooting angle of view that is to be measured for water level. The correction information storage unit 212 stores images used to correct deviations in the shooting angle of view. There may be one correction image or multiple images. The correction unit 213 performs correction processing on the input water region estimation image using the measurement region and the correction images.

[0017] Figure 2 is a block diagram showing the configuration of the water region estimation image correction unit. The measurement region storage unit 211 stores information that allows for the identification of a rectangular region in the image. Figure 9 shows an example, in which the y-coordinates of the upper and lower ends of the rectangle and the x-coordinates of the left and right ends of the rectangle are stored. When measuring the water level, this information is read from memory and used.

[0018] The correction information storage unit 212 stores information used to correct deviations in the shooting angle of view. A correction image, which serves as the reference image for correction, is required as correction information. The correction image is a portion extracted from an image taken at a predetermined angle of view (hereinafter referred to as the reference angle of view) set for water level measurement. The area to be extracted should preferably contain a fixed, distinctive object, such as a road traffic sign. There may be one or multiple correction images. The coordinates of the correction image within the reference angle of view are also necessary for correction. Figure 10 illustrates a table recording the storage location and coordinate information of the correction image. The correction information storage unit 212 stores the information in this table along with the correction image.

[0019] The correction unit 213 performs correction processing on the input water region estimation image using the measurement region and correction information.

[0020] Figure 16 is a block diagram showing an example of the hardware configuration of the water level measuring device 12 in an embodiment of the present invention.

[0021] As shown in Figure 16, the information processing device is connected via a system bus 1604 to a CPU (Central Processing Unit) 1601, ROM (Read Only Memory) 1602, RAM (Random Access Memory) 1603, input controller 1605, video controller 1606, memory controller 1607, and communication I / F controller 1608.

[0022] The CPU 1601 provides comprehensive control over all devices and controllers connected to the system bus 1604.

[0023] ROM1602 or external memory 1611 holds the BIOS (Basic Input / Output System) and OS (Operating System), which are control programs executed by the CPU 1601, as well as computer-readable and executable programs and various necessary data (including data tables) for realizing this information processing method.

[0024] RAM1603 functions as the main memory, work area, etc., of the CPU1601. The CPU1601 loads the necessary programs, etc., from ROM1602 or external memory 1611 into RAM1603, and then executes the loaded programs to perform various operations.

[0025] The input controller 1605 controls input from input devices such as a keyboard 1609 or a pointing device such as a mouse (not shown). If the input device is a touch panel, the user can give various instructions by pressing (touching with a finger, etc.) icons, cursors, or buttons displayed on the touch panel.

[0026] Furthermore, the touch panel may be a multi-touch screen or other touch panel capable of detecting the positions of multiple fingers touching it.

[0027] The video controller 1606 controls the display to an external output device such as the display 1610. The display includes the display of a notebook computer integrated with the main unit. The external output device is not limited to a display; for example, it may be a projector. Furthermore, for the aforementioned touch-enabled device, an input device is also provided.

[0028] The video controller 1606 can control the video memory (VRAM) used for display control. It can utilize a portion of the RAM 1603 as the video memory area, or it can provide a separate, dedicated video memory.

[0029] The memory controller 1607 controls access to the external memory 1611. The external memory can include an external storage device (hard disk), a flexible disk (FD), or a CompactFlash® memory connected to a PCMCIA card slot via an adapter, which stores boot programs, various applications, font data, user files, editing files, and other data.

[0030] The communication interface controller 1608 connects to and communicates with external devices via a network and performs communication control processing over the network. For example, it can handle communication using TCP / IP, telephone lines such as ISDN, and 3G mobile phone lines.

[0031] Furthermore, the CPU 1601 enables display on the display 1610 by, for example, performing the process of expanding (rasterizing) outline fonts into the display information area in RAM 1603. The CPU 1601 also enables user input via a mouse cursor (not shown) on the display 1610.

[0032] Figure 3 is a flowchart showing the processing of the water level calculation unit 124. Figure 4 is a flowchart showing the water region estimation image correction process in Figure 3. Figure 5 is a flowchart showing the positional shift correction process in Figure 4.

[0033] In Figure 3, first, the estimation result output by the water region estimation unit 121 is received, and it is determined whether a water region exists in the input image (S11). If no water region exists, a message indicating that the water level measurement failed is returned. At this point, a request to retry with a different image may be accepted. If a water region exists, the process proceeds to the next step.

[0034] Next, water region estimation image correction is performed (S12). In water region estimation image correction, the water region estimation image is corrected by correcting camera misalignment and removing water regions outside the misrecognized or measurement area, thereby improving the accuracy of water level measurement.

[0035] In Figure 4, first, positional shift correction is performed (S21). Due to errors in the PTZ adjustment function and the effects of wind, the camera's field of view may shift slightly. If the water level measurement process is performed as is, the shift in the field of view will directly result in an error in the water level measurement result. Positional shift correction corrects this shift, enabling more accurate measurements.

[0036] In Figure 5, first, the correction image and the input image are compared (S31). The correction image is a portion extracted from an image taken at a specified field of view (hereinafter referred to as the reference field of view) set for water level measurement. It is desirable that the area to be extracted contains a fixed and distinctive object, such as a road traffic sign. This extracted image is compared with the input image, and similar areas in the input image are searched for. Search methods include template matching and feature point matching. If only translation is considered, correction can be performed by template matching, and if rotation and scaling are also considered, correction can be performed by feature point matching. In this embodiment of the present invention, correction by template matching will be described.

[0037] The embodiments of this invention are used in a variety of situations, including weather and time of day. Therefore, a single correction image may not be able to detect similar areas. To address this, multiple correction images can be set to suit different situations. Here, we will explain the case where two correction images, A and B, shown in Figure 6, are used.

[0038] First, the coordinates of the correction image A within the reference field of view are (x A ,y A Let's assume that the coordinates of the rectangle are the coordinates of the top-left point of the rectangle. Then, using template matching, we find the coordinates (x') within the input image. A ,y' A The rectangle indicated by ) is calculated to be the most similar to the correction image. The similarity score at this time is s A Let's assume that.

[0039] Next, it is determined whether all correction images have been compared (S32). Here, since the comparison of correction image B has not been performed yet, the process returns to S31 to perform the comparison of correction image B. Similar to correction image A, the coordinates within the reference picture angle of correction image B are set as (x B , y B ), the position matched within the input image is set as (x' B , y' B ), and the matching score is set as s B .

[0040] After completing the comparison for all correction images, the process proceeds to determine whether the matching score exceeds the threshold (S33). The maximum value s max of the matching scores for all correction images is compared with the threshold s thr . If s max ≤ s thr , the process ends as a failure. This occurs when the measurement becomes difficult due to significant video distortion or the like. If s max > s thr , the process proceeds to the subsequent processing.

[0041] Next, the correction image with the maximum matching score is selected (S34). This enables the selection of an appropriate correction image according to the situation. For example, if two correction images, one taken during the day and the other taken at night, are used, correction during the day can be performed using the correction image taken during the day, and correction at night can be performed using the correction image taken at night. Here, the explanation is given assuming that the matching score of correction image A is the maximum.

[0042] Next, the affine matrix M is calculated from the comparison results (S35). This is for transforming the water area estimation image to the reference picture angle. In this example, it is as follows.

[0043] <​​​​​​​​Finally, the water region estimation image is transformed using this affine matrix M (S36). The reason the water region estimation image is transformed instead of the input image is that the subsequent water level measurement process will refer to the water region estimation image.

[0045] Returning to the explanation of Figure 4, first, if the positional misalignment correction fails, the process is terminated as a failure (S22). If the positional misalignment correction is successful, the process proceeds to the next steps.

[0046] Figure 7 shows an example of an input image and a corresponding water region estimation image. In the figure, the white areas are regions estimated to be water regions, and the black areas are regions estimated to be non-water regions. As can be seen in Figure 7, regions other than true water regions may be misidentified as water regions. However, in most cases, the misidentified water region is smaller than the true water region. Therefore, by retaining only the water region with the largest area (S23), the misidentified water region can be excluded.

[0047] Next, the water area outside the measurement region is excluded (S24). There are cases where only a portion of the image may be used for measurement, such as when the end of a riverbank is included within the reference field of view. Therefore, by allowing the measurement region to be set in advance and using only that region for measurement, the reference field of view can be set more flexibly.

[0048] Returning to the explanation of Figure 3, first, if the water region estimation image correction fails, the process is terminated as a failure (S13). If the water region estimation image correction is successful, the process proceeds to the next steps.

[0049] Next, it is determined whether water regions exist in the corrected water region estimation image (S14). If the correction results in the absence of water regions, the process is terminated as a failure.

[0050] Next, the water surface boundary point cluster is obtained (S15). The water surface boundary point cluster is the set of points (a,b) that are included in the water region on the line x=a in the image and have the smallest y-coordinate (when the upper left of the image is the origin). Multiple values ​​of a are taken to obtain multiple water surface boundary point clusters in the x-coordinate direction of the image. The values ​​of a may be taken at equal intervals, or according to a predetermined criterion (for example, densely in the center of the image, sparsely towards the edges, etc.). If there is no water region on the line x=a, points on that line are not included in the boundary point cluster.

[0051] Next, the elevation of the boundary point cloud of the water surface is obtained (S16). The elevation information is stored in the elevation information storage unit 123 as a two-dimensional array of the same size as the resolution of the input image, and the elevation information of the coordinates (a,b) in the reference field of view is recorded at the position (a,b) of the array. Figure 11 shows an example of elevation information when the resolution of the input image is 1920 × 1080. The elevation information may be received directly as a two-dimensional array, or it may be created by receiving elevation information for several coordinates and interpolating it through calculation. Alternatively, instead of elevation information for each coordinate, the elevation may be obtained by a calculation formula defined for a sub-region of the image (for example, one side of a levee, etc.).

[0052] The S16 process yields an elevation group of the same size as the boundary point group of the water surface. Next, a representative value is calculated for this elevation group (S17). Possible representative values ​​include the mean, mode, and median. In this embodiment of the present invention, the median is used. This is expected to eliminate the influence of outlier elevation values. The representative value calculated here becomes the final water level measurement result.

[0053] Finally, the standard deviation of the elevation group is calculated (S18). In this example, the standard deviation is used, but the variance may also be used. If the water area estimation result is distorted due to image distortion, that is, if the reliability of the water level measurement result is low, the water surface boundary also tends to be distorted. An example of this is shown in Figure 8. In this case, the standard deviation of the obtained elevation group will be large. Therefore, if the standard deviation is large, the reliability of the water level measurement result is considered to be low. For this reason, the user may set a threshold for the standard deviation and discard the measurement result if it exceeds that value. This reduces the risk of forcing measurements in situations where the measurement conditions are extremely poor.

[0054] The water level measurement results and the standard deviation of the elevation group obtained as a result of the above processing are recorded in the measurement result storage unit 125. Figure 12 shows an example of a table containing the measurement results.

[0055] Figures 13 to 15 illustrate an example of a screen that displays measurement results.

[0056] Figure 13 is an example of a measurement results list screen that displays a list of measurement results.

[0057] The measurement results list screen 1300 displays basic information 1301 and the measurement results list 1302. Basic information 1301 displays the measurement location and camera ID. The measurement results list 1302 displays the measurement results for each time period, and the measurement results include information such as the shooting time, success / failure classification, water level, and standard deviation, as well as path information to the image data, such as the input image path and the water area estimation image path.

[0058] Additionally, the measurement results list screen 1300 displays buttons (1302-1304) for jumping to the image analysis results screen, the water level estimation results screen, and the menu screen (not shown). When each button is pressed, the corresponding screen is displayed.

[0059] Figure 14 is an example of an image analysis results screen that displays the results of the image analysis.

[0060] The image analysis results screen 1400 displays basic information 1401, input image 1402, and water area estimation image 1403. Basic information 1401 displays the measurement location and camera ID from the measurement results list screen 1300, as well as the date and time, success / failure classification, and water level. Input image 1402 and water area estimation image 1403 display images obtained from the aforementioned input image path and water area estimation image path.

[0061] Additionally, the image analysis results screen 1400 displays buttons (1404-1406) for jumping to the measurement results list screen, the water level estimation results screen, and the menu screen (not shown). When each button is pressed, the corresponding screen is displayed.

[0062] Figure 15 is an example of a water level estimation results screen that displays the water level estimation results.

[0063] The water level estimation results screen 1500 displays basic information 1501, water level estimation status 1502, and input image 1503. Basic information 1501 displays the measurement location and camera ID from the measurement results list screen 1300, as well as the date and time, success / failure classification, water level, and standard deviation. Water level estimation status 1502 displays a group of boundary points on top of the water area estimation image, and the elevation for each boundary point is displayed. Input image 1503 is displayed for reference.

[0064] Additionally, the water level estimation results screen 1500 displays buttons (1504-1506) for jumping to the measurement results list screen, the image analysis results screen, and the menu screen (not shown). When each button is pressed, the corresponding screen is displayed.

[0065] As described above, it becomes possible to estimate the water area from images of rivers, calculate the water surface elevation from the estimation results, and verify those results.

[0066] Although embodiments of the present invention have been described above, the present invention can take the form of, for example, a system, apparatus, method, program, or recording medium. Specifically, it may be applied to a system consisting of multiple devices, or to an apparatus consisting of a single device.

[0067] Furthermore, the program in this invention is a program that allows a computer to execute the processing methods of each flowchart. The program in this invention may also be a separate program for each processing method of each device in each flowchart.

[0068] As described above, it goes without saying that the object of the present invention can also be achieved by supplying a recording medium containing a program that realizes the functions of the embodiments described above to a system or device, and by having the computer (or CPU or MPU) of that system or device read and execute the program stored on the recording medium.

[0069] In this case, the program read from the recording medium itself realizes the novel function of the present invention, and the recording medium on which that program is recorded constitutes the present invention.

[0070] For recording media used to supply programs, examples include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EPROMs, silicon disks, and the like.

[0071] Furthermore, it goes without saying that the functions of the aforementioned embodiments are realized not only by the computer executing the program it has read, but also by the operating system (OS) running on the computer performing some or all of the actual processing based on the instructions of that program, thereby realizing the functions of the aforementioned embodiments.

[0072] Furthermore, it goes without saying that this also includes cases where, after a program read from a recording medium is written to the memory of a function expansion board inserted into a computer or a function expansion unit connected to a computer, the CPU or other components of the function expansion board or function expansion unit perform some or all of the actual processing based on the instructions of the program code, and the functions of the aforementioned embodiments are realized through that processing.

[0073] Furthermore, the present invention may be applied to a system consisting of multiple devices or to a device consisting of a single device. It goes without saying that the present invention can also be applied when the results are achieved by supplying a program to a system or device. In this case, by reading a recording medium containing a program for achieving the present invention into the system or device, the system or device can enjoy the effects of the present invention.

[0074] Furthermore, by downloading and reading the program for achieving the present invention from a server, database, etc. on a network using a communication program, the system or device can enjoy the effects of the present invention. It should be noted that configurations combining the above-described embodiments and their variations are all included in the present invention. [Explanation of Symbols]

[0075] 11. Imaging device 12. Water level measuring device

Claims

1. A selection means for selecting multiple locations relating to the boundary between the image region of the water surface and the image region other than the water surface from an image obtained by changing the field of view of an image that captures a predetermined range including the water surface, A selection means that identifies the height information relating to the selected plurality of locations based on the height information stored in the storage means in association with the position in the image that has been changed to the predetermined field of view, and the position in the image relating to the plurality of locations selected by the selection means, A water level estimation means for estimating the height of the water surface based on the height information identified by the aforementioned identification means, An information processing device characterized by comprising:

2. The aforementioned identification means identifies the altitude information corresponding to the position in the image relating to the multiple locations by calculation based on the altitude information. The information processing apparatus according to claim 1, characterized by the following:

3. The water level estimation means estimates the height of the water surface by using statistical values ​​based on altitude information corresponding to the positions in the image relating to the multiple locations. The information processing apparatus according to claim 1, characterized by the following:

4. The aforementioned statistical value is the median. The information processing apparatus according to claim 3, characterized by the following:

5. The system further includes an evaluation means for evaluating the reliability of the estimated water surface altitude based on the variation in altitude information corresponding to the positions in the image relating to the multiple locations. The information processing apparatus according to claim 1, characterized by the following:

6. The display control means further includes a means for controlling the display to show the time the image was captured and the estimated water surface altitude. The information processing apparatus according to claim 1, characterized by the following:

7. The system further includes a display control means for controlling the display of an image that distinguishes between an image region of the water surface and an image region other than the water surface. The information processing apparatus according to claim 1, characterized by the following:

8. The display control means further includes a means for controlling the display of an image showing the selected plurality of locations and altitude information corresponding to the positions in the image relating to the identified plurality of locations. The information processing apparatus according to claim 1, characterized by the following:

9. The selection means selects the multiple locations in the image so that they are spaced at regular intervals. The information processing apparatus according to claim 1, characterized by the following:

10. The selection means includes a selection step of selecting multiple locations relating to the boundary between the image region of the water surface and the image region other than the water surface from an image obtained by changing the angle of view of an image that captures a predetermined range including the water surface, The identification means identifies the height information relating to the selected plurality of locations based on the height information stored in the storage means, which is associated with the position in the image changed to the predetermined angle of view, and the positions in the image relating to the plurality of locations selected by the selection means. The water level estimation means includes a water level estimation step of estimating the height of the water surface based on the height information identified by the identification means, A control method for an information processing device, characterized by comprising the following:

11. A program for causing at least one computer to function as one of the means of an information processing device described in any one of claims 1 to 9.

Images