River monitoring system and river monitoring method

Abstract

To determine an accurate flooding part at the time of flooring.SOLUTION: A river monitor system includes: a learning image creation unit for creating a simulated flooding image simulating an image of a flooding on the basis of a simulated flooding label showing a site where water immersion is expected to happen at the time of flooding and a taken image; a learning unit for re-learning a machine learning model to detect a water surface by using the simulated flooding image created by the learning image creation unit related to the simulated flooding label as teacher data; and a river flooding detection unit for inferring the site of a water surface from the taken image on the basis of the machine learning model re-learned by the learning unit, and detecting the presence or absence of a flooding.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a river monitoring system and a river monitoring method. [Background technology]

[0002] In recent years, river monitoring systems have been proposed that use machine learning models to detect water surfaces from river images (river footage) obtained from unmanned aerial vehicles (UAVs, drones, etc.) or satellites, and grasp the flooding situation of rivers.

[0003] Patent document 1 discloses a technology that determines whether the flow of a river shown in an image deviates from its normal state and whether there is a possibility of flooding based on the state of the river before it reaches a flood state that has been learned in advance. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-170204 Summary of the Invention [Problem to be solved by the invention]

[0005] By the way, machine learning models used for river monitoring are usually created by learning from many river images (river videos). In other words, the accuracy of the machine learning model is dramatically improved when the environment is similar to the environment in which the machine learning model will actually be used.

[0006] Machine learning models used in river monitoring are expected to be able to accurately identify flooded areas when rivers overflow. However, it is difficult to collect images of rivers that are actually flooding, and machine learning models that have been trained on many images (river videos) of rivers that are not flooding are unable to accurately identify flooded areas when rivers overflow.

[0007] The present invention has been made in view of the above, and has an object to accurately determine flooded areas when a river floods. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems and achieve the object, the present invention comprises a training image creation unit that creates pseudo-flood images that resemble images of river flooding based on pseudo-flood labels that indicate areas where flooding is predicted in the event of a river flood and photographed images; a learning unit that associates the pseudo-flood images created by the training image creation unit with the pseudo-flood labels as training data and re-trains a machine learning model for water surface detection; and a river flood detection unit that infers locations where water surfaces exist from photographed images based on the machine learning model re-trained by the learning unit and detects whether or not a river is flooding. [Effects of the Invention]

[0009] According to the present invention, it is possible to accurately determine the location of flooding when a river floods. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram illustrating a configuration of a river monitoring system according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating the hardware configuration of a computer (server). [Figure 3] FIG. 3 is a block diagram illustrating an example of the functional configuration of the river monitoring system. [Figure 4] FIG. 4 is a diagram showing an example of a captured image and a normal label in a normal state. [Figure 5] FIG. 5 is a diagram showing an example of a pseudo-flood image and a pseudo-flood label. [Figure 6] FIG. 6 is a flowchart showing the flow of river monitoring processing in the river monitoring system. DETAILED DESCRIPTION OF THE INVENTION

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of a river monitoring system and a river monitoring method will be described in detail with reference to the accompanying drawings.

[0012] Fig. 1 is a block diagram showing the configuration of a river monitoring system 100 according to an embodiment. As shown in Fig. 1, in the river monitoring system 100, a captured image accumulation server 10, a label storage server 20, a learning image creation server 30, a learning server 40, a river flood detection device 50, and at least one river monitoring camera 60 are connected via a network 70. The network 70 is, for example, a LAN (Local Area Network), Ethernet (registered trademark), or the Internet.

[0013] Next, the hardware configuration of the computers (servers) that constitute the captured image storage server 10, the label storage server 20, the learning image creation server 30, the learning server 40, and the river flood detection device 50 will be described.

[0014] Here, Figure 2 is a diagram showing the hardware configuration of a computer (server). As shown in Figure 2, the computers (servers) constituting the captured image storage server 10, label storage server 20, learning image creation server 30, learning server 40, and river flood detection device 50 each include a control unit 101 such as a CPU (Central Processing Unit) that controls the entire device, a main memory unit 102 such as a ROM (Read Only Memory) or RAM (Random Access Memory) that stores various data and programs, an auxiliary memory unit 103 such as an HDD (Hard Disk Drive) or CD (Compact Disc) drive that stores various data and programs, and a bus 104 connecting these, and can be realized with a hardware configuration using a normal computer. The CPU has a clock function that measures the current time.

[0015] In addition, the captured image storage server 10, label storage server 20, learning image creation server 30, learning server 40, and river flood detection device 50 each perform unique functions due to differences in the programs stored in the auxiliary memory unit 103.

[0016] In addition, the control unit 101 is connected via a bus 104 to a display unit 105 that displays information, an operation input unit 106 such as a keyboard or mouse that accepts user instruction input, and a communication I / F (interface) 107 that controls communication with external devices.

[0017] Next, we will explain the various functions of the river monitoring system 100 that are realized in such a hardware configuration by the control units 101 of the captured image storage server 10, the label storage server 20, the learning image creation server 30, the learning server 40, and the river flood detection device 50 executing programs stored in the auxiliary memory unit 103.

[0018] 3 is a block diagram illustrating the functional configuration of the river monitoring system 100. As shown in FIG. 3, the river monitoring system 100 includes a captured image storage unit 110, a label storage unit 120, a learning image creation unit 130, a learning unit 140, and a river flood detection unit 150.

[0019] The captured image storage unit 110 is realized by the control unit 101 of the captured image storage server 10 executing a program stored in the auxiliary storage unit 103. The label storage unit 120 is realized by the control unit 101 of the label storage server 20 executing a program stored in the auxiliary storage unit 103. The training image creation unit 130 is realized by the control unit 101 of the training image creation server 30 executing a program stored in the auxiliary storage unit 103. The learning unit 140 is realized by the control unit 101 of the learning server 40 executing a program stored in the auxiliary storage unit 103. The river flood detection unit 150 is realized by the control unit 101 of the river flood detection device 50 executing a program stored in the auxiliary storage unit 103.

[0020] The captured image storage unit 110 of the captured image storage server 10 stores the video (captured images) acquired from the river monitoring camera 60. The captured images may be arbitrarily determined by the user and stored in the captured image storage unit 110 of the captured image storage server 10, or may be automatically stored in the captured image storage unit 110 of the captured image storage server 10 when certain conditions such as the time, weather, date, etc. are met.

[0021] The label storage unit 120 of the label storage server 20 generates and stores normal labels by extracting the water surface area for each captured image stored in the captured image storage unit 110 of the captured image storage server 10. It also predicts areas that will be flooded in the event of a river flood and generates and stores pseudo-flood labels by extracting the water surface area. The normal labels and pseudo-flood labels are generated automatically or manually by the label storage unit 120 of the label storage server 20.

[0022] Fig. 4 shows an example of a captured image and a normal label under normal circumstances. Fig. 4(a) shows a captured image acquired from the river monitoring camera 60, and Fig. 4(b) shows a normal label for the captured image shown in Fig. 4(a). As shown in Fig. 4(b), the normal label is obtained by extracting the river area of ​​the captured image acquired from the river monitoring camera 60 as a water surface area.

[0023] FIG. 5 shows an example of a pseudo-flood image and pseudo-flood labels created by the learning image creation server 30, which will be described later. FIG. 5(a) shows a pseudo-flood image created by the learning image creation server 30, which will be described later, and FIG. 5(b) shows a pseudo-flood label for the pseudo-flood image shown in FIG. 5(a). As shown in FIG. 5(b), the pseudo-flood label is an extracted water surface area where flooding is predicted in the event of a river flood. In the normal label shown in FIG. 4(b) and the pseudo-flood label shown in FIG. 5(b), the water surface area is represented by filling in white and the rest with black.

[0024] Pseudo-flood labels are labels that annotate areas where rivers are likely to overflow during heavy rain or typhoons, even though no actual flooding has occurred, and river areas as water surface areas. Flooding locations are predicted using past data or by simulating them using topographical data. One method for assigning pseudo-flood labels is to use a GIS (Geographic Information System) to estimate the approximate location of a photograph from a captured image. Then, based on the estimated location, pseudo-flood labels are assigned uniformly to areas that are lower than a predetermined elevation by more than a certain height from a pre-set point. Note that multiple pseudo-flood labels may be generated based on topographical data as well as the cause of flooding (typhoons, heavy rain, dam discharge, etc.) and its scale.

[0025] The training image creation unit 130 of the training image creation server 30 receives captured images from the captured image storage unit 110 of the captured image storage server 10. The training image creation unit 130 of the training image creation server 30 also receives pseudo-flood labels from the label storage unit 120 of the label storage server 20. The training image creation unit 130 of the training image creation server 30 then creates pseudo-flood images that resemble images of river flooding and are used for machine learning, from the captured images received from the captured image storage unit 110 of the captured image storage server 10 and the pseudo-flood labels at positions corresponding to the captured images received from the label storage unit 120 of the label storage server 20.

[0026] More specifically, the learning image creation unit 130 of the learning image creation server 30 creates a pseudo-flood image by performing color conversion on the white areas of the pseudo-flood labels in the photographed images stored in the photographed image storage unit 110 of the photographed image storage server 10 (for example, by adding a color that imitates flooding in the river part to a photographed image of a normal river).

[0027] Fig. 5(a) is a diagram showing an example of a pseudo-flood image. As shown in Fig. 5(a), the pseudo-flood image is created by adding colors that mimic flooding to a photographed image of a normal river.

[0028] The training image creation unit 130 of the training image creation server 30 defines the created pseudo-flood image as a "teaching image" and the pseudo-flood label used to create the pseudo-flood image as a "correct label," and associates these to create teaching data.

[0029] Thereafter, the training image creation unit 130 of the training image creation server 30 transmits the created training data to the learning unit 140 of the learning server 40, causing additional learning of the water surface detection model. This learning can improve the robustness of the river flood detection unit 150 of the river flood detection device 50.

[0030] The learning unit 140 of the learning server 40 retrains the water surface detection model using the pseudo-flood images and pseudo-flood labels that are the training data received from the learning image creation unit 130 of the learning image creation server 30. In addition, the learning image creation unit 130 of the learning server 40 replaces the retrained water surface detection model with the water surface detection model of the currently operating river flood detection device 50, thereby completing the retraining phase, which will be described later.

[0031] The river flood detection unit 150 of the river flood detection device 50 determines the location of river flooding using video (photographed images) acquired from the river monitoring camera 60 and a water surface detection model trained by deep learning.

[0032] Deep learning is a type of machine learning method that uses a multilayer neural network (deep neural network) that mimics the neurons in the human brain, and is a method that can automatically learn the characteristics of data. In this embodiment, it is supervised deep learning that uses training data. Training data is a pair of a "training image" and a "correct label" that represents the correct classification. In this embodiment, the photographed image and the pseudo-flood image are considered "training images," and the normal label and pseudo-flood label that indicate the water surface area of ​​each image are considered "correct labels."

[0033] In its initial state, the river flood detection unit 150 of the river flood detection device 50 acquires, as a "teacher image," a video (photographed image) of the flow of a river under normal conditions captured by the river monitoring camera 60. Then, based on a water surface detection model (machine learning model) that has learned the features of the river flow using deep learning based on the captured image, which is the "teacher image," it infers the location of the water surface from the video (photographed image) captured by the river monitoring camera 60 and detects whether or not there is river flooding.

[0034] Next, the river monitoring process in the river monitoring system 100 will be described.

[0035] Fig. 6 is a flowchart showing the flow of river monitoring processing in the river monitoring system 100. As shown in Fig. 6, the river monitoring system 100 asynchronously operates an inference phase and a relearning phase when executing the river monitoring processing. When the relearning phase ends, the river monitoring system 100 replaces the water surface detection model of the river flood detection unit 150 of the river flood detection device 50 that is currently operating with the water surface detection model after relearning.

[0036] First, the re-learning phase will be described.

[0037] First, the captured image storage unit 110 of the captured image storage server 10 stores the video (captured image) acquired from the river monitoring camera 60 (step S1).

[0038] Next, the label storage unit 120 of the label storage server 20 generates and stores normal labels and pseudo-flood labels for river images to be assigned to the photographed images stored in the photographed image storage unit 110 of the photographed image storage server 10 (step S2).

[0039] Next, the learning image creation unit 130 of the learning image creation server 30 creates pseudo-flood images to be used for learning from the captured images received from the captured image storage unit 110 of the captured image storage server 10 and the pseudo-flood labels received from the label storage unit 120 of the label storage server 20 (step S3). More specifically, the learning image creation unit 130 of the learning image creation server 30 creates pseudo-flood images by performing color conversion (for example, by applying a color that imitates the flooded part of the river to a captured image of a normal river) on the areas corresponding to the water surface areas (white areas shown in Figure 6(b)) indicated by the pseudo-flood labels for the captured images stored in the captured image storage unit 110 of the captured image storage server 10.

[0040] The specific method for color conversion on the pseudo-flood label is to use the Lab color space. Specifically, for each pixel in the image at coordinates (x, y), only the part corresponding to the water surface area indicated by the pseudo-flood label is color converted using the following conversion formula:

[0041]

number

[0042] The learning image creation unit 130 of the learning image creation server 30 regards the created pseudo-flood image as a "teacher image" and the pseudo-flood label used to create the pseudo-flood image as a "correct label," and transmits the teacher data that corresponds these to the learning unit 140 of the learning server 40.

[0043] Next, the learning unit 140 of the learning server 40 receives the training data from the learning image creation unit 130 of the learning image creation server 30, and re-learns the water surface detection model (step S4).

[0044] In addition, the learning unit 140 of the learning server 40 ends the re-learning phase by replacing the re-learned water surface detection model with the water surface detection model of the river flood detection unit 150 of the currently operating river flood detection device 50 (step S5).

[0045] In this embodiment, the label storage unit 120 of the label storage server 20 generates a normal label and a pseudo-flood label for each captured image stored in the captured image storage unit 110 of the captured image storage server 10, but this is not limited to this. For example, if the shooting direction of the river monitoring camera 60 is fixed, the label storage unit 120 of the label storage server 20 may estimate the shooting location of the river monitoring camera 60 and the parts of the shooting location that are likely to be flooded due to a rise in the river water level, etc., when the river monitoring camera 60 is installed, and generate and store normal labels and pseudo-flood labels. This eliminates the need to generate normal labels and pseudo-flood labels for each captured image.

[0046] Furthermore, the captured image storage unit 110 of the captured image storage server 10 may be configured to automatically store images (captured images) acquired from the river monitoring camera 60 when certain conditions, such as time, weather, date, etc., are met. This makes it possible to acquire captured images under different shooting conditions, and to create a large amount of training data under different conditions.

[0047] Next, the inference phase will be described.

[0048] The river flood detection unit 150 of the river flood detection device 50 detects, at a predetermined timing, locations where a water surface exists from the video (photographed image) captured by the river monitoring camera 60 based on the water surface detection model (step S11).

[0049] According to this embodiment, by detecting locations where a water surface exists from a captured image based on a water surface detection model (machine learning model) created by learning pseudo-flood images that resemble images of river flooding, the accuracy of the machine learning model used for river monitoring is improved, making it possible to accurately determine flood locations when a river floods.

[0050] In addition, the programs executed by the captured image storage server 10, label storage server 20, learning image creation server 30, learning server 40 and river flood detection device 50 in this embodiment are provided as files in an installable or executable format recorded on a computer-readable recording medium such as a CD-ROM, flexible disk (FD), CD-R or DVD (Digital Versatile Disc).

[0051] The programs executed by the captured image storage server 10, label storage server 20, learning image creation server 30, learning server 40, and river flood detection device 50 of this embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The programs executed by the captured image storage server 10, label storage server 20, learning image creation server 30, learning server 40, and river flood detection device 50 of this embodiment may be provided or distributed via a network such as the Internet.

[0052] Furthermore, the river monitoring system 100 of this embodiment is equipped with a captured image storage server 10, a label storage server 20, a learning image creation server 30, a learning server 40, and a river flood detection device 50, but is not limited to this, and all of the functions of the captured image storage unit 110, the label storage unit 120, the learning image creation unit 130, the learning unit 140, and the river flood detection unit 150 may be realized by a single device, or several functions may be combined and realized by a single device. [Explanation of symbols]

[0053] 100 River Monitoring System 110 Photographed image storage unit 120 Label Storage Unit 130 Learning image creation unit 140 Learning Department 150 River flood detection unit

Claims

1. a learning image creation unit that creates pseudo-flood images that resemble images of river floods based on pseudo-flood labels that indicate areas that are predicted to be flooded when rivers flood, and on captured images; a learning unit that associates the pseudo-flood images created by the learning image creation unit with pseudo-flood labels to use them as training data and re-trains a machine learning model for water surface detection; a river flood detection unit that infers locations where water surfaces exist from photographed images based on the machine learning model retrained by the learning unit and detects whether or not a river is flooding; A river monitoring system comprising:

2. the learning image creation unit performs color conversion on areas in the captured images where flooding is predicted and which are indicated by the pseudo-flood labels; 2. The river monitoring system according to claim 1.

3. the learning image creation unit performs color conversion on the captured image to a color that imitates a flooded river portion; 3. The river monitoring system according to claim 2.

4. A river monitoring method in a river monitoring system, comprising: a learning image creation process for creating pseudo-flood images that resemble images of river floods based on pseudo-flood labels that indicate areas that are predicted to be flooded when a river floods and on photographed images; a learning process in which the pseudo-flood images created in the learning image creation process are associated with pseudo-flood labels to be used as training data, and a machine learning model for water surface detection is retrained; a river flood detection step of inferring locations where water surfaces exist from photographed images based on the machine learning model retrained in the learning step and detecting whether or not a river is flooding; A river monitoring method comprising:

Images