Image analysis device and image analysis method

Abstract

This image analysis device enables low-cost monitoring of water surface areas, thereby improving cost-effectiveness and allowing for easy and stable identification of water surface areas. [Solution] The image analysis device 10 includes an observation image data acquisition unit (step ST1) that acquires observation image data of the water surface at one or more observation points set in a body of water, a detection target extraction unit (step ST3) that extracts detection target area data from the acquired observation image data, and a determination processing unit (step ST4) that performs a determination processing of the boundary position between the detection target area and the non-detection target area based on the extracted result image data including the extracted detection target area data.

Description

Technical Field

[0001] The present invention relates to an image analysis device and an image analysis method.

Background Art

[0002] As is well known, for example, in rivers and the like, it is necessary to monitor inundation, illegal acts or nuisance acts on riverbanks, so monitoring systems are often provided. In this type of monitoring system, a plurality of monitoring cameras are installed near the river, and videos captured by these monitoring cameras are centrally monitored at a monitoring center.

[0003] By the way, when the risk of river inundation increases due to typhoons or concentrated torrential rains, river administrators stationed at the monitoring center are forced to appropriately select or organize diverse and enormous information obtained from a large number of monitoring cameras, and make quick and accurate judgments. This increases the time and effort of the river administrators. In addition, it is difficult for river administrators to constantly visually check the monitor images from a large number of installed monitoring cameras.

[0004] Accordingly, as a river monitoring technique for detecting river flooding, inland water flooding, and the like, a convolutional neural network (CNN: transfer learning) is widely used using monitor images from monitoring cameras (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of Invention

Problems to be Solved by the Invention

[0006] By the way, in the technology described in Patent Document 1 above, in order to create a large amount of training data, the labeling work necessary for determining water bodies, that is, the work of coloring the detection targets (for example, water bodies) in the image, is performed manually. As a result, it takes an enormous amount of effort and time to complete the creation of training data, and the cost is also high.

[0007] Furthermore, the image quality of surveillance camera footage varies depending on the weather and meteorological conditions at the time of shooting, which can lead to false detections in real-world operating environments. In addition, while the method used to determine the boundaries of detected objects relies on a threshold of the probability of detection, there are challenges in determining the accuracy of the determination under arbitrary shooting conditions.

[0008] Therefore, the present invention has been made in view of the above-mentioned unresolved conventional problems, and aims to provide an image analysis device and image analysis method that enable low-cost monitoring of the water surface area, thereby improving cost-effectiveness, and enabling easy and stable identification of the water surface area. [Means for solving the problem]

[0009] An image analysis device according to one embodiment of the present invention comprises: an observation image data acquisition unit that acquires observation image data of the water surface at one or more observation points set in a body of water; a detection target extraction unit that extracts detection target area data from the observation image data acquired by the observation image data acquisition unit; and a determination processing unit that performs a determination process of the boundary position between the detection target area and the non-detection target area based on the extracted result image data including the detection target area data extracted by the detection target extraction unit.

[0010] Furthermore, an image analysis apparatus according to another embodiment of the present invention includes: an observation image data acquisition unit that acquires observation image data of the water surface at one or more observation points set in a body of water; a first image processing unit that generates image data with a resolution higher than the resolution of the observation image data from the observation image data acquisition unit; a second image processing unit that corrects the brightness value in the image data output from the first image processing unit; a detection target extraction unit that extracts detection target area data from the output image data of the second image processing unit; and a determination processing unit that performs a determination process of the boundary position between the detection target area and the non-detection target area based on the extracted result image data including the detection target area data extracted by the detection target extraction unit.

[0011] Furthermore, an image analysis apparatus according to another embodiment of the present invention includes: an observation image data acquisition unit that acquires observation image data of the water surface at one or more observation points set in a body of water; a detection target extraction unit that extracts detection target area data from the observation image data acquired by the observation image data acquisition unit; a boundary position candidate generation unit that calculates a first boundary position candidate between the detection target area and the non-detection target area under a first condition and a second boundary position candidate between the detection target area and the non-detection target area under a second condition different from the first condition, based on the extracted result image data including the detection target area data extracted by the detection target extraction unit; and a determination processing unit that performs a determination processing of the final boundary position between the detection target area and the non-detection target area based on the correlation of two areas divided by the first boundary position candidate calculated by the boundary position candidate generation unit and the correlation of two areas divided by the second boundary position candidate.

[0012] Furthermore, in another embodiment of the present invention, the image analysis method acquires observational image data of the water surface at one or more observation points set in a body of water, extracts detection target area data from the acquired observational image data, and performs a determination process for the boundary position between the detection target area and the non-detection target area based on the extracted image data including the extracted detection target area data.

[0013] Furthermore, in another embodiment of the present invention, the image analysis method acquires observational image data of the water surface at one or more observation points set in a body of water, generates image data with a higher resolution than the resolution of the acquired observational image data, performs brightness value correction on the generated image data, extracts detection target area data from the image data with brightness value correction, and performs boundary position determination processing between the detection target area and the non-detection target area based on the extracted result image data including the extracted detection target area data.

[0014] Furthermore, an image analysis method according to another embodiment of the present invention acquires observational image data of the water surface at one or more observation points set in a body of water, extracts detection target area data from the acquired observational image data, calculates a first boundary position candidate between the detection target area and the non-detection target area under a first condition and a second boundary position candidate between the detection target area and the non-detection target area under a second condition different from the first condition based on the extracted image data including the extracted detection target area data, and performs a final determination process of the boundary position between the detection target area and the non-detection target area based on the correlation between the two areas divided by the first boundary position candidate and the correlation between the two areas divided by the second boundary position candidate. [Effects of the Invention]

[0015] According to one aspect of the present invention, it is possible to perform monitoring of the water surface area at low cost, thereby improving cost-effectiveness and enabling easy and stable identification of the water surface area. [Brief explanation of the drawing]

[0016] [Figure 1] This is a block diagram showing an example of a river image analysis device according to one embodiment of the present invention. [Figure 2] This flowchart shows the control procedure executed by a control unit according to one embodiment of the present invention, based on a river image analysis program. [Figure 3] This is an explanatory diagram for illustrating the operation of the present invention. [Figure 4]This is a flowchart illustrating a control procedure executed by a control unit in Comparative Example 1 of one embodiment. [Figure 5A] This is a block diagram illustrating the processing configuration of a CNN during execution of image processing in Comparative Example 1. [Figure 5B] This is a block diagram illustrating the processing configuration of a CNN during learning in Comparative Example 1. [Figure 6A] This is a diagram illustrating an example of an image prepared as a learning image for the CNN in Comparative Example 1. [Figure 6B] This is a diagram illustrating an example of an image obtained by labeling the learning image in Comparative Example 1. [Figure 6C] This is a diagram illustrating an example of an image input to the input layer of the CNN in Comparative Example 1. [Figure 6D] This is a diagram illustrating an example of an image output from the output layer of the CNN in Comparative Example 1. [Figure 6E] This is a diagram illustrating an example of a correct image to be compared with the output image in the CNN of Comparative Example 1. [Figure 7] This is a diagram (Part 1) illustrating an example of an image for explaining the problem caused by the CNN in Comparative Example 1. [Figure 8] This is a diagram (Part 2) illustrating an example of an image for explaining the problem caused by the CNN in Comparative Example 1. [Figure 9] This is a diagram (Part 3) illustrating an example of an image for explaining the problem caused by the CNN in Comparative Example 1. [Figure 10] This is a flowchart illustrating the detailed processing of step ST2 shown in Figure 2. [Figure 11A] This is a diagram for explaining an example of an image with block-shaped noise. [Figure 11B] This is a diagram for explaining an example of an image in which block-shaped noise is reduced. [Figure 12A] This is a diagram for explaining an example of an unclear image. [Figure 12B] This is a diagram for explaining an example of a sharpened image. [Figure 13]Figure 2 is a flowchart showing a detailed example of the process in step ST3. [Figure 14] This is a block diagram illustrating an image matting method, which is one embodiment of the present invention. [Figure 15] This figure shows an example of an image processed by an image matting method, which is one embodiment of the present invention. [Figure 16A] This is Figure (1) showing an example of an image processed by the SAM method, which is one embodiment of the present invention. [Figure 16B] This is Figure (2) showing an example of an image processed by the SAM method, which is one embodiment of the present invention. [Figure 17] This is a diagram illustrating entropy-based image processing, which is one embodiment of the present invention. [Figure 18] This figure illustrates an example of an image processed by entropy, which is one embodiment of the present invention. [Figure 19] Figure 2 is a flowchart showing the detailed processing of step ST4. [Figure 20A] This is a diagram (part 1) showing an example of an image for performing synthesis method 1, which is one embodiment of the present invention. [Figure 20B] This is a diagram (part 2) showing an example of an image for performing synthesis method 1, which is one embodiment of the present invention. [Figure 20C] This is Figure (3) showing an example of an image for performing synthesis method 1, which is one embodiment of the present invention. [Figure 20D] This is Figure (4) showing an example of an image for performing synthesis method 1, which is one embodiment of the present invention. [Figure 21A] This is a diagram (part 1) showing an example of an image for performing synthesis method 2, which is one embodiment of the present invention. [Figure 21B] This is a diagram (part 2) showing an example of an image for performing synthesis method 2, which is one embodiment of the present invention. [Figure 21C] This is Figure (3) showing an example of an image for carrying out synthesis method 2, which is one embodiment of the present invention. [Figure 21D] This is Figure (4) showing an example of an image for performing synthesis method 2, which is one embodiment of the present invention. [Figure 22] This figure shows a processing flow for executing synthesis method 4, which is one embodiment of the present invention. [Figure 23] This figure shows an example of an image used to perform a projection transformation, which is one embodiment of the present invention. [Figure 24] This figure illustrates how the boundary position between land and water is determined by a threshold in Comparative Example 2 of one embodiment. [Figure 25A] This figure shows an example of an image before contrast adjustment, which is one embodiment of the present invention. [Figure 25B] This figure shows an example of an image after contrast adjustment, which is one embodiment of the present invention. [Figure 26] This is a diagram illustrating Fisher's evaluation criteria, which is one embodiment of the present invention. [Figure 27] This diagram illustrates how a second amphibious boundary candidate was determined using Fisher's evaluation criteria, which is one embodiment of the present invention. [Figure 28] This figure shows an example of an image obtained by further adjusting the contrast of the image shown in Figure 27. [Figure 29] This figure illustrates how the first candidate for the land-sea boundary was determined based on the image shown in Figure 28. [Figure 30] This figure shows an example of an image processed by the final selection from candidates for the land-sea boundary, which is one embodiment of the present invention. [Modes for carrying out the invention]

[0017] Embodiments of the present invention will be described below with reference to the drawings. However, the embodiments described below are merely illustrative, and there is no intention to exclude various modifications or applications of techniques not explicitly stated below. The present invention can be implemented with various modifications (for example, by combining the embodiments) without departing from its spirit. Furthermore, the embodiments shown below illustrate devices and methods for realizing the technical concept of the present invention, and the technical concept of the present invention is not limited to the arrangement of components as described below. The technical concept of the present invention can be modified in various ways within the technical scope defined by the claims described in the patent claims.

[0018] <One Embodiment> <Configuration of the river image analysis system> Figure 1 is a block diagram showing an example of a river image analysis device 10 according to one embodiment of the present invention. The river image analysis device 10 (an example of an image analysis device) is composed of, for example, a personal computer and comprises a control unit 11, an input unit 12, a display unit 13, an input interface unit 14 (referred to as input I / F unit 14 in Figure 1), and a storage unit 15.

[0019] The control unit 11 inputs observational image data of the river surface observed at one or more river water level observation stations 20 installed along the river at a predetermined timing, such as a fixed period, and stores it in the storage unit 15. The river water level observation station 20 includes, for example, a surveillance camera 20a that photographs the river surface and a communication unit 20b that transmits the observational image data of the river surface captured by the surveillance camera 20a via wireless communication or the like. The control unit 11 communicates with each communication unit 20b via the input interface unit 14 and collects the observational image data. The control unit 11 may also input observational image data of the river surface observed at one river water level observation station 20 at a predetermined timing, such as a fixed period, and store it in the storage unit 15.

[0020] This section describes how to obtain observational image data from river water level observation stations 20, but it is sufficient to obtain observational image data from pre-configured river water level observation stations 20. For example, it may be obtained from information containing observational image data distributed by public institutions, observational image data disseminated by the general public through social media and the internet, or information transmitted by private businesses such as broadcasting stations. Alternatively, for example, observational image data obtained from river water level observation stations 20 may be combined with observational image data obtained from information transmitted by private businesses.

[0021] The control unit 11 then displays the boundary position between the water surface and land on the display unit 13 based on the current observation image data at each river water level observation station 20 stored in the memory unit 15. The control unit 11 also displays a danger level alert on the display unit 13 based on the boundary position between the water surface and land. The input unit 12 is used to input various operation instructions to the control unit 11, and is also used for inputting trimmap image data for masking areas not to be detected in the observation image data, and for inputting contrast adjustment values ​​in the observation image data.

[0022] The memory unit 15 stores the river image analysis program necessary for calculation processing in the control unit 11, as well as various parameters acquired during the calculation process. The memory unit 15 is also equipped with a trimmap image storage unit 151 that stores trimmap image data for each of the river water level observation stations 20. Furthermore, the control unit 11 may acquire observational image data of the river water surface observed at one or more river water level observation stations 20 installed along the river, as well as observational image data of locations where water surfaces are constantly present, such as coastlines, reservoirs, lakes, dams, floodplains, waterways, pipelines, water tanks, wells, rice paddies, and offshore facilities, and observational image data of one or more water level observation stations installed on roads and underpasses that may be flooded during floods. In this case, the control unit 11 displays the boundary position between the water surface and the land / wall on the display unit 13, and displays an alert on the display unit 13 indicating the degree of danger based on the boundary position between the water surface and the land / wall.

[0023] <Operation> Next, an example of the operation of the river image analysis device 10 will be described. Figure 2 is a flowchart showing the control procedure executed by the control unit 11 based on the river image analysis program.

[0024] The control unit 11 first requests the transmission of observation image data from one or more river water level observation stations 20 installed along the river, for example as shown in Figure 3, and acquires the observation image data observed at each river water level observation station 20. Then, it stores the acquired observation image data in the storage unit 15 (step ST1) (an example of an observation image data acquisition unit).

[0025] The control unit 11 then proceeds to step ST2, where it performs preprocessing on the observed image data to reduce noise and ambiguity that could hinder subsequent analysis (an example of the first and second image processing units).

[0026] The control unit 11 moves to step ST3 and extracts water surface region data, which is the region to be detected, from the image data that has been preprocessed in step ST2 (an example of a detection target extraction unit). Then, the control unit 11 moves to step ST4 and performs a determination process for the boundary position between the water surface region and the land region based on the extracted image data, which includes the water surface region data extracted in step ST3 (an example of a determination processing unit).

[0027] Thereafter, the control unit 11 displays the boundary position between the water surface area and the land area, i.e., the water level, determined in step ST4, on the display unit 13 for each river water level observation station 20, and also displays an alert on the display unit 13 indicating the degree of danger based on the boundary position between the water surface area and the land area determined in step ST4 (step ST5). This alert display data is edited to make the degree of danger easily recognizable to the user, and consists of, for example, a mark representing the degree of danger or a concise explanatory text. By repeatedly executing the series of processes shown in Figure 2 at a predetermined period, the current river water level is displayed in real time, and the degree of danger at the river water level is displayed as an alert in real time. Note that the alert display data may also be an audible notification of the degree of danger, in addition to being a visual display.

[0028] <Comparative Example 1 of One Embodiment> Next, a previously conceived river image analysis technology will be described as Comparative Example 1, representing one embodiment. Figure 4 is a flowchart showing the control procedure performed by the control unit in Comparative Example 1.

[0029] When the control unit receives observed river surface image data (Step STB1), it detects the water surface region data using a trained CNN (Convolutional Neural Network) (Step STB2). Then, it projectively transforms the extracted image data, which includes the detected water surface region data, from image data of the water surface region viewed from diagonally above to image data of the boundary between land and water viewed from the front (Step STB3). As will be described in detail in Comparative Example 2, the control unit then determines the boundary position between land and water using a threshold based on the projectively transformed image data (Step STB4).

[0030] Thereafter, the control unit displays the boundary position between the water surface area and the land area, i.e., the water level, determined in step STB5, for each river water level observation station 20, and also displays a danger level alert on the display unit based on the boundary position between the water surface area and the land area determined in step STB4 (step STB5).

[0031] The trained CNN in step STB2 described above consists of an input layer B31, a convolutional layer B32, a pooling layer B33, a deconvolutional layer B34, and an output layer B35, as shown in Figure 5A. The observed image data shown in Figure 6C, input to the input layer B31, is convolved in the convolutional layer B32 using the given parameters to generate an intermediate feature map. This intermediate feature map is then subjected to average pooling in the pooling layer B33 to generate a feature map. This feature map is then deconvolved in the deconvolutional layer B32 and output from the output layer B35 as extracted image data including water surface region data, as shown in Figure 6D.

[0032] As shown in Figure 5B, the CNN used for training is configured by adding a comparison unit B36, a loss calculation unit B37, and a parameter update unit B38 to the configuration shown in Figure 5A. In order to train the CNN shown in Figure 5B, the user needs to prepare several thousand training image data as shown in Figure 6A and label the water surface regions of the several thousand training image data as shown in Figure 6B. In Figure 6B, the labeled water surface regions are the areas indicated by the shaded areas in Figure 6B.

[0033] The extracted image data shown in Figure 6D, output from output layer B35, is compared by comparison unit B36 with the labeled ground truth image data shown in Figure 6E. In Figure 6E, the labeled water surface region is the area indicated by the diagonal lines in Figure 6E. The loss calculation unit B37 calculates a loss function for the ground truth image based on the comparison result from comparison unit B36 and outputs the calculation result to parameter update unit B38. The parameter update unit B38 updates each parameter using an optimization function based on the loss function for the ground truth image and outputs the updated parameters to convolutional layer B32.

[0034] <Problem 1 based on Comparative Example 1> Incidentally, while the CNN in Comparative Example 1 demonstrates a certain level of performance in recognizing spatial patterns, as shown in Figure 7, it has a challenge in recognizing the boundary between the water surface and land (waterfront), which becomes a source of error in water level calculation. Figure 7(a) is an observation image of a river taken by a surveillance camera. Figure 7(b) is an observation image enlarged from the area within the square frame in the river observation image shown in Figure 7(a). In the area within the square frame shown in Figure 7(a), the error in the boundary position between the water surface and land is a difference of a few pixels. However, a difference of a few pixels can result in an error of several tens of centimeters to meters, as shown by the arrow in Figure 7(b).

[0035] <Problem 2 based on Comparative Example 1> Furthermore, the water surface conditions (patterns) captured by the surveillance cameras naturally differ between normal water levels (as shown in Figure 8(a)) and high water levels (as shown in Figure 8(b)). While it is necessary to include flow conditions during high water levels when training the CNN, collecting images under diverse sunlight conditions is difficult.

[0036] <Problem 3 based on Comparative Example 1> Even at the same location or angle of view, the way the water surface is reflected changes depending on sunlight conditions, such as the angle of the rising or setting sun, and whether it's daytime or nighttime. Furthermore, even at the same time of day, the sun's altitude is not uniform across seasons.

[0037] In Comparative Example 1, the CNN misdetects areas other than the water surface (the area enclosed by the square frame in Figure 9(a)) due to differences in sunlight conditions, as shown in Figure 9(a). Furthermore, as shown in Figure 9(b), the CNN in Comparative Example 1 also misdetects areas other than the water surface, such as puddles (the area enclosed by the square frame in Figure 9(b)). Moreover, as shown in Figure 9(c), the CNN in Comparative Example 1 fails to detect anything at night. Thus, it is extremely difficult to train a CNN to be applicable to any given time (shooting conditions).

[0038] <Solution in one embodiment> <Step ST2: Pre-treatment process> In one embodiment, first, a process is performed to remove noise and ambiguity that would hinder the analysis. Figure 10 is a flowchart showing the detailed process of step ST2 described above. The preprocessing step of step ST2 described above consists of a process to improve image quality and reduce noise (step ST21) (an example of a first image processing unit) and a process to sharpen the image by equalizing the histogram (step ST22) (an example of a second image processing unit).

[0039] The observational image data shown in Figure 11A, acquired in step ST1 above, contains blocky noise. Figure 11A(1) shows the observational image with blocky noise (the square framed area in Figure 11A(1)), and Figure 11A(2) shows a magnified image of the blocky noise. Observational images captured by the surveillance camera 20a are often compressed before transmission or storage to reduce communication capacity. Image capacity compression can degrade image quality and result in blocky or mosaic-like noise.

[0040] In step ST21 described above, the observed image data shown in Figure 11A is input, and image processing is performed using the trained Real-ESRGAN to generate the high-resolution image data shown in Figure 11B. Real-ESRGAN is a super-resolution model. Figure 11B(1) shows the observed image with reduced blocky noise (the square framed area in Figure 11B(1)), and Figure 11B(2) shows a magnified image of the area where the blocky noise has been reduced.

[0041] During training, Real-ESRGAN uses a degraded image (an example of high-resolution image data) on the output side as the training image data (an example of low-resolution image data) to create a blurred image, which is then used as the input side training image data. Here, the input side training image data and the output side training image data are the training image data input to Real-ESRGAN and the training image data that should be output by Real-ESRGAN during training, respectively. During training, Real-ESRGAN learns to obtain an image that is close to the output side training image data by taking the input side training image data as input and performing image processing on it.

[0042] In step ST21 above, Real-ESRGAN can achieve high resolution while reducing block noise.

[0043] In step ST22 described above, the image data shown in Figure 12A, output from step ST21, is corrected for brightness using histogram equalization, and the image data shown in Figure 12B is output. Images taken in rainy weather may have fog (haze) and reduced visibility, as shown in Figure 12A. In addition, images may become unclear depending on shooting conditions such as nighttime.

[0044] Therefore, in step ST22, the image data output from step ST21 is subjected to histogram equalization to enhance color changes, making it easier for AI (artificial intelligence) to recognize.

[0045] <Step ST3: Extraction process of the water surface area> In one embodiment, the characteristics of the reflection (color pattern) on the water surface are captured to estimate the water surface area and support the subsequent process of determining the land-water boundary position. The water surface reflects various things due to various influences such as dropping objects into the water, reflections of land objects on the water surface, and ripples in turbid water or currents. Until now, water surface areas have been extracted using AI technology with CNNs, but the training process is costly. Therefore, in one embodiment, a combination of methods that minimize costs is used to determine the water surface area.

[0046] Figure 13 is a flowchart showing an example of the detailed processing in step ST3 described above. The water surface region extraction process in step ST3 consists of detection by image matting (step ST31), detection by SAM (Segmentation Anything Model) (step ST32), and image contour enhancement by entropy (step ST33). These three processes—detection by image matting (step ST31), detection by SAM (Segmentation Anything Model) (step ST32), and image contour enhancement by entropy (step ST33)—are executed in parallel and simultaneously.

[0047] <Step ST31: Image Matting Method> Image matting is a commonly used method in image synthesis, where objects are extracted from images. Image matting allows for the separation of the background from the object, and by focusing on the land-water boundary, it can extract the water surface region with high versatility.

[0048] Figure 14 is a block diagram of the image processing that realizes the image matting method. To realize the image matting method, the control unit 11 is equipped with a comparison and matching unit 111. The comparison and matching unit 111 generates extracted result image data (output image (C) shown in Figure 15) that includes the extracted water surface region by, for example, comparing the image data output from step ST22 (original image (A) shown in Figure 15) with the trimmap image data stored in the trimmap image storage unit 151 (trimap image (B) shown in Figure 15).

[0049] The trimmap image data stored in the trimmap image storage unit 151 is an image used to mask at least the land area of ​​the original image (A). The user operates the input unit 12 to register a single trimmap image data in the trimmap image storage unit 151, in which the same position as the river water surface identified in the original image (A) during low water is set to white as a virtual water surface area, the area identified as land during high water is set to black, and the remaining area is set to gray. In this case, the trimmap image (B) is registered in association with identification information that identifies the river water level observation station 20. In the output image (C) shown in Figure 15, the whiter (luminance value: 255), the more it is determined to be the foreground (water surface).

[0050] <Step ST32: SAM Method> The SAM method is commonly used in the field of image synthesis to extract objects from images. According to the SAM method, the background and the object can be separated, and by focusing on the land-water boundary, the water surface region can be extracted with high versatility.

[0051] Figure 16A shows an image in which the coordinates to be extracted from the water surface region of the observed image have been specified. Figure 16B shows an image in which the detection target region has been extracted by the SAM method.

[0052] The user operates the input unit 12 to specify the coordinates of the region to be separated from the water surface region to be extracted from the observation image shown in Figure 16A. In this case, for example, five coordinates P1(X1,Y1,Z1), P2(X2,Y2,Z2), P3(X3,Y3,Z3), P4(X4,Y4,Z4), and P5(X5,Y5,Z5) are specified. Then, the control unit 11 separates the water surface region containing the specified coordinates using the SAM method and extracts the region of the separated detection target object OB1 (the region shown by the diagonal lines in Figure 16B). Note that the SAM method outputs either an object (1) or an object (0).

[0053] <Step ST33: Image contour enhancement using entropy> Water surfaces generally exhibit monotonous patterns, while land areas often present more chaotic patterns. Therefore, entropy is used to determine the degree of randomness in color patterns.

[0054] The determination of vegetation areas based on entropy is performed using Equation 1 below.

number

[0055] As shown in Figure 17(a), an image reflecting the surface of water has low entropy because it is flat and featureless. As shown in Figure 17(b), an image reflecting plants and trees has high entropy because it is cluttered and featureless.

[0056] In step ST33, the control unit 11 generates extracted image data (Figure 18(b)) with the contour of the water surface region emphasized by entropy, based on the image data output from step ST22 (Figure 18(a)).

[0057] <Step ST4: Determination of the land-sea boundary position> In one embodiment, the water surface region was extracted in the previous step (evaluated for the likelihood of it being a water surface region). This is spatial information. Since the final desired result is the water level (scalar value; a single numerical value), the position of the land-water boundary is determined.

[0058] Figure 19 is a flowchart showing the detailed processing of step ST4 described above. The land-water boundary position determination process of step ST4 described above consists of a synthesis process (step ST41) (an example of an image synthesis unit), a projection transformation process (step ST42) (an example of a projection transformation unit), a first contrast adjustment process (step ST431) (an example of a first contrast adjustment unit) and a second contrast adjustment process (step ST432) (an example of a second contrast adjustment unit) executed in parallel, detection of land-water boundary candidate by applying the first Fisher evaluation criterion (step ST441) and detection of land-water boundary candidate by applying the second Fisher evaluation criterion (step ST442) (an example of a boundary position candidate generation unit), and final selection from land-water boundary candidates (step ST45) (an example of a determination processing unit).

[0059] <Step ST41: Synthesis Process> The synthesis process combines the images extracted using multiple methods in the previous step into a single image for subsequent processing. One of the synthesis methods used is two image matting methods (an example of multiple detection target extraction methods).

[0060] (Synthesis Method 1) In step ST31 described above, the comparison and matching unit 111 of the control unit 11 compares the image data output from step ST22 (the surveillance camera image shown in Figure 20A) with the trimmap image data stored in the trimmap image storage unit 151 (the trimmap image shown in Figure 20B(1)), thereby generating extracted image data including the extracted water surface region (the output result shown in Figure 20C(1)).

[0061] Furthermore, the comparison and matching unit 111 of the control unit 11 compares the image data output from step ST22 (the surveillance camera image shown in Figure 20A) with the trimmap image data stored in the trimmap image storage unit 151 (the trimmap image shown in Figure 20B(2)) to generate extracted image data including the extracted water surface region (the output result shown in Figure 20C(2)). Note that the trimmap image shown in Figure 20B(2) is a trimmap image used to mask the water surface region by inverting the water surface region and land region of the trimmap image shown in Figure 20B(1).

[0062] Then, in step ST41, the control unit 11 combines the extracted image data (output result shown in Figure 20C(1)) and the extracted image data (output result shown in Figure 20C(2)) to generate the output image shown in Figure 20D. In the output image shown in Figure 20D, the water surface boundary is clearly defined.

[0063] (Synthesis Method 2) In synthesis method 2, for example, the extracted image data obtained by the image matting method in step ST31, the extracted image data obtained by the SAM method in step ST32, and the extracted image data obtained by the entropy-based determination in step ST33 are combined. Here, if you want to change the weights for each method, you can also combine them by multiplying or adding coefficients as shown in equation 2 below (weighted average).

number

[0064] (Synthesis Method 3) In synthesis method 3, for example, the extracted image data obtained by the image matting method in step ST31 is combined with the extracted image data obtained by the entropy determination in step ST33.

[0065] The control unit 11 obtains extracted image data by masking the water surface region shown in Figure 21B using the image matting method in step ST31 from the surveillance camera image data shown in Figure 21A, and also obtains extracted image data shown in Figure 21C by determining by entropy in step ST33. If there is a tendency for undetected areas (incorrectly identified as land) to occur in one of the multiple methods, the control unit 11 can correct the undetected areas by choosing the darker area, as shown in equation 3 below.

number

[0066] (Synthesis Method 4) In synthesis method 4, for example, as shown in Figure 22, the extracted image data obtained by the image matting method in step ST31a, the extracted image data obtained by the image matting method in step ST31b, and the extracted image data obtained by the entropy determination in step ST33 are combined.

[0067] Furthermore, a combination of images (resulting images from different methods) is treated as synthesis method 3 (a method that takes the darker image), and the resulting composite image is then combined with the results from yet another method to perform synthesis method 1 (weighted average).

[0068] <Step ST42: Projection Transformation Processing Process> The projection transformation process converts the composite water surface detection result image data obtained in step ST41 into image data that shows the boundary between the water surface region and the land region as if viewed from the front. In other words, the rectangular area of ​​the water surface detection result image shown in Figure 23(a) is converted into image data shown in Figure 23(b) as if viewed from the front. Therefore, as shown in Figure 23(c), it is possible to estimate how many pixels from the top in the image space the boundary position WA1 between the water surface and the land is located at.

[0069] <Comparative Example 2 of One Embodiment> Next, the determination of the boundary position between land and water surface using the threshold in step STB4 described above will be explained as Comparative Example 2 of one embodiment. In Comparative Example 2, as shown in Figure 24, a thresholding process was applied where, for example, the probability of the water surface was tracked sequentially from bottom to top along line L1, and the point where the probability of the water surface area first fell below 50% was designated as the boundary line. In Figure 24, the vertical axis of the graph shows the probability of being at the boundary between the water surface and land, and the horizontal axis shows the position of line L1 in the image from bottom to top.

[0070] However, 50% is not always optimal depending on sunlight conditions; it often fluctuates to 25% or even 75%. Adjusting this threshold is difficult, making it challenging to ensure versatility.

[0071] <Step ST431: First contrast adjustment step> Therefore, in contrast to Comparative Example 2 described above, in one embodiment of the present invention, Fisher's evaluation criteria are applied to generate candidate boundary positions between the water surface and the land. To generate these boundary position candidates, the control unit 11, in step ST431, adjusts the contrast of the extracted result image data shown in Figure 25A, which has been projected in the projection transformation processing step, to generate extracted result image data corresponding to the first condition (the image shown in Figure 25B). The value of the contrast to be adjusted is set by the user operating the input unit 12.

[0072] <Step ST432: Second contrast adjustment step> Furthermore, in step ST432, the control unit 11 adjusts the contrast of the extracted image data obtained by projection transformation in the projection transformation processing step to a different contrast from that in step ST431, thereby generating extracted image data corresponding to the second condition.

[0073] <Step ST441: Detection of land-sea boundary candidates by applying the first Fisher evaluation criterion> In step ST441, the control unit 11 uses Fisher's evaluation criteria to determine a first land-water boundary candidate (an example of a first boundary position candidate) based on the extracted image data corresponding to the first condition generated by the first contrast adjustment step. The reference value of Fisher's evaluation criteria is determined by equation 4 below.

number

[0074] As shown in Figure 26(a), the numerator of Fisher's evaluation criterion increases as the difference between the average of the upper values ​​and the average of the lower values ​​increases. In Figure 26(a), the vertical axis represents the luminance value, and the horizontal axis represents the number of pixels in the vertical direction of the image shown in Figure 25A.

[0075] As shown in Figure 26(b), the denominator of Fisher's evaluation criterion becomes smaller as the variation in both the upper and lower values ​​decreases. In Figure 26(b), the vertical axis represents the square of the luminance value, and the horizontal axis represents the number of pixels in the vertical direction of the image shown in Figure 25A.

[0076] Therefore, the larger the Fisher evaluation criterion, the further apart the features of the upper and lower patterns are. As shown in Figure 26(c), the control unit 11 sets the position (horizontal line) where the Fisher evaluation criterion is largest as the land-water boundary candidate Lm1. In Figure 26(c), the vertical axis shows the reference value of the Fisher evaluation criterion, and the horizontal axis shows the number of pixels in the vertical direction of the image shown in Figure 25A.

[0077] <Step ST441: Detection of land-sea boundary candidates by applying the second Fisher evaluation criterion> In step ST442, the control unit 11 uses Fisher's evaluation criteria to determine a second land-sea boundary candidate Lm2 (an example of a second boundary position candidate) shown in Figure 27, based on the extracted image data corresponding to the second condition generated by the second contrast adjustment step.

[0078] <Another example of detecting potential land-water boundaries using Fisher's evaluation criteria> In step ST442, the control unit 11 may, after determining the second land-water boundary candidate Lm2 (an example of a second boundary position candidate) shown in Figure 27, adjust the contrast to generate the image shown in Figure 28, and then determine the first land-water boundary candidate Lm1 shown in Figure 29.

[0079] <Step ST45: Final selection from candidate land-sea boundary> In step ST45, the control unit 11 performs a final determination of the land-water boundary position based on the correlation between the two regions AR1 and AR2 divided by the first land-water boundary candidate Lm1 shown in Figure 30, and the correlation between the two regions AR2 and AR3 divided by the second land-water boundary candidate Lm2. Here, if the pattern distributions of regions AR1 and AR2 are similar, the control unit 11 selects the first land-water boundary candidate Lm1 as the water level. On the other hand, if the pattern distributions of regions AR2 and AR3 are similar, the control unit 11 selects the second land-water boundary candidate Lm2 as the water level. By using this two-stage determination configuration, determination inconsistencies are suppressed.

[0080] <Effects of one embodiment> As described above, according to the above embodiment, observation image data of the river water surface observed at one or more river water level observation stations 20 can be acquired, and river water surface area data that constitutes the detection target area can be extracted from this acquired observation image data. Furthermore, the boundary position between the water surface area and the land area is determined from the extracted image data including this river water surface area data, thereby performing a double determination of the water surface area. Consequently, individual learning of observation points is not required, resulting in high versatility, and the boundary position between the water surface area and the land area can be easily captured, making it suitable for waterline detection, and good determination accuracy can be expected even during high water levels. As a result, it can help users take appropriate actions in response to changes in the current river water level.

[0081] Furthermore, according to the above embodiment, when performing the image matting method, by simply storing a single trimmap image data in the trimmap image storage unit 151 once, in which the same position as the river surface identified during low water levels in the observed river surface image data is set to white as a virtual water surface area, the area identified as land during high water levels is set to black, and the remaining area is set to gray, the process from acquiring the observed image data to determining the boundary position between the water surface area and the land area can be performed without human intervention. As a result, the work of creating a large amount of training data is completely unnecessary, and it becomes possible to monitor the river surface area at a low cost. Consequently, cost-effectiveness can be improved.

[0082] Furthermore, according to the above embodiment, when the SAM method is executed, by specifying the coordinates of multiple detection target areas in the river surface observation image data, the data is separated into multiple detection target area data, and extracted result image data including the separated multiple detection target areas is generated.Therefore, since individual learning of observation points is not required, high versatility can be obtained, and the boundary position between the water surface area and the land area can be easily captured, making it suitable for waterline detection, and good judgment accuracy can be expected even during high water, and because a series of areas are extracted, it is less likely to misdetect isolated areas such as puddles.

[0083] Furthermore, according to the above embodiment, when extracting a water surface region from observational image data of a river surface, the water surface region can be extracted by using entropy to determine the randomness of the color pattern.

[0084] Furthermore, according to the above embodiment, prior to the step of extracting water surface region data from observed image data, noise can be removed using image processing with Real-ESRGAN, and by using histogram equalization, color changes can be emphasized even under shooting conditions such as nighttime or rainy weather. This eliminates noise and ambiguity that would hinder subsequent analysis.

[0085] Furthermore, according to the above embodiment, by performing a two-stage determination process in which, from the extracted image data including water surface data, a candidate for the land-water boundary between the water surface area and the land area under the first condition and a second candidate for the land-water boundary between the water surface area and the land area under a second condition different from the first condition are obtained, and the final boundary position between the water surface area and the land area is determined based on the correlation between the two areas divided by the first land-water boundary candidate and the correlation between the two areas divided by the second land-water boundary candidate, the versatility can be further increased and the accuracy of determining the final boundary position can be improved.

[0086] Furthermore, according to the above embodiment, the versatility can be further enhanced by adjusting the contrast of the extracted image data of the water surface region and using Fisher's evaluation criteria to determine the first and second candidates for the land-water boundary.

[0087] Furthermore, according to the above embodiment, by simultaneously and in parallel executing multiple detection target extraction methods, the accuracy of determining the final land-water boundary position can be further improved. If one of the multiple detection target extraction methods tends to produce undetected areas, the undetected areas can be corrected by selecting the darker one.

[0088] Furthermore, according to the above embodiment, the extracted image data of the water surface region synthesized in the synthesis process can be converted to water level by projecting the boundary position between the water surface region and the land region onto image data viewed from the front.

[0089] Although embodiments of the present invention have been described above, these embodiments are merely illustrative examples of devices and methods for realizing the technical concept of the present invention, and the technical concept of the present invention does not specify the material, shape, structure, arrangement, etc., of the components. The technical concept of the present invention can be modified in various ways within the technical scope defined by the claims described in the patent claims. [Explanation of Symbols]

[0090] 10 River Image Analysis Device 11 Control Unit 12 Input section 13 Display section 14 Input Interface Section 15 Storage section 20 River water level observation stations 20a Surveillance camera 20b Communication Department 111 Comparison and Verification Unit 151 Trimapped Image Storage Unit AR1,AR2,AR3 area B31 Input Layer B32 Convolutional layer B33 Pooling layer B34 Inverse Convolutional Layer B35 Output Layer B36 Comparison section B37 Loss calculation section B38 Parameter Update Section L1 Line Lm1: First candidate for the amphibious boundary. Lm2: Second candidate for the amphibious boundary. OB1 Separated detection target

Claims

1. An observation image data acquisition unit that acquires observation image data of the water surface at one or more observation points set in a body of water, A detection target extraction unit extracts detection target area data from the observation image data acquired by the observation image data acquisition unit, An image analysis apparatus comprising: a determination processing unit that performs a determination process for determining the boundary position between a detection target area and an undetectable area based on extracted image data including the detection target area data extracted by the detection target extraction unit.

2. The detection target extraction unit includes a trimmap image storage unit that stores trimmap image data created in advance for masking at least the undetectable area from among the detection target area, undetectable area, and unknown area different from the detection target area and the undetectable area in the water surface observation image data. The image analysis apparatus according to claim 1, which generates extracted result image data by comparing the observed image data with the trimmapped image data.

3. The image analysis device according to claim 2, wherein the trimmap image storage unit stores trimmap image data in which the same position as the water surface identified as low water level in the water surface observation image data is set as a first color as a virtual water surface region, the region identified as land during high water level is set as a second color, and the remaining region is set as a third color.

4. The image analysis apparatus according to claim 1, wherein the detection target extraction unit separates the observed image data of the water surface into a plurality of detection target region data with specified coordinates, and generates the extracted result image data including the plurality of detection target region data that have been separated.

5. The image analysis apparatus according to claim 1, wherein the detection target extraction unit generates the extraction result image data in which the contour of the detection target region is emphasized by entropy for determining the randomness of the color pattern, based on the observed image data of the water surface.

6. An observation image data acquisition unit that acquires observation image data of the water surface at one or more observation points set in a body of water, A first image processing unit generates image data with a resolution higher than the resolution of the observation image data from the observation image data acquisition unit, A second image processing unit that corrects the brightness value in the image data output from the first image processing unit, A detection target extraction unit extracts detection target area data from the output image data of the second image processing unit, An image analysis apparatus comprising: a determination processing unit that performs a determination process for determining the boundary position between a detection target area and an undetectable area based on extracted image data including the detection target area data extracted by the detection target extraction unit.

7. The first image processing unit receives the observation image data acquired by the observation image data acquisition unit, performs image processing using Real-ESRGAN, and outputs image data with a resolution higher than the resolution of the observation image data. The image analysis apparatus according to claim 6, wherein the second image processing unit corrects the brightness values ​​of the image data output from the first image processing unit using histogram equalization.

8. An observation image data acquisition unit that acquires observation image data of the water surface at one or more observation points set in a body of water, A detection target extraction unit extracts detection target area data from the observation image data acquired by the observation image data acquisition unit, A boundary position candidate generation unit calculates, based on extracted image data including the detection target area data extracted by the detection target extraction unit, a first boundary position candidate between the detection target area and the non-detection target area under a first condition, and a second boundary position candidate between the detection target area and the non-detection target area under a second condition different from the first condition. An image analysis apparatus comprising: a determination processing unit that performs a determination of the final boundary position between a detection target area and an undetectable area based on the correlation between two areas divided by the first boundary position candidate calculated by the boundary position candidate generation unit and the correlation between two areas divided by the second boundary position candidate.

9. The image analysis apparatus according to claim 8, wherein the boundary position candidate generation unit determines the first boundary position candidate and the second boundary position candidate using Fisher's evaluation criteria.

10. The aforementioned detection target extraction unit processes the observed image data using multiple detection target extraction methods, The image analysis apparatus according to claim 8, wherein the boundary position candidate generation unit comprises an image synthesis unit that synthesizes a plurality of extracted result image data obtained by image processing of each detection target extraction method into a single image data, and determines the first boundary position candidate and the second boundary position candidate based on the synthesized image data.

11. The image analysis apparatus according to claim 8, wherein the boundary position candidate generation unit includes a projection transformation unit that projects the extracted result image data onto image data viewed from the front, showing the boundary position between the detected target and the area outside the detected target.

12. The boundary position candidate generation unit includes a first contrast adjustment unit that adjusts the contrast of the extracted result image data to generate extracted result image data corresponding to a first condition, and a second contrast adjustment unit that adjusts the contrast of the extracted result image data to generate extracted result image data corresponding to a second condition. The image analysis apparatus according to claim 8, wherein a candidate for the first boundary position is determined based on the extracted image data corresponding to the first condition, and a candidate for the second boundary position is determined based on the extracted image data corresponding to the second condition.

13. Obtain observational image data of the water surface at one or more observation points set up in a body of water. From the acquired observational image data, the detection target area data is extracted. An image analysis method that performs a determination process for the boundary position between a detection target area and an undetectable area based on extracted image data including the extracted detection target area data.

14. Obtain observational image data of the water surface at one or more observation points set up in a body of water. From the acquired observational image data, image data with a higher resolution than the resolution of the aforementioned observational image data is generated. Perform brightness value correction on the generated image data. From the image data to which the brightness value has been corrected, the detection target area data is extracted. An image analysis method that performs a determination process for the boundary position between a detection target area and an undetectable area based on extracted image data including the extracted detection target area data.

15. Obtain observational image data of the water surface at one or more observation points set up in a body of water. From the acquired observational image data, extract the data of the area to be detected. Based on the extracted image data including the extracted detection target area data, a first candidate boundary position between the detection target area and the non-detection target area under the first condition and a second candidate boundary position between the detection target area and the non-detection target area under a second condition different from the first condition are calculated. An image analysis method that determines the final boundary position between a detection target area and an undetectable area based on the correlation between two areas divided by the first boundary position candidate and the correlation between two areas divided by the second boundary position candidate.

Images