Determination device, determination method, and program

Abstract

To provide a detection device that can suppress false detections of snow jams. [Solution] A determination device according to one aspect of the present disclosure includes: an acquisition unit that acquires an image which is an image including a waterway; an extraction unit that extracts snow jams based on a mask image obtained by segmenting the image on a pixel-by-pixel basis; and a determination unit that determines the presence of ice and snow in the image based on a depth image generated based on a depth estimated by a depth estimation means that estimates the depth of the shooting environment from the image, and the mask image.

Description

Technical Field

[0001] The present disclosure relates to a determination device, a determination method, and a program.

Background Art

[0002] Techniques for detecting floating substances in a water channel are known. In Patent Document 1, for a plurality of consecutive frame images obtained by photographing a water channel, detection is performed for each feature point, and based on the ratio of the number of pixels determined to have floating substances in one frame image and the number of feature points, etc., a technique for determining the presence of floating substances is disclosed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The technique described in Patent Document 1 relates to a system on the premise that there are no obstacles such as ice and snow between the water channel and the area camera. Therefore, for example, when there is ice and snow between the water channel and the area camera, and when snow adheres to the lens of the area camera, etc., over - detection of the area of snow jams occurs. At this time, for example, in the case of a device that automatically performs gate operations depending on the presence or absence of snow jams, there is a risk of malfunction of the gate. Therefore, it is desired to more accurately distinguish whether the detected object is a snow jam staying in the water channel or other snow, etc.

[0005] An object of the present disclosure is to provide a determination device capable of suppressing false detection of snow jams.

Means for Solving the Problems

[0006] A determination device according to one aspect of the present disclosure includes: an acquisition unit that acquires an image which is an image including a waterway; an extraction unit that extracts snow jams based on a mask image obtained by segmenting the image on a pixel-by-pixel basis; and a determination unit that determines the presence of ice and snow in the image based on a depth image generated based on a depth estimated by a depth estimation means that estimates the depth of the shooting environment from the image, and the mask image. [Effects of the Invention]

[0007] According to the determination device described herein, false detection of snow jams can be suppressed. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an overview of the ice and snow detection system according to the first embodiment of this disclosure. [Figure 2] This figure illustrates the imaging of a snow jam in a waterway by an area camera according to the first embodiment of this disclosure. [Figure 3] This figure shows the hardware configuration of the determination device according to the first embodiment of this disclosure. [Figure 4] This is a block diagram showing the functional configuration of a determination device according to the first embodiment of this disclosure. [Figure 5] This is a block diagram showing the configuration of the learning function for generating a mask image in the determination device according to the first embodiment of this disclosure. [Figure 6] This figure shows examples of each figure generated by the determination device according to the first embodiment of this disclosure. [Figure 7] This is a flowchart showing the determination method executed by the determination device according to the first embodiment of this disclosure. [Figure 8] This is a flowchart showing the determination method executed by the determination device according to the second embodiment of this disclosure. [Modes for carrying out the invention]

[0009] The embodiments for carrying out the invention will be described below with reference to the drawings. In each drawing, the same reference numerals are used for identical components, and redundant explanations may be omitted.

[0010] [First Embodiment] <Configuration of Ice and Snow Detection System 1> Figure 1 is a diagram showing an overview of the ice and snow detection system 1 according to the first embodiment of this disclosure. The ice and snow detection system 1 comprises a determination device 10, an area camera 21, lighting 22, a camera control device 23, a lighting control device 24, an operation input device 25, and a communication device 26. Note that the configuration of the ice and snow detection system 1 is not limited to the example shown.

[0011] Area camera 21 is an example of an imaging device. Area cameras 21 are installed individually or in multiple locations near areas where ice and snow detection is required. Based on control by camera control device 23, they capture moving images or continuous still images of the water surface in the waterway at the location where ice and snow detection is required and transmit them to the determination device 10. Note that the imaging device is not limited to area cameras 21; various devices having an image sensor such as a CCD image sensor or a CMOS image sensor and an optical system can be used.

[0012] The lighting 22 illuminates the water surface of the channel, which is imaged by the area camera 21, based on the control of the lighting control device 24. The lighting 22 is installed individually or in multiple locations near the image-captured surface of the area camera 21. The light source of the lighting 22 may be a visible light source such as a white light source, or an invisible light source such as infrared light, and may be switched between based on the control of the lighting control device 24.

[0013] The area camera 21 and lighting 22 are installed so that the area camera 21 can image the water surface of a waterway where ice and snow detection is necessary, such as a waterway to a hydroelectric power plant in a cold region, an outlet from the waterway to a reservoir upstream of a dam, or an inflow channel to a reservoir, and the lighting 22 can illuminate the imaged location. In the following, waterways include waterways to hydroelectric power plants, outlets from the waterway to a reservoir upstream of a dam, and inflow channels to reservoirs.

[0014] The determination device 10 acquires the image captured by the area camera 21, and determines the presence or absence of snow jam, which is ice and snow staying in the water channel, by performing image processing. The determination device 10 also determines the presence or absence of reflection of ice and snow between the water channel and the area camera 21. The determination device 10 outputs the determined result to the operation input device 25, the operator's terminal device, an external information processing device, etc. via the communication device 26. The reflection of ice and snow includes the reflection of ice and snow existing between the water channel and the area camera 21 and the ice and snow adhering to the lens of the area camera 21.

[0015] The camera control device 23 controls the focus, exposure, etc. of the area camera 21. When the area camera 21 is installed so that the imaging direction can be changed, the camera control device 23 may control the imaging location by the area camera 21.

[0016] The lighting control device 24 controls the irradiation of light from the lighting 22 to the imaging location. Specifically, when the irradiation light intensity, color, etc. of the lighting 22 are configured to be changeable, and when the light source of the lighting 22 is configured to be able to switch between a visible region light source such as a white light source and an invisible region light source such as infrared light, the lighting control device 24 controls these. Also, the lighting control device 24 may change the irradiation light intensity, color, etc. of the lighting 22 based on information from a sensor that detects the brightness provided around the lighting 22. The sensor for detecting brightness is a sensor provided with a sensing circuit including a photoelectric conversion element such as a photodiode and an amplification circuit, etc., and detects the brightness of the illuminated water channel 2 and its surroundings.

[0017] Also, when the light source of the lighting 22 is configured to be able to switch between a visible region light source such as a white light source and an invisible region light source such as infrared light, it is preferable that the camera control device 23 makes the imaging conditions of the area camera 21 match the state of the lighting 22. When it is possible to handle only by the control of the area camera 21 by the camera control device 23, it is not necessary to perform the control of the lighting 22 by the lighting control device 24.

[0018] The operation input device 25 is a device that remotely monitors the states of gates such as sluice gates and dust collectors provided in the water channel and automatically controls the opening and closing of each gate. The operation input device 25 is composed of, for example, an input device such as a keyboard or a touch panel, or a receiving unit that acquires information from other devices, and receives operation inputs from the user. Also, the operation input device 25 receives various information from the determination device 10. Further, the operation input device 25 transmits information such as the state of the gate to the determination device 10.

[0019] The operation input device 25 is configured to be communicable with a sluice gate opening / closing device (not shown) installed in each gate that is an operation target. The sluice gate opening / closing device is a device that operates the gate. By an operation by an operator or an automatic operation remotely, the operation input device 25 transmits a signal instructing the operation of the gate to the sluice gate opening / closing device, closes the sluice gate by arranging the door body of the sluice gate at the lowest position, and fully opens the sluice gate by arranging the door body at the uppermost position. The operation input device 25 may be provided with a control program for controlling the operations of the sluice gate and the dust collector and the like that are operation targets. The control program may transmit a signal instructing the operation of the gate to the sluice gate opening / closing device according to the output from the determination device 10 and cause the gate to operate.

[0020] The communication device 26 outputs the determination result output from the determination device 10 to the terminal device of the operator and other information processing devices that can communicate with the determination device 10. The communication means may be wired or wireless.

[0021] FIG. 2 is a diagram for explaining the imaging of the snow jam 3 in the water channel 2 by the area camera 21 according to the first embodiment of the present disclosure. As shown in the figure, the area camera 21 is installed so that the water surface at the position where it is desired to detect the snow jam 3 where ice and snow have accumulated in the water channel 2 is within the imaging range.

[0022] Snow accumulated upstream of waterway 2 flows into waterway 2, such as a river, and accumulates around the gate, causing a snow jam 3. As a result, the gate stops operating. Therefore, it is desirable to detect the occurrence of snow jam 3 as early as possible. Furthermore, since the location where snow jam 3 occurs is waterway 2 around the gate, it is usually a place where the workers who operate the gate are not present. For this reason, in this embodiment, snow jam 3 is detected using images captured by the area camera 21 of waterway 2.

[0023] <Configuration of the determination device 10> Figure 3 shows the hardware configuration of the determination device 10 according to the first embodiment of this disclosure. The determination device 10 includes an input device 101, a display device 102, an external I / F (Interface) 103, RAM 104, ROM 105, CPU (Central Processing Unit) 106, a communication I / F 107, and an HDD 108, each connected by bus B.

[0024] The input device 101 and the display device 102 may be connected as needed. In the determination device 10, the RAM 104, ROM 105, and CPU 106 realize the functions of each part of the determination device 10.

[0025] The input device 101 includes a touch panel, operation keys and buttons, a keyboard and mouse, etc., used by the user to input various signals. The display device 102 consists of a display such as a liquid crystal or organic EL that displays the screen, and a speaker that outputs sound data such as voice and music. The communication I / F 107 is an interface for the determination device 10 to communicate data via an external network.

[0026] Furthermore, the HDD108 is an example of a non-volatile storage device that stores programs and data. The programs and data stored include the OS, which is the basic software that controls the entire determination device 10, and applications that provide various functions on the OS. The determination device 10 may also use a drive device that uses flash memory as the storage medium instead of the HDD108. For example, an SSD could be used as the drive device.

[0027] External I / F 103 is an interface to an external device. An external device could be a recording medium 103a. This allows the determination device 10 to read from and write to the recording medium 103a via the external I / F 103. The recording medium 103a could be a flexible disk, CD, DVD, SD memory card, USB memory, etc.

[0028] RAM104 is an example of volatile semiconductor memory that temporarily holds programs and data. ROM105 is an example of non-volatile semiconductor memory that can retain programs and data even when the power is turned off. ROM105 stores programs and data such as the BIOS, OS settings, and network settings that are executed when the determination device 10 is started up.

[0029] The CPU 106 is an arithmetic unit that controls and implements the overall functionality of the determination device 10 by reading programs and data from storage devices such as the ROM 105 and HDD 108 onto the RAM 104 and executing processing. The determination device 10 may also be equipped with a GPU (Graphics Processing Unit) instead of the CPU 106.

[0030] Figure 4 is a block diagram showing the functional configuration of the determination device 10 according to the first embodiment of this disclosure. As shown in the figure, the determination device 10 comprises an acquisition unit 11, an extraction unit 12, a determination unit 13, and an output unit 14.

[0031] The acquisition unit 11 acquires an image that includes the waterway 2. The waterway 2 includes a waterway to facilities such as a hydroelectric power plant, an outlet from the waterway to a reservoir upstream of the dam, an inflow channel to the reservoir, etc. The image including the waterway 2 is an image captured by an area camera 21 or the like. In this embodiment, the area camera 21 is, for example, a fixed camera that captures images of the waterway 2.

[0032] The extraction unit 12 extracts snow jams 3 based on a mask image segmented pixel by pixel from the captured image acquired by the acquisition unit 11. In order to segment the image, the extraction unit 12 performs preprocessing such as cropping, resizing, rotation, compression, and normalization on the captured image acquired by the acquisition unit 11.

[0033] The extraction unit 12 may apply a rule-based algorithm to search for the boundary between the snow jam 3 and the background, and perform segmentation by dividing the image into regions corresponding to the snow jam 3 and regions corresponding to the background. Alternatively, the extraction unit 12 may use a linear method for segmentation. Furthermore, the extraction unit 12 may use a learned model for segmentation. When using a learned model, the extraction unit 12 applies the trained model based on local features obtained from the captured image and the captured image itself to recognize the snow jam 3 contained in the captured image and generate a mask image that estimates the region in the captured image corresponding to the snow jam 3. The extraction unit 12 then calculates the area of ​​the snow jam 3.

[0034] Figure 5 is a block diagram showing the configuration of the learning function for mask image generation in the determination device 10 according to the first embodiment of the present disclosure. As shown in the figure, the extraction unit 12 includes a mask image generation unit 121. The mask image generation unit 121 is configured to generate a mask indicating an image segment in an captured image based on an captured image including the waterway 2.

[0035] The mask image generation unit 121 generates a mask image using an inference model 122. The inference model 122 includes a neural network 123 that has been trained by machine learning to receive an image as input and output a mask image.

[0036] More specifically, inference model 122 is a pre-trained model that takes image features obtained from the captured image and the captured image itself as input, recognizes snow jam 3 on the captured image, and outputs a mask image.

[0037] The trained model is obtained, for example, by performing machine learning, i.e., supervised learning, on the base trained model using a training dataset. The trained model may be generated by performing machine learning on the trained model in the extraction unit 12, or by performing machine learning on the trained model in a functional unit other than the extraction unit 12 and an information processing device other than the judgment device 10. For example, each training data included in the training dataset is a combination of image features obtained from the captured image as input data or the captured image itself, and data specifying the region corresponding to the snow jam 3 on the captured image as output data.

[0038] Furthermore, the trained model is primarily composed of a deep neural network (DNN), and the DNN's machine learning is made more efficient by applying backpropagation based on training data. The trained model may also include a U-Net capable of recognizing SnowJam 3 from captured images, which are taken as input.

[0039] The inference model 122 may be located inside the mask image generation unit 121 and the extraction unit 12, as shown in the figure, or it may be located in a functional unit other than the extraction unit 12 and an information processing device other than the determination device 10. Furthermore, the inference model 122 may be provided in, for example, the storage means of the determination device 10 and other devices. Also, the devices that construct the inference model 122 are, for example, the determination device 10 and other information processing devices.

[0040] The extraction unit 12 can extract the snow jam 3 based on a mask image generated by segmenting the region containing the recognized snow jam 3 in the captured image into a region corresponding to the snow jam 3 and a region corresponding to the background.

[0041] In Figure 4, the determination unit 13 determines the presence of ice and snow in the captured image based on a depth image and a mask image generated based on the depth estimated by a depth estimation means that estimates the depth of the shooting environment from the captured image including the waterway 2. The shooting environment includes the waterway 2, snow jam 3, structures, and objects such as gates in the space captured by the area camera 21. The ice and snow presence refers to ice and snow other than the snow jam 3 in the waterway 2, such as ice and snow adhering to the lens of the area camera 21 and ice and snow present on structures between the waterway 2 and the imaging device.

[0042] The determination unit 13 generates a depth image by estimating the depth of the object indicated by each pixel in the acquired image. The depth image includes information about the depth associated with the position of each pixel. The determination unit 13 may also perform monocular depth estimation on the image using, for example, the BiFuse method. Each pixel in the depth image includes coordinate information in three-dimensional space for that pixel, allowing for visualization of a three-dimensional point cloud.

[0043] The determination unit 13 determines whether ice and snow are reflected in the image based on the depth changes between each object in the depth image in the region corresponding to the snow jam 3 extracted in the mask image. More specifically, the determination unit 13 calculates the difference in brightness values ​​between adjacent pixels in the image generated by the process of multiplying the mask image and the depth image, and determines that there is an ice and snow reflection if there is a region where the calculated difference in brightness values ​​exceeds a predetermined value.

[0044] Furthermore, the determination unit 13 determines the presence or absence of snow jam 3 based on the area of ​​the waterway 2 in the captured image and the area of ​​the snow jam 3 calculated by the extraction unit 12. If the area camera 21 is a fixed camera, the region corresponding to the waterway 2 in the captured image is set in advance, and the area of ​​the waterway 2 is calculated in advance. The determination unit 13 determines that snow jam 3 has occurred if the ratio of the area of ​​snow jam 3 to the area of ​​waterway 2 is greater than or equal to a predetermined value. The predetermined value is any value such as 10% and 20%, but it is desirable to determine it from the viewpoint of eliminating noise and excluding the detection of snow jam 3 which does not need to be removed.

[0045] The output unit 14 outputs the results of the determination by the determination unit 13 regarding the reflection of ice and snow and the presence or absence of snow jams 3. The determination results are output to the operation input device 25, the operator's terminal device, and other devices that can communicate with the determination device 10. The determination results may also be displayed on the display means provided by each device. The display means may include a display device such as a liquid crystal display or an organic EL (Electro-Luminescence) display that displays the determination results in the form of characters and images. The output unit 14 may output, for example, a mask image and a depth image as shown later in Figure 6, and the display means of each device may display these images.

[0046] Based on the information displayed on the display means of the information processing device and terminal device that can communicate with the determination device 10, the worker can determine, for example, if there is a snow jam 3 in the waterway 2 and there is reflection of ice and snow, the need to remove the ice and snow attached to the lens of the area camera 21 and the ice and snow accumulated on other structures. Also, if there is a snow jam 3 in the waterway 2 but there is no reflection of ice and snow, the worker can determine the need to operate the sluice gate and debris removal machine, etc.

[0047] The output unit 14 outputs the results of the determination unit 13's determination regarding the reflection of ice and snow and the presence or absence of snow jam 3 to the operation input device 25, which is a device that automatically controls the opening and closing of the gate installed in the waterway 2. At this time, the operation input device 25 transmits a signal to the sluice gate opening and closing device to instruct the gate to operate, according to the determination result received from the determination device 10, and the gate can be operated. The instruction to the sluice gate opening and closing device to operate the gate may also be made by a control program provided in the operation input device 25 that controls the operation of the sluice gate and the debris removal machine, etc., which are the targets of the operation.

[0048] Figure 6 shows examples of each figure generated by the determination device 10 according to the first embodiment of the present disclosure. In Figure 6, (a) is an image of the waterway 2 captured by the area camera 21, (b) is a mask image, (c) is a depth image, and (d) is an image generated by a process that combines the depth image and the mask image.

[0049] As shown in Figure 6(a), the captured image includes the waterway 2 and the snow jam 3. The image also shows snow accumulation on a structure located in front of the area camera 21, which is closer to the waterway 2. The captured image may be either black and white or color.

[0050] Figure 6(b) is a mask image obtained by segmenting the captured image in (a) on a pixel-by-pixel basis. As shown, the mask image displays only the region corresponding to snow jam 3. Figure 6(c) is a depth image generated based on the depth estimated by a depth estimation means that estimates the depth of the shooting environment from the captured image in (a). The white areas shown indicate snow accumulation on structures located in front of waterway 2, which are included in the captured image. According to the depth image, the depth changes between each object in the depth image become clear according to the brightness value of each pixel.

[0051] Figure 6(d) is an image generated by multiplying the mask image from (b) and the depth image from (c). By multiplying the mask image and the depth image, a depth image is obtained that is limited to only the area corresponding to the masked snow jam 3 in the mask image. The determination unit 13 determines that there is reflection of ice and snow in the area shown A in the figure because the difference in brightness values ​​between adjacent pixels exceeds a predetermined value.

[0052] <Processing flow executed by the determination device 10> Figure 7 is a flowchart showing the determination method performed by the determination device 10 according to the first embodiment of this disclosure. The acquisition unit 11 acquires an image, which is an image including the waterway 2, from the area camera 21 (S101). The extraction unit 12 performs preprocessing on the image acquired by the acquisition unit 11 for segmentation, such as cropping, resizing, rotating, compressing, and normalizing the image (S102).

[0053] The extraction unit 12 generates a mask image segmented pixel by pixel from the captured image (S103). The extraction unit 12 calculates the area of ​​the snow jam 3 included in the mask image (S104). The determination unit 13 determines the presence or absence of the snow jam 3 in the captured image based on the previously calculated area of ​​the waterway 2 and the area of ​​the snow jam 3 (S105).

[0054] The determination unit 13 determines that a snow jam 3 has occurred if the ratio of the area of ​​the snow jam 3 to the area of ​​the waterway 2 is greater than or equal to a predetermined value (YES in S105), and the output unit 14 outputs this result to the operation input device 25 and the operator's terminal device, etc. (S106). Also, the determination unit 13 determines that a snow jam 3 has occurred if the ratio of the area of ​​the snow jam 3 to the area of ​​the waterway 2 is less than a predetermined value (NO in S105), and the output unit 14 outputs this result to the operation input device 25 and the operator's terminal device, etc. (S107).

[0055] The determination unit 13 generates a depth image based on the depth estimated by the depth estimation means, which estimates the depth of the shooting environment from the captured image (S108). The determination unit 13 generates an image by multiplying the mask image and the depth image (S109). The determination unit 13 calculates the difference in brightness values ​​between adjacent pixels in the generated image (S110).

[0056] The determination unit 13 determines that there is reflection of ice and snow if there is an area where the difference in the calculated brightness values ​​exceeds a predetermined value (YES in S111), and the output unit 14 outputs this fact as the result of the determination to the operation input device 25 and the operator's terminal device, etc. (S112). The determination unit 13 determines that there is no reflection of ice and snow if there is no area where the difference in the calculated brightness values ​​exceeds a predetermined value (NO in S111), and the output unit 14 outputs this fact as the result of the determination to the operation input device 25 and the operator's terminal device, etc. (S113).

[0057] These steps carry out the determination method according to the first embodiment of this disclosure. However, the determination method according to the first embodiment of this disclosure may include other steps as appropriate, depending on the measurement conditions, measurement environment, etc.

[0058] <Effects of the determination device 10 according to the first embodiment> The determination device 10 according to this embodiment acquires an image that includes the waterway 2, and extracts the snow jam 3 using a mask image generated based on the image. Then, by multiplying the depth image generated by a depth estimation means that estimates the depth of the shooting environment from the image and the mask image, it determines whether or not there is reflection of ice and snow.

[0059] Therefore, the determination device 10 according to this embodiment can accurately determine whether or not ice and snow are reflected in the captured image, thereby suppressing false detection of snow jams 3 in the waterway 2. In addition, the determination device 10 can suppress the operation of gates such as sluice gates and debris removers due to false detection of snow jams 3, thereby reducing the power consumption related to the operation of the gates, etc.

[0060] [Second Embodiment] The determination device 10 according to this embodiment includes an acquisition unit 11, an extraction unit 12, a determination unit 13, and an output unit 14, similar to the determination device 10 according to the first embodiment. Unlike the first embodiment, the area camera 21 is a movable camera. Therefore, since the area of ​​the waterway 2 in the captured image is not calculated in advance, the extraction unit 12 extracts both the waterway 2 and the snow jam 3 in the captured image based on the mask image and calculates the area of ​​both the extracted waterway 2 and the snow jam 3.

[0061] <Processing flow executed by the determination device 10> Figure 8 is a flowchart showing the determination method performed by the determination device 10 according to the second embodiment of this disclosure. The acquisition unit 11 acquires an image containing the waterway 2 from the area camera 21 (S201). The extraction unit 12 performs preprocessing on the image acquired by the acquisition unit 11 for segmentation, such as cropping, resizing, rotating, compressing, and normalizing the image (S202).

[0062] The extraction unit 12 generates a mask image segmented pixel by pixel from the captured image (S203). The extraction unit 12 calculates the area of ​​the extracted waterway 2 and snow jam 3 (S204). The determination unit 13 determines the presence or absence of snow jam 3 based on the area of ​​waterway 2 and snow jam 3 in the captured image (S205).

[0063] The determination unit 13 determines that a snow jam 3 has occurred if the ratio of the area of ​​the snow jam 3 to the area of ​​the waterway 2 is greater than or equal to a predetermined value (YES in S205), and the output unit 14 outputs this result to the operation input device 25 and the operator's terminal device, etc. (S206). Also, the determination unit 13 determines that a snow jam 3 has occurred if the ratio of the area of ​​the snow jam 3 to the area of ​​the waterway 2 is less than a predetermined value (NO in S205), and the output unit 14 outputs this result to the operation input device 25 and the operator's terminal device, etc. (S207).

[0064] The determination unit 13 generates a depth image based on the depth estimated by the depth estimation means, which estimates the depth of the shooting environment from the captured image (S208). The determination unit 13 generates an image by multiplying the mask image and the depth image (S209). The determination unit 13 calculates the difference in brightness values ​​between adjacent pixels in the generated image (S210).

[0065] The determination unit 13 determines that there is reflection of ice and snow if there is an area where the difference in the calculated brightness values ​​exceeds a predetermined value (YES in S211), and the output unit 14 outputs this as the result of the determination to the operation input device 25 and the operator's terminal device, etc. (S212). The determination unit 13 determines that there is no reflection of ice and snow if there is no area where the difference in the calculated brightness values ​​exceeds a predetermined value (NO in S211), and the output unit 14 outputs this as the result of the determination to the operation input device 25 and the operator's terminal device, etc. (S213).

[0066] These steps carry out the determination method according to the second embodiment of this disclosure. However, the determination method according to the second embodiment of this disclosure may include other steps as appropriate, depending on the measurement conditions, measurement environment, etc.

[0067] <Effects of the determination device 10 according to the second embodiment> In the determination device 10 according to this embodiment, the extraction unit 12 extracts the waterway 2 and snow jam 3 in the captured image based on the mask image and calculates the area of ​​the extracted waterway 2 and snow jam 3. Therefore, the determination device 10 can calculate the area of ​​the waterway 2 even when there is a change in the area of ​​the waterway 2 due to changes in water level, etc., and even when the area camera 21 is movable, and can accurately determine whether or not ice and snow are reflected in the captured image, thereby suppressing false detection of snow jam 3 in the waterway 2.

[0068] As described above, embodiments have been explained, but these embodiments are presented as examples only, and the present invention is not limited by these embodiments. The above embodiments can be implemented in various other forms, and various combinations, omissions, substitutions, and modifications are possible without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents.

[0069] Each of the functions of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute each of the functions described above. [Explanation of Symbols]

[0070] 1. Ice and Snow Detection System 2 waterways 3 Snow Jam 10 Judgment device 11 Acquisition Department 12 Extraction part 13 Judgment section 14 Output section 21 Area Cameras 25. Operation Input Device

Claims

1. An acquisition unit that acquires an image that includes a waterway, An extraction unit extracts snow jams based on a mask image obtained by segmenting the captured image at the pixel level, A determination unit determines whether ice and snow are reflected in the captured image based on a depth image generated based on the depth estimated by a depth estimation means that estimates the depth of the shooting environment from the captured image, and the mask image. A determination device equipped with this device.

2. The determination unit determines the presence of ice and snow based on the depth changes between each object in the depth image in the region corresponding to the snow jam extracted in the mask image. The determination device according to claim 1.

3. The determination unit calculates the difference in brightness values ​​between adjacent pixels in the image generated by the process of multiplying the mask image and the depth image, and determines that there is reflection of ice and snow if there is a region where the calculated difference in brightness values ​​exceeds a predetermined value. The determination device according to claim 1.

4. The determination unit determines the presence or absence of the snow jam based on the area of ​​the waterway and the area of ​​the snow jam in the captured image. The determination device according to claim 1.

5. The extraction unit extracts the waterway in the captured image based on the mask image and calculates the area of ​​the waterway. The determination device according to claim 1.

6. The system further includes an output unit that outputs the results of the determination unit and the determination regarding the presence or absence of ice and snow reflections and snow jams. A determination device according to any one of claims 1 to 5.

7. The output unit outputs the result of the determination to a device that automatically controls the opening and closing of the gate installed in the waterway. The determination device according to claim 6.

8. A determination method performed by a determination device, The steps include: acquiring an image that includes a waterway, The steps include extracting snow jams based on a mask image obtained by segmenting the captured image at the pixel level, The steps include determining whether ice and snow are reflected in the captured image based on a depth image generated based on the depth estimated by a depth estimation means that estimates the depth of the shooting environment from the captured image, and the mask image, A determination method that includes this.

9. A program that makes the judgment device function, The process of acquiring an image that includes a waterway, The process involves extracting snow jams based on a mask image obtained by segmenting the aforementioned captured image at the pixel level, A process to determine whether ice and snow are reflected in the captured image, based on a depth image generated based on the depth estimated by a depth estimation means that estimates the depth of the shooting environment from the captured image and the mask image, A program to execute.

Images