Dirt retention amount grasping system, dirt retention amount grasping method, and dirt retention amount grasping program

Abstract

To provide a dirt retention amount grasping system, a dirt retention amount grasping method, and a dirt retention amount grasping program that can accurately grasp the amount of dirt retained on the lake to recover the dirt in a timely manner.SOLUTION: In grasping the amount of dirt 11 retained in a predetermined search area on the upstream side of a boom 8 installed on the lake surface 7, a dirt retention amount grasping system calculates the occupancy of dirt in the search area by using a first photographed image obtained by photographing the search area from the sky using an aerial drone 2, a second photographed image obtained by photographing the search area from the sky using an air moving body when there is no dirt in the search area, and an upstream side boundary of dirt retained in a search area that is specified by moving a water drone 4 along the boom 8 on the upstream side of the boom 8 on the lake surface, and continuously or intermittently measuring the time until the return of a reflected wave of a detection wave emitted by the water drone 4 horizontally toward the boom 8 in the process of the moving.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a dust retention amount grasping system, a dust retention amount grasping method, and a dust retention amount grasping program capable of accurately grasping the retention amount of dust staying on the surface of a dam lake or the like.

Background Art

[0002] In a dam lake, a net field (dam fence) is provided to block dust floating on the water surface such as driftwood flowing in and protect facilities such as water intake facilities, power generation facilities, and water discharge facilities. Since the dust staying on the upstream side of this net field hinders the water intake of the dam lake, removal is required when a certain amount accumulates.

[0003] Therefore, conventionally, the maintenance staff of the dam visually inspected the state of dust retention, and when the necessity of dust removal was recognized, a dust cleaning boat was operated to perform the dust removal work. For this reason, there were cases where the removal of the accumulated dust could not be carried out in a timely manner depending on the dust retention situation, and there was a risk of problems occurring in the operation of the reservoir.

[0004] In this regard, conventionally, in order to grasp the dust staying on the water surface, an aquatic monitoring device was installed, the analog signal obtained by the imaging means in the aquatic environment was waveform-analyzed to detect whether or not there was a suspended substance, and when a suspended substance was detected, information regarding the size, shape, and brightness of the suspended substance was obtained by image processing, or the water quality pollution level was diagnosed based on water surface measurement information such as the color of the water surface and water quality, and a purification means for purifying the aquatic environment corresponding to the level of pollution was provided (Patent Document 1), or a device that changes the position where laser light is emitted onto the water surface and detects an oil film based on the amount of reflected light (Patent Document 2), etc. are known.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, in the former case, (1) Since the chromaticity and water quality of the water on the dam lake surface are not constant due to upstream weather (rainy or sunny days) and seasonal variations, etc., it is difficult to uniformly judge the floating substances on the dam lake surface based on the chromaticity and water quality of the water surface. (2) Also, among the water surface measurement information, there is the accumulation degree of floating substances. However, in actual implementation, in a vast dam lake, it is necessary to set up several measurement points. To obtain highly accurate observation results, it is necessary to install a large number of measurement devices, and it is also necessary to appropriately select the installation locations of the measurement devices. (3) Furthermore, in the case of a turbid water lake surface, it is difficult to detect the dust floating just below the water surface.

[0007] Also, in the latter case, Since it is necessary to install on land a pedestal for fixing a detection device or a lifting device equipped with functions such as the emission and reception of laser light, it is difficult to detect a vast area such as a dam lake. Also, with a structure in which the detection device moves up and down so as to always maintain a certain distance from the water surface, the reflected light of the laser light from the water surface is always received by the light receiving means. This technology enables measurement under certain conditions regardless of water surface fluctuations and can improve detection accuracy. However, since the laser light is emitted on the premise of a fixed range on the water surface, the measurement range becomes narrow. Also, in this technology, although it is possible to detect an oil film in principle, it is difficult to detect the dust floating just below the water surface.

[0008] The present invention has been made in view of such circumstances, and the main object is to provide a dust retention amount grasping system, a dust retention amount grasping method, and a dust retention amount grasping program capable of accurately grasping the retention amount of dust staying on the lake surface and timely collecting the staying dust.

Means for Solving the Problem

[0009] In order to achieve the above object, the dust retention amount grasping system according to the present invention is a dust retention amount grasping system that grasps the retention amount of dust staying in a predetermined search area on the upstream side of a net field installed on a lake surface, and a primary calculation means for calculating the occupancy rate of dust staying in the search area based on information obtained by comparing a first captured image of the search area taken from above by an airborne vehicle with a second captured image of the search area taken from above by the airborne vehicle when there is no dust in the search area; a secondary calculation means for specifying the upstream boundary of the dust staying in the search area by continuously or intermittently measuring the time until the reflected wave of a detection wave horizontally emitted from the waterborne vehicle toward the net field returns in the process of moving the waterborne vehicle along the net field on the upstream side of the net field on the lake surface at almost the same time as the shooting time of the first captured image, and correcting the occupancy rate of the dust calculated by the primary calculation means based on the upstream boundary; It is characterized by comprising the above.

[0010] Therefore, by the primary calculation means, a first captured image of the search area taken from above by an airborne vehicle is compared with a second captured image of the search area taken from above by the airborne vehicle when there is no dust in the search area, and the occupancy rate of the dust staying in the search area is temporarily calculated. That is, in the search area of the lake surface, when the presence of dust not shown in the second captured image is recognized in the comparison between the first captured image and the second captured image, or when a range with a different color tone from the second captured image is recognized, etc., it is assumed that there is some dust in the area of the different part, and the ratio occupied in the search area of that area is calculated primarily.

[0011] Thereafter, in order to confirm whether the occupancy rate of the dust grasped in the comparison between the first captured image and the second captured image is accurate, or in order to confirm the presence or absence of dust that cannot be grasped in the comparison between the first captured image and the second captured image, the secondary calculation means continuously or intermittently measures the time until the reflected wave of the detection wave horizontally emitted from the watercraft toward the net field returns, and identifies the upstream boundary of the dust staying in the search area obtained thereby, and corrects the occupancy rate of the dust calculated by the primary calculation means based on the upstream boundary. Therefore, it becomes possible to more accurately grasp the occupancy rate (retention amount) of the dust in the search area, and based on this, it becomes possible to perform the recovery work of the lake surface dust in a timely manner.

[0012] Note that when grasping the occupancy rate (retention amount) of the dust in the search area, instead of a configuration in which the occupancy rate of the dust grasped based on the captured image taken by the aerial vehicle is corrected based on the upstream boundary of the retained dust measured by the watercraft, a configuration may be adopted in which the occupancy rate of the dust grasped based on the upstream boundary of the retained dust measured by the watercraft is corrected based on the captured image taken by the aerial vehicle. That is, a dust retention amount grasping system for grasping the retention amount of dust staying in a predetermined search area upstream of a net field installed on a lake surface includes a primary calculation means that moves a watercraft along the net field on the upstream side of the net field on the lake surface, and continuously or intermittently measures the time until the reflected wave of the detection wave horizontally emitted from the watercraft toward the net field returns during the moving process, thereby identifying the upstream boundary of the dust staying in the search area, and calculating the occupancy rate of the dust staying in the search area based on the upstream boundary; and a secondary calculation means that corrects the occupancy rate of the dust calculated by the primary calculation means based on information obtained by comparing a first captured image taken by an aerial vehicle from above the search area at approximately the same time as the time when the watercraft is moved along the net field on the lake surface with a second captured image taken by the aerial vehicle from above the search area when there is no dust in the search area. may be configured to include.

[0013] In such a configuration, based on the upstream boundary of the dust staying in the search area obtained by continuously or intermittently measuring, by the primary calculation means, the time until the reflected wave of the detection wave horizontally emitted from the watercraft towards the net field returns, the occupancy rate of the dust in the search area is temporarily calculated. That is, if the upstream boundary of the dust staying in the net field is specified, the occupancy rate of the dust in the search area is calculated on the premise that the dust stays without a gap between the net field and the upstream boundary.

[0014] However, even if the upstream boundary can be specified, there may be a case where there is no dust between the upstream boundary and the net field, and the calculation of the occupancy rate of the dust based only on the upstream boundary of the staying dust does not accurately grasp the state of the dust behind the upstream boundary (on the net field side).

[0015] Therefore, in order to confirm the presence or absence of dust that cannot be grasped by the detection wave from the watercraft, based on the information obtained by comparing the first captured image of the search area taken from above by the airborne vehicle at approximately the same time as the time when the watercraft is moved along the net field on the lake surface by the secondary calculation means, with the second captured image of the search area taken from above by the airborne vehicle when there is no dust in the search area, the occupancy rate of the dust staying in the search area is corrected.

[0016] Thereby, it is possible to accurately grasp, based on the captured image from the airborne vehicle, the state of the dust deposited upstream of the net field that cannot be fully grasped from the upstream boundary, so that it becomes possible to perform the recovery work of the staying dust in a timely manner.

[0017] Furthermore, in order to grasp the occupancy rate of the lake surface dust in the search area, it may be inferred using a learning model. That is, a dust retention amount grasping system for grasping the retention amount of the dust staying in a predetermined search area upstream of the net field installed on the lake surface A lake surface image database that stores a first captured image of the search area taken from above by an aerial vehicle when there is garbage in the search area and a second captured image of the search area taken from above by the aerial vehicle when there is no garbage in the search area. A stationary garbage upstream boundary database that stores data on the upstream boundary of the garbage staying in the search area obtained by continuously or intermittently measuring the time until the reflected wave of the detection wave horizontally emitted from the surface vehicle towards the net field returns while moving the surface vehicle along the net field on the upstream side of the net field on the lake surface at approximately the same time as the capture time of the first captured image. Comprising A learning model storage unit that stores a learning model obtained by machine learning the correlation between input data including comparison data obtained by comparing the first captured image and the second captured image stored in the lake surface image database, and data on the upstream boundary of the garbage staying in the search area stored in the stationary garbage upstream boundary database obtained at approximately the same time as the capture time of the first captured image, and output data including the occupancy rate of the garbage when the garbage staying in the search area is confirmed at approximately the same time as the capture time of the first captured image. An input data acquisition unit that acquires estimation input data including comparison data obtained by comparing an estimation captured image of the search area taken from above by an aerial vehicle and the second captured image stored in the lake surface image database, and data on the upstream boundary of the garbage staying in the search area obtained by continuously or intermittently measuring the time until the reflected wave of the detection wave horizontally emitted from the surface vehicle towards the net field returns while moving the surface vehicle along the net field on the upstream side of the net field on the lake surface at approximately the same time as the capture time of the estimation captured image. A garbage occupancy rate estimation unit that inputs the estimation input data acquired by the input data acquisition unit into the learning model to estimate the occupancy rate of the garbage staying in the search area on the lake surface. It may be configured to have.

[0018] In such a system, an estimated captured image obtained by photographing a search area from above by an aerial vehicle, and a second captured image previously captured by an aerial vehicle from above the search area when there is no debris in the search area, and the upstream boundary of the debris staying in the search area obtained by a water vehicle at approximately the same time as the time of capturing the estimated captured image, from which the occupancy rate of the debris in the search area of the lake surface is estimated using a learning model, so that it is possible to accurately estimate the occupancy rate of the debris.

[0019] In each of the above systems, it is desirable that the detection wave is composed of laser light emitted on the water surface along the lake surface and ultrasonic waves emitted into the water along the lake surface.

[0020] The state of the debris staying above the lake surface (the distance to the nearest debris staying above the lake surface, that is, the upstream boundary of the debris staying above the lake surface) can be grasped by the laser light emitted on the water surface, and the state of the debris staying below the lake surface (the distance to the nearest debris staying below the lake surface, that is, the upstream boundary of the debris staying below the lake surface) can be grasped by the ultrasonic waves emitted into the water.

[0021] In addition, a reflector may be fixed to the upper end side surface on the upstream side of the net field, and this reflector may be erected from above the water surface into the water and arranged toward the upstream side, and may be arranged continuously or at predetermined intervals along the net field to enable reflection of the detection wave horizontally emitted from the water vehicle toward the net field.

[0022] In such a configuration, if the detection wave emitted from the water vehicle does not hit the debris (if there is no debris between the water vehicle and the net field), it is reflected by the reflector, so the time until the reflected wave returns is a value determined by the distance between the water vehicle and the reflector, and it can be used as a reference value for grasping the staying state of the debris.

[0023] In addition, regarding the dust collection operation, if the occupancy rate of dust in the search area is below a predetermined value, no dust collection request is made, and if it exceeds the predetermined value, a dust collection request is made.

[0024] Also, if there is a rainfall forecast within a predetermined time near the lake surface or on the upstream side of the lake surface, and at least no dust collection operation has been performed within the past predetermined time, a collection request may be made.

Effect of the Invention

[0025] As described above, according to the present invention, it is possible to accurately grasp the retention amount of dust staying on the lake surface (occupancy rate of dust in the search area) and collect the dust in a timely manner.

Brief Description of the Drawings

[0026]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Embodiments for Carrying Out the Invention

[0027] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0028] In FIG. 1, a schematic configuration in which a dust retention amount grasping system 1 according to the present invention is applied to a dam lake is shown.

[0029] This dust retention amount grasping system 1 includes an aerial drone 2 as an airborne vehicle, a lake surface image database 3 that stores captured images taken by this aerial drone 2, a waterborne drone 4 as a waterborne vehicle, and a retention dust upstream boundary database 5 that stores data on the upstream boundary of the retention dust measured by this waterborne drone 4, and an occupancy rate calculation unit 6 that calculates the occupancy rate of the dust staying on the lake surface based on the captured images stored in the lake surface image database 3 and the data on the upstream boundary of the retention dust stored in the retention dust upstream boundary database 5.

[0030] The aerial drone 2 and the waterborne drone 4 are used to grasp the retention amount of the dust 11 staying in the search area 10 set upstream of the net field 8 installed on the lake surface 7 of the dam lake 70 shown in FIG. 2.

[0031] As also shown in FIG. 3, the net field 8 is configured by connecting floats 8b to a rope 8a stretched across the upstream side of the water intake 9 of the dam lake 70 and suspending a net 8c below the water surface. The search area 10 is a virtual area set upstream from the fishing net area 8 of the dam lake 70. It has a width that spans the entire length of the fishing net area 8 (the length across the dam lake 70), with the fishing net area 8 as the downstream boundary line, and a virtual line is set as the upstream boundary line at a predetermined distance upstream from the fishing net area 8. The example shown in FIG. 2 is a schematic diagram conceptually showing the dam lake 70. In this example, when the vertical direction on the paper surface across the dam lake 70 is taken as the X-axis direction and the horizontal direction on the paper surface from the upstream side connected to the river 12 of the dam lake 70 towards the downstream side with the water intake 9 is taken as the Y-axis direction, the search area 10 is shown as a rectangular area on the X-Y plane.

[0032] The aerial drone 2 is a known one equipped with a camera 22. By this aerial drone 2, a first captured image of the search area 10 taken from above by the aerial drone 2 when there is debris 11 in the search area 10 and a second captured image of the search area 10 taken from above by the aerial drone 2 when there is no debris 11 in the search area 10 are captured. The second captured image is pre-captured when almost all the debris 11 has been collected and disappeared, and it becomes the image data serving as a reference when determining the presence or absence of debris 11.

[0033] The water drone 4 moves the lake surface parallel to the fishing net area 8 at the upstream edge portion of the search area 10 (while being separated from the fishing net area 8 by a predetermined distance upstream). And in the process of moving, the water drone 4 horizontally emits a detection wave towards the fishing net area 8 and is capable of receiving the reflected wave of the detection wave.

[0034] As the detection wave, laser light and ultrasonic waves are used. The water drone 4 is provided with a light projecting unit 40a that projects laser light along the lake surface above the lake surface towards the fishing net area, a light receiving unit 40b that receives the reflected light of the projected laser light, an oscillation unit 40c that oscillates ultrasonic waves along the lake surface below the lake surface towards the fishing net area, and a receiving unit 40d that receives the reflected wave of the oscillated ultrasonic waves. It is preferable that both the laser light projected towards the fishing net area 8 and the ultrasonic waves oscillated towards the fishing net area 8 are emitted in a direction perpendicular to the traveling direction of the water drone 4. As shown in Fig. 3, a reflector 13 is fixed to the upper end side surface on the upstream side of the netting area 8. This reflector 13 is, for example, rectangular, vertically erected from above the water surface into the water, and arranged with the reflecting surface facing the upstream side. Also, a plurality of reflectors 13 are provided across the entire width of the lake surface (across the entire length of the netting area), and they may be arranged continuously along the netting area 8 or at predetermined intervals.

[0035] Then, the water drone 4 is moved on the X-axis while maintaining a predetermined distance from the netting area 8 (moving the upstream end of the search area 10), and in this process, detection waves (laser light and ultrasonic waves) are continuously or intermittently emitted from the water drone 4 toward the reflector 13 of the netting area 8. When there is no debris 11 between the water drone 4 and the netting area 8, the detection waves emitted from the water drone 4 reach the reflector 13 installed in the netting area 8, are reflected here, and if it is laser light, it is received by the light receiving unit 40b provided on the water drone 4, and if it is ultrasonic waves, it is received by the receiving unit 40d provided on the water drone 4. Also, when there is debris 11 between the water drone 4 and the reflector 13 provided in the netting area 8, the detection waves emitted from the water drone 4, if it is laser light, are reflected by the debris floating above the water surface at the closest distance from the water drone 4, and a part of the reflected wave is returned to the light receiving unit 40b, and if it is ultrasonic waves, it is reflected by the debris staying below the water surface (in the water) at the closest distance from the water drone 4, and a part of the reflected wave is returned to the receiving unit 40d. Therefore, by continuously or intermittently emitting detection waves from the water drone 4 toward the reflector 13 of the netting area 8 while moving the water drone 4 on the lake surface, and continuously or intermittently measuring the time until the reflected wave of the detection wave returns, data on the upstream boundary of all the debris staying in the search area can be obtained. That is, from the upstream boundary of the staying debris obtained in the process of moving the water drone 4, as shown by the dashed-dotted line in Fig. 2, the upstream boundary line of the debris existing in the search area (not only the upstream boundary line of the debris emerging above the lake surface but also the upstream boundary line of the debris existing in the water) is specified.

[0036] Using the above configuration, the process of automatically grasping the amount of waste staying in the search area 10 and requesting the operation of the waste cleaning boat as needed is preferably appropriately performed in consideration of the weather and the operation status of the waste cleaning boat in the past. Therefore, the presence or absence of the necessity of detecting waste in the search area is determined using the determination method shown in FIG. 4. That is, using a worker terminal (not shown), the weather data within the next 24 hours near the dam lake and in the upper reaches of the dam lake 70 is confirmed, and the operation status of the waste cleaning boat within the past 24 hours is confirmed (step S01). Based on this information, for example, the necessity of grasping the amount of waste staying in the search area is determined from the determination conditions shown in FIG. 5 (step S02). As a result, when it is determined that there is a necessity to detect waste, requests for the operation of the water drone and the aerial drone are made (step S03). In response to this, the aerial drone 2 and the water drone 4 are automatically operated, and based on the information obtained from these drones, in the occupancy rate calculation unit 6, the following process for grasping the amount of waste 11 staying in the search area 10 is executed. Note that the aerial drone and the water drone can also be manually operated regardless of the presence or absence of the necessity of grasping the amount of waste staying in the search area based on weather information.

[0037] The grasping of the amount of waste 11 staying in the search area 10 is performed by calculating the occupancy rate of the waste 11 in the search area 10. In the occupancy rate calculation unit 6, for example, the process shown in FIG. 6 is performed.

[0038] That is, the occupancy rate calculation unit 6 inputs the image data of the first captured image (the captured image captured by the aerial drone 2 when there is waste) obtained by the aerial drone 2 and the second captured image captured when there is no waste (step S11). The first captured image and the second captured image are compared, waste 11 is identified from objects or shadows that have not moved to the second captured image within the search area, and a primary calculation for primarily calculating the occupancy rate of the waste 11 in the search area 10 is performed from the horizontal projected area of the waste 11 staying in the search area 10 (step S12).

[0039] At this time, in the comparison between the first captured image and the second captured image, preprocessing such as appropriately processing both images to remove noise may be performed before the comparison. At this stage of the primary calculation, since the occupancy rate is calculated based only on information in one direction from above, it is assumed that there may be cases where accurate determination of dust and dirt becomes difficult due to the influence of the weather on the day, oil films on the water surface, muddy water after rain, etc. Also, with only image data from above, it is difficult to discriminate dust and dirt floating under the water surface (not exposed above the water surface). Therefore, a secondary calculation is performed to correct the occupancy rate of the dust and dirt calculated primarily based on the upstream boundary of the dust and dirt 11 in the search area 10 obtained by the water drone 4. That is, the data of the upstream boundary of the dust and dirt obtained by the water drone 4 is input (step S13). If the upstream boundary of the dust and dirt obtained by the water drone matches the upstream boundary of the dust and dirt grasped from the image data, the occupancy rate calculated primarily is used as it is. If the upstream boundary of the dust and dirt obtained by the water drone is different from the upstream boundary of the dust and dirt grasped from the image data, the occupancy rate calculated primarily is corrected based on the upstream boundary of the dust and dirt obtained by the water drone. Therefore, it becomes possible to grasp the occupancy rate of the dust and dirt 11 in the search area 10 more accurately compared to the case of using only the captured image from above, and based on this occupancy rate, it becomes possible to collect the dust and dirt in a timely manner.

[0040] That is, in this example, if the occupancy rate of the dust and dirt 11 in the search area 10 obtained by the secondary calculation is 10% or less, it is considered within the allowable range, and no operation request for the dust and dirt cleaning boat is made (steps S15, S16). If it exceeds 10% and is 20% or less, since the dust and dirt has accumulated to a certain extent, the operation of one dust and dirt cleaning boat is requested (steps S15, S17, S18). If the occupancy rate exceeds 20%, since there is a high request to immediately collect the dust and dirt to protect facilities such as water intake facilities, the operation of two dust and dirt cleaning boats is requested (steps S17, S19).

[0041] In the above operation, after determining the occupancy rate based on the captured image obtained from the aerial drone 2, an example of correcting the occupancy rate based on the upstream boundary of the debris obtained from the water drone 4 was shown. However, after determining the occupancy rate based on the upstream boundary of the debris obtained from the water drone 4, the occupancy rate may be corrected based on the captured image obtained from the aerial drone 2.

[0042] That is, for example, as shown in FIG. 7, the occupancy rate calculation unit 6 inputs the data of the upstream boundary of the debris 11 obtained by the water drone 4 (step S21), and based on the upstream boundary of the debris 11 in the search area 10 obtained by the water drone 4, the occupancy rate of the debris 11 in the search area 10 is primarily calculated (step S22). This primary calculation is based on the premise that in the dam lake 70, since water slowly moves from the upstream connected to the river 12 to the downstream river where the water intake 9 is provided, most of the debris is gathered at the net field 8 and stays in the upstream direction from the net field 8 without gaps. If the upstream boundary of the debris obtained by the detection wave emitted horizontally from the water drone is grasped, the occupancy rate of the debris can be calculated approximately accurately. However, the debris does not stay without gaps, and there are also quite a few non - existence areas where there is no debris. Therefore, simply calculating the occupancy rate based only on the upstream boundary of the debris obtained by the detection wave emitted horizontally from the water drone 4 does not mean that the calculation accuracy of the occupancy rate is sufficient. Therefore, the image data of the first captured image obtained by the aerial drone (the captured image taken by the aerial drone when grasping the retention amount of the debris) and the second captured image taken when there is no debris are input (step S23). These first and second captured images are compared to identify the existence state of the debris on the back side of the upstream boundary line from objects or shadows that have not moved to the second captured image within the search area. Based on that information, a secondary calculation is performed to correct the primarily calculated occupancy rate of the debris (step S24). That is, if the occupancy rate of dust in the search area identified by comparing the captured image obtained by the aerial drone with the captured image when there is no dust matches the occupancy rate grasped from the upstream boundary line, the primary calculated occupancy rate is used as it is. If the occupancy rate of dust in the search area grasped from the image data differs from the occupancy rate grasped from the upstream boundary line, the primary calculated occupancy rate is corrected based on the information obtained by comparing the captured image (the first captured image) obtained from the aerial drone 2 with the captured image (the second captured image) when there is no dust. Therefore, it becomes possible to accurately grasp the occupancy rate of the dust 11 within the search area 10, and based on this occupancy rate, it becomes possible to timely collect the dust 11.

[0043] That is, if the occupancy rate of the dust 11 in the search area 10 obtained by the secondary calculation is 10% or less, it is considered to be within the allowable range, and no operation request for the dust cleaning boat is made (steps S25, S26). If it exceeds 10% and is 20% or less, since the dust has accumulated to a certain extent, the operation of one dust cleaning boat is requested (steps S25, S27, S28). If the occupancy rate exceeds 20%, since there is a high request to immediately collect the dust and protect facilities such as water intake facilities, the operation of two dust cleaning boats is requested (steps S27, S29). In the above configuration, the calculation of the occupancy rate of the dust 11 in the search area 10 is to improve the accuracy of the occupancy rate of the dust 11 by correcting one of the occupancy rate calculated by comparing the captured image taken by the aerial drone 2 with the captured image when there is no dust in the search area 10 and the occupancy rate calculated based on the upstream boundary of the dust 11 in the search area 10 measured by the water drone 4 with the other. However, the calculation of the occupancy rate of the dust 11 in the search area 10 (grasping the retention amount in the search area) may be accurately performed using a learning model. Such a configuration example is shown in FIGS. 8 to 12. Hereinafter, a dust retention amount grasping system 1 that estimates the occupancy rate of the dust 11 in the search area 10 using a learning model will be described. As shown in FIG. 8, this dust retention amount grasping system 1 includes an aerial drone 2 (the aerial drone 2a used in the learning phase and the aerial drone 2b used in the estimation phase may be the same or different), a water drone 4 (the water drone 4a used in the learning phase and the water drone 4b used in the estimation phase may be the same or different), a lake surface image database 3, a retained dust upstream boundary database 5, an image analysis device 14 (an image analysis device 14a for analyzing an image taken by the aerial drone and an image analysis device 14b for analyzing an image taken by the aerial drone), a machine learning device 15, a dust occupancy rate estimation device 16, and a display 17.

[0044] Since the aerial drone 2 and the water drone 4 are the same as those shown in FIGS. 1 to 3, their description is omitted. Also, since the lake surface image database 3 and the retained dust upstream boundary database 5 are the same as those shown in FIG. 1, their description is omitted. The image data stored in the lake surface image database 3 and the data of the upstream boundary of the dust stored in the retained dust upstream boundary database 5 are used as needed when forming the learning model 18 in the learning phase. Also, an image (second captured image) captured when there is no dust in the search area 10 stored in the lake surface image database 3 is used to compare with the captured image of the search area 10 captured by the aerial drone 2 (2b) in the estimation phase. In the learning phase, the captured image taken by the aerial drone 2 is temporarily stored in the lake surface image database 3 to keep its record, but it may also be directly sent to the image analysis device 14a. Also, in the estimation phase, the image taken by the aerial drone 2b is directly sent to the image analysis device 14b, but it may also be temporarily stored in an image database (not shown) for preprocessing such as removing unnecessary image parts.

[0045] The image analysis device 14 is used to compare a captured image (first captured image) of the search area 10 taken by the aerial drone 2 when there is garbage 11 in the search area 10 with a captured image (second captured image) of the search area taken in advance when there is no garbage (for example, after collecting the garbage in the search area 10), and to obtain comparison result information (such as the shadow of an object, a discolored area, and their regions) where a change is recognized in the search area 10. The image analysis device 14a used in the learning phase compares a past first captured image with a second captured image to obtain comparison result information, and is used to output this comparison result information to the machine learning device 15. Also, the image analysis device 14b used in the estimation phase compares the captured image at the time when the first captured image is taken with the second captured image stored in the lake surface image database 3 to obtain comparison result information, and is used to output this comparison result information to the garbage occupancy rate estimation device 16.

[0046] The machine learning device 15 operates as the main body in the learning phase, and generates a learning model 18 used for estimating the occupancy rate of the garbage 11 in the search area 10 by machine learning from the comparison result information obtained by comparing a captured image (first captured image) of the search area 10 taken by the aerial drone 2 when there is garbage in the search area 10 with a captured image (second captured image) when there is no garbage in the search area 10, and the upstream boundary of the garbage 11 in the search area 10 obtained by the water drone 4.

[0047] The garbage occupancy rate estimation device 16 operates as the main body in the estimation phase, and estimates the occupancy rate of the garbage in the search area based on the comparison result information obtained by comparing the captured image taken by the aerial drone with the second captured image using the learned learning model 18 generated by the machine learning device 15, and the upstream boundary of the garbage in the search area measured by the water drone.

[0048] The display 17 can be substituted by the display screen of various terminals, and displays information on the occupancy rate of the garbage estimated by the garbage occupancy rate estimation device 16, the determination result on the necessity of garbage collection based on the estimated occupancy rate of the garbage, a request for operation of the garbage cleaning boat, etc.

[0049] The machine learning device 15 is configured by a general-purpose or dedicated computer. This computer may be a stationary computer or a portable computer, and may also be a client computer, a server computer, or a cloud computer.

[0050] As shown in FIG. 9, the machine learning device 15 includes a learning data acquisition unit 51, a learning data storage unit 52, a machine learning unit 53, and a learned model storage unit 54.

[0051] The learning data acquisition unit 51 functions as an interface unit that acquires learning data including input data and output data via the communication network 20. The learning data acquisition unit 51 is connected to the image analysis device 14a, the operator terminal 21 used by the operator, etc. via the communication network 20.

[0052] The learning data storage unit 52 is a database that stores a plurality of sets of the learning data acquired by the learning data acquisition unit 51.

[0053] The machine learning unit 53 performs machine learning using the learning data stored in the learning data storage unit 52. The machine learning unit 53 inputs a plurality of sets of learning data into the learning model 18, causing the learning model 18 to learn the correlation between the input data included in the learning data and the output data corresponding to the input data, and generating a learned model.

[0054] There are various methods for generating a learned model. For example, it is advisable to adopt a neural network in which the machine learning unit 53 performs supervised learning.

[0055] The learned model storage unit 54 is a database that stores the learned learning model 18 generated by the machine learning unit 53. The learning model 18 stored in the learned model storage unit 54 is provided to the dust occupancy rate estimation device 16 via an arbitrary communication network or storage medium.

[0056] Note that the learning data storage unit 52 and the learning model storage unit 54 may be configured by a single storage device or by separate storage devices.

[0057] As learning data, as input data, the first captured image (an image captured from above the search area by an aerial drone when there is debris in the search area) stored in the lake surface image database 3, and the second captured image (an image captured from above the search area by an aerial drone when there is no debris in the search area), and comparison result information obtained by comparing them, and data on the upstream boundary of the debris 11 in the search area 10 stored in the stagnant debris upstream boundary database 5 (when the water drone 4 is moved along the net field 8 on the upstream side of the net field 8 on the lake surface at approximately the same time as the shooting time of the first captured image, and in the process of moving it, the time until the reflected wave of the detection wave horizontally emitted from the water drone 4 returns toward the net field is continuously or intermittently measured), and at least includes the data of the upstream boundary of the debris in the search area obtained in this way.

[0058] Here, the captured image from above the search area 10 only reflects the state of the debris downstream of the upstream debris based on the data of the upstream boundary of the debris obtained by the water drone. Therefore, in order to confirm or complement the analysis result based only on the horizontal data, it is necessary to increase the accuracy of the occupancy rate of the debris by taking into account information from other angles (information from above).

[0059] Similarly, the data on the upstream boundary of the debris 11 in the search area 10 may make it difficult to uniformly determine the debris on the lake surface due to weather, season, etc. based only on the data of the captured image of the search area taken from the air. Therefore, in order to confirm or complement the analysis result based only on the one-directional data taken from above, it is necessary to increase the accuracy of the occupancy rate of the debris by taking into account information from other angles (information from the horizontal direction).

[0060] (Machine learning method) The machine learning device 15 forms a learning model using, for example, a neural network model. This neural network model is a model known per se, which inputs input data included in learning data into an input layer, and learns the correlation between the input data and the output data by comparing the output data output from the output layer as an estimation result with the output data (teacher data) included in the learning data.

[0061] First, as a preliminary preparation for causing the machine learning device 15 to perform machine learning, the learning data acquisition unit 51 acquires a desired number of pieces of learning data, and stores the acquired plurality of sets of learning data in the learning data storage unit 52. The number of pieces of learning data to be prepared here is appropriately set in consideration of the estimation accuracy required for the learning model 18.

[0062] The learning data can be prepared in various ways. For example, a captured image (first captured image) from above the search area by the aerial drone 2 at the time of collecting past dust, and a captured image (second captured image) of the search area taken in advance by the aerial drone when there is no dust in the search area 10 are compared to obtain comparison result information (shadows of objects, discolored areas, their regions, etc.), and at approximately the same time as the capture time of the first captured image, the water drone 4 is moved along the net field 8 on the upstream side of the net field on the lake surface, and the time until the reflected wave of the detection wave horizontally emitted from the water drone 4 towards the net field 8 returns is continuously or intermittently measured to obtain data on the upstream boundary of the dust in the search area, which is used as input data, and the occupancy rate of the dust in the search area confirmed when actually performing the recovery operation of the dust staying in the search area is used as output data, and these are associated with the capture time of the first captured image to create a set of learning data. After that, a large number of such pieces of learning data are prepared and stored in advance in the learning data storage unit 52 as learning data for generating a learning model.

[0063] After the above preparations, the machine learning unit 53 performs a series of processes in the flowchart shown in FIG. 10, for example, to generate a learned model.

[0064] In step S100, the machine learning unit 53 acquires a set of learning data from the plurality of sets of learning data stored in the learning data storage unit 52. The acquisition of the learning data may be performed in a predetermined order or randomly.

[0065] Next, in step S110, the machine learning unit 53 performs supervised machine learning. That is, the machine learning unit 53 inputs the input data included in the acquired set of learning data into the prepared learning model 18 to output an estimation result, compares this estimation result with the output data (teacher data) included in the learning data acquired in step S100, and performs machine learning by a known neural network method. Thereby, the machine learning unit 53 causes the learning model 18 to learn the correlation between the input data and the output data (occupancy rate of dust and dirt).

[0066] Thereafter, using the learning model being learned, the above operations are performed on the remaining learning data prepared, and the learning in the machine learning unit 53 is continued. At this time, the machine learning unit 53 determines whether to continue the machine learning based on the error between the output data and the teacher data, the number of learning times, or the remaining number of unlearned learning data stored in the learning data storage unit 52 (step S120).

[0067] That is, in step S120, when the machine learning unit 53 determines to continue the machine learning (YES), the processes of steps S100 to S110 are performed on the learning model 18 being learned using the unlearned learning data. On the other hand, when the machine learning unit 53 determines not to continue the machine learning (NO) in step S120, in step S130, the generated learned learning model 18 is stored in the learning model storage unit 54, and the machine learning is terminated.

[0068] (Dust and Dirt Occupancy Rate Estimation Device) Next, a garbage occupancy estimation device 16 that estimates the occupancy rate of garbage 11 in the search area 10 using the learned learning model 18 generated by the above-described method will be described.

[0069] The garbage occupancy estimation device 16 is configured by a general-purpose or dedicated computer. This computer may be a stationary computer or a portable computer, and may also be a client-type computer, a server-type computer, or a cloud-type computer. As shown in FIG. 11, the garbage occupancy estimation device 16 includes an input data acquisition unit 61, an estimation unit 62, a learning model storage unit 63, and an output processing unit 64.

[0070] The input data acquisition unit 61 compares, by the image analysis device 14b, a captured image (estimated captured image) of the search area 10 taken by the aerial drone with a captured image (second captured image) of the search area 10 when there is no garbage in the search area 10 stored in the lake surface image database 3, and obtains a set of data including the comparison data and the data of the upstream boundary of the garbage 11 in the search area 10 obtained by the water drone 4 at approximately the same time as the capture time of the estimated captured image as input data.

[0071] When a set of input data is input by the input data acquisition unit 61, the estimation unit 62 inputs the input data acquired by the input data acquisition unit 61 into the learning model 18 stored in the learning model storage unit 63 and performs an estimation process (estimates the garbage occupancy rate in the search area 10). This estimation process uses the learned learning model 18 learned by the machine learning device 15.

[0072] The estimation unit 62 may include not only a function of performing an estimation process using the learning model 18, but also a preprocessing function of adjusting the input data acquired by the input data acquisition unit 61 to a desired format or the like and inputting it into the learning model 18 as a preprocessing of the estimation process.

[0073] The learning model storage unit 63 is a database that stores the learned learning model 18 used in the estimation process of the estimation unit 62. Note that the number of learning models 18 stored in the learning model storage unit 63 is not limited to one. For example, when the number of input data is different or when the machine learning methods are different, a plurality of learning models 18 may be stored and appropriately selected and used.

[0074] Also, as a post - process of the estimation process, based on the estimated occupancy rate of dust and dirt, a determination shown in FIG. 12 (a determination process equivalent to the processes of steps S15 - S19 in FIG. 6 and steps S25 - S29 in FIG. 7) for determining the necessity of dust and dirt collection may be performed.

[0075] Therefore, according to the dust and dirt retention amount grasping system 1 described above, using the learning model learned based on the past captured images of the search area from the aerial drone or the upstream boundary of the dust and dirt measured by the water - based drone, the occupancy rate of the dust and dirt 11 in the search area 10 is estimated, so that it is possible to accurately estimate the occupancy rate of the dust and dirt.

[0076] Note that, as a specific method of machine learning by the machine learning unit 53 above, the case of adopting a neural network has been described. However, the machine learning unit 53 may adopt any other machine learning method. For example, tree - type such as decision tree and regression tree, ensemble learning such as bagging and boosting, neural network - type (including deep learning) such as recursive neural network and convolutional neural network, clustering - type such as hierarchical clustering, non - hierarchical clustering, k - nearest neighbor method, k - means method, multivariate analysis such as principal component analysis, factor analysis, logistic regression, support vector machine, etc. may be adopted.

[0077] In addition, each step of the dust retention amount grasping system described above can also be provided in the form of a program for executing the same. Further, the above-described machine learning device 15 and the dust occupancy rate estimation device 16 can also be provided in the form of a program (machine learning program and retention amount grasping program) for executing each step included in the above-described machine learning method and estimation method.

Explanation of Signs

[0078] 1 Dust retention amount grasping system 2 Aerial drone 3 Lake surface image database 4 Water drone 5 Retained dust upstream boundary database 7 Lake surface 8 Net field 10 Search area 11 Dust 18 Learning model 54,63 Learning model storage unit 61 Input data acquisition unit

Claims

1. A dust retention amount grasping system for grasping the dust retention amount of dust staying in a predetermined search area on the upstream side of a net field installed on a lake surface, a primary calculation means for calculating the occupancy rate of dust staying in the search area based on information obtained by comparing a first captured image of the search area taken from above by an airborne vehicle with a second captured image of the search area taken from above by the airborne vehicle when there is no dust in the search area; a secondary calculation means for specifying the upstream boundary of the dust staying in the search area by continuously or intermittently measuring the time until the reflected wave of a detection wave horizontally emitted from the waterborne vehicle toward the net field returns during the process of moving the waterborne vehicle along the net field on the upstream side of the net field on the lake surface at approximately the same time as the shooting time of the first captured image, and correcting the occupancy rate of the dust calculated by the primary calculation means based on the upstream boundary; A dust retention amount grasping system, characterized by comprising the above.

2. A dust retention amount grasping system for grasping the dust retention amount of dust staying in a predetermined search area on the upstream side of a net field installed on a lake surface, a primary calculation means for specifying the upstream boundary of the dust staying in the search area by continuously or intermittently measuring the time until the reflected wave of a detection wave horizontally emitted from the waterborne vehicle toward the net field returns during the process of moving the waterborne vehicle along the net field on the upstream side of the net field on the lake surface, and calculating the occupancy rate of the dust staying in the search area based on the upstream boundary; a secondary calculation means for correcting the occupancy rate of the dust calculated by the primary calculation means based on information obtained by comparing a first captured image of the search area taken from above by an airborne vehicle with a second captured image of the search area taken from above by the airborne vehicle when there is no dust in the search area at approximately the same time as the time when the waterborne vehicle moves along the net field on the lake surface, A dust retention amount grasping system, characterized by comprising the above.

3. A dust retention amount grasping system for grasping the retention amount of dust staying in a predetermined search area on the upstream side of a net field installed on a lake surface, A lake surface image database that stores a first captured image obtained by photographing the search area from above by an aerial vehicle when there is dust in the search area and a second captured image obtained by photographing the search area from above by the aerial vehicle when there is no dust in the search area, At approximately the same time as the shooting time of the first captured image, a water vehicle is moved along the net field on the upstream side of the net field on the lake surface, and the time until the reflected wave of the detection wave horizontally emitted from the water vehicle towards the net field returns is continuously or intermittently measured. A retention dust upstream boundary database that stores data on the upstream boundary of the dust staying in the search area obtained thereby, Comprising, A learning model storage unit that stores a learning model obtained by machine learning the correlation between input data including comparison data obtained by comparing the first captured image and the second captured image stored in the lake surface image database and data on the upstream boundary of the dust staying in the search area stored in the retention dust upstream boundary database obtained at approximately the same time as the shooting time of the first captured image, and output data including the occupancy rate of the dust when the dust staying in the search area is confirmed at approximately the same time as the shooting time of the first captured image, An input data acquisition unit that acquires estimation input data including comparison data obtained by comparing an estimation captured image obtained by photographing the search area from above by an aerial vehicle with the second captured image stored in the lake surface image database, and data on the upstream boundary of the dust staying in the search area obtained by continuously or intermittently measuring the time until the reflected wave of the detection wave horizontally emitted from the water vehicle towards the net field returns during the process of moving the water vehicle along the net field on the upstream side of the net field on the lake surface at approximately the same time as the shooting time of the estimation captured image, A dust occupancy rate estimation unit that inputs the input data for estimation acquired by the input data acquisition unit into the learning model to estimate the occupancy rate of dust staying in the search area on the lake surface. A dust retention amount grasping system, characterized by comprising the same.

4. The detection wave is composed of laser light emitted onto the water surface along the lake surface and ultrasonic waves emitted into the water along the lake surface, and the dust retention amount grasping system according to any one of claims 1 to 3.

5. A reflector is fixed to the upper end side surface on the upstream side of the net field, and this reflector is erected from above the water surface into the water with the reflecting surface facing the upstream side, and is arranged continuously or at predetermined intervals along the net field, so that the detection wave horizontally emitted from the watercraft toward the net field can be reflected, and the dust retention amount grasping system according to any one of claims 1 to 3.

6. A method for grasping the retention amount of dust staying in a predetermined search area on the upstream side of a net field installed on the lake surface, comprising: A primary calculation step of calculating the occupancy rate of dust staying in the search area based on information obtained by comparing a first captured image captured by an airborne vehicle from above the search area with a second captured image captured by the airborne vehicle from above the search area when there is no dust in the search area; At a time approximately the same as the shooting time of the first captured image, a watercraft is moved along the net field on the upstream side of the lake surface of the net field, and the upstream boundary of the dust staying in the search area is specified by continuously or intermittently measuring the time until the reflected wave of the detection wave horizontally emitted from the watercraft toward the net field returns during the moving process, and a secondary calculation step of correcting the occupancy rate of the dust calculated by the primary calculation step based on the upstream boundary; A method for grasping the retention amount of dust, characterized by comprising the same.

7. A method for grasping the amount of debris staying in a predetermined search area on the upstream side of a net field installed on a lake surface, moving a water vehicle along the net field on the upstream side of the net field on the lake surface, and continuously or intermittently measuring the time until the reflected wave of the detection wave horizontally emitted from the water vehicle toward the net field returns during the moving process to identify the upstream boundary of the debris staying in the search area, and a primary calculation step of calculating the occupancy rate of the debris staying in the search area based on the upstream boundary; a secondary calculation step of correcting the occupancy rate of the debris calculated by the primary calculation step based on information obtained by comparing a first captured image of the search area taken from above by an air vehicle at almost the same time as the time when the water vehicle moves along the net field on the lake surface with a second captured image of the search area taken from above by the air vehicle when there is no debris in the search area; A method for grasping the amount of debris staying, characterized by comprising the above.

8. a lake surface image database for storing a first captured image of the search area taken from above by an air vehicle when there is debris in a predetermined search area on the upstream side of a net field installed on the lake surface and a second captured image of the search area taken from above by the air vehicle when there is no debris in the search area; a staying debris upstream boundary database for storing data of the upstream boundary of the debris staying in the search area obtained by moving a water vehicle along the net field on the upstream side of the net field on the lake surface at almost the same time as the time of capturing the first captured image and continuously or intermittently measuring the time until the reflected wave of the detection wave horizontally emitted from the water vehicle toward the net field returns during the moving process; a method for grasping the amount of debris staying in the search area on the lake surface using the above, An input data acquisition step of acquiring estimation input data including comparison data obtained by comparing an estimation captured image obtained by photographing the search area from above by an airborne vehicle with the second captured image stored in the lake surface image database, and data on the upstream boundary of the debris staying in the search area obtained by continuously or intermittently measuring the time until the reflected wave of a detection wave horizontally emitted from the waterborne vehicle toward the net field returns along the net field on the upstream side of the net field on the lake surface at substantially the same time as the shooting time of the estimation captured image. Using a learning model obtained by machine learning the correlation between input data including comparison data obtained by comparing the first captured image and the second captured image stored in the lake surface image database, and data on the upstream boundary of the debris staying in the search area stored in the upstream boundary database of the staying debris obtained at substantially the same time as the shooting time of the first captured image, and output data including the occupancy rate of the debris when the debris staying in the search area is confirmed at substantially the same time as the shooting time of the first captured image, input the estimation input data acquired in the input data acquisition step into the learning model to estimate the occupancy rate of the debris staying in the search area of the lake surface. This is a debris occupancy rate estimation step. A method for grasping the amount of debris staying, characterized by comprising the above steps.

9. The method for grasping the amount of debris staying according to any one of claims 6 to 8, characterized in that the processing of each step is executed when there is a rainfall forecast in the vicinity of the lake surface or on the upstream side of the lake surface within a predetermined time, and at least no debris collection work has been carried out within the past predetermined time.

10. The method for grasping the amount of debris staying according to any one of claims 6 to 8, further comprising a step of determining whether debris collection is required or not, in which if the occupancy rate of the debris staying in the search area is equal to or less than a predetermined value, no request for debris collection is made, and if it exceeds the predetermined value, a request for debris collection is made.

11. A dust retention amount grasping program for causing a computer to execute each step included in the dust retention amount grasping method according to any one of claims 6 to 8.

Images