Water environment remote sensing device, water environment remote sensing system, and method for generating time lapse video

Abstract

To generate a time-lapse video suitable for monitoring water environment.SOLUTION: A sensor node 110-1 is equipped with: a camera 111-1 that captures an image at predetermined time intervals; and a water level sensor 113-1. A master station 102-1 includes a water level determination unit 114, and shortens a capturing interval of the camera 111 - 1 when a water level measured by the water level sensor 113 - 1 exceeds a predetermined level. The master station 102-1 includes a time-lapse video image generation unit 105, and when generating a time-lapse video image from an image captured in the past stored in a storage device 106, generates the time-lapse video image in which reproduction time of each frame of the time-lapse video image is uniform, meaning that a time compression rate is uniform, using capturing time or capturing interval information of the captured image.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to a remote sensing device using a camera for use in an aqueous environment, a remote sensing system, and a method for generating a time lapse video. [Background technology]

[0002] In the water environment domain, represented by water supply and sewerage systems, waterways, rivers, lakes, and marshes, there is an increasing need for sensing of water levels and water quality from the perspective of flood prevention, environmental improvement, and pollution prevention. Generally, water environments are characterized by the fact that the areas and facilities to be measured are large, access to power sources and communication cables is poor, measurement points are not suitable for installing information devices due to the risk of submersion or corrosion, and workers have difficulty accessing the areas, making them often unsuitable for installing information devices. In such cases, remote sensing is used, in which sensors or sensor nodes are placed at the observation points and the measurement results are transmitted to a main unit (parent unit) placed at a distance.

[0003] Water environment indices measured by such remote sensing vary depending on the purpose, including water level, water quality (pH, turbidity, salinity, electrical conductivity, various ion concentrations, etc.), water temperature, flow velocity, flow rate, etc. In recent years, cameras that capture still images and videos for the purpose of monitoring water levels, the presence or absence of foreign objects or intruders, structural abnormalities, etc., have also become an important monitoring method, and are being introduced to monitor rivers, harbors, weirs, etc.

[0004] Fig. 1 shows an example of the configuration of a water environment monitoring device equipped with a conventional camera, as disclosed in JP 2014-70998 A "Flowing Water Monitoring Device" (Patent Document 1). The flowing water monitoring device (main body case) 300 is a device that is installed in a sewer pipe and operates for a long period of time. Its main power source is an internal battery 308, and this battery is supplementarily charged by the electromotive force of a generator 309 that generates electricity from the flow of sewage. The power charged in the battery is supplied via a power supply circuit 306 to a main board 301 equipped with a main CPU and a low-power microcomputer 305. This monitoring device is equipped with a monitoring camera 302 and a water level sensor 307 as sensors, and the measured sewage water level and captured images are recorded in an external storage medium 303, and can be taken out from a communication module 304 via a wireless line or the like as necessary.

[0005] This monitoring device operates on limited power from batteries and hydroelectric power generated by the sewage, so if the frequency of image capture, which consumes a lot of power, is increased, the battery will be consumed in a short period of time, and the device's operating time will be shortened, which is a problem. For this reason, the monitoring camera 302 is usually started at long intervals, such as every hour, to capture images inside the sewer pipe, and when the capture is completed, the main board 301 or the monitoring camera 302 is turned off to save power. On the other hand, the low-power microcomputer 305 is started at shorter intervals of 10 minutes to observe the sewer water level using the water level sensor 307, and when the sewer water level exceeds a specified value due to rainfall, etc., the main board and the monitoring camera are started to capture images. In this way, a method is described in which the frequency of image capture is dynamically changed from the usual one-hour interval to the 10-minute interval when the water level rises. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2014-70998 A Summary of the Invention [Problem to be solved by the invention]

[0007] In addition to the operating power consumption mentioned above, a major issue with the image monitoring of water environments using cameras is the sudden increase in the load of evaluating the captured images, making effective use of the images difficult. Increasing the frequency of capture provides more detailed and real-time monitoring information, but at the same time, the number of captured images increases to 144 images per day when captured every 10 minutes, and 1,440 images per day when captured every minute. The task of visually checking such a large number of similar images one by one is time-consuming and mentally taxing for monitors, and mistakes such as overlooking abnormalities and changes increase. Increasing the frequency of capture in this way only increases the burden on workers and does not necessarily improve monitoring efficiency.

[0008] Another issue is the difficulty of providing the large amount of images captured by remote sensing devices equipped with cameras in a format that is easy for users to use in real time. There are various purposes for image monitoring of water environments, such as "I want to know the water level fluctuations over the past few hours," "I want to see the behavior of the water level during a specific rainfall in the past," and "I want to get an overview of the operating status of water environment facilities on the previous day." Each purpose requires viewing hundreds to thousands of images taken during the relevant period, but even with a large number of images, it is difficult to grasp trends and changes in the situation in a short period of time, resulting in poor user convenience. [Means for solving the problem]

[0009] The above-mentioned problems of the present invention can be solved by providing a camera-mounted water environment remote sensing device with a function for generating time lapse videos.

[0010] Furthermore, if the camera is equipped with a camera that can change the shooting interval in order to save power or improve shooting efficiency, changes in shooting frequency will result in uneven playback speed of the time lapse video, i.e., uneven time compression rate. However, this can be solved by having a function to generate a time lapse video using the shooting time or shooting interval information of each captured image, and making the playback time of each frame image of the generated time lapse video approximately uniform in time with respect to the shooting time, i.e., making the time compression rate uniform.

[0011] Furthermore, the uniformity of the playback speed (time compression rate) of time-lapse videos as described above is particularly effective when the sensor node or other sensor node equipped with a camera has a function for measuring water environment information such as water level and water quality, and has a mechanism for shortening the camera's shooting interval when any of these measured values ​​exceeds a first predetermined value and for extending the camera's shooting interval when any of these measured values ​​falls below a second predetermined value. This also applies when the camera, sensor node, or higher-level device has an image processing function for measuring water level and flow rate from camera images and varies the measurement interval according to the measured water level and flow rate, when the device has a function for receiving water environment information or external commands from the outside and varies the camera's shooting interval according to the received water environment information or commands, or when the camera, sensor node, or higher-level device has an image processing function for analyzing camera images taken, and has a function for shortening the shooting interval during and after the detection of a foreign object or an abnormal image is detected in the image.

[0012] In addition, the present invention can generate a time lapse video in which the playback time of each frame image is uniform with respect to the shooting time by first varying the frame rate during playback in proportion to the shooting interval of the still images. However, in reality, the frame rate cannot be set to a completely arbitrary value due to restrictions such as video standards and playback applications, so if there are no images captured at the exact shooting time corresponding to each frame when generating the variable frame video, images captured at the nearest previous or next time can be copied, and if there are too many still images, still images can be thinned out, thereby generating a time lapse video in which the playback time is approximately uniform with respect to the shooting time, i.e., the time compression rate is approximately uniform.

[0013] As a second method, the time lapse video to be generated can be a fixed frame rate video, and each frame image that makes up the video can be generated by selecting and copying an image captured at a shooting time determined by the product of the playback time from the beginning of the video and the time compression rate of the time lapse video, or an image captured immediately before or after that shooting time.

[0014] In addition, when providing time-lapse videos, the time of shooting or the measurement value of the water environment information and the time of measurement can be burned into each frame image as a string of characters, making it easier to recognize the time the image was shot or the time of measurement, thereby increasing its effectiveness.

[0015] As described above, the sensing system of the present invention can effectively utilize captured images by generating and providing a time-lapse video of any place and time according to the user's request. Such a mechanism for providing a video can be realized by generating a time-lapse video of a specified time from images captured by a specified camera from images captured in the past, in response to a user's request sent via a network or communication path, and returning the video to the requesting user or making the video viewable via a network. It can also be achieved by providing a function for generating a time-lapse video in which the playback time of each frame image constituting the video is approximately uniform with respect to the shooting time, i.e., the time compression rate is approximately uniform, and automatically generating a time-lapse video using images captured by a specified camera from images captured in the past at a predetermined or specified time interval, and storing the video in a predetermined location or making the video viewable via a network. Effect of the Invention

[0016] The present invention has the effect of significantly reducing the time required to check the captured images and understand trends by providing a large number of images obtained by surveillance cameras of water environments as time-lapse videos, thereby reducing the burden on workers.

[0017] In addition, when the camera's shooting frequency is changed, the playback time of each frame of the generated time-lapse video is made uniform to the shooting time, i.e., the time compression rate is made approximately uniform. This makes it possible to correctly convey trends of change, such as the rate of water level rise and the rapid increase, to viewers, making it easier to predict operational information such as dam overflow and the time until the water level reaches the storage limit.

[0018] In addition, by converting images captured over a period from the past to the present specified by the user into a time-lapse video and providing it to the user, it is possible to significantly improve the viewability, convenience, and usability of the captured images. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a configuration diagram showing a water environment monitoring device equipped with a conventional camera. [Diagram 2] FIG. 11 is an explanatory diagram showing the relationship between the shooting time and the playback time in a time lapse camera of related art having a mechanism for varying the shooting frequency. [Diagram 3] FIG. 1 is a configuration diagram showing a first embodiment. [Figure 4] 4A to 4C are explanatory diagrams showing a method of generating a time lapse video with a variable frame rate in the first embodiment. [Diagram 5] FIG. 11 is a configuration diagram showing a second embodiment. [Figure 6] FIG. 11 is an explanatory diagram showing a method for generating a time lapse video at a fixed frame rate in the second embodiment. [Figure 7] FIG. 11 is a configuration diagram of a water environment sensing system in a third embodiment. [Figure 8] FIG. 13 is an explanatory diagram showing an example of writing shooting time and water environment measurement data to a time lapse video in Example 4. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Hereinafter, several embodiments of the present invention will be described with reference to the drawings. EXAMPLES

[0021] Time-lapse video is considered to be an effective way to view a large number of images taken by cameras mounted on remote monitoring devices for aquatic environments. Time-lapse video is a technique that connects still images taken at regular intervals (or still images extracted at specified intervals from a long video) and converts them into a short video, allowing long periods of video footage, such as several hours to several days, to be observed in a short time as a fast-forwarded video of a few minutes to a few seconds. Time-lapse video has the advantages of being able to view a large number of images in a short time, and making it easy to grasp changes and trends in the images, and is used for monitoring events such as the appearance of animals at night captured by fixed cameras, and for time-saving observations of the slow growth of plants and the movement of celestial bodies.

[0022] Fig. 3 is a configuration diagram showing Example 1 of the present specification, and shows a basic embodiment. Sensor nodes 110-1 and 110-2 are placed near monitoring points of the water environment, such as the water surface at key points in sewer pipes, confluences and dams of sewer pipes, outlets into rivers, and levees. This example shows an example in which one camera 111-1 is connected to the sensor node 110-1, and two cameras 111-2 and 111-3 are connected to the sensor node 110-2 to monitor images of the monitoring points.

[0023] The parent station 102-1 is a higher-level management device connected to multiple sensor nodes 110-1, 110-2 via a node connection cable 112, and sends measurement and photography commands to each sensor node, manages the status of each sensor node, and collects and stores measurement data. The remote monitoring terminal (RT) 100 is a higher-level device installed in a local government or management company that monitors the water environment. The remote monitoring terminal 100 is connected to multiple parent stations 102-1, 102-2 via a network 101, and has the role of viewing the measurement data and images collected by each parent station and remotely managing each device.

[0024] Additionally, the master station 102-1 includes a CPU 103, a real-time clock 104, a time lapse video generator 105, and a storage device 106. The CPU 103 sends image capture commands to the sensor nodes 110-1 and 110-2 at predetermined intervals based on the real-time clock 104.

[0025] When the sensor node 110-1 receives an image capture command, it captures an image with the camera 111-1, receives the captured image 107-1 from the camera, and sends it back to the parent station 102-1. The CPU 103 in the parent station 102-1 stores the captured image 107-2 sent from the sensor node 110-1 in the storage device 106 together with the individual numbers of the sensor node and camera that captured the image (110-1, 111-1 in this example), and information such as the capture time or capture interval obtained from the real-time clock 104.

[0026] The time lapse video generator 105 generates a time lapse video by using previously captured images stored in the storage device 106 and their capture time information and capture interval information. The timing and period for generating the time lapse video can be set arbitrarily. For example, a time lapse video for a predetermined period may be automatically generated at a predetermined timing, such as once a day for one day, or a time lapse video may be generated each time according to a date, time period, etc. specified in a command sent from the remote monitoring terminal 100.

[0027] In addition, the node connection cable 112 in the figure can be a wired electric cable such as a general electric multi-core cable or an Ethernet cable (Ethernet is a registered trademark), and this cable can be used to transmit uplink and downlink communication signals between the parent station and the sensor node. In addition to this, optical fiber lines or wireless communication can also be used, and in the latter case, the node connection cable 112 would not actually exist.

[0028] The operating power of the sensor node can be supplied using the node connection cable 112, or other means can be used in combination. For example, a power supply wire can be provided in a multi-core communication cable, or the communication cable can be used for power supply as is, as in PoE (Power over Ethernet). Optical power supply technology can also be used, which uses optical fiber to transmit power from the parent station to the sensor node using a strong laser light, and the sensor node can be battery-operated. Any combination of the above means can also be used, such as wireless communication and battery operation, or optical fiber communication and power supply via an electric cable.

[0029] When using existing fixed cameras to monitor events such as the appearance of animals, mechanisms that increase the frequency of shooting by detecting animals using a motion sensor or abnormalities in the camera image may be used to save power for shooting and image storage capacity. Changing the shooting frequency in this way is also an effective function for aquatic environment remote sensing devices, in terms of reducing the storage capacity required to store images and reducing operating power. This is particularly important when the sensor node is battery-operated or when using a power supply method with limited power supply capacity, such as optical fiber power supply or environmental harvesting.

[0030] Figure 2 shows the relationship between image capture by a time-lapse camera, a related technology with a mechanism for varying the capture frequency, and the playback images. The horizontal axis in Figure 2(a) shows the actual capture time of the images. The inverted triangles indicate the timing of image capture, and #1 to #15 are the sequential numbers of the captured images. In this figure, images #1 to #4 were captured at 10-minute intervals, and for #4 to #12, the capture interval was shortened to 30 seconds when an event such as the appearance of an animal was detected, and then after the event ended in #12, the capture interval was returned to 10 minutes.

[0031] Meanwhile, Figure 2(b) shows the playback of a time-lapse video created from captured images #1 to #15. The horizontal axis indicates the playback time of each image (frame), i.e., the display timing. In this example, by displaying each image at 1-second intervals (frame rate 1 fps (frames per second)), all images captured over a little over an hour are compressed into a 15-second video, making it possible to view it in a short time.

[0032] In this case, even though the original shooting time for the section of images #4 to #12, where the shooting interval was shortened, was only 8 minutes in total, due to the large number of shots, the playback time in the time-lapse video accounts for the majority of 14 seconds, at 8 seconds out of 14. In event detection such as animal observation, the focus of interest is the time when the subject is in the picture, and images taken at other times are thought to be unnecessary, so it can be said that extending the playback time for the section where the shooting frequency was increased in this way is actually preferable.

[0033] However, when applying such time-lapse videos to an aquatic environment sensing device with a variable camera shooting speed, the playback time is extended in the areas where the shooting frequency is reduced, resulting in a major problem in that the correspondence between the playback time of each frame image and the actual shooting time, i.e., the time compression rate, becomes uneven.

[0034] When using time-lapse video in water environments, it is possible to estimate the overflow of a dam, the time when water reaches the top of a levee, and the storage limit of a reservoir from the past trends of changes in water level and flow rate. In such applications, if the playback time of each frame of a time-lapse video does not correspond to the actual shooting time or the time compression rate is uneven, the urgency of important events and the expected occurrence time may be misinterpreted, leading to incorrect operation operations and serious accidents such as failure of pumps and treatment facilities, backflow of sewage, and flooding of streets. In particular, when monitoring water levels, the more frequently surveillance cameras need to shoot when the water level rises due to rainfall, etc., but the playback speed for this period is slow in the above-mentioned time-lapse video generation method, and there is a risk that dangerous information such as a sudden rise in water level will be overlooked.

[0035] For this reason, in one embodiment of the present specification, a time lapse video with a substantially uniform time compression ratio is generated, in which the playback time of each frame corresponds to a substantially uniform (including completely uniform) time with respect to the shooting time, and the rate at which the water level rises in a water environment, the time until the weir overflows, or the duration of the overflowing state, etc. are animated without compromising the time intervals. As a result, it is possible to prevent errors by operators of pumps and weirs and enable them to make accurate judgments.

[0036] In one embodiment of the present specification, frame images of a time lapse video are created from captured images obtained from a sequence of captured images by a camera, and a playback time of each frame image is determined so that the difference in time compression rate between the beginning of the time lapse video and the playback time of each frame image in the time lapse video is small. This makes it possible to make the correspondence between the playback time of the frame images and the actual shooting time closer to uniform, and to make the time compression rate approximately uniform. A relationship between a fixed or variable frame rate of the time lapse video and the playback time of each frame image is determined so that the difference in time compression rate between each part of the time lapse video is minimized. This makes it possible to effectively make the correspondence between the playback time of the frame images and the actual shooting time closer to uniform.

[0037] Fig. 4 is an explanatory diagram showing a method for generating a time lapse video with a variable frame rate according to this embodiment, in which the shooting time and the playback time correspond to each other almost uniformly, in the time lapse video generating unit 105 in Fig. 3. The horizontal axis in Fig. 4(a) is the actual shooting time of the camera 111-1, the inverted triangles indicate the timing of image shooting, and #1 to #15 indicate the numbers of the captured images in order. The shooting interval of the camera 111-1 in this embodiment is variable. In this figure, images #1 to #4 are shot at 10-minute intervals, and for #4 to #12, the shooting interval is shortened to 30 seconds when an event such as a rise in water level is detected as a trigger, and after #12 when the event detection ends, the shooting interval is returned to 10 minutes.

[0038] FIG. 4(b) shows an example of a time lapse video with a substantially uniform time compression ratio, in which the playback time of each frame generated by this embodiment is uniform with respect to the shooting time. In this example, this is achieved by varying the frame rate of the time lapse video. That is, the time compression ratio of the time lapse video with respect to the actual shooting time is set to 1 / 300. In the portion where the shooting interval is 10 minutes, the frame rate is set to 0.5 fps so that the frame interval of the video is 10 minutes / 300=2 seconds, and in the portion where the shooting interval is 30 seconds, the frame rate is accelerated to 10 fps so that the frame interval of the video is 30 seconds / 300=0.1 seconds. This makes it possible to make the correspondence between the shooting time and the playback time uniform.

[0039] Note that the frame rate of a video cannot necessarily be set to an arbitrary value due to restrictions on video standards and the software used to play and generate the video. On the other hand, it is desirable to be able to set parameters such as the shooting time and shooting interval of the original image, and the length of the time-lapse video (the time required for playback) arbitrarily according to the user's requests. In this case, it is not necessarily the case that there are images captured at times that exactly match each frame of the time-lapse video.

[0040] Fig. 4(c) shows a method for generating a time lapse video with a substantially uniform time compression ratio, in which the playback time of each frame is substantially uniform with respect to the shooting time. In this example, the frame rate of the time lapse video is set to 1 fps instead of 0.5 fps during a period in which the shooting interval is 10 minutes, and the frame rate is set to 12 fps instead of 10 fps during a period in which the shooting interval is shortened to 30 seconds. The time compression ratio of the time lapse video remains at 1 / 300, and one actual captured image is required every 300 seconds during a section in which the time lapse video is 1 fps (frame interval of 1 second), but the actual shooting interval is 10 minutes (600 seconds).

[0041] Therefore, in FIG. 4(c), in the even-numbered frames from the beginning where there is no actual photographed image (black inverted triangles), the immediately preceding actual photographed image is copied and used as a substitute. For example, image #1' used in the second frame is image #1, and image #2' used in the fourth frame is image #2'. This is a copy of #2.

[0042] In addition, in the section where the frame rate was increased to 12 fps, the frame interval was 0.083 seconds, and there were no images with exactly the same actual shooting time. In each case, the previous or most recent image was used, or there was a shortage of actual images, as in #6' / #9' in Figure 4(c). In such a case, the time lapse video of this embodiment can be generated by appropriately copying and using the immediately preceding actually captured image.

[0043] In the example of Figure 4(c) above, the video frame rates are set to 1 fps and 12 fps, but these are merely examples, and in practice it is desirable to select a frame rate that corresponds to the type of video to be created (e.g. MJPEG / MP4 / H.264, etc.) and the equipment and software to be used for playback. Also, for frames for which no corresponding image exists, the image immediately preceding the frame is copied, but it is possible to generate a roughly equivalent time-lapse video by selecting the image immediately following or the image closest in time.

[0044] In addition, although there are two shooting intervals and frame rates, long and short, these can also be shortened or extended stepwise or continuously as necessary. In each case, as in Figure 4(c), for each frame of the time-lapse video, the image captured closest in time to the actual captured image can be selected so that the time compression rate is approximately uniform.

[0045] 3, the time lapse video generating unit 105 in the parent station 102-1 is depicted as a component separate from the CPU 103, but this function may be realized by hardware external or internal to the CPU 103, such as a different CPU from the CPU 103 or an internal GPU, as shown in the figure, or may be software, such as a part of the parent station control program executed by the CPU 103. The time lapse video generating function may be implemented in a sensor node. This is the same in the other embodiments. EXAMPLES

[0046] 5 is a block diagram showing a second embodiment of the present specification, which is an example in which the interval between camera shots is changed according to the water level of a sewer or river. In the figure, sensor node 110-1 is equipped with camera 111-1 and water level sensor 113-1, and sensor node 110-2 is equipped with camera 111-2 and water level sensor 113-2.

[0047] In this example, the CPU 103 in the parent station 102-1 sends measurement commands to the sensor nodes 110-1 and 110-2 at regular intervals, such as every 10 seconds, to measure the water level using the water level sensors 113-1 and 113-2. In parallel with this, the CPU 103 sends image capture commands to the sensor nodes 110-1 and 110-2, usually at 10 minute intervals, to capture images using the cameras 111-1 and 111-2.

[0048] The water level measurement values ​​of each sensor are used to determine in the water level determination unit 114 whether the water level measured by the water level sensor 113-1 has exceeded a specified value A or fallen below a specified value B, and the CPU 103 shortens the shooting interval of the camera 111-1 in the former case and extends the shooting interval in the latter case. Generally, it is possible to prevent frequent changes in the image shooting interval by setting a hysteresis such that A>B. It is also possible to prevent malfunctions due to noise or abnormal values ​​in the measurement values ​​by changing the shooting frequency on the condition that the water level has exceeded a certain value A or fallen below B for a certain period of time.

[0049] In the above, the interval between shots of the camera 111-1 is changed based on the water level measured by the sensor node 110-1, but the frequency of shots of the camera mounted on the other node 110-2 may be changed at the same time. Also, the reference water levels A and B for the frequency of shots may be set independently for each camera, or the frequency of shots of the camera mounted on another node may be changed based on the measured water level of another water level sensor obtained from another node or parent station, and may be set arbitrarily according to the state of the water environment to be measured. As an example of such a case, for example, the water level may be measured several meters to several kilometers upstream of a river or sewer pipe, and the interval between shots of the camera installed on the downstream weir or pumping station inlet pipe may be changed.

[0050] Furthermore, the water environment measurement quantity used as a trigger for changing the photography interval does not have to be limited to the water level. For example, it is possible to use the drop in water temperature caused by rainwater inflow as a trigger to shorten the photography interval when the inverse of the water temperature exceeds a certain value (when the water temperature drops below a certain value). Similarly, measurements such as pH, electrical conductivity, and turbidity can also be used to change the photography frequency.

[0051] The water level may be obtained from an image captured by a camera without using the water level sensors 113-1 and 113-2. In this case, the water level determination unit 114 performs image processing for each captured image, calculates the water level from the water surface height obtained from the image, and determines whether the water level exceeds the specified value A or falls below the specified value B.

[0052] The object to be measured and determined may be the flow rate. The master station 102-1 may perform image processing for each captured image, calculate the flow rate from the water surface height obtained from the image, and determine whether the flow rate exceeds a specified value A or falls below a specified value B. In another example, the master station 102-1 may have an image processing function for analyzing the captured camera image, and when an abnormality is detected in the image, may shorten the capture interval during the detection of the abnormality and for a predetermined time after the detection is completed. The image processing function may use, for example, a machine learning model or a rule-based model. The image processing function may be implemented in the sensor node or the camera instead of the master station.

[0053] Similarly to the time lapse video generating unit 105 described above, the water level determining unit 114 can also be implemented as hardware or software such as part of the master station control program.

[0054] FIG. 6 is an explanatory diagram showing a method for generating a time lapse video with a fixed frame rate with uniform capture time and playback time, generated by the time lapse video generating unit 105 in FIG. 5. In general, in digital videos, there are more devices and software that support fixed frame rate videos than variable frame rates, and they are more widely and generally used. FIG. 6(a) shows the capture timing of images by the camera 111-1, similar to FIG. 4(a). FIG. 6(b) is an example of a time lapse video with a substantially uniform time compression rate, in which the playback time of each frame is uniform with respect to the capture time, according to an embodiment of this specification, and in this example, the frame rate of the time lapse video is a fixed value of 10 fps.

[0055] As in the previous example, if the time compression rate of the time lapse video is 1 / 300, the image update frequency in a time lapse video with a shooting interval of 10 minutes corresponds to 2 second intervals, and in a shooting interval of 30 seconds, it corresponds to 0.1 second intervals. Therefore, in the former interval, the shot image #1 is placed in the first frame of the time lapse video, and the #1 image is copied (image #1') in the 19th frame after that. In the latter case, the captured images are simply placed in frames at 0.1 second intervals, allowing the creation of a time-lapse video with a fixed frame rate of 10 fps.

[0056] Note that Figure 6(c) shows an example where the fixed frame rate is set to 12 fps, and the image capture timing does not match the frame timing of the time lapse video. In such a case, as in Figure 4(c) above, by appropriately selecting, copying and allocating the captured images that correspond to each frame timing of the time lapse video, it is possible to generate a time lapse video with any fixed frame rate.

[0057] In this application, including this embodiment, an example has been described in which the camera 111 mainly captures still images and the time lapse video is created from still images stored in the storage device 106 in the parent station 102, but some or all of these still images may be generated using still images extracted from the video.

[0058] Even when shooting videos with the camera 111, since there are limitations on the power or storage capacity of the sensing device, it is difficult to constantly shoot videos that consume a lot of power and have a large amount of data and record everything. For this reason, use cases are possible in which only still images are shot normally and one minute of video is intermittently shot every five minutes only when the water level rises, or one minute of video is intermittently shot every ten minutes normally and continuously shot when the water level rises, or a case in which video is shot at a reduced frame rate all the time and the frame rate is increased only when the water level rises.

[0059] In such cases, changing the frequency of intermittent shooting or the video frame rate corresponds to changing the shooting frequency as defined in the present application, and the desired time lapse video can be generated by assigning the captured still images, or one or more still images extracted from each video, to each frame of a time lapse video with a variable or fixed frame rate, as in the examples of Figures 4 and 6 described above.

[0060] If the sensor node operates using battery power, it is desirable to consider not only the sensor measurements mentioned above but also the remaining battery power when changing the camera shooting frequency. This also applies when the power supply to the sensor node is limited by the power supply capacity of environmental power generation, optical fiber power supply, etc., and a battery is used in combination.

[0061] Generally, a camera is a sensor that requires a large amount of power, and the power required for shooting is even greater when it is necessary to illuminate the image by turning on an LED flash or the like when shooting in a dark place. For this reason, increasing the shooting frequency may result in a sudden increase in battery consumption, so it is desirable to take measures such as returning the camera's shooting frequency to a steady state or reducing it further than normal when the remaining battery level falls below a specified value. Even in cases where the shooting frequency is variable due to factors such as the remaining battery level, the method of creating a time-lapse video with a substantially uniform time compression rate, in which the playback time of image frames corresponds uniformly to the shooting time, according to the embodiments of this specification is effective. EXAMPLES

[0062] 7 is a configuration diagram showing a third embodiment of the present specification, which is a configuration diagram of a water environment sensing system in which the time lapse video generation function and video viewing function of the embodiment of the present specification are arranged in a host server on a network. In this configuration, when an increase in water level due to rainfall is predicted, the administrator can operate the remote monitoring terminal 100 in advance to send a shooting frequency change command 127 to the parent station 102-1, and change the shooting interval of the camera 111-1 mounted on the sensor node 110-1 to a specified period, for example, from 12 hours to 48 hours.

[0063] In another example, the master station 102-1 may receive water environment information from an external source, shorten the image capture interval of the camera when the value indicated by the water environment information exceeds a predetermined first specified value, and extend the image capture interval when the value indicated by the water environment information falls below a predetermined second specified value. The water environment information may be information about the weather or information about the water level or flow rate at a location. Note that instead of the master station, a sensor node may receive commands or water environment information from an external source and control the image capture frequency.

[0064] On the other hand, the image / video server 120 automatically receives image data 124 stored in the storage device 106 in the parent station 102-1 via the network 101 at any timing, generates the time lapse video of the present application in the time lapse video generator 105, and stores this in the mass storage device 121.

[0065] These time lapse videos can be displayed on a web page on the server 122 as needed, making them viewable from the remote monitoring terminal 100 or other user PCs 123 connected to the network 101. In this way, it is possible to convert images captured each day into time lapse videos and store them, making it possible to view past time lapse videos at any time.

[0066] It is also possible to request creation of a time lapse movie of images taken with a specified camera during a desired period of time by directly sending a movie request command 125 from the remote monitoring terminal 100 or the user PC 123 to the image / movie server 120, or by inputting necessary information into a web page on the server 122. In this case, the image / movie server 120 obtains the necessary image data from its own mass storage device 121 or the storage device 106 of the master station 102-1 or the like, and generates the requested time lapse movie in the time lapse movie generator 105.

[0067] The created time lapse video can be sent from the server 122 to the requesting monitoring terminal 100 or user PC 123 in the form of a file or streaming video data 126, or can be stored in a mass storage device 121 and a link to it posted on a web page on the server 122, making it viewable by the requesting user. EXAMPLES

[0068] When performing fixed-point observation using images in places where the images do not change much, such as in water environments, there are issues such as the timing of image changes being difficult to discern on a time-lapse video, and most of the measurement information being missing when water environment information measured at intervals shorter than the image capture interval is displayed on the images. In general, since images in a time-lapse video continue to change at regular intervals, another issue is that it is difficult for a user to notice if the playback speed of a time-lapse video changes midway, as in the examples of this specification.

[0069] Furthermore, because time-lapse videos have no means of conveying changes in information that are shorter than the interval between image captures, there is a possibility that a sudden change in an important water environment indicator that occurs within a time period shorter than the capture interval at that time (e.g., 10 minutes) (e.g., a deterioration in water quality for 5 minutes due to illegal wastewater) may be overlooked.

[0070] Fig. 8 is an explanatory diagram showing an example of writing the shooting time and water environment measurement data to a time lapse video in Example 4 of this specification. In this example, the configuration of Fig. 6 described above is assumed, and Fig. 8(a) shows a graph of the shooting timing of camera 111-1, and Fig. 8(b) shows a graph of the water level measured by water level sensor 113-1. The horizontal axes of Fig. 8(a) and Fig. 8(b) each have the left end of the graph set to 0s, and the horizontal axis titles are shooting time (T) and measurement time (T), respectively, but both are the same and are actually the current time indicated by real-time clock 104.

[0071] The shooting interval of camera 111-1 is normally set to 10 minutes, but when the water level measured by sensor 113-1 exceeds specified value A, the shooting interval is shortened to 30 seconds, and when it falls below water level B, the shooting interval is returned to 10 minutes. Meanwhile, some of the points where the water level is measured are indicated by circles in Fig. 8(b), and in this example, the water level is constantly measured at 15-second intervals, which is shorter than the image shooting interval.

[0072] FIG. 8(c) is an example of the image captured in FIG. 8(a) converted into a time lapse video with a fixed frame rate of 10 fps, and FIG. 8(d) to FIG. 8(f) show examples of images displayed at playback time t of the time lapse video. In the time lapse video generating unit 105, the actual shooting time is imprinted in seconds on the top left first line of each image. For example, the frames with playback time t=0.0 seconds in FIG. 8(d), playback time t=0.1 seconds in FIG. 8(e), and playback time t=0.2 seconds in FIG. 8(f) each use image #1 obtained at shooting time 0s. Therefore, the shooting time is imprinted with a common value of "0s".

[0073] Also, the frames with playback time t=2.0s in Fig. 8(g), shooting time t=2.1s in Fig. 8(h), and shooting time t=2.2s in Fig. 8(i) each use the second shot image #2 captured at shooting time 600s. Therefore, the shooting time imprinted into the image is set to "600s." By doing this, it is possible to accurately display the shooting time of the image even for similar images with little change.

[0074] In this example, for convenience, the shooting time is burned in units of seconds, with the shooting start time being 0s. However, in practice, the shooting time may be burned in any format, such as year, month, day, hour, minute, and second.

[0075] In addition, it is possible to write in the image any number of measurements of the water environment obtained by the same sensor node as the camera or by a nearby node. A representative example is the water level, but as mentioned above, the intervals at which the water level is measured do not necessarily match the intervals at which images are taken. Therefore, in this example, more detailed water level data corresponding to the actual time is written in each frame of Figure 8(d), Figure 8(e), Figure 8(f), Figure 8(g), Figure 8(h), and Figure 8(i), which display the same image in the time-lapse video. In this example, the time compression rate of the time-lapse video is 1 / 300, and the time frame t=0.1 s in the time-lapse video of Figure 8(e) and the time frame t=0.2 s in the time-lapse video of Figure 8(f) correspond to 30 seconds and 60 seconds, respectively, in the actual measurement time in Figure 8(b).

[0076] For this reason, in this embodiment, the water level at the measurement time corresponding to each frame is burned into the second line of the burn-in character string, as in the time frames of Figures 8(d), 8(e), and 8(f), as water levels W(0s), W(30s), and W(60s), respectively (the actual water level is displayed as a number, such as "45cm"). This makes it possible to grasp detailed changes in important water environment measurement values, such as water level, on the time-lapse video, independent of the image capture interval. Note that only the capture time or the measurement value and measurement time may be burned in.

[0077] The present invention is not limited to the above-mentioned embodiment, but includes various modified examples. For example, the above-mentioned embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to those including all of the configurations described. It is also possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0078] In addition, the above-mentioned configurations, functions, processing units, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card or SD card.

[0079] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other. [Industrial Applicability]

[0080] The present invention can be applied to a water environment sensing device that uses sensors and cameras to monitor water environments such as water supply and sewerage systems, treatment plants, and rivers. [Explanation of symbols]

[0081] 100: remote monitoring terminal, 101: network, 102: parent station 103: CPU, 104: real-time clock, 105: time-lapse video generator 106: storage device, 107: captured image 110: sensor node, 111: camera, 112: node connection cable 113: Water level sensor, 114: Water level determination unit 120: image / video server, 121: mass storage device, 122: server 123: User PC, 124: Image data, 125: Video request command 126: Video data, 127: Shooting frequency change command 300: running water monitoring device (main body case), 301: main board, 302: monitoring camera 303: external storage medium, 304: communication module, 305: low-power microcomputer 306: power supply circuit, 307: water level sensor, 308: battery, 309: generator

Claims

1. One or more sensor nodes equipped with cameras; A host device, The sensor node or the higher-level device is a water environment remote sensing device that generates a time lapse video using images captured by the camera.

2. The water environment remote sensing device according to claim 1, The camera has a variable shooting interval, The sensor node or the higher-level device generating the time lapse video using information on the shooting times or shooting intervals of the captured images; A water environment remote sensing device, wherein a time compression rate of the playback times of the frame images of the generated time lapse video relative to the shooting times of the captured images is approximately uniform.

3. The water environment remote sensing device according to claim 2, A water environment remote sensing device in which the sensor node equipped with the camera or another sensor node measures the water environment, shortens the shooting interval of the camera when the measured value of the water environment exceeds a first specified value, and extends the shooting interval when the measured value of the water environment falls below a second specified value.

4. The water environment remote sensing device according to claim 2, The camera, the sensor node, or the higher-level device, Measure the water level or flow rate from the captured image; A water environment remote sensing device that shortens the shooting interval of the camera when the measured water level or flow rate exceeds a first specified value, and extends the shooting interval when the measured water level or flow rate falls below a second specified value.

5. The water environment remote sensing device according to claim 2, The sensor node or the higher-level device Receive water environment information from outside, A water environment remote sensing device that shortens the shooting interval of the camera when the value indicated by the received water environment information exceeds a first specified value, and extends the shooting interval when the value indicated by the received water environment information falls below a second specified value.

6. The water environment remote sensing device according to claim 2, The sensor node or the higher-level device The water environment remote sensing device shortens the image capture interval of the camera when an image capture interval shortening command is received from outside, and extends the image capture interval when an image capture interval extension command is received from outside.

7. The water environment remote sensing device according to claim 2, The camera, the sensor node, or the higher-level device, The water environment remote sensing device analyzes the captured images and shortens the capture interval of the camera during detection of anomalies and for a predetermined period of time after detection is completed.

8. The water environment remote sensing device according to claim 2, The sensor node or the higher-level device generating the time lapse video in which the playback time of the frame images has a substantially uniform time compression rate relative to the shooting time of the captured images by changing a frame rate during playback in proportion to the shooting interval of the camera; A water environment remote sensing device that generates a time lapse video by copying the most recently captured image, either before or after, when there is no captured image corresponding to the playback time of a particular frame of the time lapse video.

9. The water environment remote sensing device according to claim 2, the time lapse video is a fixed frame rate video; A water environment remote sensing device in which each frame image that constitutes the time lapse video is a copy of an image captured at a shooting time determined by the product of the playback time from the beginning of the time lapse video and the time compression rate of the time lapse video, or a copy of the most recent image captured either before or after the shooting time.

10. The water environment remote sensing device according to claim 2, The sensor node or the higher-level device imprints the shooting time, or one or more water environment measurement values ​​and the measurement time, as a character string into a frame image included in the time-lapse video.

11. The water environment remote sensing device according to claim 1, The sensor node or the higher-level device creating frame images of the time lapse video from captured images obtained from the sequence of captured images by the camera; a water environment remote sensing device that determines the playback time of each of the frame images in the time lapse video so that a difference in time compression rate from the beginning of the time lapse video to the playback time of each of the frame images is small.

12. The water environment remote sensing device according to claim 11, A water environment remote sensing device in which the sensor node or the higher-level device determines the relationship between a fixed or variable frame rate of the time lapse video and the playback time of each of the frame images so as to minimize the difference in the time compression rate.

13. One or more sensor nodes equipped with cameras; one or more higher-level devices; A water environment remote sensing system in which the one or more higher-level devices, in response to a user request, generate a time-lapse video at a specified time using images taken by a specified camera from past accumulated images, and return the video to the user or process the user's viewing via a network.

14. The water environment remote sensing system according to claim 13, A water environment remote sensing system in which the one or more higher-level devices automatically generate a time-lapse video using images captured by the designated camera from previously accumulated captured images, and store the video in a designated location or make it viewable via a network.

15. A method for generating a time-lapse video by a water environment remote sensing device, comprising: The water environment remote sensing device includes a camera installed in the water environment; The method further comprises the steps of: Acquire past photographed images taken by the camera at varying photographing intervals; generating the time lapse video using information on the shooting times or shooting intervals of the captured images; a playback time of the frame images of the generated time lapse video has a substantially uniform time compression rate with respect to the capture time of the captured images.

Images