Site monitoring system

Abstract

To provide a site monitoring system capable of easily performing monitoring with an unmanned aircraft even in a site where a satellite positioning system does not function.SOLUTION: A site monitoring system performs: receiving a video captured with a camera of a mobile body; receiving time information during a time when the video is captured; receiving the time when the mobile body passes a plurality of markers and information on the respective markers; acquiring route data inside a site; detecting one or more objects from the received video; identifying the type of each one of the detected objects and a detection time of the object on a video reproduction time axis; allowing a user terminal to display a detection result list where a marker passage time and its on-site position of each one of the markers, the detection time of each one of the objects, an on-site position of the object, a marker passage time and position of the marker, and the detection time and position of the object are displayed as a list; and allowing a processed video to be reproduced and displayed on the user terminal.SELECTED DRAWING: Figure 2

Description

Technical Field

[0006] , , , ,

[0001] The present invention relates to a on-site monitoring system.

Background Art

[0002] It is known to monitor a site using an unmanned aerial vehicle (for example, Patent Document 1). In Patent Document 1, a drone identifies its own position by radio waves from a GNSS (Global Navigation Satellite System) such as GPS (Global Positioning System), and has autonomous movement means for autonomous flight (autonomous movement) along a three-dimensional flight route (movement path) according to a patrol inspection plan, and while flying along a predetermined route R, it patrols a plurality of devices, for example, it discloses photographing sensors (sensor display screens) installed on each device.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, for example, in a site where global positioning such as a satellite positioning system like GNSS does not function due to reasons such as signal interruption, there is a problem that it is difficult to monitor with an unmanned aerial vehicle.

[0005] ​​​​​​​A field monitoring system according to one embodiment is a field monitoring system that provides a field display screen to a user terminal for monitoring a field, using a mobile body that moves to sequentially pass through a plurality of markers installed along a movement path at a field, and captures video of the field with a camera mounted on it, and includes: video receiving means for receiving video captured by the camera of the mobile body; shooting time receiving means for receiving time information of at least one specific point in time during the time the camera of the mobile body is capturing video; marker passing data receiving means for receiving marker passing data including the marker passing time and identification information of each marker at the time when the mobile body passes each of the plurality of markers; path data acquisition means for acquiring path data including location information of at least one specific point on the movement path within the field; and detecting one or more objects from the received video, identifying the type of each of the one or more detected objects and the time information of the detection point of each object on the playback time axis of the video. Furthermore, the system includes: a detection means that processes a video to create a processed video so that the user can visually recognize one or more objects in the video; a time and location identification means that uses time information of a specific point in time received by a shooting time receiving means, marker passage data for each of a plurality of markers received by a marker passage data receiving means, path data acquired by a path data acquisition means, and the type and time information of each of the one or more objects identified by the detection means to identify the marker passage time of a plurality of markers, the location of the plurality of markers at the site, the detection time corresponding to the shooting time at the time of detection of one or more objects, and the location of one or more objects at the site; a detection result list display means that displays a list of detection results on the user terminal, which is a list of the marker passage times and locations of the plurality of markers identified by the time and location identification means, and the detection time and location of one or more objects; and a video display means that plays and displays the processed video created by the detection means on the user terminal.

[0007] According to one embodiment, the detection result list displayed on the user terminal includes multiple rows displaying the marker passage time and position of multiple markers, and one or more rows displaying the detection time and position of one or more objects, arranged in order according to the order of marker passage time and detection time.

[0008] According to one embodiment, the user can select any one of multiple markers and one or more objects from the detection result list displayed on the user terminal. When the user selects a marker or object from the detection result list, the video display means starts playing the processed video on the user terminal from a position on the playback time axis of the processed video corresponding to the marker passage time of the selected marker or the detection time of the selected object.

[0009] According to one embodiment, the positions of the multiple markers and one or more objects, which are identified by the time and position identification means, are the relative positions of the multiple markers and one or more objects with respect to a reference point within the site.

[0010] According to one embodiment, the relative positions of multiple markers and one or more objects with respect to a reference point are the distances measured from the respective reference points of the multiple markers and one or more objects.

[0011] According to one embodiment, the distance measured from each of the multiple markers and one or more objects to a reference point is the sum of the distance between a specific point on the movement path and the reference point, and the distance measured along the movement path from the specific point to each of the multiple markers and one or more objects.

[0012] In one embodiment, the site is a tunnel, the reference point is one of the tunnel entrances, and the specific point is the starting point of the moving object.

[0013] According to one embodiment, the detection result list display means and the video display means display a single on-site display screen on the user terminal, which arranges the detection result list and the processed video side by side. [Brief explanation of the drawing]

[0014] [Figure 1] A diagram showing one embodiment of the field monitoring system according to the present invention and the overall hardware configuration of the peripheral system operating in conjunction with it. [Figure 2] Diagram showing the site configuration (example of monitoring at a tunnel construction site) [Figure 3] Functional block diagram of the on-site monitoring server. [Figure 4] Diagram showing an example of the on-site display screen displayed on the user terminal [Figure 5] Diagram showing the control flow of the flight operation department [Figure 6] Diagram showing an example of the marker passing history [Figure 7] Control flow of the object detection unit [Figure 8] Diagram showing an example of the object detection history [Figure 9] Control flow of the detection time conversion unit [Figure 10] Diagram showing the control flow of the marker time conversion unit [Figure 11] Diagram showing the control flow of the distance calculation unit [Figure 12] Diagram showing an example of the flight history [Figure 13] Diagram showing the control flow of the on-site display unit

Mode for Carrying Out the Invention

[0015] Hereinafter, this embodiment will be described with reference to the drawings. Note that the embodiments described below do not limit the invention according to the claims, and not all of the elements and combinations thereof described in the embodiments are essential for the solution means of the invention.

[0016] In the drawings for explaining the embodiments, the same reference numerals are given to portions having the same function, and the repeated explanation thereof is omitted.

[0017] Also, in the following description, the expressions such as simply "xxx" or "xxx list" may be used as an example of information, but the data structure of the information may be any type. That is, in order to indicate that the information does not depend on the data structure, "xxx" or "xxx list" can be referred to as "xxx data" or "xxx table". And in the following description, the configuration of each information is an example, and the information may be divided and held, or combined and held.

[0018] In the following explanation, the subject of a process may be "the program (e.g., the XX section)." However, since a program is executed by a processor (e.g., a CPU (Central Processing Unit)) and performs defined processes using memory resources (e.g., memory) and / or communication interface devices (e.g., ports) as appropriate, the program may be the subject of the process. A process described with the program as the subject may also be a process performed by the processor or a computer having that processor.

[0019] Figure 1 shows the overall hardware configuration of the field monitoring system according to this embodiment and the peripheral systems that operate in conjunction with it. Reference numerals are underlined in this figure.

[0020] The field monitoring server 1 is a device capable of various information processing functions, such as a computer or other information processing device. The field monitoring server 1 is a computer system having a processor (CPU) 10, memory 11, communication interface (communication I / F) 12, and storage 13, etc.

[0021] The processor is shown as an example, a CPU (Central Processing Unit) 10, but it may also be a GPU (Graphics Processing Unit), FPGA (Field-Programmable Gate Array), ASIC (Application Specific Integrated Circuit), etc. The memory 11 includes, for example, magnetic storage media such as an HDD (Hard Disk Drive), semiconductor storage media such as RAM (Random Access Memory), ROM (Read Only Memory), and SSD (Solid State Drive). A combination of an optical disc such as a DVD (Digital Versatile Disk) and an optical disc drive can also be used as a storage unit. In addition, known storage media such as magnetic tape media can also be used as memory 11.

[0022] Memory 11 can be used by the CPU 10 as a workspace. When the field monitoring server 1 starts up (for example, when powered on), the CPU 10 reads a program from this memory 11 and executes it. In addition to programs, memory 11 may also store data necessary for each process of the field monitoring server 1, and data 14 and programs 15 from storage 13 can also be stored in memory.

[0023] Storage 13 stores data 14 necessary for the operation of the field monitoring server 1 in this embodiment, such as various histories and videos, and programs 15 for performing various functions described later. Details of the data 14 and programs 15 will be described later.

[0024] Furthermore, the field monitoring server 1 in this embodiment may be configured as a so-called cloud, in which multiple information processing devices are configured to communicate with each other via a communication network.

[0025] The field monitoring server 1 can communicate with one or more remote user terminals 25 through the communication interface 12.

[0026] The user terminal 25 can be connected to the field monitoring server via a communication network such as the Internet or directly. The user terminal 25 may be, for example, a personal computer, smartphone, or tablet device, and is equipped with an input unit and a display unit, and inputs data and work instructions necessary for the work performed by the field monitoring server 1, and displays videos, history, etc., created by the field monitoring server 1 on its screen.

[0027] Furthermore, the site monitoring server 1 can communicate wirelessly with one or more unmanned aerial vehicles (drones in this example) 23 located at the site 2, which is the target of monitoring, via the communication interface 12.

[0028] Site 2 is a spatial area to be monitored, which can be arbitrarily selected by the user, and may be an area where global positioning is not available, such as a tunnel construction site or the interior of a building. Site 2 may utilize, for example, an area coordinator 21 and a drone (unmanned mobile device, e.g., an unmanned aerial vehicle) 23. Site 2 is usually assumed to be located remotely from the site monitoring server 1, but is not limited to this.

[0029] The area coordinator 21 is a computer system for relaying communication between the drone 23 and the field monitoring server 1. In other words, the area coordinator 21 can communicate with both the drone 23 and the field monitoring server 1. The area coordinator 21 has a processor, memory, communication interface, storage, etc. (not shown in the diagram). The area coordinator 21 receives instructions from the field monitoring server 1, controls the drone 23, and transmits various data acquired by the drone 23, such as captured videos, to the field monitoring server 1.

[0030] In the following explanation, it is assumed that data 14 and program 15 are stored in storage 13, but they may also be stored in memory 11, or on external storage or the cloud.

[0031] Figure 2 shows an example of the configuration of site 2 according to this embodiment. In this embodiment, a tunnel construction site is shown as an example of site 2, but it is not limited to this.

[0032] In the example of site 2 shown in Figure 2, a tunnel T, ground G, object O to be detected, takeoff position SP, landing position EP, drone 23, and multiple markers MP are shown. The flight path FR of drone 23 is shown by a dotted line between takeoff position SP and landing position EP. The flight path FR can be freely designed in the form of a flight plan on the site monitoring server 1 by the user accessing the site monitoring server 1 from the user terminal 25.

[0033] The user places markers MP at any number of locations along the desired flight path FR. In the illustrated example, markers MP are placed at multiple locations on the wall of tunnel T along the flight path FR. Markers MP may also be placed at the takeoff position SP and landing position EP, as in the illustrated example. Alternatively, markers MP may not be placed at the takeoff position SP and landing position EP.

[0034] Each marker MP is described in a manner that allows the drone 23 to automatically read a unique identification code (e.g., a number). For example, the marker is a two-dimensional code such as an ArUco marker that can be recognized by a camera, but it is not limited to this, as long as the drone's camera can read the unique identification code.

[0035] For example, a user can connect to the field monitoring server 1 from a user terminal 25 and use the field monitoring server 1 to create a flight plan.

[0036] In the flight plan, the flight path FR is defined as follows: The flight plan defines, for example, the takeoff position SP and the landing position EP in the form of the distance from the tunnel entrance TE to each point, and further includes instructions (movement (flight) instructions for the drone 23) that define how the drone 23 will move (fly) through each section of the flight path FR, such as the section from the takeoff position SP to the first marker MP, the section from each marker MP to the next marker MP, and the section from the last marker MP to the landing position EP. The instructions may include various control data, such as the flight altitude of the drone 23 and the distance from the wall of the tunnel T.

[0037] The drone 23 can store a flight plan in its own memory (not shown) and fly autonomously according to that flight plan. Possible methods for providing the flight plan to the drone 23 include, for example, the drone 23 receiving the flight plan from the field monitoring server 1 via the area coordinator 21, or the user terminal 25 receiving the flight plan from the field monitoring server 1 and providing it to the drone 23.

[0038] The drone 23 has a marker camera (not shown) for recognizing marker MP, a field camera (not shown) for recording video of the site, and one or more positioning sensors (not shown). The marker camera may be, for example, a camera capable of reading 2D barcodes. The positioning sensors are, for example, sensors for detecting whether the drone (itself) is flying at a position (e.g., distance from the tunnel wall or altitude from the ground) according to the instructions of the flight plan. Specifically, the positioning sensors may be various sensors such as an altitude sensor that measures the distance from the ground, i.e., altitude, and a wall distance sensor that measures the distance from the tunnel wall, or a sensor that combines these functions.

[0039] The flight of the drone 23 is performed as follows: The user places the drone 23 at the landing site and activates the drone 23. Upon activation, the drone 23 begins taking pictures with its marker camera at takeoff and discovers each marker MP by taking pictures of the spatial area where marker MPs may exist (e.g., tunnel walls) until landing. The drone 23 flies from the takeoff position SP to the first marker MP, then from each marker MP discovered to the next marker MP, and finally from the last marker MP to the landing position EP, using the sensing results of the positioning sensor according to the instructions of the flight plan. The drone 23 performs this operation from the takeoff position to the landing position, and as a result, the drone 23 flies autonomously along the above flight path FR.

[0040] During flight, each time the drone 23 passes over a marker MP, it acquires the identification code (marker number) of that marker MP and the time it passed over that marker MP (hereinafter referred to as the marker passage time, which is expressed in the standard time of the area of ​​flight, for example, Japan Standard Time in Japan), and records this information in the memory 11. The drone 23 also wirelessly transmits the identification code and marker passage time of each marker MP it has passed over to the field monitoring server 1 via the area coordinator 21, either in real time during flight or in a batch after flight.

[0041] Furthermore, the drone 23 activates its on-site camera upon takeoff and continues to photograph the area to be monitored at the site with that camera until landing, recording the captured video in memory 11. The drone 23 also wirelessly transmits the video captured by the on-site camera to the on-site monitoring server 1 via the area coordinator 21, either in real time during flight or in batches after flight.

[0042] Figure 3 is a functional configuration diagram of the field monitoring server 1 according to this embodiment, showing the functional elements of the field monitoring server 1 and the data flow between these elements. Reference numerals are indicated with underlined numbers in this figure.

[0043] Each function is realized, for example, by the processor (CPU 10) calling and executing a program 15 stored in the storage 13 shown in Figure 1.

[0044] Referring to Figure 3, the field monitoring server 1 includes, as functional elements, a flight operation unit 310, a model creation unit 311, an object detection unit 312, a detection time conversion unit 313, a field display unit 314, a marker time conversion unit 315, and a distance calculation unit 316.

[0045] Furthermore, this figure shows the original video 320, object detection history 321, processed video 322, flight history 323, marker passage history 324, flight plan 325, and route data 330 as data exchanged between these functions. This data may be created, updated, edited, deleted, etc. by each function and stored in the memory 11 or storage 13 within the field monitoring server 1. Note that the data may also be stored outside the field monitoring server 1, such as in external storage or the cloud, or data may be exchanged directly between the function and the drone 23 or user terminal 25. In the following, it may be explained that data is exchanged (sent and received) between functions, but the data may be stored (temporarily) in the memory 11, storage 13 or externally by the transmitting function and retrieved by the receiving function.

[0046] The flight operations unit 310 exchanges various data with, for example, the drone 23 (communication with the drone 23 on site is conducted via the area coordinator 21; the same applies hereinafter), the user terminal 25, the object detection unit 312, the marker time conversion unit 315, the detection time conversion unit 313, the distance calculation unit 316, etc. The flight operations unit 310 creates and saves the flight plan 325. Specifically, for example, the flight operations unit 310 creates the flight plan 325 according to user instructions coming from the user terminal 25. The flight plan 325 includes, for example, route data defining the flight path FR and control data such as flight control instructions for the drone 23. The flight operations unit 310 can transmit the flight plan 325 to the drone 23 or the user terminal 25, etc., in order to provide the flight plan 325 to the drone 23.

[0047] Furthermore, the flight operations unit 310 receives, for example, information from each drone 23 indicating that the drone 23 has passed each marker MP (hereinafter referred to as marker passage information, which includes, for example, the identification number of the marker MP and the time of marker passage) and the video footage of the scene that was captured (original video 320).

[0048] Furthermore, the flight operations unit 310 creates and saves a marker passage history 324 based on the marker passage information received from the drone 23. The marker passage history 324 records the takeoff time, the identification number and marker passage time of each marker MP, and the landing time for a single flight according to the drone 23's flight plan.

[0049] The model creation unit 311 creates a program (hereinafter referred to as the detection model, which is, for example, a neural network model that has finished learning how to detect a predetermined object) to detect an object desired by the user from a video of the site that has been filmed (original video 320). The object to be detected can be arbitrarily set by the user from the user terminal 25, and for example, in the case of monitoring a tunnel construction site, it could be, for example, a water leak in the tunnel, various equipment and pylons placed in the tunnel, etc.

[0050] The object detection unit 312 uses the detection model created by the model creation unit 311 to detect a predetermined object in the video of the site (original video 320) captured by the drone 23, and creates an object detection history 321 representing the detection result. The object detection unit 312 also applies a predetermined processing to the area of ​​the original video 320 in which the detected object is shown, so that the user can visually recognize where each detected object is shown in the original video 320 when the video is played back. This processing may include, for example, filling the image area in which the detected object is shown with a predetermined color, adding an outline or border to the image area in which the detected object is shown, or adding an arrow-like mark to indicate that image area. The video after such processing has been applied to the original video 320 is hereinafter referred to as the processed video 322. The object detection unit 312 saves the processed video 322.

[0051] Furthermore, the object detection unit 312 identifies the type of object detected from the original video 320 and the elapsed time since the object was detected, and creates an object detection history 321 that records this data. Here, the elapsed time refers to the elapsed time of video playback from the start of playback of the original video 320 (which is also the start of playback of the processed video 322) to the time when the object was detected, that is, the time when the object was detected with the start of the video being considered as time zero.

[0052] The detection time conversion unit 313 acquires the marker passage history 324 and the object detection history 321, and uses the takeoff time recorded in the marker passage history 324 (which is also the playback disclosure time of the processed video 322) to convert the elapsed time of each detected object recorded in the object detection history 321 to standard time (that is, the time according to standard time at the time each object was detected, hereinafter referred to as the detection time), and then records the converted detection time of each detected object in the flight history 323, which will be described later.

[0053] The marker time conversion unit 315 acquires the marker passage history 324, converts the marker passage time (standard time) of each marker recorded in the marker passage history 324 into the elapsed time of video playback from the start of playback of the original video 320, and then records each elapsed time in the flight history 323.

[0054] The distance calculation unit 316 calculates the distance to each detected object and the distance to each marker, measured from a predetermined reference point within the site 2 (for example, the tunnel entrance). Specifically, the distance calculation unit 316 acquires, for example, the flight history 323, the marker passage history 324, and the flight path data 330 (takeoff position and landing position, for example, the distance from the reference point (tunnel entrance) of the takeoff point and the reference point (distance from the tunnel entrance) of the landing point), and calculates the distance from the reference point to each marker and each detected object from the marker passage time and marker identification number of each marker in the marker passage history 324, the detection time of each detected object in the flight history 323, and the path data (takeoff position, landing position (or distance from the takeoff position to the landing position)). The distance calculation unit 316 records the calculated distance in the flight history 323. Here, the reference point can be set arbitrarily by the user, for example, the entrance or exit of the tunnel, or the takeoff position or landing position of the drone, but in this embodiment, as an example, the reference point is the tunnel entrance.

[0055] The field display unit 314 provides the user terminal 25 with a field display screen based, for example, the flight history 323 and the processed video 322.

[0056] Figure 4 shows an example of the on-site display screen 4 shown on the user terminal 25.

[0057] The on-site display screen 4 shows, for example, a video file 41 indicating the filename of the processed video 322 selected by the user, a detection result video 42 which is a playback of that processed video 322, and a detection result list 43 based on the flight history 323.

[0058] The detection result list 43 displays, for example, the type (or marker identification number) of each detected object and each marker (hereinafter referred to as "object" in this description), the detection (or passage) time of each object, the distance from the reference point to each object, and jump buttons for playing the processed video from the point in time when each object was detected or passed. Specifically, the detection result list 43 has display items such as the time of object detection or marker passage 431, the distance from the (tunnel) entrance 432, the detected object or marker number 433, and jump buttons 434. The detection result list 43 is arranged in chronological order, with data for detected objects and markers.

[0059] The object detection or marker passage time 431 indicates the time when each object was detected or passed. The distance from the entrance 432 indicates the distance from the tunnel entrance to each object. The detected object or marker number 433 indicates the object type or marker number (identification number) of each object. When the user clicks the jump button 434 for each object, the scene of the processed video 322, which is played on the detection result video 42, jumps to the scene at the time each object was detected or passed.

[0060] The processed video 322 is played and displayed in the detection result video 42. In the example shown in Figure 4, a border 422 is added to the area of ​​the detected object (e.g., engine) 421 in the playback scene, so that the user can easily visually recognize where the detected object 421 is located in the site 2. In addition, the date and time 423 in standard time of the playback scene is displayed along with the detection result video 42, so that the user can know what time period the playback scene represents.

[0061] As described above, by viewing the detection result video 42, users can easily understand what was present, when, and where at the site.

[0062] Furthermore, in the detection results list 43, the detection (passage) time and distance from the reference point for each detected object and marker are displayed in chronological order. By looking at this, the user can easily understand the location of each detected object within the site by referring to the position of the marker. For example, in the example in Figure 4, it can be seen that there is an engine between marker 1 and marker 2. Since the user placed marker 1 and marker 2 themselves, they can easily recall the positions of marker 1 and 2 within the site, and from the fact that there is an engine between them, they can easily understand where the engine is located within the site.

[0063] Furthermore, in the detection result list 43, by using the jump button 434 to that scene, a link is established between the detected or passed object and the scene in which that object was captured. As a result, the user can immediately see the scene in which the object was shown, making it even easier to understand the location of the detected object.

[0064] Figure 5 shows the control flow of the flight operations unit 310.

[0065] The flight operations unit 310 determines whether or not the user terminal 25 has requested the creation of a flight plan (step S50). If the creation of a flight plan has been requested (YES in step S50), the flight operations unit 310 creates and saves the flight plan according to the instructions from the user terminal 25 (step S51). Next, the flight operations unit 310 provides the route data 330 based on the flight plan to the distance calculation unit 316 (step S53) and terminates the process.

[0066] Furthermore, the flight operations unit 310 determines whether or not a request for transmission of a flight plan has been received from the user terminal 25 or the drone 23 (step S52). If a request for transmission of a flight plan has been received (YES in step S52), the flight operations unit 310 transmits the flight plan to the requesting user terminal 25 or drone 23 (step S54) and terminates the process.

[0067] The flight operations unit 310 also determines whether or not it has received video (step S55). If it has received video (YES in step S55), the flight operations unit 310 saves the video (step S56) and terminates the process.

[0068] Furthermore, the flight operations unit 310 determines whether or not it has received marker passage data from the drone 23 (step S57). Here, the marker passage data received from the drone 23 includes the user ID of the user who conducted the flight of the drone 23, the flight plan number which is the identifier of the flight plan used in that flight, the flight number which is the identifier of that flight (instead of or in addition to this, the file name of the on-site video (original video) taken during that flight may be included), the marker number which is the identifier of the marker passed, and the time of marker passage. The marker passage data may be received from the drone 23 via the area coordinator 21 or via the user terminal 25.

[0069] If marker passage data is received (YES in step S57), the flight operations unit 310 records and saves the marker passage data in the marker passage history 324 (step S58).

[0070] Figure 6 shows an example of marker passage history 324.

[0071] As described above, the marker passage history 324 is created by the flight operations unit 310 for each flight based on the marker passage data.

[0072] The marker passage history 324 records, for example, the user ID 61 of the user who performed the flight, the flight plan number 62 of the flight plan used for that flight, the flight number 63 of that flight, and the marker number 64 and time of passage for each marker passed. The marker passage history also records the takeoff and landing positions of the flight, with the same information as marker number 64. If markers are also placed at the actual takeoff and landing positions, the takeoff and landing positions can be treated exactly the same as markers along the flight path.

[0073] Figure 7 shows the control flow of the object detection unit 312.

[0074] The object detection unit 312 identifies, for each flight, a frame in the original video 320 taken during that flight that contains an object to be detected, and identifies the image area of ​​the object within the identified frame and the type of the object (step S71). The original video 320 also includes data items such as user ID, flight plan number, and flight number, similar to the marker passage data mentioned above.

[0075] The object detection unit 312 modifies each identified frame in the video to clearly indicate the region of the identified object image, and records the modified video as a modified video (step S72). Here, the modification to clearly indicate the region of the object image may be, for example, filling the region with a specific color, adding an outline or frame around the region, or adding a shape such as an arrow pointing to the region. In short, the original video 320 is modified so that when the user plays the modified video, the user can easily visually recognize where the detected object is located in the image.

[0076] The object detection unit 312 records the elapsed time of the identified frame and the type of object in the object detection history 321, and then terminates processing.

[0077] Figure 8 shows an example of object detection history 321.

[0078] The object detection history 321 is created by the flight operations unit 310, as described above.

[0079] The object detection history 321 records, for example, the user ID 81 of the user who performed the flight, the flight plan number 82 of the flight plan used, the flight number 83 of that flight, the detected object 84 indicating the type of each detected object, and the elapsed time 85 representing the elapsed time for each detected object, for each flight. Here, the elapsed time for each detected object is, as mentioned above, the video playback time from the start of video playback (the start of shooting, i.e., the drone's takeoff) to the time when the object is detected in that video. Therefore, by adding the elapsed time for each object to the time of the start of playback (standard time), the detection time (standard time) of that object can be obtained.

[0080] Figure 9 shows the control flow of the detection time conversion unit 313.

[0081] The detection time conversion unit 313 calculates the detection time for each detected object (step S91) by adding the time the drone took off (the time 65 at marker number 64 "takeoff", i.e., the time when the video playback started) in the marker passage history 324 to the elapsed time 85 at each detected object 84 in the acquired object detection history 321, based on the object detection history 321 and marker passage history 324. This calculates the detection time (standard time) for each detected object.

[0082] The detection time conversion unit 313 records, for example, the data in the object detection history 321 of each detected object (for example, user ID 81, flight plan number 82, flight number 83, detected object 84, elapsed time 85) and the detection time of each detected object calculated in S91 into the flight history 323 (step S92), and then terminates the process.

[0083] Figure 10 shows the control flow of the marker time conversion unit 315.

[0084] The marker time conversion unit 315 calculates the elapsed time for each marker (step S101) by subtracting the time the drone took off (the time the drone took off corresponding to marker number 64 "takeoff") from the time the drone took off (the time the drone took off corresponding to marker number 64 "takeoff") based on the marker passage history 324. The elapsed time for each marker is, as described above, the playback time of the operation from the start of the video to the time the drone took off from the time the drone passed over each marker.

[0085] The marker time conversion unit 315 records the data in the marker passage history 324 for each marker (for example, user ID 61, flight plan number 62, flight number 63, marker number 64, passage time 65) and the elapsed time calculated in step S101 into the flight history 323 (step S102), and then terminates the process.

[0086] Figure 11 shows the control flow of the distance calculation unit 316.

[0087] The distance calculation unit 316 calculates the flight distance by subtracting the takeoff position (for example, the distance from the tunnel entrance at the takeoff point) from the landing position (for example, the distance from the tunnel entrance at the landing point) in the route data 330 (step S111).

[0088] The distance calculation unit 316 calculates the flight time by subtracting the takeoff time from the landing time in the marker passage history 324 (step S112).

[0089] The distance calculation unit 316 calculates the average speed by dividing the flight distance calculated in step S11 by the flight time calculated in step S112 (step S113).

[0090] The distance calculation unit 316 calculates the distance from the takeoff position of each detected object and each marker by multiplying the elapsed time of each detected object and each marker in the flight history 323 by the average speed calculated in step S113 (step S114).

[0091] The distance calculation unit 316 calculates the distance from a reference point (e.g., the tunnel entrance at the landing site) to each detected object and each marker by adding the takeoff position (e.g., the distance from the takeoff position of each detected object and each marker calculated in step S114) (step S115).

[0092] The distance calculation unit 316 records the distances calculated in steps S114 and S115 from the takeoff position of each detected object and each marker, as well as the distance from the reference point, in the flight history 323 (step S116), and then terminates the process.

[0093] Figure 12 shows an example of flight history 323. Note that some of the reference symbols are indicated with underlined numbers in this figure.

[0094] The flight history 323 is created by the detection time conversion unit 313, the marker time conversion unit 315, and the distance calculation unit 316, etc. For each flight, the flight history 323 records data for items such as the landing position (for example, distance from the tunnel entrance which serves as a reference point) 1211, the landing position (for example, distance from the tunnel entrance which serves as a reference point) 1212, the flight distance 1213, the flight time 1214, the average speed 1215, the user ID 1216 of the user who performed the flight, the flight plan number 1217, the flight number 1218, the marker number 1219 indicating the number of each marker, the detected object 1220 indicating the type of each detected object, the elapsed time for each detected object and each marker 1221, the detection / passage time for each detected object and each marker 1222, the distance from the takeoff position of each detected object and each marker 1223, and the distance from the entrance of each detected object and each marker 1224.

[0095] By creating a flight history, data for each flight, as well as data for each detected object and marker in each flight, are acquired.

[0096] Figure 13 shows the control flow of the field display unit 314.

[0097] The field display unit 314 reads from the flight history 323 the detection / passage time data 1222 for all marker numbers 1219 and detected objects 1220, as well as the distance data from the tunnel entrance, which is the reference point, and displays this data in chronological order in the detection result list 43 on the field display screen 4 (see Figure 4) (step S131).

[0098] The field display unit 314 determines whether or not one of the "Jump" buttons in the Jump Button 434 column of the detection result list 43 in the field display screen 4 has been clicked (step S132). If a "Jump" button is clicked (YES in step S132), the field display unit 314 plays the processed video from the point in time corresponding to the detected object or marker of the clicked "Jump" button and displays it on the detection result video 42 in the field display screen 4 (step S133). On the other hand, if a "Jump" button is not clicked (NO in step S132), the field display unit 134 proceeds to step S135.

[0099] Furthermore, while the field display unit 314 is playing the processed video and displaying it on the detection result video 42, it converts the playback position of the processed video to a date and time in standard time notation and displays the date and time 423 in standard time notation on the detection result video 42 (step S134).

[0100] The field display unit 314 determines whether the user has requested video playback control (pause, fast forward, reverse, resume playback, etc.) (step S135). If video playback control is requested (YES in step S135), the field display unit 314 controls video playback according to the user's request (step S136).

[0101] On the other hand, if no request for video playback control is made (NO in step S135), the field display unit 314 determines whether or not the user has requested that the display be terminated (step S137). If the user has requested that the display be terminated (YES in step S137), the field display unit 314 terminates processing.

[0102] On the other hand, if no request is made to end the display (NO in step S137), the field display unit 314 returns to step S132 (A).

[0103] The on-site display unit 314 displays the detected objects and markers in the detection result list 43 in the order of the time they were detected or passed. This allows the user to easily understand which markers each detected object is located between. In particular, since the markers can be placed arbitrarily by the user, the user can easily understand the approximate location of the detected object within the site based on the placement of markers at the site that they remember. In addition, the distance of the detected object and marker from the reference point is also displayed in the detection result list 43. By referring to this distance as well, the user can more accurately determine the location of the detected object at the site.

[0104] Furthermore, the detection result list 43 displays the time (standard time) when the detected object and marker were detected or passed, and the detection result video 42 also displays the standard time of the playback position. This makes it easy for the user to understand what state the situation shown in the detection result video 42 was in at what time.

[0105] In this way, users can easily determine the location of objects (detected objects) and markers at the site of their interest, as well as the standard time when these objects and markers were captured in the video footage taken by the drone. This helps users to properly understand the site and take timely action.

[0106] The embodiments described above are merely illustrative examples for explaining the present invention. The present invention can be implemented in a variety of other forms.

[0107] For example, in the embodiment described above, the calculation was performed using the average speed of the entire flight path FR from takeoff to landing of the drone 23, but it is not limited to this. Alternatively, for example, the position of each marker (e.g., distance from the tunnel entrance) may be set in advance and stored in a server, and the calculation may be performed using the position of each marker and the average speed of the section between each marker and the next marker. This would allow for more accurate distance calculation.

[0108] Furthermore, although the above-described embodiment was explained using a tunnel as the site, this is merely an example, and the present invention can also be applied to facilities or areas other than tunnels.

[0109] Furthermore, in the above-described embodiment, the drone flies along the direction in which the tunnel extends, and the position of the object (detected object) is indicated by the distance along that direction, i.e., a one-dimensional indicator, but it is not limited to this. Alternatively, for example, the planar (two-dimensional) position of the marker, or the three-dimensional position including altitude information, may be stored in a server, and the position of the detected object may be indicated by a two-dimensional or three-dimensional indicator using the stored two-dimensional or three-dimensional position of the marker, along with the flight time and speed between each marker and the next marker.

[0110] Furthermore, in the above-described embodiment, the drone is an aerial vehicle, but it is not limited to that. It may also be a vehicle that travels on the surface of a solid or liquid, or even one that submerges in a liquid, and can be appropriately selected depending on the site. [Explanation of Symbols]

[0111] 1…Site monitoring server 2…Site 4…Site display screen 5… 10…CPU 11…Memory 12…Communication interface 13…Storage 14…Data 15…Program 21…Area coordinator 23…Drone 25…User terminal 310…Flight operation unit 311…Model creation unit 312…Object detection unit 313…Detection time conversion unit 314…Site display unit 315…Marker time conversion unit 316…Distance calculation unit 321…Object detection history 322…Processed video 323…Flight history 324…Marker passage history 325…Flight plan 330…Route data

Claims

1. A site monitoring system that assists a user in monitoring a site by using a mobile device that moves sequentially through multiple markers installed along a route at the site, and records video of the site with a camera mounted on it, A video receiving means for receiving the video captured by the camera of the mobile body, A shooting time receiving means that receives time information for at least one specific point in time during the time when the camera of the moving object is shooting the video, A marker passage data receiving means that receives marker passage data including the time of passage of each of the plurality of markers when the moving body passes each of the plurality of markers and identification information of each marker, Route data acquisition means for acquiring route data including location information of at least one specific point on the travel route within the site, A detection means that detects one or more objects from the received video, identifies the type of each of the detected objects and the time information of the detection point of each object on the playback time axis of the video, and further processes the video to create a processed video so that the user can visually recognize the one or more objects in the video. A time and location identification means that uses the time information of the specific point in time received by the shooting time receiving means, the marker passage data of each of the plurality of markers received by the marker passage data receiving means, the route data acquired by the route data acquisition means, and the type and time information of each of the one or more objects identified by the detection means to identify the marker passage time of the plurality of markers, the location of the plurality of markers at the site, the detection time corresponding to the shooting time of the one or more objects at the detection point, and the location of the one or more objects at the site, A detection result list display means displays on a user terminal a list of detection result information, which includes the time and position of the multiple markers identified by the time and position identification means, and the detection time and position of one or more objects. A video display means that plays and displays the processed video created by the detection means on the user terminal, A field monitoring system equipped with this system.

2. In the detection result list displayed on the user terminal, multiple rows displaying the marker passage time and position of the multiple markers, and one or more rows displaying the detection time and position of the one or more objects, are arranged in an order according to the order of the marker passage time and the detection time. The field monitoring system according to claim 1.

3. On the detection result list displayed on the user terminal, the user can select any one of the multiple markers and the one or more objects. When a marker or object is selected by the user from the detection result list, the video display means starts playback of the processed video on the user terminal from a position on the playback time axis of the processed video corresponding to the time the selected marker passes over the marker or the detection time of the selected object. The field monitoring system according to claim 1.

4. The positions of the plurality of markers and the one or more objects, as determined by the time and position determination means, are the relative positions of the plurality of markers and the one or more objects with respect to a reference point within the site. The field monitoring system according to claim 1.

5. The relative positions of the plurality of markers and the one or more objects with respect to the aforementioned reference point are the distances measured from the aforementioned reference point for each of the plurality of markers and the one or more objects. The on-site monitoring system according to claim 4.

6. The distance measured from the reference point to each of the plurality of markers and the one or more objects is the sum of the distance between the specific point on the movement path and the reference point, and the distance measured along the movement path from the specific point to each of the plurality of markers and the one or more objects. The on-site monitoring system according to claim 4.

7. The aforementioned site is a tunnel. The aforementioned reference point is one of the entrances to the tunnel, The aforementioned specific point is the starting point of the moving object. The on-site monitoring system according to claim 5.

8. The detection result list display means and the video display means display a single on-site display screen on the user terminal, which arranges the detection result list and the processed video side by side. A field monitoring system according to any one of claims 1 to 7.

Images