Control system, remote monitoring and control system, and patrol inspection planning system
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- KK TOSHIBA
- Filing Date
- 2025-01-22
- Publication Date
- 2026-08-03
Smart Images
Figure 2026125430000001_ABST
Abstract
Description
Technical Field
[0005] ,
[0001] Embodiments of the present invention relate to a control system, a remote monitoring control system, and a patrol inspection planning system.
Background Art
[0002] Plants such as hydroelectric power plants scattered in mountainous areas, which take time to access, are usually unmanned, but workers regularly go to the site to conduct patrols and inspections. It is desired to reduce the number of worker dispatchments and reduce O&M (Operation & Maintenance) costs by automatically patrolling some of such operations with robots placed inside the power plant.
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Patent Document 2
Non-Patent Documents
Non-Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0005] For example, patrols and inspections using drones as robots have been considered. In a hydroelectric power plant in a mountainous area, the facilities to be patrolled and inspected are arranged indoors such as underground, and there is a problem that GPS (Global Positioning System) signals generally used by outdoor drones cannot be received.
[0006] Therefore, embodiments of the present invention provide a control system, a remote monitoring and control system, and a patrol and inspection planning system that enable patrols or inspections even indoors. [Means for solving the problem]
[0007] According to one embodiment, the control system includes a mapping unit used to estimate the self-position of a drone that patrols and inspects an indoor area, and which creates an environmental map dividing the indoor area into one or more regions. Furthermore, the control system includes a scenario creation unit that creates a scenario indicating the flight path and shooting locations based on input of patrol plan data indicating the plan for the patrol and inspection to be performed by the drone. Furthermore, the control system includes a drone integrated management unit that transmits the environmental map and the scenario to the drone and instructs it to perform the patrol and inspection. [Brief explanation of the drawing]
[0008] [Figure 1] This is a schematic diagram of the inspection planning system in this embodiment. [Figure 2] This is another schematic diagram of the patrol inspection planning system in this embodiment. [Figure 3] This is yet another schematic diagram of the patrol inspection planning system in this embodiment. [Figure 4] This is a hardware configuration diagram of the drone and charging port in this embodiment. [Figure 5] This is an example of an environmental map created by the map creation unit in this embodiment. [Figure 6] This is an example of switching the environmental map using a drone in this embodiment. [Figure 7] This is an example of a user interface for the remote monitoring and control system in this embodiment. [Figure 8] This is an example of a flowchart of the control system in this embodiment. [Figure 9]This is an example of a flowchart for the remote monitoring and control system in this embodiment. [Figure 10] This is a hardware configuration diagram of the control system in this embodiment. [Modes for carrying out the invention]
[0009] Embodiments of this disclosure will be described below with reference to the drawings. These embodiments are not intended to limit the present invention. The drawings are schematic or conceptual, and the proportions of each part may not necessarily be the same as those of actual objects. In the specification and drawings, elements similar to those described above with respect to previously shown drawings are denoted by the same reference numerals, and detailed descriptions are omitted as appropriate.
[0010] Furthermore, the X, Y, and Z axes described below represent mutually orthogonal axes, with the Z direction being the direction of gravity, and the X and Y directions being in a horizontal plane perpendicular to the direction of gravity. Also, the +Z direction corresponds to the upward direction, and the -Z direction corresponds to the downward direction. As an example, the +X direction corresponds to the forward direction, and the -X direction corresponds to the backward direction. Also, the +Y direction corresponds to the right direction, and the -Y direction corresponds to the left direction. The forward and backward directions may be reversed, and the right and left directions may be reversed. The X direction is an example of the first direction, the Y direction is an example of the second direction, and the Z direction is an example of the third direction.
[0011] Furthermore, in the following embodiments, the terms "greater than or equal to" and "less than or equal to" can be replaced with "greater than" and "less than," respectively.
[0012] Figure 1 is a schematic diagram of the patrol inspection planning system 100 in this embodiment.
[0013] The patrol inspection planning system 100 is a system for patrolling and inspecting facilities arranged indoors using a drone 3 in a facility constructed in a location that takes time to access from a management office where workers are stationed. In this embodiment, a hydroelectric power plant 200 in the mountains is taken as such a facility for explanation. A worker is an example of a user of the patrol inspection planning system 100.
[0014] The patrol inspection planning system 100 creates 3D map data of a substation based on camera images captured by the drone 3. The drone 3 conducts patrol or inspection of the hydroelectric power generation facilities indoors in the hydroelectric power plant 200 based on the created 3D map data and a patrol plan selected by a worker. The patrol plan is a plan for patrol and inspection that the drone 3 performs through a series of flights. Also, the results of the patrol inspection are remotely displayed on a display device such as a display installed in the management office together with the trajectory of the flight path of the drone 3. Hereinafter, the 3D map data created based on the camera images captured by the drone 3 is also referred to as an environmental map. Also, hereinafter, when simply referred to as a patrol, it may include only a patrol or may include a patrol and inspection.
[0015] In this embodiment, the hydroelectric power plant 200 is taken as an inspection location to explain the patrol inspection planning system 100, but the applicable locations of this system are not limited to this. For example, the patrol inspection planning system 100 can be applied to the patrol and inspection of indoor facilities in locations where GPS signals do not reach, such as other power plants, substations, water treatment plants, or tunnels, or indoor facilities in locations with poor access from a management office.
[0016] The patrol inspection planning system 100 includes a control system 1 and a remote monitoring control system 2. It also includes a drone 3, a charging port 4, an in-site communication device 5, and an out-site communication device 6. In addition, in the hydroelectric power plant 200, in addition to the control system 1, a drone 3, a charging port 4, an in-site communication device 5, and an out-site communication device 6 are installed. Further, in the data center 300, the remote monitoring control system 2 is installed. The configuration of the patrol inspection planning system 100 and the arrangement of the devices are examples, and as will be described later, various configurations can be adopted in consideration of the line environment and the like of the patrol and inspection locations.
[0017] First, the facilities on the hydroelectric power plant 200 side in the present embodiment will be described.
[0018] In the present embodiment, the drone 3 is equipped with a 360° camera and a tracking camera capable of shooting 360° around the flight area. For example, it can estimate its own position using 360° VSLAM (Visual Simultaneous Localization and Mapping).
[0019] In this embodiment, the drone 3 patrols and inspects the interior of the hydroelectric power plant 200. The drone 3 flies along a patrol plan pre-registered in the work management unit 22, passing through waypoints. However, in order to fly accurately, it is desirable to estimate its own position with high accuracy and without time delay. Since GPS cannot be properly received inside the hydroelectric power plant 200, the drone 3 in this embodiment estimates its own position using images from two cameras: a 360° camera and a tracking camera. The drone 3 estimates its own position with high accuracy and without time delay on the environmental map by interpolating the estimated position from the 360° camera, which has a relatively long update cycle, with the relative movement amount from the tracking camera, which has a relatively short update cycle. Hereinafter, the estimation result from the 360° camera will also be referred to as the first estimation result, and the estimation result from the tracking camera will also be referred to as the second estimation result. The 360° camera is an example of the first imaging device, and the tracking camera is an example of the second imaging device. In this embodiment, the example described is when the drone 3 is equipped with a 360° camera and a tracking camera, but the configuration is not limited to these cameras, as long as it has a configuration that can estimate the drone 3's own position. For example, an INS (Inerial Navigation System) that calculates the relative position from IMU (Electronic Speed Controller) data may be used. A device that estimates its own position without using radio wave assistance, such as an IMU, is an example of a self-position estimation device.
[0020] Furthermore, Drone 3 is equipped with a 2D laser distance sensor, which allows it to determine the distance to obstacles in its surroundings and fine-tune its flight path accordingly.
[0021] Charging port 4 is a charging device that charges drone 3 via wireless power transfer (WPT). When conducting patrols and inspections, drone 3 starts its flight with charging port 4 as its initial position. Charging port 4 is equipped with an AR marker recognition camera, and after completing patrols and inspections, it recognizes the position of drone 3 in the air from the AR marker attached to the bottom of drone 3. Charging port 4 transmits the recognized position information of drone 3 to the drone integrated management unit 12. Based on this position information, the drone integrated management unit 12 generates commands such as speed commands and transmits them to drone 3. Drone 3 uses these commands and estimated self-position information to land at charging port 4.
[0022] Control system 1 is a system that centrally manages the operation of drone 3 based on input of patrol plan data corresponding to the patrol plan selected by the worker and other operation instructions. Control system 1 comprises a map creation unit 11, a drone integrated management unit 12, and a scenario creation unit 13.
[0023] Control system 1 is, for example, a PC (Personal Computer) and has hardware resources such as a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), HDD (Hard Disk Drive), and GPU (Graphics Processing Unit). Control system 1 can be implemented by installing a program for control system 1. The CPU within control system 1 executes this program, thereby realizing the functions of each functional block. Furthermore, control system 1 may be implemented not only as a single device but also by multiple PCs.
[0024] The mapping unit 11 receives 360° camera footage from the drone 3 and extracts feature points from this footage. As the drone moves the 360° camera, the feature points change, so the mapping unit 11 calculates the distance to the feature points from the changes and creates an environmental map. The mapping unit 11 also transmits the created environmental map to the drone 3. The drone 3 estimates its own position on the environmental map using both the 360° camera and the tracking camera. The environmental map may be transmitted using the frequency band used by the drone 3 for video transmission, or it may be transmitted using the frequency band of a private area communication device 5, such as a router that uses Wi-Fi (registered trademark) for communication within the premises. As will be described in detail later, when creating the environmental map, the mapping unit 11 divides the indoor area into one or more areas and creates the environmental map. The following example describes an example in which the mapping unit 11 divides the environmental map into multiple areas. Furthermore, while we will explain using the first environmental map and the second environmental map, which is a continuous area partially overlapping with the first environmental map, as examples of environmental maps created by dividing the indoor area into multiple regions, the map creation unit 11 may create environmental maps by further dividing the indoor area into even more regions. For example, the map creation unit 11 may create environmental maps by further dividing the area into even more regions, such as a third environmental map and a fourth environmental map.
[0025] The drone integrated management unit 12 transmits the environmental map created by the mapping unit 11 to the drone. The drone 3, upon receiving the environmental map, registers the environmental map. The drone integrated management unit 12 also receives input of patrol plan data and individual operation instructions for the drone 3 from the remote monitoring and control system 2, transmits the corresponding scenario to the drone 3, and issues commands to the drone 3 regarding autonomous flight, such as registering flight paths and shooting locations. The drone integrated management unit 12 also transmits the captured images, which are inspection results received from the drone 3 via the mapping unit 11, to the remote monitoring and control system 2 along with their location information. The drone integrated management unit 12 may also receive status notifications from the drone 3 indicating the status of the drone 3, such as information indicating that patrol and inspection are underway, information indicating the battery level, or information indicating whether or not there is a malfunction.
[0026] The scenario creation unit 13 uses the patrol plan data and operation instructions received by the drone integrated management unit 12 to create a scenario corresponding to this data. The scenario is created, for example, by listing the flight path and shooting locations. The start command for the created scenario is transmitted to the drone 3 via the drone integrated management unit 12. As a result, the flight path and imaging locations are registered to the drone 3, and the drone 3 begins patrolling and inspection.
[0027] The control system 1 transmits data to or receives data from the remote monitoring and control system 2 via an external communication device 6 such as a router. In this embodiment, a wireless public network such as a 4G or 5G network, such as LTE (Long Term Evolution), is used as the communication line, but various other lines may be used depending on the network environment of the patrol and inspection location. For example, in the case of a hydroelectric power plant 200 in a mountainous area, these wireless public networks may be outside the coverage area and unusable. Therefore, a dedicated line such as an optical cable, metal cable, satellite link, or μ wireless may be used.
[0028] Next, the equipment on the data center 300 side in this embodiment will be described.
[0029] The remote monitoring and control system 2 is a system that remotely monitors and controls the drone 3. The worker selects a patrol plan or inputs other operation instructions from the user interface (UI) of the remote monitoring and control system 2. The remote monitoring and control system 2 transmits the patrol plan data corresponding to the patrol plan and other operation instructions to the control system 1. As described above, based on this transmission, the patrol and inspection by the drone 3 is started, and the flight path trajectory, which is the result of the patrol by the drone 3, along with the captured images, which are the result of the inspection, are displayed on the user interface of the remote monitoring and control system 2. The remote monitoring and control system 2 includes a patrol planning unit 21, a work management unit 22, a display unit 23, a media management unit 24, and an analysis unit 25.
[0030] The remote monitoring and control system 2, like the control system 1, is, for example, a PC and has various hardware resources. The functions of each functional block of the remote monitoring and control system 2 are realized by installing programs for the remote monitoring and control system 2. The work management unit 22 and the media management unit 24 are built on, for example, an auxiliary storage device on an HDD.
[0031] Alternatively, the remote monitoring and control system 2 may be remotely controlled from a control center where workers are stationed. In such cases, the remote monitoring and control system 2 is remotely controlled using an input device installed at the control center.
[0032] The patrol planning unit 21 displays a list of patrol plans for the drone 3 on the user interface, allowing the worker to select one. A concrete example of a patrol plan is a sequence of commands such as moving between waypoints. The worker may create a patrol plan list in advance before patrolling or inspecting, for example, when introducing or modifying the remote monitoring and control system 2. The patrol planning unit 21 may also accept individual operation instructions from the worker that are not part of the patrol plan. In this embodiment, an example of inspection and patrolling using an indoor drone 3 with a relatively small battery capacity will be described. From the standpoint of battery capacity, it will be described as inspecting and patrolling only a portion of the equipment inside the building, selected from the list, rather than inspecting and patrolling all the equipment inside the building.
[0033] The work management unit 22 is a memory area that stores the created patrol plans, and is implemented, for example, as a database. When the program of the remote monitoring and control system 2 is running, this information is loaded into memory.
[0034] The display unit 23 displays the patrol and inspection results from the drone 3 on the user interface. Specifically, the drone displays the flight path trajectory and captured images on the map data. At this time, the self-position estimation result from the drone 3 is used as position information, and this trajectory is displayed as the flight path trajectory. The captured images are displayed at positions corresponding to the position information at the time of capture. In this embodiment, the display unit 23 uses a floor plan of the building acquired as electronic data to display 2D map data as the user interface. Other data may be used for the 2D map data, but it is preferable to use existing management materials in terms of effort and cost. Also, when displaying 3D map data as the user interface, the environmental map created by the map creation unit 11 may be used as map data. The floor plan and 3D map are examples of drawings. The display unit 23 also creates and displays the flight path trajectory from the position information of the drone 3 acquired via the drone integrated management unit 12, and displays captured images at the imaging locations along the trajectory.
[0035] Furthermore, in hydroelectric power plants 200, the building is often divided into multiple floors, and the equipment subject to inspection and monitoring is often located in multiple separate machine rooms. In this embodiment, the user interface generated by the display unit 23 accepts input from workers for switching floor plans, allowing the building's floor plans to be switched. For example, a worker can switch the building's floor plan using a pull-down menu.
[0036] Furthermore, by linking the floor plan of the building with the estimated self-position on the environmental map and the waypoints during the flight of drone 3, the trajectory of the flight path can be displayed in correspondence with the floor plan of the building. Since the structure of the building is not expected to change much after construction, the linking may be performed using this data after the environmental map is created by the map creation unit described above and before the patrol and inspection are carried out. This linking may also be implemented using a layer structure, such as a layer for the floor plan and a layer for displaying the trajectory of the flight path and captured images.
[0037] The media management unit 24 stores floor plans of the building, captured images, and location information, as well as coordinate transformation data for converting the relative coordinates of the drone 3 to absolute coordinates. Since the location information acquired from the drone 3 is relative location information, it is converted to absolute coordinates and linked to the floor plan of the building. The media management unit 24 may also store the acquired relative coordinates of the location information in correspondence with absolute coordinates.
[0038] Furthermore, the media management unit 24 may receive and store status notifications from the drone 3 via the drone integrated management unit 12. In this case, the display unit 23 may display this data together with the plan view on the user interface.
[0039] The analysis unit 25 analyzes data using at least one of the captured images, location information, status notifications, floor plans, and coordinate transformation data stored in the media management unit 24. The analysis unit 25 performs the analysis using, for example, a machine learning model. For example, the analysis unit 25 may perform the analysis using a machine learning model that has been trained using the results of determining whether or not there is a malfunction in the equipment captured in the images as training data.
[0040] Figure 2 is another schematic diagram of the patrol inspection planning system 100 in this embodiment.
[0041] In contrast to Figure 1, Figure 2 shows a different configuration of the hydroelectric power plant 200. In Figure 1, the mapping unit 11, the drone integrated management unit 12, and the scenario creation unit 13 were implemented as a single device as software functional blocks, whereas in this configuration diagram, these functional blocks are implemented as separate devices: the mapping device 11a, the drone integrated management device 12a, and the scenario creation device 13a. As in this example, the control system 1 may be implemented by combining multiple devices. The mapping device 11a, the drone integrated management device 12a, and the scenario creation device 13a are examples of the mapping unit 11, the drone integrated management unit 12, and the scenario creation unit 13, respectively. Similarly, the remote monitoring and control system 2 may also be implemented by combining multiple devices.
[0042] Figure 3 is yet another schematic configuration diagram of the patrol inspection planning system 100 in this embodiment.
[0043] Compared to Figure 1, Figure 3 shows a different configuration of the hydroelectric power plant 200 and the data center 300. In Figure 1, the control system 1 is installed within the hydroelectric power plant 200, whereas in this configuration diagram, the functions equivalent to the control system 1 are built on the data center 300 side. In this configuration diagram, the functions of the control system 1 and the remote monitoring and control system 2 are implemented as a single device.
[0044] In this example, captured images, location information, status notifications, and camera footage are transmitted from the drone 3 to the patrol and inspection planning system 100 in the data center 300 via the off-site communication device 6. For example, if the line connecting the hydroelectric power plant 200 and the data center 300 has sufficient capacity to transmit camera footage and environmental maps, the information acquired by the drone 3 is processed on the data center 300 side.
[0045] For example, the map creation unit 11 creates an environmental map based on camera images transmitted by the drone 3 via the off-site communication device 6. The created environmental map is transmitted from the drone integrated management unit 12 to the drone 3 via the off-site communication device 6 and used for estimating its own position.
[0046] Furthermore, the drone 3 transmits captured images, location information, and status notifications to the data center 300 via the off-site communication device 6. The processing of this information at the data center 300 is the same as described above.
[0047] Since the processing of each functional block in Figures 2 and 3 is mainly the same as in Figure 1, the following explanation will use the configuration of Figure 1.
[0048] Figure 4 is a hardware configuration diagram of the drone 3 and charging port 4 in this embodiment.
[0049] Drone 3 includes a motor and propeller 310, an ESC (electronic speed controller) 311, a battery 312, a power receiving unit 313, a flight controller 314, a companion computer 315, a tracking camera 316, a 360° camera 317, an inspection camera 318, a 2D laser distance sensor 319, a receiver 320, a transmitter 321, LED lighting 322, and an AR marker 323. Charging port 4 includes an AR marker recognition camera 410, LED lighting 411, and a power transmission unit 412. The functions of each part of drone 3 and charging port 4 will be described below.
[0050] The motor and propeller 310 generate lift for the drone 3. In this embodiment, the drone 3 has multiple propellers (e.g., four) and motors (e.g., four) for each propeller. The ESC 311 is an electronic device for controlling the rotational speed of the motors.
[0051] The battery 312 is a power source for supplying power to the drone 3. In this embodiment, since the drone 3 is for indoor use, a drone 3 with a small battery capacity may be used. For example, a drone 3 with a battery capacity sufficient to fly for about 7 or 8 minutes during a single patrol and inspection may be used. The power receiving unit 313 is a device for automatically supplying power to the battery 312 without contact. The charging method may be contact-type.
[0052] The flight controller 314 incorporates an IMU (Electronic Speed Controller) including an accelerometer and gyroscope to stabilize the flight of the drone 3. The flight controller 314 also instructs the ESC 311 to output the motors. The companion computer 315 is a computer that processes data received from sensors such as the 2D laser distance sensor 319 and performs collision avoidance and self-position estimation processing. The companion computer 315 is connected to the drone integrated management unit 12 via a wireless connection such as Wi-Fi.
[0053] The tracking camera 316 is a camera used to estimate its own position from video data obtained through shooting. Similarly, the 360° camera 317 is also a camera used to estimate its own position from video data obtained through shooting. Self-position estimation using the tracking camera 316 has a short update cycle and small time delay, but because it is done by accumulating the relative movement amount, the relative position drifts. In this embodiment, the drift of the self-position estimated by the tracking camera 316 is corrected by interpolating the position estimated by the 360° camera, which has a relatively long update cycle, with the relative movement amount of the tracking camera, which has a relatively short update cycle.
[0054] The inspection camera 318 is a camera used to capture images and photographs. For example, the drone 3 uses the inspection camera 318 to take pictures at the location indicated in the scenario. The 2D laser distance sensor 319 is a sensor for obstacle detection. While in flight, the drone 3 uses the 2D laser distance sensor 319 to detect the distance to obstacles within the hydroelectric power plant 200 and corrects its direction of movement. Also, if an anomaly occurs in the estimation of its own position, the drone 3 uses this sensor to detect surrounding obstacles and lands while avoiding them.
[0055] The receiver 320 is a device for receiving data from and transmitting data to the transmitter 321 for manual operation. The receiver 320 is connected to the transmitter 321 by a wireless link using a predetermined frequency. The captured images, location information, status notifications, and camera footage of the drone 3 may be transmitted to the drone integrated management unit 12 via the companion computer 315, or to the transmitter 321 via the receiver 320. In such a configuration, for example, the camera footage is transmitted to the transmitter 321 and then input to the mapping unit 11.
[0056] The LED light 322 is a lighting device for ensuring visibility of the equipment being inspected and the flight environment of the drone 3. The AR marker 323 is attached to the bottom of the drone 3 and used so that the charging port 4 can recognize the position of the drone 3. The drone 3 may automatically turn on the LED light 322 when it reaches a place with a poor working environment, such as a narrow staircase or passageway, or depending on the ambient light level.
[0057] The AR marker recognition camera 410 is a camera used to recognize the position of the drone 3 in the air using the AR marker 323 after the drone 3 has completed its patrol and inspection. The LED lighting 411 is a lighting device to ensure the visibility of the AR marker 323. The power transmission unit 412 is a charging device that charges the drone 3 by wireless power transmission. The power transmission unit 412 takes AC100V as input, converts it to a predetermined DC voltage for charging the battery 312, and supplies it.
[0058] Figure 5 shows an example of an environmental map created by the map creation unit 11 in this embodiment.
[0059] As mentioned above, in hydroelectric power plants 200, the building is often divided into multiple floors, and the equipment to be inspected and monitored is often located in multiple separate machine rooms. When drone 3 flies over a wider area, spanning multiple rooms and floors, it uses environmental maps for 360° VSLAM corresponding to each location to estimate its own position, which increases the memory load and can make real-time estimation of its own position difficult.
[0060] Furthermore, hydroelectric power plants such as the 200 may contain rooms with similar structures, such as Building 1 or Building 2. When Drone 3 performs self-position estimation using 360° VSLAM in such buildings, it may extract similar feature points, resulting in unstable estimation results.
[0061] In this embodiment, the mapping unit 11 divides the area within the hydroelectric power plant 200 into multiple areas and creates an environmental map. The mapping unit 11 divides the indoor area, for example, to include a room or a floor within a building, and creates an environmental map. The mapping unit 11 also overlaps parts of each environmental map created by dividing the indoor area to create an overall environmental map. The mapping unit 11 sets a map switching boundary within the overlapping area. The drone integrated management unit 12 registers the environmental map with the drone 3. When the drone 3's estimated position reaches a map switching boundary located within the overlapping area, it switches the environmental map. Information indicating the drone 3's flight direction may be used to determine whether to switch maps.
[0062] This diagram shows two environmental maps, the first and second, which are part of an environmental map that divides the indoor area into multiple sections. These two maps overlap in some areas. Furthermore, a map switching boundary, set via the map creation unit 11, exists within this overlapping area.
[0063] Figure 6 shows an example of switching the environmental map using the drone 3 in this embodiment.
[0064] Figure 6 illustrates the process by which Drone 3 switches from the first environmental map to the second environmental map during flight. Figure 6(A) shows the state before Drone 3 switches the environmental map. When Drone 3 is estimating its own position using the first environmental map during flight, for example, the second environmental map has not yet been loaded into memory and is in a hiatus state. In this figure, the second environmental map is shown with a dashed line for clarity of explanation. On the other hand, Figure 6(B) shows the state after Drone 3 switches the environmental map. When Drone 3 reaches a map switching boundary located in the overlapping area of the first and second environmental maps during flight, it switches from the first environmental map to the second environmental map. For example, the second environmental map is loaded from auxiliary storage at this time, and the first environmental map is hiatus from memory to auxiliary storage. In this way, the increase in load associated with loading a large-capacity environmental map can be suppressed and it does not hinder real-time self-position estimation.
[0065] The map creation unit 11 may, for example, set a map switching boundary in a passageway that forms the boundary between one machine room and another machine room. Alternatively, the map creation unit 11 may set a map switching boundary in a staircase that connects one floor to another floor.
[0066] Furthermore, this map switching boundary may be automatically set by the map creation unit 11 in the overlapping area by comparing the two environmental maps before registering the environmental map to the drone 3, or it may be set in advance by an operator using an input device via the user interface of the map creation unit 11. In addition, the map switching boundary may be taught to the drone 3 by some method other than those described above.
[0067] Furthermore, if the difference between the self-position estimated by the 360° camera and the self-position estimated by the tracking camera on the environmental map is large, drone 3 will determine that an anomaly has occurred and land. At this time, drone 3 will detect obstacles in the surrounding area using the position information immediately before the anomaly was determined and various onboard sensors, and will land while avoiding the obstacles. This difference is a design value and may be set to any threshold depending on the location and type of equipment being inspected.
[0068] Figure 7 shows an example of the user interface of the remote monitoring and control system 2 in this embodiment.
[0069] This embodiment shows an example of unifying and displaying the user interface created by the patrol planning unit 21 and the user interface created by the display unit 23 on a display device. The first user interface displays the patrol plan list created by the patrol planning unit 21, and the second user interface displays the flight path of the drone 3 and inspection results created by the display unit 23. In this example, the upper section is shown as an example of the first user interface, and the lower section is shown as an example of the second user interface.
[0070] The upper section of Figure 7 shows a patrol plan list with fields for plan ID, flight location, plan title, scheduled execution date and time, start date and time, end date and time, plan execution status, execution result, and remarks. The patrol planning unit 21 displays this user interface on the display device. The worker selects a patrol plan from the patrol plan list. The worker may also specify the scheduled execution date and time when deciding on a patrol plan.
[0071] The Plan ID is information that identifies the patrol plan for Drone 3, and a different ID is assigned to each patrol plan. The Flight Location indicates the area where Drone 3 will fly according to the patrol plan. The Plan Title indicates the title of the patrol plan.
[0072] The scheduled execution date and time indicates the scheduled execution date of the patrol plan in question. The start date and time indicates the date and time when the corresponding patrol plan started, and the end date and time indicates the date and time when the corresponding patrol plan ended. The plan execution status indicates the execution status of the patrol plan by Drone 3. For example, if the inspection plan is in progress, the string "In Progress" will be displayed on the user interface. Also, since Drone 3 transmits the captured images as inspection results to the media management unit 24 along with their location information, the string "Upload Complete" will be displayed on the user interface when the transmission of the inspection results is complete. The plan execution status information may also be created using status notification information, such as whether Drone 3 is in flight.
[0073] The execution result will display the string "Execution Successful" if the patrol plan is completed. For example, after drone 3 finishes its patrol, completes the transmission of captured images, and lands at charging port 4, it sends a signal to remote monitoring and control system 2 as a status notification indicating that the patrol was successful. The patrol planning unit 21 uses this information to display the string "Execution Completed" on the user interface.
[0074] The remarks column allows workers to freely add comments. Alternatively, this column may be configured to automatically display comments using information received from Drone 3.
[0075] The lower part of Figure 7 displays the flight path of drone 3 on a map data using a floor plan of the building. This figure shows the patrol and inspection of the first and second machine rooms in progress (plan ID corresponds to 0001). Clicking on a location on the flight path where the inspection camera 318 was used will display the captured image. This captured image may also have a function to be enlarged by clicking a button or mouseover.
[0076] Furthermore, this user interface displays the first and second machine rooms as floor plans, and the floor plans can be switched using the "Floor Plan Switching" pull-down menu in the upper right corner of the floor plan. In this example, the floor plan is switched manually by the worker. As mentioned above, since the floor plan inside the building is linked to the position information on the environmental map estimated by the drone 3, if the drone's position on the environmental map falls outside a predetermined range on the floor plan inside the building, the display unit 23 may automatically switch the floor plan on the user interface.
[0077] On the other hand, as described above, when drone 3 reaches a map switching boundary located within the overlapping area of the first and second environment maps during flight, it switches the environment map internally. In this example, the first machine room and corridor area are designated as the first environment map, and the second machine room and corridor area are designated as the second environment map. When drone 3 reaches the corridor area, which is part of the overlapping area, it performs the process of switching between the first and second environment maps.
[0078] Figure 8 is an example of a flowchart of the control system 1 in this embodiment.
[0079] This flowchart describes the process by which the control system 1 generates an environmental map using camera footage received from the drone 3, and then registers this map with the drone 3. This process is performed before the execution of the patrol plan, for example, when the patrol inspection planning system 100 is introduced. In this embodiment, the example of generating an environmental map using camera footage received from the drone 3 is used, but the camera footage used to generate the environmental map is not limited to footage received from the drone 3. For example, an environmental map may be generated using existing camera footage, or camera footage captured by a device other than the drone 3 may be used. Furthermore, the camera footage referred to here may be captured in units that divide the environmental map, such as a first environmental map or a second environmental map, or it may be captured as a single video.
[0080] In step S1, the mapping unit 11 receives 360° camera footage from the drone 3. The camera footage may be, for example, footage taken inside the building of the hydroelectric power plant 200 by a worker operating the drone 3. Alternatively, the camera footage may be footage automatically captured by the autonomous flight of the drone 3. Furthermore, this camera footage may be captured in units that divide the environmental map, such as a first environmental map or a second environmental map, or it may be captured as a single video.
[0081] In step S2, the mapping unit 11 creates an environmental map using the received camera footage. The mapping unit 11 extracts feature points from the camera footage. As the drone moves the 360° camera, the feature points change, so the mapping unit 11 calculates the distance to the feature points from the changes and creates an environmental map. The mapping unit 11 also divides the hydroelectric power plant 200 into multiple areas and creates environmental maps for each area. The mapping unit 11 divides the area into regions such as corridors and staircases that serve as map switching boundaries, and creates multiple environmental maps so that these regions overlap. The mapping unit 11 may also automatically recognize regions such as corridors and staircases that serve as map switching boundaries from feature points in a single camera image and divide the environmental map accordingly. Alternatively, it may receive multiple camera images and create environmental maps from each image.
[0082] In step S3, the map creation unit 11 sets a map switching boundary in the overlapping area of the multiple environment maps created. The map creation unit 11 may compare the two environment maps and automatically set the map switching boundary in the overlapping area. Alternatively, the map creation unit 11 may set the map switching boundary using information entered by an operator on the user interface of the map creation unit 11.
[0083] In step S4, the drone integrated management unit 12 transmits the environmental map created by the map creation unit 11 to the drone 3 and issues a registration instruction. Upon receiving the environmental map, the drone 3 registers the environmental map along with the map switching boundary. This allows the drone 3 to perform self-position estimation and environmental map switching.
[0084] Figure 9 is an example of a flowchart of the remote monitoring and control system 2 in this embodiment.
[0085] This flowchart describes the process in which Drone 3 conducts patrols and inspections within the hydroelectric power plant 200 based on a patrol plan selected by a worker on the user interface, and displays the results on the user interface. It also assumes that the patrol plan list has been created in advance of this process, and that the environmental map has been registered in Drone 3 in advance.
[0086] In step S21, the patrol planning unit 21 displays the patrol plan list stored in the work management unit 22 on the user interface. The patrol planning unit 21 displays the list, for example as shown in Figure 7, so that workers can select a patrol plan.
[0087] In step S22, the patrol planning unit 21 transmits patrol plan data corresponding to the patrol plan decided by the worker from the patrol plan list to the control system 1. The worker may specify the scheduled date and time of execution when deciding on the patrol plan. Upon receiving the patrol plan data, the control system 1's scenario creation unit 13 uses the patrol plan data received by the drone integrated management unit 12 to create a scenario corresponding to this data. The scenario is created, for example, by listing the flight path and shooting locations. The start command for the created scenario is transmitted to the drone 3 via the drone integrated management unit 12. As a result, the flight path and imaging locations are registered to the drone 3, and the drone 3 begins patrolling and inspection.
[0088] In step S23, the media management unit 24 stores the captured images and their location information transmitted from the drone 3. The media management unit 24 also stores location information transmitted from the drone 3 as needed to track the flight path.
[0089] In step S24, the display unit 23 displays the flight path of the drone 3 and the captured images along with a floor plan of the building. The display unit 23 uses the captured images, location information, floor plan, and coordinate transformation data stored in the media management unit 24 to display the flight path of the drone 3 and the captured images on the floor plan. The display unit 23 converts the relative coordinates of the location information acquired from the drone 3 into absolute coordinates and uses this information to display the flight path and captured images on the floor plan. The flight path of the drone 3 is constantly updated and displayed on the user interface. In addition, if a worker switches the floor plan of the building using the pull-down menu, the display unit 23 switches to the corresponding floor plan. The flight path of the drone 3 and the captured images are then displayed on the switched floor plan.
[0090] Figure 10 is a hardware configuration diagram of the control system 1 in this embodiment.
[0091] The control system 1 in Figure 10 comprises a processor 52 such as a CPU, a main memory 53 such as RAM, an auxiliary storage device 54 such as an HDD, a network interface 55 such as a LAN (Local Area Network) board, a device interface 56 such as memory slots and memory ports, and a bus 57 that connects these devices to each other. The control system 1 is, for example, a computer such as a PC, and is equipped with external input devices such as a keyboard and mouse, and a display device such as an LCD monitor.
[0092] In this embodiment, a program for causing a computer to perform information processing on the control system 1 is installed in the auxiliary storage device 54. The control system 1 loads this program into the main storage device 53 and executes it using the processor 52. This enables the functions of the map creation unit 11, the drone integrated management unit 12, and the scenario creation unit 13 shown in Figure 1 to be realized within the control system 1, making it possible to control the drone 3 as described in this embodiment. Data generated by this information processing, such as environmental maps, is temporarily held in the main storage device 53 or stored in the auxiliary storage device 54.
[0093] Furthermore, the control system 1 is connected to the on-site communication device 5 and the off-site communication device 6 via the network interface 55. The drone integrated management unit 12 controls the network interface 55 and transmits and receives various types of information.
[0094] The drone control program for control system 1 can be installed, for example, by attaching an external device 58 containing the program to the device interface 56 and storing the program from the external device 58 to the auxiliary storage device 54. An example of the external device 58 is a computer-readable recording medium or a recording device that incorporates such a recording medium. Examples of recording media include CD-ROM (Compact Disk Read Only Memory), CD-R (Compact Disk Recordable), flexible disk, DVD-ROM (Digital Versatile Disk Read Only Memory), and DVD-R (Digital Versatile Disk Recordable), while an example of a recording device is an HDD. The program can also be installed, for example, by downloading it via the network interface 55.
[0095] Furthermore, the hardware configuration diagram of the remote monitoring and control system 2 is the same as that of the control system 1. A program for causing the computer to execute information processing for the remote monitoring and control system 2 is installed in the auxiliary storage device 54. The remote monitoring and control system 2 loads this program into the main storage device 53 and executes it using the processor 52. This enables the functions of the patrol planning unit 21, work management unit 22, display unit 23, media management unit 24, and analysis unit 25 shown in Figure 1 to be realized within the control system 1, making it possible to monitor and control the drone 3 described in this embodiment.
[0096] According to this embodiment, the drone 3 estimates its own position using an environmental map created by the control system 1 and performs patrols and inspections inside the hydroelectric power plant 200 according to a patrol plan selected from a remote remote monitoring and control system 2. This allows the drone 3 to automatically perform patrols and inspections even in indoor areas that are difficult to access and where GPS signals cannot reach. Furthermore, since there is no need for workers to go to the site for patrols and inspections, O&M costs can be reduced.
[0097] Furthermore, according to this embodiment, the remote monitoring and control system 2 displays a patrol plan list on the user interface and allows the worker to select the contents of the patrol plan to be performed by the drone 3. This makes it possible to remotely and efficiently perform tasks with predetermined contents, such as daily patrols and daily inspections.
[0098] Furthermore, according to this embodiment, the remote monitoring and control system 2 displays the floor plan of the hydroelectric power plant 200 building and the flight path of the drone 3 linked together on the user interface. This allows workers to visually confirm where the drone 3 is flying, even indoors where GPS signals cannot reach.
[0099] Furthermore, according to this embodiment, the remote monitoring and control system 2 switches the floor plan of the hydroelectric power plant 200 building manually or automatically. This allows the worker to visually confirm where the drone 3 is flying by checking the floor plan after the switch, even if the drone 3 has moved from the floor plan before the switch.
[0100] Furthermore, according to this embodiment, the control system 1 creates separate environmental maps of the hydroelectric power plant 200 and registers them with the drone 3. The drone 3 then flies while switching between multiple environmental maps when conducting patrols and inspections. This reduces the load on the drone 3 compared to estimating its own position by reading the environmental map of the entire building.
[0101] Furthermore, according to this embodiment, if the difference between the drone 3's self-position estimated by the 360° camera and the self-position estimated by the tracking camera on the environmental map is large, the drone 3 will determine that there is an anomaly and land while avoiding obstacles. This prevents accidents such as the drone 3 colliding with surrounding equipment or becoming entangled in rotating machinery, even if an anomaly occurs during self-position estimation.
[0102] Furthermore, according to this embodiment, the drone 3 is equipped with LED lighting 322. This ensures that the drone 3 is visible during inspections, even in poor working environments such as basements, narrow stairwells, or passageways.
[0103] Although several embodiments have been described above, these embodiments are presented only as examples and are not intended to limit the scope of the invention. The novel patrol inspection planning system 100 described herein can be implemented in a variety of other forms. Furthermore, various omissions, substitutions, and modifications can be made to the forms of the patrol inspection planning system 100 described herein, without departing from the spirit of the invention. The appended claims and equivalents are intended to include such forms and modifications included in the scope and spirit of the invention. [Explanation of symbols]
[0104] 1: Control system, 2: Remote monitoring and control system, 3: Drone 4: Charging port, 5: On-premises communication device, 6: Off-premises communication device, 11: Map creation department, 11a: Map creation device, 12: Drone integrated management department, 12a: Drone integrated management device, 13: Scenario creation unit, 13a: Scenario creation device, 21: Patrol Planning Department, 22: Work Management Department, 23: Display Department, 24: Media Management Department, 25: Analysis unit, 52: Processor, 53: Main memory, 54: Auxiliary memory, 55: Network interface, 56: Device interface, 57: Bus, 58: External equipment, 100 patrol inspection planning system, 200: Hydroelectric power plants, 300: Data centers, 310: Motors and propellers, 311: ESC, 312: Battery, 313: Power receiving unit, 314: Flight controller, 315: Companion computer, 316: Tracking camera, 317: 360° camera, 318: Inspection camera, 319: 2D laser distance sensor, 320: Receiver, 321: Transmitter, 410: Camera for AR marker recognition, 411: LED lighting, 412: Power transmission unit
Claims
1. A mapping unit used to estimate the self-position of a drone that patrols and inspects indoors, and which creates an environmental map that divides the indoor area into one or more regions, A scenario creation unit creates a scenario indicating the flight path and shooting locations based on input of data from a patrol plan that shows the patrol and inspection plans to be performed by the drone. The system includes a drone integrated management unit that transmits the aforementioned environmental map and the aforementioned scenario to the drone and instructs it to perform the aforementioned patrol and inspection. Control system.
2. The aforementioned environmental map includes a first environmental map and a second environmental map, The control system according to claim 1, wherein a map switching boundary is provided in the overlapping portion between the first environmental map and the second environmental map.
3. The control system according to claim 1, wherein the estimation of the self-position is performed by 360° VSLAM (Visual Simultaneous Localization and Mapping) using a first imaging device provided on the drone.
4. The control system according to claim 3, wherein the estimation of the self-position is further performed using a second imaging device different from the first imaging device provided on the drone.
5. The control system according to claim 4, wherein an abnormality in the self-position estimation result by the drone 3 is determined based on the difference between the self-position estimated using the first imaging device and the self-position estimated using the second imaging device.
6. The control system according to claim 3, wherein the estimation of the self-position is further performed using lighting provided on the drone.
7. The control system according to claim 5, wherein the estimation of the self-position is further performed using lighting provided on the drone.
8. The control system according to claim 2, wherein the first environmental map and the second environmental map are switched and used when the drone reaches the map switching boundary.
9. A patrol planning unit allows the user to select a patrol plan for a drone that patrols and inspects indoors, via a first user interface. The system includes a display unit that displays the results of the patrol and inspection on a second user interface, The second user interface displays the flight path based on the drone's self-position estimation and the images captured by the drone on the indoor drawing. Remote monitoring and control system.
10. The remote monitoring and control system according to claim 9, wherein the display unit displays the flight path on the second user interface by linking the self-position estimation result and waypoints during flight by the drone with the drawing.
11. The remote monitoring and control system according to claim 9, wherein the display unit receives input of a switching command to switch the drawing in response to the user's operation on the second user interface.
12. The remote monitoring and control system according to claim 10, wherein the display unit automatically switches the drawing on the second user interface when the self-position estimated by the drone falls outside a predetermined range on the drawing.
13. A control system for controlling a drone that patrols and inspects indoors, The drone is equipped with a remote monitoring and control system that monitors and controls the drone from a distance. The control system is A mapping unit used to estimate the drone's own position and to create an environmental map dividing the indoor area into one or more regions, A scenario creation unit creates a scenario indicating the flight path and shooting locations based on input of data from a patrol plan that shows the patrol and inspection plans to be performed by the drone. Includes a drone integrated management unit that transmits the environmental map and the scenario to the drone and instructs it to perform the patrol and inspection, The remote monitoring and control system is, A tour planning unit that allows the user to select the aforementioned tour plan on the first user interface, It includes a display unit that displays the results of the patrol and inspection on a second user interface, The second user interface displays the flight path based on the drone's self-position estimation and the images captured by the drone on the indoor drawing. Inspection and patrol planning system.