Monitoring device

Abstract

To solve the problem in which: it may be difficult to appropriately recover a foreign substance.SOLUTION: A monitoring device has: a detection unit that detects a foreign substance according to image data with a monitoring target area as a subject; and a recovery instruction unit that instructs an external device to recover the foreign substance detected by the detection unit. The recovery instruction unit instructs the external device to recover the foreign substance according to the priority determined in advance, for an area where the foreign substance is present, of areas obtained by dividing the monitoring target area.SELECTED DRAWING: Figure 13

Description

[Technical Field]

[0001] The present invention relates to a monitoring device, a monitoring method, and a program. [Background technology]

[0002] There are known techniques used to recover detected foreign matter and the like.

[0003] For example, Patent Document 1 discloses a system including a radar device that detects objects, a radar monitoring device that controls the radar device, and a foreign object collection device that moves to the location of the object based on the object's location information and collects the object. According to Patent Document 1, after collecting the object, the foreign object collection device transmits a foreign object collection notification to the radar monitoring device indicating that collection of the object has been completed. Furthermore, if the radar monitoring device confirms after receiving the foreign object collection notification that the radar device can no longer detect the object within its detection range, it transmits a collection confirmation notification to the foreign object collection device indicating that collection of the object has been confirmed. Then, after receiving the collection confirmation notification, the foreign object collection device withdraws from the scene. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2023-096269 Summary of the Invention [Problem to be solved by the invention]

[0005] The impact of the presence of foreign matter may vary depending on the location of the foreign matter, and simply collecting foreign matter as described in Patent Document 1 may not always be able to collect the foreign matter in a way that takes into account the impact of the foreign matter. As a result, there has been a problem in that it may be difficult to collect foreign matter appropriately, such as there being a delay in the collection of foreign matter that has a large impact.

[0006] Therefore, an object of the present invention is to provide a monitoring device, a monitoring method, and a program that can solve the above-mentioned problems. [Means for solving the problem]

[0007] In order to achieve this object, a monitoring device according to one embodiment of the present disclosure comprises: a detection unit that detects foreign objects based on image data of a monitored area; a collection instruction unit that instructs an external device to collect the foreign matter detected by the detection unit; and The collection instruction unit instructs the external device to collect the foreign matter in accordance with a priority determined in advance for an area where the foreign matter exists among the divided areas of the monitoring area. The structure is as follows.

[0008] Furthermore, a monitoring method according to another aspect of the present disclosure includes: The information processing device Detects foreign objects based on image data of the monitored area, Instructing an external device to collect the detected foreign matter, When issuing an instruction to collect foreign matter, the external device is instructed to collect foreign matter according to a priority determined in advance for an area where foreign matter exists among the divided areas of the monitoring area. The structure is as follows.

[0009] Furthermore, a program according to another aspect of the present disclosure includes: In the information processing device, Detects foreign objects based on image data of the monitored area, Instructing an external device to collect the detected foreign matter, When issuing an instruction to collect foreign matter, the external device is instructed to collect foreign matter according to a priority determined in advance for an area where foreign matter exists among the divided areas of the monitoring area. It is a program for realizing the processing. [Effects of the Invention]

[0010] According to the above-described configurations, foreign matter can be more appropriately collected in accordance with the detected information. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of the overall configuration of a monitoring system. [Figure 2] FIG. 2 is a block diagram showing an example configuration of an unmanned aerial vehicle. [Figure 3] FIG. 3 is a diagram showing an example of detection information shown in FIG. 2. [Figure 4] FIG. 10 is a diagram illustrating an example of a priority determination process. [Figure 5] FIG. 10 is a diagram illustrating an example of a collection instruction. [Figure 6] FIG. 10 is a diagram illustrating an example of setting a flight path. [Figure 7] FIG. 10 is a diagram illustrating an example of setting a flight path. [Figure 8] FIG. 2 illustrates an example of the configuration of a management device. [Figure 9] FIG. 10 is a sequence diagram illustrating an example of the operation of the monitoring system. [Figure 10] 10 is a flowchart illustrating an example of the operation of an unmanned aerial vehicle when setting a flight path. [Figure 11] 10 is a flowchart showing an example of the operation of an unmanned aerial vehicle when performing priority determination. [Figure 12] FIG. 10 is a diagram illustrating an example of a hardware configuration of a monitoring device according to a second embodiment of the present disclosure. [Figure 13] FIG. 2 is a block diagram illustrating an example of the configuration of a monitoring device. [Figure 14] 10 is a flowchart illustrating an example of the operation of the monitoring device. DETAILED DESCRIPTION OF THE INVENTION

[0012] [First embodiment] A first embodiment of the present invention will be described with reference to FIGS. 1 to 11. FIG. 1 is a diagram illustrating an example of the overall configuration of monitoring system 100. FIG. 2 is a block diagram illustrating an example of the configuration of unmanned aerial vehicle 200. FIG. 3 is a diagram illustrating an example of detection information 234. FIG. 4 is a diagram illustrating an example of priority determination processing. FIG. 5 is a diagram illustrating an example of a recovery instruction. FIGS. 6 and 7 are diagrams illustrating an example of setting a flight path. FIG. 8 is a diagram illustrating an example of the configuration of management device 400. FIG. 9 is a sequence diagram illustrating an example of the operation of monitoring system 100. FIG. 10 is a flowchart illustrating an example of the operation of unmanned aerial vehicle 200 when setting a flight path. FIG. 11 is a flowchart illustrating an example of the operation of unmanned aerial vehicle 200 when performing priority determination. It should be noted that the drawings in this disclosure may be associated with one or more of the embodiments.

[0013] In a first embodiment of the present disclosure, a monitoring system 100 is described that monitors foreign objects and the like using an unmanned aerial vehicle 200, such as a drone, as a monitoring device and, if necessary, collects the foreign objects using a collection device 300 or the like. As described below, the monitoring system 100 divides a monitoring area into multiple regions using predetermined information and information collected using the unmanned aerial vehicle 200 or the like, and calculates a priority for collecting foreign objects for each divided region. For example, the monitoring system 100 divides a monitoring area into multiple regions using the various information described above, and calculates a priority for each region based on the magnitude of the impact of the presence of the foreign object, the possibility of foreign object leakage, and other risks. Furthermore, when a foreign object is detected using the unmanned aerial vehicle 200 or the like, the monitoring system 100 instructs the collection device 300 to collect the detected foreign object. At this time, the monitoring system 100 can instruct the collection device 300 to collect the foreign object according to the priority corresponding to the region in which the foreign object is present. For example, the monitoring system 100 may instruct the collection device 300 to collect foreign objects present in areas with higher priority earlier. For example, with such a configuration, the monitoring system 100 can perform appropriate collection depending on the presence of the detected foreign object. Note that the areas for which the priority is calculated may be predetermined, instead of being divided by the monitoring system 100.

[0014] Furthermore, monitoring system 100 can be configured to set the flight path of unmanned aerial vehicle 200 used as a monitoring device using the foreign object detection results and information collected using unmanned aerial vehicle 200. For example, monitoring system 100 can set the flight path so that it passes over the location where the foreign object was detected, or can set the flight path so that the monitoring time for the foreign object detection location is increased by circling unmanned aerial vehicle 200 at the location where the foreign object was detected. In addition to the above example, monitoring system 100 may also be configured to set the flight path using information collected when determining the priority of collection.

[0015] In this disclosure, a case where roads such as expressways and railroads are monitored will be described as an example of a monitored area. However, the monitored area is not limited to roads and railroads, and may be any other location. For example, the monitored area may be a coastal area, a border, or any other location. As will be described later, the information collected by unmanned aerial vehicle 200 and the type of foreign object to be detected may be determined according to the monitored area. For example, if the monitored area is a highway, unmanned aerial vehicle 200 can collect information such as the speed and speed changes of vehicles traveling on the road, whether there are traffic jams, road conditions such as uphill slopes and sharp curves, and other information. Unmanned aerial vehicle 200 can also be configured to detect foreign objects, such as objects falling from vehicles traveling on the road.

[0016] FIG. 1 shows an example configuration of a monitoring system 100. Referring to FIG. 1, monitoring system 100 includes, for example, an unmanned aerial vehicle 200 that functions as a monitoring device in the system, a collection device 300 that collects foreign objects, and a management device 400. For example, as shown in FIG. 1, unmanned aerial vehicle 200 and collection device 300 can communicate with each other using wireless communication or the like. Furthermore, unmanned aerial vehicle 200 and management device 400 can communicate with each other using wireless communication or the like. Similarly, collection device 300 and management device 400 may be connected so as to be able to communicate with each other using wireless communication or the like.

[0017] The number of each component included in the monitoring system 100 may be adjusted as desired. For example, the monitoring system 100 may include one or more unmanned aerial vehicles 200. The monitoring system 100 may also include one or more collection devices 300. Similarly, the monitoring system 100 may also include one or more management devices 400. As will be described later, the monitoring system 100 may also include different types of collection devices 300, such as vehicles, drones, or any other robots.

[0018] Unmanned aerial vehicle 200 is a monitoring device that monitors a predetermined monitoring area by, for example, acquiring image data while flying over the area along a set flight path. For example, unmanned aerial vehicle 200 can fly over the area along a set flight path at regular intervals. While flying over the area, unmanned aerial vehicle 200 can collect specific information and detect foreign objects based on the image data. Furthermore, unmanned aerial vehicle 200 can notify management device 400 of the collected information and foreign object detection results, and instruct collection device 300 to collect the foreign object. In this case, unmanned aerial vehicle 200 can issue a collection instruction to collection device 300 based on the determined priority.

[0019] Fig. 2 shows an example configuration of unmanned aerial vehicle 200. Referring to Fig. 2, unmanned aerial vehicle 200 has, as main components, for example, an imaging unit 210, a communication interface unit 220, a memory unit 230, and an arithmetic processing unit 240. Note that unmanned aerial vehicle 200 may have configurations other than those exemplified above, such as a GPS (Global Positioning System) sensor used to determine the position of unmanned aerial vehicle 200, an acceleration sensor or gyro sensor used during flight, and various other sensors used to collect arbitrary information.

[0020] The imaging unit 210 acquires image data of a monitored area or the like as a subject. For example, the imaging unit 210 can acquire time-series image data at predetermined intervals. The imaging unit 210 may be an imaging device such as a camera. Note that various setting values ​​used by the imaging unit 210 when acquiring image data may be configured to be adjustable according to information associated with information indicating the flight path, etc.

[0021] The communication interface unit 220 is composed of an antenna, a data communication circuit, etc. The communication interface unit 220 performs data communication with external devices such as the collection device 300 and the management device 400.

[0022] The storage unit 230 is a storage device such as a hard disk or memory. The storage unit 230 stores processing information and programs 235 required for various processes in the arithmetic processing unit 240. The programs 235 are read into the arithmetic processing unit 240 and executed to realize various processing units. The programs 235 are read in advance from an external device or recording medium via a data input / output function such as the communication interface unit 220, and are stored in the storage unit 230. Main information stored in the storage unit 230 includes, for example, monitoring area information 231, image information 232, collected information 233, and detected information 234.

[0023] The monitoring area information 231 includes information indicating the monitoring target area, such as map information, and information indicating the flight route within the monitoring target area. The monitoring area information 231 may also include information indicating the priority of each region obtained by dividing the monitoring target area. At least a portion of the information included in the monitoring area information 231 is acquired in advance, for example, by receiving information from an external device via the communication interface unit 220, and is stored in the storage unit 230. Furthermore, the information included in the monitoring area information 231 indicating the flight route is updated, for example, when the flight route setting unit 248 (described later) sets a flight route. Furthermore, the priority of each region included in the monitoring area information 231 is updated, for example, when the priority determination unit 244 determines the priority.

[0024] The image information 232 includes image data acquired by the imaging section 210. For example, the image information 232 includes time-series image data acquired by the imaging section 210. The image information 232 is updated in response to the imaging section 210 acquiring image data, etc.

[0025] The collected information 233 includes information collected by the unmanned aerial vehicle 200. For example, the collected information 233 includes at least a portion of various information according to the characteristics of the area to be monitored, such as information indicating the speed and speed changes of traveling vehicles, information indicating the time when it is determined that a traffic jam has occurred and the scale of the traffic jam, information indicating the occurrence of an accident, information indicating road conditions such as uphill slopes and sharp curves, and any other information indicating defects that have occurred in the road, such as potholes. The collected information 233 is updated as the information collection unit 243 collects information according to image data, etc.

[0026] The detection information 234 includes information about foreign matter detected from the image data, and is updated in response to the detection unit 245 detecting a foreign matter based on the image data.

[0027] FIG. 3 shows an example of detection information 234. Referring to FIG. 3, detection information 234 associates detected object information with location information. Here, detected object information includes an identifier of the detected foreign object, etc. The detected object information may also include information indicating the type of foreign object, etc. Location information is information indicating the location of the detected foreign object, such as latitude and longitude. The location of the foreign object may be identified based on the location of unmanned aerial vehicle 200 when the image data was acquired, the location of the foreign object on the image data, monitored area information 231, etc.

[0028] The arithmetic processing unit 240 has an arithmetic device such as a CPU (Central Processing Unit) and its peripheral circuits. The arithmetic processing unit 240 reads and executes a program 235 from the storage unit 230, thereby causing the above hardware and the program 235 to cooperate to realize various processing units. Major processing units realized by the arithmetic processing unit 240 include, for example, a flight control unit 241, an image data acquisition unit 242, an information collection unit 243, a priority determination unit 244, a detection unit 245, a notification unit 246, a recovery instruction unit 247, and a flight path setting unit 248.

[0029] In addition, the arithmetic processing unit 240 may have a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an MPU (Micro Processing Unit), an FPU (Floating point number Processing Unit), a PPU (Physics Processing Unit), a TPU (Tensor Processing Unit), a quantum processor, a microcontroller, or a combination thereof, instead of the above-mentioned CPU.

[0030] The flight control unit 241 controls the unmanned aerial vehicle 200 so that the unmanned aerial vehicle 200 flies along the flight path indicated by the monitoring area information 231. The flight control unit 241 may perform the above control using any method. For example, the flight control unit 241 can control the flight direction of the unmanned aerial vehicle 200 so that the unmanned aerial vehicle 200 moves along the flight path indicated by the monitoring area information 231 while confirming the position of the unmanned aerial vehicle using a GPS sensor or the like.

[0031] The image data acquisition unit 242 acquires image data using the imaging unit 210. For example, the image data acquisition unit 242 acquires time-series image data at predetermined intervals while the unmanned aerial vehicle 200 flies along the flight path. The image data acquisition unit 242 also stores the acquired image data in the storage unit 230 as image information 232.

[0032] The information collection unit 243 collects predetermined information about the area to be monitored using the acquired image data and other sensors of the unmanned aerial vehicle 210. For example, the information collection unit 243 collects information while the unmanned aerial vehicle 200 flies along the flight path. The information collection unit 243 may collect information using image data acquired in advance at timings other than those exemplified above.

[0033] For example, the information collection unit 243 collects various information necessary to determine the magnitude of the impact and danger of a foreign object in the monitored area. The information collection unit 243 may collect the various information in accordance with the characteristics of the monitored area. For example, if the monitored area is a highway, the presence of a foreign object will have a significant impact on vehicles traveling on the road. Furthermore, the impact of a foreign object will be greater if there is heavy traffic, few lanes, or no bypasses. Therefore, the information collection unit 243 collects information indicating the speed and speed changes of traveling vehicles, information indicating traffic volume per unit time, information indicating the time when a congestion is determined to have occurred and the scale of the congestion, information indicating the occurrence of an accident, information indicating road conditions such as uphill slopes and sharp curves, information indicating the number of passable lanes, and any other information. The information collection unit 243 also stores the collected information in the storage unit 230 as collected information 233.

[0034] The information collection unit 243 may collect the information using any means. For example, the information collection unit 243 may calculate the speed of vehicles in each predetermined section by detecting and tracking vehicles from time-series image data, or may calculate information indicating traffic volume by counting the number of vehicles using the image data. The information collection unit 243 may also measure the number of lanes by referring to the monitoring area information 231, or may measure the number of usable lanes by checking lanes that are unavailable due to construction, accidents, or other reasons using the image data. The information collection unit 243 may also detect the occurrence of an accident or other events based on the image data. The information collection unit 243 may collect various information by inputting the image data into a pre-trained model, or other events. In addition to using image data, the information collection unit 243 may collect various information, such as those described above, that can determine the magnitude of the impact of a foreign object, using various sensors included in the unmanned aerial vehicle 200. For example, the information collection unit 243 may collect various information using at least some of the above-mentioned methods or methods other than those mentioned above.

[0035] The priority determination unit 244 determines the priority for collecting foreign objects using the monitoring area information 231, the collected information 233, etc. Furthermore, the priority determination unit 244 can divide the monitoring target area into multiple regions using the monitoring area information 231, the collected information 233, etc. The priority determination unit 244 may determine the priority at any timing, such as at a predetermined interval, or when new information is collected by the information collection unit 243, such as when a decrease in lanes is confirmed. Furthermore, the timing at which the priority determination unit 244 divides the regions may be the same as or different from the timing at which the priority determination is made.

[0036] For example, the priority determination unit 244 can determine the priority by checking the impact that the presence of a foreign object has on vehicles traveling on the road, based on the collected information 233, etc. Furthermore, the priority determination unit 244 can divide the area to be monitored into multiple regions in accordance with the above-mentioned checking.

[0037] For example, the impact of foreign objects on vehicles traveling on a road varies significantly depending on the number of lanes, such as the number of lanes. Therefore, as shown in FIG. 4, the priority determination unit 244 can divide the monitoring area into multiple regions depending on the number of available lanes, including temporary lanes due to construction, accidents, etc. For example, the priority determination unit 244 may divide the monitoring area into multiple regions so that each region has a different number of available lanes. Furthermore, the priority determination unit 244 can determine the priority based on the number of lanes, such as by assigning a higher priority value to an area with fewer lanes. Naturally, the priority determination unit 244 may assign a lower priority value to an area with more lanes. Furthermore, the priority determination unit 244 may assign a higher priority value when the number of lanes is less than a certain threshold. Note that such a threshold may be specified by a user of the monitoring system 100 via various input means, or may be calculated based on various information acquired by the monitoring system 100.

[0038] Furthermore, the impact of foreign objects is greater in areas where the vehicle speed or average speed is higher or the traffic volume is heavier. Therefore, the priority determination unit 244 can divide the monitoring area into multiple regions based on the vehicle speed, traffic volume, and the like. For example, the priority determination unit 244 can divide the monitoring area into multiple regions by checking to which of multiple predetermined numerical ranges the vehicle speed, etc., belongs. In other words, the priority determination unit 244 may divide the monitoring area based on locations where the average vehicle speed changes. Furthermore, the priority determination unit 244 can determine the priority based on the speed or traffic volume, such that the faster the vehicle speed or the heavier the traffic volume, the higher the priority value. Note that checking whether a value belongs to one of multiple numerical ranges may involve determining only whether the value falls within a certain range, or may involve determining whether the value falls within one of multiple ranges.

[0039] Furthermore, on expressways and the like, the impact of foreign objects on the shoulders of roads varies depending on the location, such as near the center of the road on which vehicles are traveling, near the median strip, or near the shoulders, with the impact being smaller. Therefore, the priority determination unit 244 may refer to the monitoring area information 231 or the like to divide the monitoring target area according to the portions corresponding to the shoulders of roads, and determine the priorities. For example, the priority determination unit 244 may determine the priorities so that the portions corresponding to the shoulders of roads are given a lower priority. In addition to the above examples, the priority determination unit 244 may determine the priorities according to the width of the lanes, etc.

[0040] For example, the priority determination unit 244 can divide the area to be monitored and determine the priority using any one of the methods exemplified above or a combination thereof. The priority determination unit 244 may also divide the area to be monitored and determine the priority using collected information 233 other than those exemplified above. For example, if a foreign object is present at the edge of a bridge or the like, there is a risk that the foreign object may fall. Therefore, the priority determination unit 244 may divide the edge of a predetermined location such as a bridge into a separate area and determine the priority. Furthermore, in addition to the methods exemplified above, the priority determination unit 244 may divide the area to be monitored and determine the priority for each area in response to inputting information indicating the area to be monitored, collected information 233, etc. to a pre-trained model.

[0041] The detection unit 245 uses image data to detect foreign objects that are not normally present in the monitored area. The detection unit 245 may detect foreign objects and estimate the type of foreign object. For example, the detection unit 245 may detect foreign objects and estimate the type of foreign object by inputting image data into an object detection model such as YOLO (You Only Look Once). As an example, the detection unit 245 may detect objects that are not normally present on the road, such as objects that have fallen from a vehicle, natural objects such as fallen trees and branches, and other arbitrary debris, as foreign objects. The detection unit 245 may also store the detection results and the like in the storage unit 230 as detection information 234.

[0042] Furthermore, the detection unit 245 may be configured to detect a foreign object and also estimate the position of the detected foreign object. For example, the detection unit 245 may estimate the position of the detected foreign object by referring to the position of the unmanned aerial vehicle 200 when the image data was acquired, the position of the foreign object in the image data, map information indicated by the monitored area information 231, or other arbitrary information. The detection unit 245 may estimate the position of the foreign object using a method other than those exemplified above, such as using a model learned in advance.

[0043] The notification unit 246 notifies the management device 400 of at least a portion of the various types of information included in the collected information 233, the detected information 234, etc. The notification unit 246 may make the notification at any timing.

[0044] For example, in response to the detection of a foreign object by the detection unit 245, the notification unit 246 notifies the management device 400 of the detection of a foreign object, information indicating the type of the detected foreign object, information indicating the location of the foreign object, and the like. The notification unit 246 may notify the management device 400 of the detection of a foreign object, as well as information regarding the area in which the foreign object was detected and the priority level, etc. Furthermore, the notification unit 246 can notify the management device 400 of information included in the collected information 233 at a predetermined timing, such as a predetermined timing. The notification unit 246 may be configured to notify the management device 400 of information that is predefined to have a major impact, such as a reduction in passable lanes due to construction, an accident, or the like, when the information collection unit 243 collects the information.

[0045] In response to detection of a foreign object by the detection unit 245, the collection instruction unit 247 instructs the collection device 300 to collect the detected foreign object. At this time, the collection instruction unit 247 can instruct the collection device 300 to collect the foreign object according to the priority determined for the area where the foreign object is present. Note that the collection instruction unit 247 may issue any instruction to the collection device 300 required to collect the foreign object, such as an instruction including information indicating the location where the foreign object is present, such as latitude and longitude.

[0046] For example, the collection instruction unit 247 can instruct the collection device 300 to collect foreign objects that exist in areas with higher priority first. Furthermore, if the detection unit 245 detects a foreign object that exists in an area with higher priority while the collection device 300 is heading to collect the foreign object, the collection instruction unit 247 may again instruct the collection device 300 that is heading to collect it to collect the foreign object that exists in the area with higher priority before the foreign object that is heading to be collected.

[0047] 5, for example, foreign matter 1 is present in an area with a higher priority than foreign matter 2. Therefore, the collection instruction unit 247 can instruct the collection device 300 to collect foreign matter 1 and foreign matter 2 so that foreign matter 2 is collected after foreign matter 1 is collected.

[0048] The monitoring system 100 may include different types of collection devices 300, such as drones, vehicles, or any other robots. The collection instruction unit 247 may select a device to instruct to collect the foreign object based on the type of foreign object estimated by the detection unit 245, and then instruct the selected device to collect the foreign object based on a priority. For example, the collection instruction unit 247 may be configured to select a device to instruct to collect the foreign object based on the weight or danger level of the foreign object estimated based on the estimated type of foreign object.

[0049] The collection instruction unit 247 may also be configured to modify the priority according to the area where the foreign object exists according to the type of foreign object, and then issue an instruction to the collection device 300. For example, the collection instruction unit 247 may specify a priority according to the area where the foreign object exists, and then correct the specified priority according to the type of foreign object, such as a dangerous object, and then instruct the collection device 300 to perform collection according to the corrected priority.

[0050] The flight path setting unit 248 sets a flight path in accordance with the detection information 234, the collection information 233, etc. The flight path setting unit 248 can set a flight path when a predetermined condition is met, such as at night or when flight along the flight path is not being performed. The conditions under which the flight path setting unit 248 sets a flight path may be determined arbitrarily.

[0051] 6, the flight path setting unit 248 can set a flight path so that the unmanned aerial vehicle 200 flies over the location where a foreign object has been detected by the detection unit 245. The flight path setting unit 248 may be configured to identify locations where foreign object detections are frequent or locations where there is a high possibility of foreign object detection, based on the detection information 234, etc., and then set a flight path so that the unmanned aerial vehicle 200 flies over the identified locations.

[0052] Furthermore, the flight path setting unit 248 may set a flight path to increase the monitoring time at the location where a foreign object has been detected, such as by circling over the location where a foreign object has been detected by the detection unit 245. As in the case described above, the flight path setting unit 248 may identify locations where foreign object detection has occurred frequently or locations where the possibility of foreign object detection is high, based on the detection information 234, and then set a flight path to increase the monitoring time above the identified locations.

[0053] For example, the flight path setting unit 248 can set a flight path according to the foreign object detection results using any one or a combination of the above-exemplified methods. The flight path setting unit 248 may set a flight path by referring to the collected information 233, the determination results by the priority determination unit 244, and the like, in addition to the detection information 234. For example, the flight path setting unit 248 may set a flight path so as to increase the monitoring time in areas with higher priority, such as areas with fewer lanes or faster speeds. The flight path setting unit 248 may also set various setting values ​​in the image capture unit 210 in addition to setting a flight path, such as by adjusting the angle of view or zoom function above the foreign object detection position to acquire more detailed image data. For example, the flight path setting unit 248 may associate a position on the flight path with a change amount of a setting value, etc. In addition to the above-exemplified methods, the flight path setting unit 248 may be configured to set a flight path in response to inputting the detection information 234, etc., into a pre-trained model.

[0054] The above is an example of the configuration of unmanned aerial vehicle 200.

[0055] The collection device 300 is a device that collects the specified foreign object in response to an instruction from the unmanned aerial vehicle 200. The collection device 300 may be any device capable of collecting foreign objects, such as a drone, a vehicle, or any other robot.

[0056] For example, the collection device 300 waits at a predetermined location until it receives an instruction from the unmanned aerial vehicle 200. Furthermore, the collection device 300 heads to collect the foreign object in response to an instruction from the unmanned aerial vehicle 200. At this time, the collection device 300 collects the foreign object according to the priority instructed by the unmanned aerial vehicle 200. Furthermore, once the collection device 300 has collected the foreign object, it can transport the collected foreign object to a predetermined location. Thereafter, the collection device 300 may transmit information indicating that collection has been completed to the unmanned aerial vehicle 200, the management device 400, or the like.

[0057] In the present disclosure, there are no particular limitations on the configuration of the recovery device 300. The recovery device 300 may recover foreign objects using any means, such as a robot arm.

[0058] Management device 400 is an information processing device that receives notifications from unmanned aerial vehicle 200. Management device 400 may be configured to determine countermeasures in response to the received notifications.

[0059] Fig. 8 shows an example of the configuration of the management device 400. Referring to Fig. 8, the management device 400 has, as main components, for example, an operation input unit 410, a screen display unit 420, a communication interface unit 430, a storage unit 440, and an arithmetic processing unit 450.

[0060] 8 illustrates an example in which the functions of management device 400 are realized using one information processing device. However, at least some of the functions of management device 400 may be realized using multiple information processing devices, for example, on the cloud. Furthermore, management device 400 may not include some of the components illustrated above, such as not having operation input unit 410 or screen display unit 420, or may have a component other than those illustrated above.

[0061] The operation input unit 410 is made up of operation input devices such as a keyboard, a mouse, etc. The operation input unit 410 detects operations by an operator who operates the management device 400 and outputs the operations to the arithmetic processing unit 450.

[0062] The screen display unit 420 is composed of a screen display device such as a liquid crystal display, an organic EL (electro-luminescence) display, etc. The screen display unit 420 can display various information stored in the storage unit 440 on the screen in response to an instruction from the arithmetic processing unit 450.

[0063] Communication interface unit 430 is composed of a data communication circuit etc. Communication interface unit 430 performs data communication with external devices such as unmanned aerial vehicle 200.

[0064] The storage unit 440 is a storage device such as a hard disk or memory. The storage unit 440 stores processing information and a program 442 required for various processes in the arithmetic processing unit 450. The program 442 is read into the arithmetic processing unit 450 and executed to realize various processing units. The program 442 is read in advance from an external device or recording medium via a data input / output function such as the communication interface unit 430, and is stored in the storage unit 440. Main information stored in the storage unit 440 includes, for example, received information 441.

[0065] Received information 441 includes information notified from unmanned aerial vehicle 200. For example, received information 441 may include at least a portion of collected information 233, detected information 234, etc. in unmanned aerial vehicle 200. Received information 441 is updated in response to receiving unit 451 receiving a notification from unmanned aerial vehicle 200, etc.

[0066] The arithmetic processing unit 450 has an arithmetic device such as a CPU and its peripheral circuits. The arithmetic processing unit 450 reads and executes a program 442 from the storage unit 440, thereby causing the above hardware and the program 442 to work together to realize various processing units. Main processing units realized by the arithmetic processing unit 450 include, for example, a receiving unit 451, a countermeasure proposing unit 452, and an output unit 453. Note that the arithmetic processing unit 450 may have the above-mentioned GPU or the like instead of the above-mentioned CPU.

[0067] Receiving unit 451 receives a notification from unmanned aerial vehicle 200. Furthermore, receiving unit 451 can store information included in the received notification in storage unit 440 as received information 441.

[0068] The countermeasure proposal unit 452 determines and proposes necessary countermeasures using at least a portion of the information included in the received information 441. For example, when the unmanned aerial vehicle 200 detects a dangerous object as a foreign body, the countermeasure proposal unit 452 can propose countermeasures such as setting up a no-entry area, or when it is determined that retrieval by the retrieval device 300 is difficult due to the weight, size, etc., sending a person to retrieve the object or dispatching a person to the accident site, etc.

[0069] For example, the countermeasure proposing unit 452 can be configured to store proposals in advance for each type of information that may be collected or each type of foreign object to be detected, and to propose a specific countermeasure in accordance with the information included in the received information 441. The countermeasure proposing unit 452 may also be configured to propose a countermeasure using a pre-trained model such as an LLM (Large Language Model). For example, the countermeasure proposing unit 452 may acquire information indicating a countermeasure in accordance with inputting at least a portion of the information included in the received information 441, such as information on the type of foreign object or information on the occurrence of an accident, into a trained model.

[0070] The output unit 453 outputs information indicating the countermeasures proposed by the countermeasure proposal unit 452. For example, the output unit 453 can display the information indicating the countermeasures on the screen display unit 420 or transmit it to an external device via the communication interface unit 430. In addition, the output unit 453 may display various notifications required when implementing the countermeasures on the screen display unit 420 or transmit them to an external device.

[0071] In addition to the information exemplified above, the output unit 453 may output at least a part of the information included in the received information 441. For example, the output unit 453 may be configured to output related information included in the received information 441 together with information indicating the countermeasure proposed by the countermeasure proposal unit 452.

[0072] The above is an example configuration of monitoring system 100. Note that monitoring system 100 may have a configuration other than that illustrated. For example, among the functions possessed by unmanned aerial vehicle 200 described with reference to FIG. 2 , at least some of the processes, such as collection of information according to image data by information collection unit 243, determination of priority by priority determination unit 244, collection instruction by collection instruction unit 247, and setting of a flight path by flight path setting unit 248, may be executed on management device 400 instead of unmanned aerial vehicle 200.

[0073] Next, an example of the operation of the monitoring system 100 will be described with reference to Fig. 9 to Fig. 11. First, an example of the operation of the monitoring system 100 will be described with reference to Fig. 9. Note that the processing shown in Fig. 9 may be started at any timing.

[0074] Fig. 9 is a sequence diagram showing an example of the operation of monitoring system 100. Referring to Fig. 9, flight control unit 241 starts flying unmanned aerial vehicle 200 along the flight path indicated by monitoring area information 231. Furthermore, while unmanned aerial vehicle 200 flies along the flight path, image data acquisition unit 242 acquires image data. Furthermore, information collection unit 243 collects predetermined information in the monitoring area (step S101).

[0075] The detection unit 245 uses the image data to detect a foreign object that is not normally present in the monitored area (step S102). If a foreign object is detected (step S102, Yes), the notification unit 246 notifies the management device 400 that a foreign object has been detected (step S103).

[0076] In response to the notification in step S103, the countermeasure proposing unit 452 of the management device 400 can determine and propose necessary countermeasures (step S201). For example, the countermeasure proposing unit 452 may propose countermeasures by inputting the received notification into a trained model.

[0077] Furthermore, if a foreign object is detected in the processing of step S102, the collection instruction unit 247 instructs the collection device 300 to collect the detected foreign object (step S104). At this time, the collection instruction unit 247 can instruct the collection device 300 to collect the foreign object according to the priority.

[0078] In response to the instruction in step S104, collection device 300 collects the instructed foreign object (step S301). At this time, collection device 300 can collect the foreign object according to the priority. After collecting the foreign object by the processing of step S301, collection device 300 may transmit a notification that collection has been completed to at least one of unmanned aerial vehicle 200 and management device 400.

[0079] The process of step S103 and the process of step S104 may be performed in any order, or may be performed in parallel.

[0080] If no foreign object is detected according to the image data (step S102, No), the processing of steps S103 and S104 is not performed. Furthermore, until flight along the flight path is completed (step S105, No), detection unit 245 continues to detect foreign objects according to the image data (step S102). On the other hand, after flight along the flight path is completed (step S105, Yes) and if a predetermined condition is met (step S106, Yes), flight path setting unit 248 sets a flight path (step S107). Furthermore, if the predetermined condition is not met (step S106, No) or after the processing of step S107, unmanned aerial vehicle 200 terminates processing.

[0081] The above is an example of the operation of monitoring system 100. Note that Fig. 9 shows an example of the operation of monitoring system 100, and the operation of monitoring system 100 is not limited to the example shown in Fig. 9. For example, unmanned aerial vehicle 200 does not have to proceed to the process of step S106 immediately after the process of step S105. For example, unmanned aerial vehicle 200 may perform the process of step S106 at any timing after completing the flight.

[0082] Next, the process of step S107 will be described in more detail with reference to FIG.

[0083] 10 is a flowchart showing an example of the operation of unmanned aerial vehicle 200 when setting a flight path. Referring to Fig. 10, flight path setting unit 248 refers to detection information 234 and the like (step S111). Flight path setting unit 248 may refer to collection information 233 and the like in addition to detection information 234.

[0084] The flight path setting unit 248 sets a flight path in accordance with the position where the foreign object was detected, as indicated by the detection information 234 (step S112). For example, the flight path setting unit 248 can set a flight path so that the aircraft flies over the area where the foreign object was detected, or can set a flight path so that the aircraft stays in the area where the foreign object was detected for an extended period of time.

[0085] The above is an example of the operation of unmanned aerial vehicle 200 when setting a flight path. Next, an example of the operation of unmanned aerial vehicle 200 when determining priority will be described with reference to Fig. 11 .

[0086] Fig. 11 shows an example of the operation of unmanned aerial vehicle 200 when determining priority. Referring to Fig. 11, when a predetermined condition such as a predetermined timing is met (step S121, Yes), priority determination unit 244 determines the priority for collecting the foreign object using collected information 233 and the like (step S122). Furthermore, priority determination unit 244 can divide the area to be monitored into multiple regions using collected information 233 and the like.

[0087] The above is an example of the operation of the priority determination unit 244. The processing shown in Fig. 11 may be performed at any timing depending on the conditions. For example, the priority determination unit 244 can perform priority determination at any timing depending on the conditions during flight along the flight path in the processing exemplified in Fig. 9. Furthermore, the priority determination unit 244 may perform priority determination at any timing depending on the conditions after flight along the flight path has ended.

[0088] Thus, unmanned aerial vehicle 200 has priority determination unit 244 and collection instruction unit 247. With this configuration, collection instruction unit 247 can, in response to the detection of a foreign object, instruct the collection of the foreign object according to the priority determined by priority determination unit 244. As a result, it is possible to achieve more appropriate collection of the foreign object according to the magnitude of the impact, danger, etc.

[0089] Furthermore, unmanned aerial vehicle 200 has information collection unit 243. With this configuration, priority determination unit 244 can use the information collected by information collection unit 243 to divide the area to be monitored into multiple regions and determine the priority of each division. As a result, priority can be determined more appropriately. This allows for more appropriate collection of foreign objects.

[0090] Furthermore, unmanned aerial vehicle 200 has a flight path setting unit 248. With this configuration, flight path setting unit 248 can set a flight path according to the results of foreign object detection, etc. As a result, a more appropriate flight path can be set, and more appropriate monitoring can be achieved.

[0091] In the present disclosure, an example has been given in which monitoring system 100 includes unmanned aerial vehicle 200 as a monitoring device. However, monitoring system 100 may include monitoring devices other than unmanned aerial vehicle 200, such as a monitoring camera, any other radar device, or an autonomous vehicle. Furthermore, the monitoring device may not only function as an imaging device itself, but may also be configured to detect foreign objects based on image data acquired from an external imaging device.

[0092] As described above, the area to be monitored may be along a coastline, a border, or any other location. When the area to be monitored is along a coastline or a border, the priority determination unit 244 may divide the area to be monitored and determine the priority according to the characteristics of each area to be monitored. For example, the priority determination unit 244 may set a region along the coastline or set a higher priority if there is a risk of foreign objects being washed away along the coastline. Furthermore, a higher priority may be set for an area with thick trees, such as along a border. The priority determination unit 244 may perform any determination according to the characteristics of the area to be monitored, in addition to the above examples.

[0093] [Second embodiment] Next, a second embodiment of the present disclosure will be described with reference to Fig. 12 to Fig. 14. Fig. 12 is a diagram illustrating an example of the hardware configuration of a monitoring device 500. Fig. 13 is a block diagram illustrating an example of the configuration of the monitoring device 500. Fig. 14 is a flowchart illustrating an example of the operation of the monitoring device 500.

[0094] In the second embodiment of the present disclosure, a monitoring device 500 that is an information processing device that detects foreign matter according to image data will be described. Fig. 12 shows an example of the hardware configuration of the monitoring device 500. Referring to Fig. 12, the monitoring device 500 has, as an example, the following hardware configuration. ·CPU (Central Processing Unit) 501 (computing unit) ROM (Read Only Memory) 502 (storage device) RAM (Random Access Memory) 503 (storage device) Programs 504 loaded into RAM 503 A storage device 505 for storing a group of programs 504 A drive device 506 that reads and writes data from a recording medium 510 outside the information processing device A communication interface 507 for connecting to a communication network 511 outside the information processing device Input / output interface 508 for inputting and outputting data Bus 509 connecting each component

[0095] 13. The monitoring device 500 can realize the functions of a detection unit 521 and a collection instruction unit 522 shown in Fig. 13 by having the CPU 501 acquire and execute the program group 504. The program group 504 is stored in advance in, for example, the storage device 505 or the ROM 502, and is loaded into the RAM 503 or the like by the CPU 501 for execution as needed. The program group 504 may be supplied to the CPU 501 via the communication network 511, or may be stored in advance in the recording medium 510, and the drive device 506 may read out the programs and supply them to the CPU 501.

[0096] 12 shows an example of the hardware configuration of the monitoring device 500. The hardware configuration of the monitoring device 500 is not limited to the above-described case. For example, the monitoring device 500 may be configured with only a part of the above-described configuration, such as excluding the drive device 506. Furthermore, the CPU 501 may be a GPU or the like exemplified in the first embodiment.

[0097] The detection unit 521 detects a foreign object based on image data of the monitored area as a subject. The detection unit 521 may detect a foreign object using any means, such as using a pre-trained object detection model.

[0098] The collection instruction unit 522 instructs an external device to collect the foreign matter detected by the detection unit 521. For example, the collection instruction unit 522 instructs an external device to collect the foreign matter according to a priority determined in advance for an area in which a foreign matter exists among the divided areas of the monitoring target area.

[0099] The above is an example of the configuration of the monitoring device 50. Next, an example of the operation of the monitoring device 500 will be described with reference to FIG.

[0100] Fig. 14 is a flowchart showing an example of the operation of monitoring device 500. Referring to Fig. 14, detection unit 521 detects a foreign object based on image data of a monitoring target area (step S401).

[0101] The collection instruction unit 522 instructs an external device to collect the foreign matter detected by the detection unit 521 (step S402). For example, the collection instruction unit 522 instructs an external device to collect the foreign matter according to a predetermined priority for an area in which a foreign matter exists, among the areas into which the monitoring target area is divided.

[0102] The above is an example of the operation of the monitoring device 500.

[0103] As described above, the monitoring device 500 includes the detection unit 521 and the collection instruction unit 522. With this configuration, the collection instruction unit 522 can instruct an external device to collect foreign objects according to a predetermined priority for an area in which a foreign object is present among the divided regions of the monitoring target area. As a result, foreign objects can be collected appropriately according to the priority.

[0104] The monitoring device 500 described above can be realized by incorporating a predetermined program into an information processing device such as the monitoring device 500. Specifically, a program according to another aspect of the present invention is a program for causing an information processing device such as the monitoring device 50 to perform the following process: detect foreign matter based on image data of a monitored area as a subject; instruct an external device to collect the detected foreign matter; and, when instructing the external device to collect the foreign matter, instruct the external device to collect the foreign matter according to a predetermined priority for an area in which the monitored area is divided and in which the foreign matter is present.

[0105] Furthermore, a monitoring method executed by an information processing device such as the above-mentioned monitoring device 500 is a method in which the information processing device detects foreign objects based on image data of a monitored area, instructs an external device to collect the detected foreign objects, and when instructing the external device to collect the foreign objects, instructs the external device to collect the foreign objects according to a predetermined priority for an area in which a foreign object is present among the areas into which the monitored area is divided.

[0106] Even if the invention is a program having the above-described configuration, or a computer-readable recording medium having the program recorded thereon, or a monitoring method, it can achieve the same functions and effects as the above-described monitoring device 500, and therefore can achieve the above-described objective of the present disclosure.

[0107] <Additional Notes> A part or all of the above-described embodiments can be described as follows: The following provides an overview of the monitoring device and the like according to the present invention. However, the present invention is not limited to the following configuration.

[0108] (Appendix 1) a detection unit that detects foreign objects based on image data of a monitored area; a collection instruction unit that instructs an external device to collect the foreign matter detected by the detection unit; and The collection instruction unit instructs the external device to collect the foreign matter in accordance with a priority determined in advance for an area where the foreign matter exists among the divided areas of the monitoring area. monitoring equipment. (Appendix 2) a determination unit that determines a priority for collecting detected foreign matter in each of the divided areas of the monitoring area in accordance with at least one of pre-stored information and information collected in the monitoring area, The collection instruction unit instructs the external device to collect the foreign matter in accordance with the priority determined by the determination unit for the area where the foreign matter exists. 10. The monitoring device described in Appendix 1. (Appendix 3) The determining unit determines the priority by checking the impact that the presence of a foreign object has on the area to be monitored based on the collected information. 10. The monitoring device described in Appendix 2. (Appendix 4) The determination unit divides the area to be monitored into a plurality of regions according to the collected information and determines the priority of each divided region. 10. The monitoring device of claim 2 or 3. (Appendix 5) The surveillance device is an unmanned aerial vehicle, The determination unit determines the priority of collecting detected foreign objects in each of the divided areas of the monitoring area, based on information collected while the unmanned aerial vehicle is flying along a predetermined flight path. 10. The monitoring device according to claim 2, wherein the monitoring device is a (Appendix 6) the detection unit is configured to detect a foreign object and a type of the foreign object; The collection instruction unit corrects the priority determined by the determination unit in accordance with the type of foreign matter, and then instructs the external device to collect the foreign matter in accordance with the corrected priority. 10. The monitoring device according to any one of claims 2 to 5. (Appendix 7) The collection instruction unit instructs an external device to collect foreign objects present in an area with a higher priority first when the detection unit detects a plurality of foreign objects. 10. The monitoring device according to claim 1, wherein the monitoring device is a (Appendix 8) The surveillance device is an unmanned aerial vehicle, a setting unit that sets a flight path when monitoring the target area in accordance with the detection result by the detection unit; 10. The monitoring device according to any one of claims 1 to 7. (Appendix 9) The information processing device Detects foreign objects based on image data of the monitored area, Instructing an external device to collect the detected foreign matter, When issuing an instruction to collect foreign matter, the external device is instructed to collect foreign matter according to a priority determined in advance for an area where foreign matter exists among the divided areas of the monitoring area. Monitoring method. (Appendix 10) In the information processing device, Detects foreign objects based on image data of the monitored area, Instructing an external device to collect the detected foreign matter, When issuing an instruction to collect foreign matter, the external device is instructed to collect foreign matter according to a priority determined in advance for an area where foreign matter exists among the divided areas of the monitoring area. A program to realize the processing.

[0109] Note that some or all of the configurations described in Supplementary Notes 2 to 8 that are dependent on the monitoring device described in Supplementary Note 1 may also be dependent in a similar dependent relationship on the monitoring method described in Supplementary Note 9 and the program described in Supplementary Note 10. Furthermore, not limited to Supplementary Notes 9 and 10, some or all of the configurations described as Supplements may also be dependent on various hardware, software, various recording means for recording software, or systems within the scope of the above-mentioned embodiments.

[0110] The programs described in the above embodiments and appendices may be stored in a storage device or a computer-readable recording medium, such as a portable medium such as a flexible disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0111] Although the present invention has been described above with reference to the above-mentioned embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. [Explanation of symbols]

[0112] 100 Surveillance System 200 unmanned aircraft 210 Imaging unit 220 Communication interface section 230 Storage section 231 Monitoring Area Information 232 Image Information 233 Collected Information 234 Detection Information 235 Programs 240 Processing Unit 241 Flight Control Unit 242 Image data acquisition unit 243 Information Gathering Department 244 Priority judgment section 245 Detector 246 Notification Department 247 Collection Instructions Section 248 Flight Path Setting Unit 300 Recovery Device 400 Management device 410 Operation input unit 420 Screen display section 430 Communication Interface Unit 440 Storage section 441 Received Information 442 Programs 450 Processing Unit 451 Receiving unit 452 Countermeasures Proposal Department 453 Output Unit 500 monitoring equipment 501 CPU 502 ROM 503 RAM 504 Programs 505 Storage device 506 Drive unit 507 Communication Interface 508 Input / Output Interface 509 Bus 510 Recording media 511 Communication Network 521 Detector 522 Collection Instructions Section

Claims

1. a detection unit that detects foreign objects based on image data of a monitored area; a collection instruction unit that instructs an external device to collect the foreign matter detected by the detection unit; and The collection instruction unit instructs the external device to collect the foreign matter in accordance with a priority determined in advance for an area where the foreign matter exists among the divided areas of the monitoring area. Monitoring equipment.

2. a determination unit that determines a priority for collecting detected foreign matter in each of the divided areas of the monitoring area in accordance with at least one of pre-stored information and information collected in the monitoring area, The collection instruction unit instructs the external device to collect the foreign matter in accordance with the priority determined by the determination unit for the area where the foreign matter exists. The monitoring device of claim 1 .

3. The determining unit determines the priority by checking the impact that the presence of a foreign object has on the area to be monitored based on the collected information. The monitoring device according to claim 2 .

4. The determination unit divides the area to be monitored into a plurality of regions according to the collected information and determines the priority of each divided region. The monitoring device according to claim 2 .

5. The surveillance device is an unmanned aerial vehicle, The determination unit determines the priority of collecting detected foreign objects in each of the divided areas of the monitoring area, based on information collected while the unmanned aerial vehicle is flying along a predetermined flight path. The monitoring device according to claim 2 .

6. the detection unit is configured to detect a foreign object and a type of the foreign object; The collection instruction unit corrects the priority determined by the determination unit in accordance with the type of foreign matter, and then instructs the external device to collect the foreign matter in accordance with the corrected priority. The monitoring device according to claim 2 .

7. The collection instruction unit instructs an external device to collect foreign objects present in an area with a higher priority first when the detection unit detects a plurality of foreign objects. The monitoring device of claim 1 .

8. The surveillance device is an unmanned aerial vehicle, a setting unit that sets a flight path when monitoring the target area in accordance with the detection result by the detection unit; The monitoring device of claim 1 .

9. The information processing device Detects foreign objects based on image data of the monitored area, Instructing an external device to collect the detected foreign matter, When issuing an instruction to collect foreign matter, the external device is instructed to collect foreign matter according to a priority determined in advance for an area where foreign matter exists among the divided areas of the monitoring area. Monitoring method.

10. In the information processing device, Detects foreign objects based on image data of the monitored area, Instructing an external device to collect the detected foreign matter, When issuing an instruction to collect foreign matter, the external device is instructed to collect foreign matter according to a priority determined in advance for an area where foreign matter exists among the divided areas of the monitoring area. A program to realize the processing.

Images