Flight path generation device, flight path generation method, and flight path generation program

Abstract

In the present invention, an imaging-region-specifying unit (22) specifies, as an imaging region, a region of an object being observed by a worker from line-of-sight information indicating the line of sight of the worker during visual confirmation work performed by the worker. An imaging-position-specifying unit (23) specifies an imaging position for imaging the imaging region identified by the imaging region identification unit (22). A flight path generation unit (24) uses the imaging position specified by the imaging-position-specifying unit (23) to generate a flight path for an unmanned aerial vehicle in order to image the imaging region using a camera mounted on the unmanned aerial vehicle.

Description

Flight path generation device, flight path generation method, and flight path generation program

[0001] The present disclosure relates to a technique for generating flight paths for unmanned aerial vehicles.

[0002] Equipment inspection work has traditionally been carried out by workers through visual inspection. In recent years, inspection work has sometimes been carried out using image data obtained by photographing the areas to be visually inspected with a camera. Inspection work using image data can also be achieved through computer image analysis, etc. By performing inspection work using image data, the burden of inspection work can be reduced. However, it can be difficult to install a camera at each inspection point. Furthermore, installing a camera at each inspection point increases costs. Therefore, it is considered to use an unmanned aerial vehicle equipped with a camera to photograph the inspection points and obtain image data of the inspection points.

[0003] The simplest method of using unmanned aerial vehicles to photograph inspection locations is for a pilot to fly the unmanned aerial vehicle and photograph each inspection location each time an inspection is carried out. However, it is time-consuming for a pilot to fly the unmanned aerial vehicle each time an inspection is carried out. Furthermore, depending on the location of the object being inspected, advanced piloting skills may be required, making it difficult to obtain appropriate image data. Therefore, it is possible to set a flight path for the unmanned aerial vehicle and automate the photographing of inspection locations. When automating the photographing of inspection locations, it is necessary to set an appropriate flight path that can photograph each inspection location.

[0004] Patent Literature 1 describes a method for setting a flight path for photographing designated photographing locations. In Patent Literature 1, a three-dimensional image of a structure is displayed, and a photographing target area on the structure and a photographing direction of a camera toward the photographing target area are displayed while being changed according to a user's instructions, and the photographing target area and photographing direction for each inspection location are specified. In Patent Literature 1, a flight path is generated that passes through positions specified from the photographing target area and photographing direction in order.

[0005] Japanese Patent Application Laid-Open No. 2020-149255

[0006] The method described in Patent Document 1 requires manual operation of a computer to set the area to be photographed and the direction of photography for each inspection location, which is time-consuming. The present disclosure aims to easily generate a flight path for an unmanned aerial vehicle to photograph locations that are being visually inspected.

[0007] The flight path generation device according to the present disclosure includes a photographing area identification unit that identifies an area of ​​an object that a worker is gazing at as a photographing area from gaze information indicating the line of sight of the worker during visual confirmation work; a photographing position identification unit that identifies a photographing position for photographing the photographing area identified by the photographing area identification unit; and a flight path generation unit that uses the photographing position identified by the photographing position identification unit to generate a flight path for the unmanned aircraft to photograph the photographing area with a camera mounted on the unmanned aircraft.

[0008] In the present disclosure, a photographing area is identified from line-of-sight information indicating the line of sight of a worker during visual confirmation work. Then, a photographing position is identified from the photographing area, and a flight path is generated using the photographing position. As a result, a flight path for photographing the area being visually confirmed is generated simply by the worker performing visual confirmation work. Therefore, a flight path can be generated easily.

[0009] 1 is a configuration diagram of a flight path generation device 10 according to a first embodiment. FIG. 2 is a flowchart showing the processing flow of the flight path generation device 10 according to the first embodiment. FIG. 3 is an explanatory diagram of gaze time identification processing according to the first embodiment. FIG. 4 is an explanatory diagram of shooting area identification processing according to the first embodiment. FIG. 5 is an explanatory diagram of shooting position identification processing according to the first embodiment. FIG. 6 is an explanatory diagram of flight path generation processing according to the first embodiment. FIG. 7 is a configuration diagram of a flight path generation device 10 according to a second embodiment. FIG. 8 is a flowchart showing the processing flow of the flight path generation device 10 according to the second embodiment. FIG. 9 is an explanatory diagram of clustering processing according to the second embodiment. FIG. 10 is an explanatory diagram of representative position identification processing according to the second embodiment. FIG. 11 is a flowchart showing the processing flow of the flight path generation device 10 according to a third embodiment. FIG. 12 is an explanatory diagram of clustering processing according to the second embodiment. FIG. 13 is an explanatory diagram of representative position identification processing according to the third embodiment. FIG. 14 is a configuration diagram of a flight path generation device 10 according to the third embodiment. FIG. 15 is a flowchart showing the processing flow of the flight path generation device 10 according to the third embodiment.

[0010] First Embodiment *** Description of Configuration *** The configuration of a flight path generation device 10 according to the first embodiment will be described with reference to Figure 1. The flight path generation device 10 is a computer. The flight path generation device 10 comprises the following hardware: a processor 11, a memory 12, a storage 13, and a communication interface 14. The processor 11 is connected to other hardware via signal lines and controls this other hardware.

[0011] The processor 11 is an IC that performs processing. IC stands for Integrated Circuit. Specific examples of the processor 11 include a CPU, a DSP, and a GPU. CPU stands for Central Processing Unit. DSP stands for Digital Signal Processor. GPU stands for Graphics Processing Unit.

[0012] The memory 12 is a storage device that temporarily stores data. Specific examples of the memory 12 include SRAM and DRAM. SRAM stands for Static Random Access Memory. DRAM stands for Dynamic Random Access Memory.

[0013] The storage 13 is a storage device that stores data. A specific example of the storage 13 is an HDD. HDD is an abbreviation for Hard Disk Drive. The storage 13 may also be a portable recording medium such as an SD (registered trademark) memory card, CompactFlash (registered trademark), NAND flash, a flexible disk, an optical disk, a compact disk, a Blu-ray (registered trademark) disk, or a DVD. SD is an abbreviation for Secure Digital. DVD is an abbreviation for Digital Versatile Disk.

[0014] The communication interface 14 is an interface for communicating with external devices. Specific examples of the communication interface 14 include Ethernet (registered trademark), USB, and HDMI (registered trademark) ports. USB stands for Universal Serial Bus. HDMI stands for High-Definition Multimedia Interface.

[0015] Flight path generation device 10 includes, as functional components, a gaze time identification unit 21, a shooting area identification unit 22, a shooting position identification unit 23, and a flight path generation unit 24. The functions of each functional component of flight path generation device 10 are realized by software. Storage 13 stores a program that realizes the function of each functional component of flight path generation device 10. This program is loaded into memory 12 by processor 11 and executed by processor 11. In this way, the function of each functional component of flight path generation device 10 is realized.

[0016] The storage 13 stores three-dimensional image data 31. The three-dimensional image data 31 is three-dimensional image data of an object 41 that is the target of visual inspection work such as inspection work. Here, the three-dimensional image data 31 is stored in the storage 13. However, the three-dimensional image data 31 may also be stored in a storage device external to the flight path generation device 10.

[0017] 1 shows only one processor 11. However, there may be multiple processors 11, and the multiple processors 11 may cooperate to execute programs that realize the respective functions.

[0018] ***Description of Operation*** The operation of flight path generation device 10 according to embodiment 1 will be described with reference to Figures 2 to 6. The operational procedure of flight path generation device 10 according to embodiment 1 corresponds to the flight path generation method according to embodiment 1. Furthermore, the program that realizes the operation of flight path generation device 10 according to embodiment 1 corresponds to the flight path generation program according to embodiment 1.

[0019] Referring to FIG. 2 , the processing flow of the flight path generation device 10 according to the first embodiment will be described. The processing shown in FIG. 2 is premised on a visual confirmation task performed by an operator. At this time, the operator performs the visual confirmation task while wearing a sensor device such as smart glasses or a HUD. HUD stands for Head-Up Display. The sensor device collects gaze information 32, which indicates the operator's gaze over time during the visual confirmation task. Specifically, the gaze information 32 is time-series data indicating the position of the operator's head relative to an object 41 and the orientation of the operator's head relative to the object 41 over time. The position of the operator's head is represented, for example, by three-dimensional coordinates of latitude, longitude, and altitude. The orientation of the operator's head is represented, for example, by a quaternion. Furthermore, the gaze information 32 may include information indicating the operator's gaze itself, instead of information indicating the orientation of the operator's head. If smart glasses or the like are equipped with a function for collecting information about the iris of the eye, the direction of the operator's gaze can be identified.

[0020] (Step S11: Gaze Time Identification Process) The gaze time identification unit 21 identifies a gaze time 42 from the gaze information 32. The gaze time 42 is the time during which the worker gazes at the object 41. Specifically, the gaze time 42 is the time during which the amount of change in the gaze indicated by the gaze information 32 is equal to or less than a threshold, and is equal to or greater than a reference period. This will be described in detail with reference to FIG. 3. FIG. 3 shows the change over time in the gaze movement speed. The gaze movement speed represents the amount of change in the gaze per unit time. Therefore, the gaze time identification unit 21 identifies the time during which the amount of time during which the gaze movement speed is equal to or less than a threshold, as the gaze time 42. In FIG. 3, two gaze times 42, a gaze time 42A and a gaze time 42B, are identified.

[0021] (Step S12: Photographing area identification process) The photographing area identification unit 22 identifies an area of ​​the object 41 that the worker gazes at as the photographing area 43. Specifically, the photographing area identification unit 22 identifies an area indicated by the line of sight during the gaze time 42 identified in step S11 as the photographing area 43. As a result, one or more photographing areas 43 are identified.

[0022] A specific description will be given with reference to FIG. 4 . First, the imaging area identification unit 22 generates mesh data 46 of the object 41 from the three-dimensional image data 31. The mesh data is data that represents the surface of the object 41 using a planar mesh such as a plurality of triangles. The mesh data 46 can be generated using point cloud data included in the three-dimensional image data 31. Next, the imaging area identification unit 22 identifies imaging target meshes 47 from the mesh data 46. Specifically, the imaging area identification unit 22 identifies, from among the meshes constituting the mesh data 46, meshes indicated by the line of sight during the gaze time 42 identified in step S1 as imaging target meshes 47. The mesh indicated by the line of sight means a mesh that intersects with the line of sight vector. The imaging area identification unit 22 sets each of the identified one or more imaging target meshes 47 as the imaging area 43. Here, multiple imaging target meshes 47 may be identified from one gaze time 42. The gaze time 42 is a time during which the amount of change in the line of sight is small, but the line of sight may change gradually. Therefore, in one observation time 42, a plurality of meshes may be observed, and each of the observed meshes may become a mesh 47 to be photographed.

[0023] (Step S13: Photographing position identification process) The photographing position identification unit 23 identifies a photographing position 44 for photographing the photographing area 43 identified in step S12. At this time, the photographing position identification unit 23 sets each photographing area 43 identified in step S12 as a target photographing area 43. Then, the photographing position identification unit 23 identifies a photographing position 44 for photographing the target photographing area 43. As a result, a photographing position 44 corresponding to each of the one or more photographing areas 43 identified in step S12 is identified.

[0024] A specific description will be given with reference to FIG. 5 . The photographing position identifying unit 23 sets each of the one or more photographing areas 43 identified in step S22 as the photographing area 43 of the target. In other words, each of the one or more photographing target meshes 47 is set as the photographing area 43 of the target. The photographing position identifying unit 23 identifies a photographing position 44 corresponding to the photographing area 43 of the target set in step S31. Specifically, the photographing position identifying unit 23 identifies, as the photographing position 44, a position that is a reference distance away in the normal direction from the photographing target mesh 47 identified as the photographing area 43. At this time, the photographing position identifying unit 23 identifies, as the photographing direction from the photographing position 44, the direction of the perpendicular line drawn from the photographing position 44 to the photographing target mesh 47. The photographing position identifying unit 23 associates the photographing position 44 with the photographing direction from the photographing position 44.

[0025] (Step S14: Flight path generation process) Using the photographing position 44 identified in step S13, the flight path generation unit 24 generates a flight path 45 for the unmanned aerial vehicle to photograph the photographing area 43 with a camera mounted on the unmanned aerial vehicle. Here, a specific example of the unmanned aerial vehicle is a drone.

[0026] A specific description will be given with reference to FIG. 6 . The flight path generation unit 24 generates a flight path 45 by sequentially connecting one or more photographing positions 44 identified in step S13. That is, the flight path 45 is information that sequentially connects the photographing positions 44. Here, each photographing position 44 is associated with a photographing direction. Therefore, the flight path 45 includes information indicating the photographing direction at each photographing position 44. In this case, the flight path generation unit 24 generates the flight path 45 by connecting the photographing positions 44 so as to minimize the path. Alternatively, the flight path generation unit 24 generates the flight path 45 by connecting the photographing positions 44 in the order in which they are gazed upon by the operator. This is not a limitation, and the flight path generation unit 24 may generate the flight path 45 by connecting the photographing positions 44 according to some rule. Note that if there is a waiting position for the unmanned aerial vehicle, the flight path generation unit 24 may complete the flight path 45 by connecting the waiting position with the first photographing position 44 on the flight path 45, and by connecting the last photographing position 44 on the flight path 45 with the waiting position.

[0027] ***Effects of First Embodiment*** As described above, flight path generation device 10 according to the first embodiment identifies photographing area 43 from line-of-sight information 32 that indicates the line of sight of the worker during visual confirmation work. Flight path generation device 10 then identifies photographing position 44 from photographing area 43, and generates flight path 45 using photographing position 44. In this way, flight path 45 is generated for photographing the area being visually confirmed simply by the worker performing visual confirmation work. Therefore, flight path 45 can be generated easily.

[0028] ***Other Configurations*** <Modification 1> In the first embodiment, each functional component is realized by software. However, in Modification 1, each functional component may be realized by hardware. The differences between Modification 1 and the first embodiment will be described below.

[0029] When each functional component is realized by hardware, the flight path generation device 10 includes an electronic circuit instead of the processor 11, the memory 12, and the storage 13. The electronic circuit is a dedicated circuit that realizes the functions of each functional component, the memory 12, and the storage 13.

[0030] Possible electronic circuits include a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, a logic IC, a GA, an ASIC, and an FPGA. GA stands for Gate Array. ASIC stands for Application Specific Integrated Circuit. FPGA stands for Field-Programmable Gate Array. Each functional component may be realized by a single electronic circuit, or each functional component may be distributed across multiple electronic circuits.

[0031] <Modification 2> As a modification 2, some of the functional components may be realized by hardware, and other functional components may be realized by software.

[0032] The processor 11, the memory 12, the storage 13, and the electronic circuitry are collectively referred to as a processing circuit. In other words, the functions of the respective functional components are realized by the processing circuit.

[0033] Furthermore, the term "unit" in the above description may be read as a "circuit," "step," "procedure," "process," or "processing circuit."

[0034] Embodiment 2. Embodiment 2 differs from embodiment 1 in that the number of shooting positions 44 is reduced by clustering the line of sight information 32. In embodiment 2, this difference will be explained, and explanation of the same points will be omitted.

[0035] ***Description of Configuration*** The configuration of flight path generation device 10 according to embodiment 2 will be described with reference to Figure 7. Flight path generation device 10 differs from flight path generation device 10 shown in Figure 1 in that it includes, as functional components, a clustering unit 25 and a parameter modification unit 26. The functions of clustering unit 25 and parameter modification unit 26, like the other functional components, are realized by software or hardware.

[0036] ***Description of Operation*** The operation of flight path generation device 10 according to embodiment 2 will be described with reference to Figures 8 to 10. The operational procedure of flight path generation device 10 according to embodiment 2 corresponds to the flight path generation method according to embodiment 2. Furthermore, the program that realizes the operation of flight path generation device 10 according to embodiment 2 corresponds to the flight path generation program according to embodiment 2.

[0037] The processing flow of flight path generation device 10 according to the second embodiment will be described with reference to Figure 8. As with the processing shown in Figure 2, the processing shown in Figure 8 also presupposes that an operator performs a visual confirmation task and collects line-of-sight information 32. Furthermore, the processing from step S21 to step S22 in Figure 8 is the same as the processing from step S11 to step S12 in Figure 2.

[0038] The processes from step S23 to step S27 are executed for each of the gaze times 42 identified in step S21.

[0039] (Step S23: Gaze Information Acquisition Process) The clustering unit 25 acquires the gaze information 32 for the gaze time 42 of the target.

[0040] (Step S24: Clustering Process) As shown in FIG. 9 , the clustering unit 25 clusters the gaze information 32 acquired in step S23 and classifies the information into one or more clusters 48. Specifically, the clustering unit 25 clusters the gaze information 32 according to parameters that define the criteria for clustering. The parameters are, for example, reference values ​​for the similarity of the gaze information 32. In this case, the clustering unit 25 uses a certain piece of gaze information 32 as a reference and classifies gaze information 32 whose similarity is higher than the reference value into the same cluster 48 as the reference gaze information 32. Note that when the process of step S24 is first executed for the target gaze time 42, initial values ​​are set for the parameters.

[0041] (Step S25: Representative Position Identification Processing) The shooting position identification unit 23 sets each cluster 48 obtained by clustering in step S24 as a target cluster 48. Then, as shown in FIG. 10 , the shooting position identification unit 23 identifies a shooting position 44 for photographing the shooting area 43 identified from the line of sight information 32 classified into the target cluster 48 as a representative position 49. Specifically, the shooting position identification unit 23 selects one representative piece of line of sight information 32 from the line of sight information 32 classified into the target cluster 48. As a specific example, the shooting position identification unit 23 selects the line of sight information 32 located in the center of the line of sight information 32 classified into the target cluster 48. Instead of selecting one representative piece of line of sight information 32, the shooting position identification unit 23 may calculate the representative line of sight information 32 by calculating statistics from the line of sight information 32 classified into the target cluster 48. As a specific example, the shooting position identification unit 23 may calculate the representative line of sight information 32 by averaging the line of sight information 32 classified into the target cluster 48. The shooting position identifying unit 23 identifies a representative mesh, which is a mesh indicated by the line-of-sight information 32 that is representative of the mesh data 46, as the shooting area 43 identified from the line-of-sight information 32. Note that the mesh data 46 is as described in step S12 of the first embodiment and is generated in step S22. The shooting position identifying unit 23 identifies the shooting position 44, which is a position a reference distance away from the identified mesh in the normal direction, as the representative position 49.

[0042] (Step S26: Photographing Criteria Determination Processing) The parameter modification unit 26 sets each cluster 48 obtained by clustering in step S24 as a target cluster 48. The parameter modification unit 26 assumes that the photographing area 43 identified from the line-of-sight information 32 classified into the target cluster 48 is photographed from the representative position 49 identified for the target cluster 48 in step S25. In this case, the parameter modification unit 26 determines whether or not there is a photographing area 43 that does not satisfy the photographing criteria. If there is no photographing area 43 that does not satisfy the photographing criteria for all clusters 48, the parameter modification unit 26 ends the processing for the target gaze time 42. On the other hand, if there is a photographing area 43 that does not satisfy the photographing criteria for even one cluster 48, the parameter modification unit 26 proceeds to step S27.

[0043] A specific example of the shooting criterion is the upper limit of the shooting angle. In this case, the parameter modification unit 26 determines whether the shooting angle is within the upper limit when the shooting area 43 is shot from the representative position 49. If the shooting angle is within the upper limit, the parameter modification unit 26 determines that the shooting condition is met. On the other hand, if the shooting angle is greater than the upper limit, the parameter modification unit 26 determines that the shooting condition is not met. The shooting angle is the angle formed by a perpendicular line cast from the representative position 49 to the representative mesh and a perpendicular line cast from the representative position 49 to the mesh that is the shooting area 43.

[0044] (Step S27: Parameter Changing Process) The parameter changing unit 26 changes the parameters that determine the clustering criteria used in step S24. Specifically, the parameter changing unit 26 changes the parameters so that the objects are less likely to be classified into the same cluster 48. For example, if the parameters are reference values ​​for similarity, the parameter changing unit 26 increases the reference value. Then, the parameter changing unit 26 returns the process to step S24 to perform the clustering again and identify the representative positions 49 again.

[0045] (Step S28: Flight path generation process) The flight path generation unit 24 generates the flight path 45 using the representative position 49 finally identified for each gaze time 42. Specifically, the flight path generation unit 24 generates the flight path 45 by using the representative position 49 instead of the shooting position 44 in step S14 of FIG.

[0046] ***Effects of Embodiment 2*** As described above, flight path generation device 10 according to Embodiment 2 reduces the number of image capture positions 44 by clustering line-of-sight information 32. This makes it possible to generate flight path 45 that enables efficient image capture of visually confirmed locations without repeated small movements.

[0047] ***Other Configurations*** <Modification 3> In the second embodiment, the line of sight information 32 is clustered to reduce the number of shooting positions 44. However, after identifying the shooting positions 44 using a method similar to that of the first embodiment, the shooting positions 44 may be clustered to reduce the number of shooting positions 44.

[0048] A specific description will be given with reference to Fig. 11. As with the process shown in Fig. 2, the process shown in Fig. 11 also presupposes that a visual confirmation task is performed by an operator and line-of-sight information 32 is collected. Furthermore, the processes from steps S31 to S33 in Fig. 11 are the same as the processes from steps S11 to S13 in Fig. 2. The processes from steps S37 to S39 are the same as the processes from steps S26 to S28 in Fig. 8.

[0049] The processes of steps S34 to S38 are executed for each of the gaze times 42 identified in step S31.

[0050] (Step S34: Photographing Position Acquisition Processing) The clustering unit 25 acquires the photographing positions 44 identified in step S33 for the photographing area 43 identified from the line of sight during the target gaze time 42. For example, in step S32, the photographing area 43 is associated with the gaze time 42 that was the source of identification and stored in the memory 12. Also, in step S33, the photographing positions 44 are associated with the gaze time 42 associated with the photographing area 43 that was the source of identification and stored in the memory 12. Then, the clustering unit 25 acquires the photographing positions 44 associated with the target gaze time 42. This makes it possible to acquire the photographing positions 44 identified in step S33 for the photographing area 43 identified from the line of sight during the target gaze time 42.

[0051] (Step S35: Clustering Process) As shown in FIG. 12 , the clustering unit 25 clusters the photography positions 44 acquired in step S34 and classifies them into one or more clusters 48. Specifically, the clustering unit 25 clusters the photography positions 44 according to parameters that define the criteria for clustering. The parameters are, for example, reference values ​​for the distance between photography positions 44. In this case, the clustering unit 25 uses a certain photography position 44 as a reference and classifies photography positions 44 that are closer than the reference value into the same cluster 48 as the reference photography position 44. Note that when the process of step S35 is first executed for the target gaze time 42, initial values ​​are set for the parameters.

[0052] (Step S36: Representative Position Identification Process) The shooting position identifying unit 23 sets each cluster 48 obtained by clustering in step S35 as a target cluster 48. Then, as shown in FIG. 13 , the shooting position identifying unit 23 identifies a representative position 49 from the shooting positions 44 classified into the target cluster 48. Specifically, the shooting position identifying unit 23 selects one representative shooting position 44 from the shooting positions 44 classified into the target cluster 48 to set it as the representative position 49. As a specific example, the shooting position identifying unit 23 selects the shooting position 44 located in the center of the shooting positions 44 classified into the target cluster 48. Instead of selecting one representative shooting position 44, the shooting position identifying unit 23 may calculate a statistical value from the shooting positions 44 classified into the target cluster 48 to calculate the representative shooting position 44. As a specific example, the shooting position identifying unit 23 may calculate the representative shooting position 44 by averaging the positions of the shooting positions 44 classified into the target cluster 48.

[0053] Embodiment 3. Embodiment 3 differs from Embodiments 1 and 2 in that a flight path 45 is generated excluding a no-fly area 50 in which the unmanned aerial vehicle cannot fly. In Embodiment 3, this difference will be explained, and explanation of the same points will be omitted. In Embodiment 3, a case where a modification is made to Embodiment 1 will be explained. However, it is also possible to make modifications to Embodiment 2.

[0054] ***Description of Configuration*** The configuration of a flight path generation device 10 according to the third embodiment will be described with reference to FIG. 14 . Flight path generation device 10 differs from flight path generation device 10 shown in FIG. 1 in that it includes, as functional components, a flight feasibility determination unit 27 and an unusable area identification unit 28. The functions of flight feasibility determination unit 27 and unusable area identification unit 28, like the other functional components, are realized by software or hardware. Flight path generation device 10 also differs from flight path generation device 10 shown in FIG. 2 in that surrounding information 33 is stored in storage 13. The surrounding information 33 is information indicating the positions of other objects present around the target object 41. A specific example of the surrounding information 33 is a three-dimensional image of the area around the target object 41.

[0055] ***Description of Operation*** The operation of flight path generation device 10 according to embodiment 3 will be described with reference to Figures 15 and 16. The operational procedure of flight path generation device 10 according to embodiment 3 corresponds to the flight path generation method according to embodiment 3. Furthermore, the program that realizes the operation of flight path generation device 10 according to embodiment 3 corresponds to the flight path generation program according to embodiment 3.

[0056] The processing flow of flight path generation device 10 according to embodiment 3 will be described with reference to Figure 15. As with the processing shown in Figure 2, the processing shown in Figure 15 is premised on the fact that an operator performs a visual confirmation task and collects line-of-sight information 32. Furthermore, the processing from step S41 to step S44 in Figure 15 is the same as the processing from step S11 to step S14 in Figure 2.

[0057] (Step S45: Flight feasibility determination process) The flight feasibility determination unit 27 determines whether the flight path 45 generated in step S34 includes a no-fly area 50 in which the unmanned aerial vehicle cannot fly. Specifically, the flight feasibility determination unit 27 references the surrounding information 33 and sets the no-fly area 50 to a reference range around objects present around the target object 41. For example, the flight feasibility determination unit 27 sets the no-fly area 50 using a range that may interfere with surrounding objects as a reference range, taking into account the size of the unmanned aerial vehicle. The flight feasibility determination unit 27 then determines whether the flight path 45 includes the no-fly area 50. If the flight path 45 includes the no-fly area 50, the flight feasibility determination unit 27 proceeds to step S46. On the other hand, if the flight path 45 does not include the no-fly area 50, the flight feasibility determination unit 27 ends the process.

[0058] (Step S46: Flight path correction process) The flight path generation unit 24 corrects the flight path 45 generated in step S44 so as to avoid the prohibited fly area 50. For example, as shown in FIG. 16 , the flight path generation unit 24 corrects the flight path 45 so that the flight path 45 follows the flight path 45 for the portion of the flight path 45 that overlaps with the prohibited fly area 50.

[0059] (Step S47: Unavailable area identification process) The unavailable area identification unit 28 identifies an uncaptureable area that cannot be photographed along the flight path 45 corrected in step S46. Specifically, the unavailable area identification unit 28 determines whether the photographing position 44 is included in the range corrected in step S46. If the photographing position 44 is included in the corrected range, the unavailable area identification unit 28 identifies the photographing area 43 photographed from the photographing position 44 as an uncaptureable area. If the photographing position 44 is not included in the corrected range, the unavailable area identification unit 28 determines that there is no uncaptureable area.

[0060] When an unphotographable area is identified, the unphotographable area identifying unit 28 may output the unphotographable area to a user terminal or the like together with the flight path 45. In this way, the user may be notified of the unphotographable area.

[0061] ***Effects of Embodiment 3*** As described above, flight path generation device 10 according to embodiment 3 generates flight path 45 by excluding no-fly area 50 where unmanned aerial vehicles cannot fly. This makes it possible to generate flight path 45 that avoids objects present around target object 41.

[0062] Furthermore, flight path generation device 10 according to the third embodiment identifies any uncaptureable areas that cannot be photographed as a result of generating flight path 45 excluding uncaptureable areas 50. This makes it possible to consider countermeasures, etc.

[0063] ***Other Configurations*** <Variation 4> In the third embodiment, if the shooting position 44 is included in the corrected range, the shooting area 43 to be shot from the shooting position 44 is identified as an unphotographable area. However, there are cases where the shooting area 43 to be shot from the shooting position 44 included in the corrected range can be shot from the corrected flight path 45. Therefore, the unphotographable area may be identified after determining whether the shooting area 43 to be shot from the shooting position 44 included in the corrected range can be shot from the corrected flight path 45. Specifically, if the shooting position 44 is included in the corrected range, the unphotographable area identifying unit 28 sets the shooting position 44 as the target shooting position 44. When shooting the shooting area 43 to be shot from the target shooting position 44, the unphotographable area identifying unit 28 determines whether a position that meets the shooting criteria is included in the flight path 45. If a position that meets the shooting criteria is included in the flight path 45, the unphotographable area identifying unit 28 sets the position as the shooting position 44 of the shooting area 43. On the other hand, if the flight path 45 does not include a position that satisfies the shooting criteria, the unusable area specifying unit 28 specifies the shooting area 43 as an unusable area.

[0064] The embodiments and modifications of the present disclosure have been described above. Some of these embodiments and modifications may be combined and implemented. Furthermore, one or more of them may be implemented partially. Note that the present disclosure is not limited to the above embodiments and modifications, and various modifications are possible as needed.

[0065] 10 Flight path generation device, 11 Processor, 12 Memory, 13 Storage, 14 Communication interface, 21 Gaze time determination unit, 22 Shooting area determination unit, 23 Shooting position determination unit, 24 Flight path generation unit, 25 Clustering unit, 26 Parameter modification unit, 27 Flight possibility determination unit, 28 Unavailable area determination unit, 31 Three-dimensional image data, 32 Line of sight information, 33 Peripheral information, 41 Object, 42 Gaze time, 43 Shooting area, 44 Shooting position, 45 Flight path, 46 Mesh data, 47 Shooting target mesh, 48 Cluster, 49 Representative position, 50 Unavailable area.

Claims

1. A flight path generation device comprising: a photographing area identification unit that identifies an area of ​​an object that a worker is gazing at as a photographing area from line-of-sight information indicating the line of sight of the worker during visual confirmation work; a photographing position identification unit that identifies a photographing position for photographing the photographing area identified by the photographing area identification unit; and a flight path generation unit that uses the photographing position identified by the photographing position identification unit to generate a flight path for the unmanned aerial vehicle to photograph the photographing area with a camera mounted on the unmanned aerial vehicle.

2. The flight path generation device according to claim 1, further comprising a gaze time determination unit that determines a gaze time during which the amount of change in the gaze indicated by the gaze information is below a threshold value and is equal to or greater than a reference period, and the shooting area determination unit determines the area indicated by the gaze during the gaze time determined by the gaze time determination unit as the shooting area.

3. The flight path generation device according to claim 1 or 2, wherein the shooting area identification unit identifies the target mesh, which is the mesh that the worker is gazing at, as the shooting area from among mesh data that represents the surface of the object by combining meshes that are multiple planes, and the shooting position identification unit identifies, as the shooting position, a position that is a reference distance away in the normal direction from the target mesh identified as the shooting area.

4. The flight path generation device described in claim 3, wherein the shooting position identification unit identifies the direction of a perpendicular line drawn from the shooting position to the mesh to be photographed as the shooting direction from the shooting position, and the flight path generation unit generates the flight path indicating the shooting direction from the shooting position.

5. The flight path generation device according to any one of claims 1 to 4, further comprising: a clustering unit that clusters the gaze information when the worker is gazing; the shooting position identification unit, for each cluster obtained by clustering by the clustering unit, identifies a shooting position for photographing the shooting area identified from the gaze information classified into that cluster as a representative position; and the flight path generation unit generates the flight path using the representative position.

6. The flight path generation device according to any one of claims 1 to 4, further comprising a clustering unit that clusters the shooting locations identified by the shooting location identification unit, and the flight path generation unit generates the flight path for each cluster obtained by clustering by the clustering unit using a representative location identified from the shooting locations classified into that cluster.

7. The flight path generation device according to claim 5 or 6, further comprising: a parameter change unit that changes the clustering parameters if the shooting area includes an area that does not satisfy the shooting criteria when photographed from the representative position identified by the shooting position identification unit; and the clustering unit redoes clustering in accordance with the parameters changed by the parameter change unit.

8. The flight path generation device described in any one of claims 1 to 7, further comprising a flight feasibility determination unit that determines whether the flight path generated by the flight path generation unit includes an unflyable area in which the unmanned aerial vehicle cannot fly, and when the flight feasibility determination unit determines that the flight path includes an unflyable area, the flight path generation unit modifies the flight path to avoid the unflyable area.

9. The flight path generation device according to claim 8, further comprising an unflyable area identification unit that, when the flight feasibility determination unit determines that the flight path includes an unflyable area, identifies an unflyable area within the shooting area that cannot be photographed on the revised flight path.

10. A flight path generation method in which a computer identifies an area of ​​an object that a worker is gazing at as a photographing area from line-of-sight information indicating the line of sight of the worker during visual confirmation work, the computer identifies a photographing position for photographing the photographing area, and the computer uses the photographing position to generate a flight path for an unmanned aerial vehicle to photograph the photographing area with a camera mounted on the unmanned aerial vehicle.

11. A flight path generation program that causes a computer to function as a flight path generation device that performs the following processes: a photography area identification process that identifies the area of ​​an object that a worker is gazing at as a photography area from line-of-sight information indicating the line of sight of the worker during visual confirmation work; a photography position identification process that identifies a photography position for photographing the photography area identified by the photography area identification process; and a flight path generation process that generates a flight path for an unmanned aerial vehicle to photograph the photography area with a camera mounted on the unmanned aerial vehicle using the photography position identified by the photography position identification process.

Images