Flight Route Generation Device, Flight Route Generation Method, and Flight Route Generation Program

Abstract

The imaging area specifying unit (22) specifies, as the imaging area, the area on the object that the operator is gazing at from the line-of-sight information indicating the operator's line of sight during the operator's visual confirmation work. The imaging position specifying unit (23) specifies the imaging position for imaging the imaging area specified by the imaging area specifying unit (22). The flight path generation unit (24) generates a flight path of the unmanned aircraft for imaging the imaging area with a camera mounted on the unmanned aircraft, using the imaging position specified by the imaging position specifying unit (23).

Description

Technical Field

[0001] The present disclosure relates to a technique for generating a flight path of an unmanned aircraft.

Background Art

[0002] Inspection work of facilities and the like has been performed by visual inspection by workers. In recent years, inspection work may be performed using image data obtained by photographing locations to be visually inspected with a camera. Inspection work using image data can also be realized by image analysis by a computer or the like. By performing inspection work using image data, the load of inspection work can be reduced. However, it may be difficult to install a camera for each inspection location. In addition, installing a camera for each inspection location increases the cost. Therefore, it is conceivable to use an unmanned aircraft equipped with a camera to photograph inspection locations and acquire image data of the inspection locations.

[0003] As a method of photographing inspection locations using an unmanned aircraft, the simplest method is for an operator to operate the unmanned aircraft to photograph each inspection location each time an inspection is performed. However, it is time-consuming for an operator to operate the unmanned aircraft each time an inspection is performed. In addition, depending on the installation location of the object to be inspected, advanced technology is required for operation, and it may be difficult to acquire appropriate image data. Therefore, it is conceivable to set a flight path for the unmanned aircraft and automate the photographing of inspection locations. When automating the photographing of inspection locations, it is necessary to set an appropriate flight path that can photograph each inspection location.

[0004] Patent Document 1 describes a method of setting a flight path for photographing a designated photographing location. In Patent Document 1, a three-dimensional image of a structure is displayed, and while changing the photographing target area on the structure and the photographing direction of the camera toward the photographing target area in accordance with a user's instruction, the photographing target area and the photographing direction are designated for each inspection location. In Patent Document 1, a path passing through the positions specified from the photographing target area and the photographing direction in order is generated as the flight path.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] In the method described in Patent Document 1, it is necessary to manually operate a computer to set the imaging target area and imaging direction for each inspection point, which is time-consuming. An object of the present disclosure is to enable easy generation of a flight path of an unmanned aircraft for photographing a location being visually inspected.

Means for Solving the Problems

[0007] The flight path generation device according to the present disclosure includes: an imaging area specifying unit that specifies, as an imaging area, an area of an object that the operator has gazed at from line-of-sight information indicating the line of sight of the operator during the visual inspection work of the operator; an imaging position specifying unit that specifies an imaging position for imaging the imaging area specified by the imaging area specifying unit; a flight path generation unit that generates a flight path of the unmanned aircraft for imaging the imaging area by a camera mounted on the unmanned aircraft using the imaging position specified by the imaging position specifying unit and includes.

Effects of the Invention

[0008] In the present disclosure, the imaging area is specified from the line-of-sight information indicating the line of sight of the operator during the visual inspection work. Then, the imaging position is specified from the imaging area, and the flight path is generated using the imaging position. As a result, just by the operator performing the visual inspection work, a flight path for imaging the location being visually inspected is generated. Therefore, the flight path can be generated easily.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Modes for Carrying Out the Invention

[0010] Embodiment 1. ***Description of Configuration*** Referring to FIG. 1, the configuration of the flight path generation device 10 according to Embodiment 1 will be described. The flight path generation device 10 is a computer. The flight path generation device 10 includes hardware such as a processor 11, a memory 12, a storage 13, and a communication interface 14. The processor 11 is connected to other hardware via signal lines and controls these other hardware.

[0011] The processor 11 is an IC that performs processing. IC is an abbreviation for Integrated Circuit. As specific examples, the processor 11 is a CPU, a DSP, or a GPU. CPU is an abbreviation for Central Processing Unit. DSP is an abbreviation for Digital Signal Processor. GPU is an abbreviation for Graphics Processing Unit.

[0012] The memory 12 is a storage device that temporarily stores data. As specific examples, the memory 12 is an SRAM or a DRAM. SRAM is an abbreviation for Static Random Access Memory. DRAM is an abbreviation for Dynamic Random Access Memory.

[0013] The storage 13 is a storage device that stores data. As a specific example, the storage 13 is an HDD. HDD is an abbreviation for Hard Disk Drive. Also, the storage 13 may be a portable recording medium such as an SD (registered trademark) memory card, a CompactFlash (registered trademark), a NAND flash, a flexible disk, an optical disk, a compact disk, a Blu-ray (registered trademark) disk, or a DVD. SD is an abbreviation for Secure Digital. DVD is an abbreviation for Digital Versatile Disk.

[0014] The communication interface 14 is an interface for communicating with an external device. As specific examples, the communication interface 14 is a port for Ethernet (registered trademark), USB, or HDMI (registered trademark). USB is the abbreviation of Universal Serial Bus. HDMI is the abbreviation of High-Definition Multimedia Interface.

[0015] The flight path generation device 10 includes, as functional components, a fixation time determination unit 21, a shooting area determination unit 22, a shooting position determination unit 23, and a flight path generation unit 24. The functions of the respective functional components of the flight path generation device 10 are realized by software. The storage 13 stores a program for realizing the functions of the respective functional components of the flight path generation device 10. This program is read into the memory 12 by the processor 11 and executed by the processor 11. Thereby, the functions of the respective functional components of the flight path generation device 10 are realized.

[0016] The storage 13 stores 3D image data 31. The 3D image data 31 is 3D image data of an object 41 that is the target of a visual inspection operation such as an inspection operation. Here, it is assumed that the 3D image data 31 is stored in the storage 13. However, the 3D image data 31 may be stored in a storage device external to the flight path generation device 10.

[0017] In FIG. 1, only one processor 11 is shown. However, there may be a plurality of processors 11, and the plurality of processors 11 may execute in cooperation a program for realizing each function.

[0018] ***Description of the operation*** With reference to FIGS. 2 to 6, the operation of the flight path generation device 10 according to the first embodiment will be described. The operation procedure of the flight path generation device 10 according to Embodiment 1 corresponds to the flight path generation method according to Embodiment 1. Also, the program for realizing the operation of the flight path generation device 10 according to Embodiment 1 corresponds to the flight path generation program according to Embodiment 1.

[0019] Referring to FIG. 2, the processing flow of the flight path generation device 10 according to Embodiment 1 will be described. As a premise for the processing shown in FIG. 2, visual confirmation work by an operator is performed. At this time, the operator wears a sensor device such as smart glasses or HUD to perform the visual confirmation work. HUD is an abbreviation for Head-Up Display. The sensor device collects line-of-sight information 32 that shows the operator's line of sight in time series during the visual confirmation work. As a specific example, the line-of-sight information 32 is time-series data that shows the position of the operator's head with respect to the object 41 and the posture of the operator's head with respect to the object 41 in time series. The position of the operator's head is represented by, for example, three-dimensional coordinates of latitude, longitude, and altitude. The posture of the operator's head is represented by, for example, a quaternion. Also, the line-of-sight information 32 may include information indicating the operator's line of sight itself instead of the information indicating the posture of the operator's head. If a function for collecting information on the black eyes is installed in smart glasses or the like, the direction of the operator's line of sight can be specified.

[0020] (Step S11: Fixation time specifying process) The fixation time specifying unit 21 specifies the fixation time 42 from the line-of-sight information 32. The fixation time 42 is the time when the operator fixates on the object 41. Specifically, the fixation time 42 is the time when the period during which the amount of change in the line of sight indicated by the line-of-sight information 32 is below the threshold is equal to or longer than the reference period. This will be specifically described with reference to FIG. 3. FIG. 3 shows the time change in the moving speed of the line of sight. The moving speed of the line of sight represents the amount of change in the line of sight per unit time. Therefore, the fixation time specifying unit 21 specifies as the fixation time 42 the time when the period during which the moving speed of the line of sight is below the threshold is equal to or longer than the reference period. In FIG. 3, two fixation times 42, namely, fixation time 42A and fixation time 42B, are specified.

[0021] (Step S12: Shooting area identification process) The shooting area identification unit 22 identifies the area on the object 41 that the operator is looking at as the shooting area 43. Specifically, the shooting area identification unit 22 identifies the area indicated by the line of sight at the fixation time 42 identified in step S11 as the shooting area 43. As a result, one or more shooting areas 43 are identified.

[0022] A specific description will be given with reference to FIG. 4. First, the shooting area identification unit 22 generates mesh data 46 of the object 41 from the three-dimensional image data 31. The mesh data is data representing the surface of the object 41 with a mesh of a plurality of planes such as triangles. It is possible to generate the mesh data 46 by using the point cloud data included in the three-dimensional image data 31. Next, the shooting area identification unit 22 identifies the shooting target mesh 47 from the mesh data 46. Specifically, the shooting area identification unit 22 identifies, as the shooting target mesh 47, the mesh indicated by the line of sight at the fixation time 42 identified in step S1 among the meshes constituting the mesh data 46. The mesh indicated by the line of sight means a mesh that intersects the line-of-sight vector. The shooting area identification unit 22 sets each of the identified one or more shooting target meshes 47 as the shooting area 43. Here, there may be a case where a plurality of shooting target meshes 47 are identified from one fixation time 42. The fixation time 42 is a time when the change amount of the line of sight is small, but the line of sight may change little by little. Therefore, in one fixation time 42, when a plurality of meshes are fixated, each fixated mesh may become the shooting target mesh 47.

[0023] (Step S13: Shooting position identification process) The imaging position specifying unit 23 specifies an imaging position 44 for imaging the imaging area 43 specified in step S12. At this time, the imaging position specifying unit 23 sets each imaging area 43 specified in step S12 as the target imaging area 43. Then, the imaging position specifying unit 23 specifies an imaging position 44 for imaging the target imaging area 43. Thereby, the imaging positions 44 corresponding to each of the one or more imaging areas 43 specified in step S12 are specified.

[0024] A specific description will be given with reference to FIG. 5. The imaging position specifying unit 23 sets each of the one or more imaging areas 43 specified in step S22 as the target imaging area 43. In other words, each of the one or more imaging target meshes 47 is set as the target imaging area 43. The imaging position specifying unit 23 specifies an imaging position 44 corresponding to the target imaging area 43 set in step S31. Specifically, the imaging position specifying unit 23 specifies, as the imaging position 44, a position that is separated from the imaging target mesh 47 specified as the imaging area 43 by a reference distance in the normal direction. At this time, the imaging position specifying unit 23 specifies the direction of the perpendicular line dropped from the imaging position 44 to the imaging target mesh 47 as the imaging direction from the imaging position 44. The imaging position specifying unit 23 associates the imaging position 44 with the imaging direction from the imaging position 44.

[0025] (Step S14: Flight path generation process) The flight path generation unit 24 generates a flight path 45 of the unmanned aircraft for imaging the imaging area 43 by using the imaging position 44 specified in step S13 with a camera mounted on the unmanned aircraft. Here, as a specific example, the unmanned aircraft is a drone.

[0026] A specific description will be given with reference to FIG. 6. The flight path generation unit 24 generates a flight path 45 by sequentially connecting one or more shooting positions 44 specified in step S13. That is, the flight path 45 is information obtained by sequentially connecting the shooting positions 44. Here, a shooting direction is associated with each shooting position 44. Therefore, the flight path 45 includes information indicating the shooting direction at each shooting position 44. At this time, the flight path generation unit 24 generates the flight path 45 by connecting the shooting positions 44 so that the path is the shortest. Alternatively, the flight path generation unit 24 generates the flight path 45 by connecting the shooting positions 44 in the order in which they are gazed at by the operator. Without being limited to this, the flight path generation unit 24 may generate the flight path 45 by connecting the shooting positions 44 according to some rule. When there is a standby position of the unmanned aircraft, the flight path generation unit 24 may connect the standby position and the first shooting position 44 in the flight path 45, and connect the last shooting position 44 in the flight path 45 and the standby position to complete the flight path 45.

[0027] ***Effects of Embodiment 1*** As described above, the flight path generation device 10 according to Embodiment 1 specifies the shooting area 43 from the line-of-sight information 32 indicating the operator's line of sight during the visual confirmation operation. Then, the flight path generation device 10 specifies the shooting position 44 from the shooting area 43, and generates the flight path 45 using the shooting position 44. Thereby, just by the operator performing the visual confirmation operation, the flight path 45 for shooting the location being visually confirmed is generated. Therefore, the flight path 45 can be generated simply.

[0028] ***Other Configurations*** <Modification 1> In Embodiment 1, each functional component is realized by software. However, as Modification 1, each functional component may be realized by hardware. The differences from Embodiment 1 will be described for this Modification 1.

[0029] When each functional component is implemented by hardware, the flight path generation device 10 includes an electronic circuit instead of the processor 11, the memory 12, and the storage 13. The electronic circuit is a dedicated circuit that realizes the functions of each functional component, the memory 12, and the storage 13.

[0030] Examples of the electronic circuit include a single circuit, a composite circuit, a programmed processor, a parallel-programmed processor, a logic IC, a GA, an ASIC, and an FPGA. GA is an abbreviation for Gate Array. ASIC is an abbreviation for Application Specific Integrated Circuit. FPGA is an abbreviation for Field-Programmable Gate Array. Each functional component may be realized by one electronic circuit, or each functional component may be realized by being distributed among a plurality of electronic circuits.

[0031] <Modification Example 2> As Modification Example 2, some of the functional components may be realized by hardware and the other functional components may be realized by software.

[0032] The processor 11, the memory 12, the storage 13, and the electronic circuit are referred to as a processing circuit. That is, the functions of each functional component are realized by the processing circuit.

[0033] Also, in the above description, the "section" may be read as "circuit", "step", "procedure", "process", or "processing circuit".

[0034] Embodiment 2. Embodiment 2 is different from Embodiment 1 in that the number of shooting positions 44 is reduced by clustering the line-of-sight information 32. In Embodiment 2, this different point will be described, and the description of the same points will be omitted.

[0035] ***Description of Configuration*** With reference to FIG. 7, the configuration of the flight path generation device 10 according to Embodiment 2 will be described. The flight path generation device 10 differs from the flight path generation device 10 shown in FIG. 1 in that it includes a clustering unit 25 and a parameter change unit 26 as functional components. The functions of the clustering unit 25 and the parameter change unit 26 are realized by software or hardware, similar to other functional components.

[0036] ***Description of Operations*** With reference to FIGS. 8 to 10, the operation of the flight path generation device 10 according to Embodiment 2 will be described. The operation procedure of the flight path generation device 10 according to Embodiment 2 corresponds to the flight path generation method according to Embodiment 2. Also, the program for realizing the operation of the flight path generation device 10 according to Embodiment 2 corresponds to the flight path generation program according to Embodiment 2.

[0037] With reference to FIG. 8, the processing flow of the flight path generation device 10 according to Embodiment 2 will be described. Regarding the processing shown in FIG. 8, similar to the processing shown in FIG. 2, as a premise, the operator's visual confirmation work is performed and the line-of-sight information 32 is collected. Also, the processing from step S21 to step S22 in FIG. 8 is the same as the processing from step S11 to step S12 in FIG. 2.

[0038] Regarding each fixation time 42 specified in step S21, the processing from step S23 to step S27 is executed.

[0039] (Step S23: Line-of-Sight Information Acquisition Processing) The clustering unit 25 acquires the line-of-sight information 32 at the target fixation time 42.

[0040] (Step S24: Clustering Processing) As shown in FIG. 9, the clustering unit 25 clusters the line-of-sight information 32 acquired in step S23 and classifies it into one or more clusters 48. Specifically, the clustering unit 25 clusters the line-of-sight information 32 according to a parameter that defines the clustering criterion. The parameter is, for example, a reference value of the similarity of the line-of-sight information 32. In this case, the clustering unit 25 classifies the line-of-sight information 32 with a similarity higher than the reference value into the same cluster 48 as the reference line-of-sight information 32 with a certain line-of-sight information 32 as a reference. In addition, when the process of step S24 is first executed for the target fixation time 42, an initial value is set for the parameter.

[0041] (Step S25: Representative position identification process) The imaging position identification unit 23 sets each cluster 48 obtained by being clustered in step S24 as the target cluster 48. Then, as shown in FIG. 10, the imaging position identification unit 23 identifies the imaging position 44 for imaging the imaging region 43 specified from the line-of-sight information 32 classified into the target cluster 48 as the representative position 49. Specifically, the imaging position identification unit 23 selects one piece of representative line-of-sight information 32 from the line-of-sight information 32 classified into the target cluster 48. As a specific example, the imaging position identification unit 23 selects the line-of-sight information 32 located at the center of the line-of-sight information 32 classified into the target cluster 48. Instead of selecting one piece of representative line-of-sight information 32, the imaging position identification unit 23 may calculate a statistical value from the line-of-sight information 32 classified into the target cluster 48 to calculate the representative line-of-sight information 32. As a specific example, the imaging position identification unit 23 may calculate the average of the line-of-sight information 32 classified into the target cluster 48 to calculate the representative line-of-sight information 32. The imaging position identification unit 23 identifies the representative mesh, which is the mesh indicated by the representative line-of-sight information 32 among the mesh data 46, as the imaging region 43 specified from the line-of-sight information 32. The mesh data 46 is as described in step S12 of Embodiment 1 and is generated in step S22. The imaging position identification unit 23 identifies the imaging position 44, which is a position separated from the specified mesh by a reference distance in the normal direction, as the representative position 49.

[0042] (Step S26: Imaging criterion determination process) The parameter changing unit 26 sets each cluster 48 obtained by clustering in step S24 as the target cluster 48. The parameter changing unit 26 assumes that, from the representative position 49 specified in step S25 for the target cluster 48, the imaging area 43 specified from the line-of-sight information 32 classified into the target cluster 48 is imaged. In this case, the parameter changing unit 26 determines whether there is an imaging area 43 that does not satisfy the imaging criteria. If there is no imaging area 43 that does not satisfy the imaging criteria for all the clusters 48, the parameter changing unit 26 ends the process for the target fixation time 42. On the other hand, if there is an imaging area 43 that does not satisfy the imaging criteria even for one cluster 48, the parameter changing unit 26 advances the process to step S27.

[0043] As a specific example, the imaging criteria is the upper limit value of the imaging angle. In this case, the parameter changing unit 26 determines whether the imaging angle is within the upper limit value when imaging the imaging area 43 from the representative position 49. If the imaging angle is within the upper limit value, the parameter changing unit 26 determines that the imaging conditions are satisfied. On the other hand, if the imaging angle is greater than the upper limit value, the parameter changing unit 26 determines that the imaging conditions are not satisfied. The imaging angle is the angle formed by the perpendicular line dropped from the representative position 49 to the representative mesh and the perpendicular line dropped from the representative position 49 to the mesh that is the imaging area 43.

[0044] (Step S27: Parameter change process) The parameter changing unit 26 changes the parameter that determines the clustering criteria used in step S24. Specifically, the parameter changing unit 26 changes the parameter so that it becomes difficult to be classified into the same cluster 48. For example, when the parameter is the reference value of similarity, the parameter changing unit 26 increases the reference value. Then, the parameter changing unit 26 returns the process to step S24 to perform clustering again and re-specify the representative position 49.

[0045] (Step S28: Flight path generation process) The flight path generation unit 24 generates a flight path 45 using the representative positions 49 finally specified for each fixation time 42. Specifically, the flight path generation unit 24 generates the flight path 45 by using the representative positions 49 instead of the shooting positions 44 in step S14 of FIG. 2.

[0046] ***Effects of Embodiment 2*** As described above, the flight path generation device 10 according to Embodiment 2 reduces the number of shooting positions 44 by clustering the gaze information 32. Thereby, it is possible to generate a flight path 45 that can efficiently shoot the visually confirmed locations without repeating fine movements.

[0047] ***Other Configurations*** <Modification Example 3> In Embodiment 2, the gaze information 32 is clustered to reduce the number of shooting positions 44. However, after specifying the shooting positions 44 in the same manner as in Embodiment 1, the shooting positions 44 may be clustered to reduce the number of shooting positions 44.

[0048] This will be specifically described with reference to FIG. 11. Regarding the process shown in FIG. 11, similar to the process shown in FIG. 2, it is premised that the operator performs a visual confirmation operation and the gaze information 32 is collected. Also, the processes from step S31 to step S33 in FIG. 11 are the same as the processes from step S11 to step S13 in FIG. 2. The processes from step S37 to step S39 are the same as the processes from step S26 to step S28 in FIG. 8.

[0049] For each fixation time 42 specified in step S31, the processes from step S34 to step S38 are executed.

[0050] (Step S34: Shooting Position Acquisition Process) The clustering unit 25 acquires the shooting positions 44 specified in step S33 for the shooting areas 43 specified from the gaze at the target fixation time 42. For example, in step S32, the shooting area 43 is stored in the memory 12 in association with the fixation time 42 that is the source of identification. Also, in step S33, the shooting area 43 that is the source of identification of the shooting position 44 is stored in the memory 12 in association with the fixation time 42 associated with the shooting area 43. Then, the clustering unit 25 acquires the shooting position 44 associated with the target fixation time 42. Thereby, it is possible to acquire the shooting position 44 specified in step S33 for the shooting area 43 specified from the line of sight at the target fixation time 42.

[0051] (Step S35: Clustering process) As shown in FIG. 12, the clustering unit 25 clusters the shooting positions 44 acquired in step S34 and classifies them into one or more clusters 48. Specifically, the clustering unit 25 clusters the shooting positions 44 according to a parameter that defines the clustering criterion. The parameter is, for example, a reference value of the distance between the shooting positions 44. In this case, the clustering unit 25 classifies the shooting positions 44 whose distance is closer than the reference value to a certain shooting position 44 into the same cluster 48 as the reference shooting position 44. When the process of step S35 is first executed for the target fixation time 42, an initial value is set for the parameter.

[0052] (Step S36: Representative position identification process) The shooting position identification unit 23 sets each cluster 48 obtained by being clustered in step S35 as the target cluster 48. Then, as shown in FIG. 13, the shooting position identification unit 23 identifies a representative position 49 from the shooting positions 44 classified into the target cluster 48. Specifically, the shooting position specifying unit 23 selects one representative shooting position 44 from the shooting positions 44 classified into the target cluster 48 as the representative position 49. As a specific example, the shooting position specifying unit 23 selects the shooting position 44 located at the center of the shooting positions 44 classified into the target cluster 48. Instead of selecting one representative shooting position 44, the shooting position specifying unit 23 may calculate a statistical value from the shooting positions 44 classified into the target cluster 48 to calculate the representative shooting position 44. As a specific example, the shooting position specifying unit 23 may calculate the representative shooting position 44 by averaging the positions of the shooting positions 44 classified into the target cluster 48.

[0053] Embodiment 3. Embodiment 3 is different from Embodiments 1 and 2 in that a flight path 45 is generated excluding a no - flight area 50 where the unmanned aircraft cannot fly. In Embodiment 3, this different point will be described, and the description of the same points will be omitted. In Embodiment 3, the case of making a change to Embodiment 1 will be described. However, it is also possible to make a change to Embodiment 2.

[0054] ***Description of the configuration*** With reference to FIG. 14, the configuration of the flight path generation device 10 according to Embodiment 3 will be described. The flight path generation device 10 is different from the flight path generation device 10 shown in FIG. 1 in that it includes a flight - ability determination unit 27 and a no - area specifying unit 28 as functional components. The functions of the flight - ability determination unit 27 and the no - area specifying unit 28 are realized by software or hardware, similar to other functional components. Also, the flight path generation device 10 is different from the flight path generation device 10 shown in FIG. 2 in that the peripheral information 33 is stored in the storage 13. The peripheral information 33 is information indicating the positions of other objects existing around the object 41. As a specific example, the peripheral information 33 is a three - dimensional image around the object 41.

[0055] ***Description of the operation*** Referring to FIGS. 15 and 16, the operation of the flight path generation device 10 according to Embodiment 3 will be described. The operation procedure of the flight path generation device 10 according to Embodiment 3 corresponds to the flight path generation method according to Embodiment 3. Also, the program for realizing the operation of the flight path generation device 10 according to Embodiment 3 corresponds to the flight path generation program according to Embodiment 3.

[0056] Referring to FIG. 15, the processing flow of the flight path generation device 10 according to Embodiment 3 will be described. Regarding the processing shown in FIG. 15, similar to the processing shown in FIG. 2, as a premise, the operator's visual confirmation work is performed, and the line-of-sight information 32 is collected. Also, the processing from step S41 to step S44 in FIG. 15 is the same as the processing from step S11 to step S14 in FIG. 2.

[0057] (Step S45: Flight feasibility determination process) The flight feasibility determination unit 27 determines whether the flight path 45 generated in step S34 includes a non-flightable area 50 where the unmanned aircraft cannot fly. Specifically, the flight feasibility determination unit 27 refers to the surrounding information 33 and sets the surrounding reference range of the objects existing around the target object 41 as the non-flightable area 50. For example, the flight feasibility determination unit 27 sets the non-flightable area 50 with a range that may interfere with the surrounding objects in consideration of the size of the unmanned aircraft as the reference range. Then, the flight feasibility determination unit 27 determines whether the flight path 45 includes the non-flightable area 50. If the flight feasibility determination unit 27 determines that the flight path 45 includes the non-flightable area 50, the process proceeds to step S46. On the other hand, if the flight feasibility determination unit 27 determines that the flight path 45 does not include the non-flightable area 50, the process ends.

[0058] (Step S46: Flight path correction process) The flight path generation unit 24 corrects the flight path 45 generated in step S44 so as to avoid the no - fly area 50. For example, as shown in FIG. 16, the flight path generation unit 24 corrects the flight path 45 along the flight path 45 for the portion of the flight path 45 that overlaps with the no - fly area 50.

[0059] (Step S47: No - fly area identification process) The no - fly area identification unit 28 identifies a non - photographable area that cannot be photographed on the flight path 45 corrected in step S46. Specifically, the no - fly area identification unit 28 determines whether the photographing position 44 is included in the range corrected in step S46. When the photographing position 44 is included in the corrected range, the no - fly area identification unit 28 identifies the photographing area 43 to be photographed from the photographing position 44 as a non - photographable area. When the photographing position 44 is not included in the corrected range, the no - fly area identification unit 28 determines that there is no non - photographable area.

[0060] Note that when the non - photographable area is identified, the no - fly area identification unit 28 may output the non - photographable area to a user terminal or the like together with the flight path 45. Thereby, the user may be notified of the non - photographable area.

[0061] ***Effects of Embodiment 3*** As described above, the flight path generation device 10 according to Embodiment 3 generates the flight path 45 excluding the no - fly area 50 where the unmanned aircraft cannot fly. Thereby, it becomes possible to generate the flight path 45 that avoids the objects existing around the object 41.

[0062] In addition, as a result of generating the flight path 45 excluding the no - fly area 50, when there is a non - photographable area that cannot be photographed, the flight path generation device 10 according to Embodiment 3 identifies the non - photographable area. Thereby, it becomes possible to consider countermeasures and the like.

[0063] ***Other configurations*** <Modification Example 4> In Embodiment 3, when the shooting position 44 is included in the modified range, the shooting area 43 to be shot from the shooting position 44 is specified as a non-shootable area. However, there may be a case where the shooting area 43 shot from the shooting position 44 included in the modified range can also be shot from the modified flight path 45. Therefore, it may be determined whether the shooting area 43 shot from the shooting position 44 included in the modified range can be shot from the modified flight path 45, and then the non-shootable area may be specified. Specifically, when the shooting position 44 is included in the modified range, the non-shootable area specifying unit 28 sets the shooting position 44 as the target shooting position 44. When shooting the shooting area 43 shot from the target shooting position 44, the non-shootable area specifying unit 28 determines whether a position satisfying the shooting criterion is included in the flight path 45. When a position satisfying the shooting criterion is included in the flight path 45, the non-shootable area specifying unit 28 sets that position as the shooting position 44 of the shooting area 43. On the other hand, when a position satisfying the shooting criterion is not included in the flight path 45, the non-shootable area specifying unit 28 specifies the shooting area 43 as a non-shootable area.

[0064] The embodiments and modification examples of the present disclosure have been described above. Some of these embodiments and modification examples may be combined and implemented. Also, any one or some of them may be partially implemented. Note that the present disclosure is not limited to the above embodiments and modification examples, and various changes can be made as needed.

Description of Reference Numerals

[0065] 10 Flight path generation device, 11 Processor, 12 Memory, 13 Storage, 14 Communication interface, 21 Fixation time determination unit, 22 Shooting area determination unit, 23 Shooting position determination unit, 24 Flight path generation unit, 25 Clustering unit, 26 Parameter change unit, 27 Flight permission determination unit, 28 No-fly area determination unit, 31 3D image data, 32 Line-of-sight information, 33 Surrounding information, 41 Object, 42 Fixation time, 43 Shooting area, 44 Shooting position, 45 Flight path, 46 Mesh data, 47 Shooting target mesh, 48 Cluster, 49 Representative position, 50 No-fly area.

Claims

1. A shooting area identification unit identifies each of the multiple meshes that the worker is fixated on from mesh data, which represents the surface of the object as a mesh of multiple planes of a standard shape, based on line-of-sight information indicating the worker's gaze during the visual inspection work, and A shooting position identification unit for identifying a shooting position for photographing the shooting area identified by the shooting area identification unit, wherein for each cluster obtained by clustering the line of sight information, the shooting area indicated by representative line of sight information classified in that cluster is designated as a representative mesh, and the shooting position for photographing the representative mesh is designated as a representative position, or, for each cluster obtained by clustering the shooting positions, the shooting position indicated by representative shooting positions classified in that cluster is designated as a representative position, and the shooting area corresponding to the representative position is designated as a representative mesh, A flight path generation unit generates a flight path for an unmanned aerial vehicle to photograph the shooting area with a camera mounted on the unmanned aerial vehicle, using the representative position identified by the shooting position identification unit. For each cluster, if, when the representative position identified for that cluster by the shooting position identification unit photographs the line of sight information classified for that cluster or other shooting areas identified from the shooting position, an area that does not meet the shooting criteria is included, the parameter modification unit modifies the clustering parameters, and if the angle formed by the perpendicular line drawn from the representative position to the representative mesh and the perpendicular line drawn from the representative position to the mesh that is the other shooting area is greater than the upper limit, the parameter modification unit determines that an area that does not meet the shooting criteria is included. A clustering unit that redoes clustering according to the parameters changed by the parameter changing unit. A flight path generation device equipped with the following features.

2. The aforementioned flight path generation device further, A gaze time identification unit identifies a gaze time for which the amount of change in gaze direction indicated by the gaze information is below a threshold, and the period is equal to or greater than the reference period. Equipped with, The shooting area identification unit identifies the area indicated by the line of sight during the gaze time identified by the gaze time identification unit as the shooting area. The flight path generation device according to claim 1.

3. The shooting position identification unit identifies a position as the shooting position that is located a reference distance away in the normal direction from the mesh identified as the shooting area. The flight path generation device according to claim 1.

4. The aforementioned shooting position identification unit identifies the direction of the perpendicular line drawn from the shooting position to the mesh as the shooting direction from the shooting position. The flight path generation unit generates the flight path that indicates the shooting direction from the shooting position. The flight path generation device according to claim 3.

5. The aforementioned flight path generation device further, Flight feasibility determination unit determines whether the flight path generated by the flight path generation unit includes an area where the unmanned aircraft cannot fly. Equipped with, If the flight feasibility determination unit determines that the flight path includes an area where flight is prohibited, the flight path generation unit modifies the flight path to avoid the area where flight is prohibited. The flight path generation device according to claim 1.

6. The aforementioned flight path generation device further, If the flight feasibility determination unit determines that the flight path includes an area where flight is not permitted, the unpermitted area identification unit identifies the areas within the shooting area that cannot be photographed using the modified flight path. The flight path generation device according to claim 5, comprising:

7. The computer uses line-of-sight information indicating the worker's gaze during the visual inspection process to identify each of the meshes that the worker focused on, from among the mesh data representing the surface of the object as a mesh of multiple planes of a standard shape, as the shooting area. The computer identifies a shooting position for capturing the shooting area, and for each cluster obtained by clustering the line-of-sight information, it designates the shooting area indicated by representative line-of-sight information classified in that cluster as a representative mesh, and identifies a shooting position for capturing the representative mesh as a representative position, or, for each cluster obtained by clustering the shooting positions, it designates a representative shooting position among the shooting positions classified in that cluster as a representative position, and identifies the shooting area corresponding to the representative position as a representative mesh. The computer uses the representative position to generate a flight path for the unmanned aerial vehicle to photograph the shooting area with a camera mounted on the unmanned aerial vehicle. When the computer captures the line-of-sight information classified in the cluster or other shooting areas identified from the shooting position for each cluster from the representative position identified for that cluster, if areas that do not meet the shooting criteria are included, the clustering parameters are changed, and if the angle between the perpendicular line drawn from the representative position to the representative mesh and the perpendicular line drawn from the representative position to the mesh that is the other shooting area is greater than the upper limit, it is determined that areas that do not meet the shooting criteria are included. A method for generating flight paths in which a computer re-performs clustering according to the modified parameters.

8. A shooting area identification process that identifies each of the multiple meshes that the worker is fixated on as a shooting area from mesh data, which represents the surface of the object as a mesh of multiple planes of a standard shape, based on line-of-sight information indicating the worker's gaze during the visual inspection work, and A shooting position identification process for identifying a shooting position for photographing the shooting area identified by the shooting area identification process, wherein for each cluster obtained by clustering the line of sight information, the shooting area indicated by representative line of sight information classified in that cluster is designated as a representative mesh, and a shooting position for photographing the representative mesh is designated as a representative position, or, for each cluster obtained by clustering the shooting positions, the shooting position indicated by representative shooting position classified in that cluster is designated as a representative position, and a shooting area corresponding to the representative position is designated as a representative mesh, A flight path generation process that generates a flight path for the unmanned aerial vehicle to photograph the shooting area with a camera mounted on the unmanned aerial vehicle using the representative position identified by the shooting position identification process, For each cluster, if, when photographing the line-of-sight information classified in that cluster or other shooting areas identified from the shooting position, as determined by the shooting position identification process for that cluster, an area that does not meet the shooting criteria is included, a parameter modification process is performed to modify the clustering parameters, wherein if the angle formed by the perpendicular line drawn from the representative position to the representative mesh and the perpendicular line drawn from the representative position to the mesh that is the other shooting area is greater than the upper limit, it is determined that an area that does not meet the shooting criteria is included. A clustering process that redoes clustering according to the parameters changed by the parameter modification process, and A flight path generation program that makes a computer function as a flight path generation device.

Images