Flight route normalization device, flight route normalization method, and flight route normalization program

Abstract

To normalize flight path data of a UAV represented by relative coordinates with a preset reference point as an origin into flight path data of absolute coordinates without manual operation.SOLUTION: The image recognition unit 132 recognizes a predetermined part from the image data obtained from the UAV. A pixel latitude and longitude conversion part 133 converts a pixel position constituting the image of the recognized prescribed part into latitude and longitude in consideration of the height to the prescribed part, the photographing position and the azimuth information. Based on the obtained latitude and longitude, the reference point identifying unit 134 identifies the latitude and longitude of the reference point. A flight route normalization part 138 converts the relative flight route data of a relative flight route data file 109 into automatic flight data of absolute coordinates by normalization processing composed of rotational conversion and parallel movement based on a reference azimuth with the latitude and longitude of a reference point as an origin.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] This invention relates to an apparatus, method, and program for normalizing the flight path of a UAV (Unmanned Aerial Vehicle) expressed in relative coordinates with a predetermined reference point as the origin to a flight path expressed in absolute coordinates (latitude, longitude). [Background technology]

[0002] Various technologies have been devised to enable appropriate use of data in different coordinate systems. For example, Patent Document 1, described below, discloses an invention relating to an image processing method for integrating map data in an XY coordinate system with satellite image data in a line-pixel coordinate system. The invention disclosed in Patent Document 1 converts map data in an XY coordinate system into a unified coordinate system, and also converts satellite image data in a line-pixel coordinate system into the unified coordinate system, finds the difference between the two converted data, and embeds the satellite image data to update the map data.

[0003] Furthermore, Patent Document 2, which will be described later, discloses an invention relating to a position interpretation device that quickly and accurately compares a map of the affected area before the disaster with the current damage situation in real time from images taken by a camera mounted on an aircraft. The invention disclosed in Patent Document 2 drops and installs multiple markers that wirelessly transmit their own positions in the affected area. Based on the marker position information from the camera image, a corrected image is generated in which geometric correction, scale correction, and orientation correction have been performed, and this is then overlaid on the map image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 9-15331 [Patent Document 2] Japanese Patent Application Laid-Open No. 2004-157397 Summary of the Invention [Problem to be solved by the invention]

[0005] In recent years, one example of a situation where data in different coordinate systems can be used appropriately is the normalization of a UAV flight path expressed in relative coordinates to a flight path expressed in absolute coordinates. UAVs, also known as drones, are unmanned aerial vehicles equipped with cameras that make them ideal for inspecting high-altitude structures. For this reason, UAVs are being used to inspect a variety of tall structures, such as gantry cranes and transfer cranes used to load and unload containers in port areas, steel towers supporting overhead (elevated) power lines, bridges, and buildings.

[0006] Gantry cranes and transfer cranes are mobile common structures, meaning that they can be moved within a limited area such as a port area or container yard, and there are multiple structures with similar structures. Steel towers for overhead power transmission lines are fixed common structures, meaning that they are located in fixed positions, but there are multiple structures with similar structures. Bridges and buildings are often fixed unique structures, meaning that they are located in fixed positions and are generally unique in themselves.

[0007] In the case of fixed unique structures such as bridges, it is conceivable to determine the photographing position (latitude, longitude, altitude (height)) for photographing the designated inspection area, photograph the inspection area from that photographing position, and classify the photographed images based on the photographing position. However, in the case of mobile common structures, since they move even within a limited area, it is not possible to uniquely determine the photographing position. For this reason, for mobile common structures, it becomes necessary to determine the photographing position each time they are inspected.

[0008] Furthermore, when there are multiple movable common structures, their respective locations are different, so even though the common structures have similar structures and the inspection areas are common, it is necessary to determine a photography position for each movable common structure and inspect them, which is time-consuming.Similarly, in the case of fixed common structures, even though the common structures have similar structures and the inspection areas are common, it is necessary to determine a photography position for each fixed common structure and inspect them, which is time-consuming, because the locations of the structures are different.

[0009] For this reason, in the case of a mobile common structure or a fixed common structure, it is possible to determine the shooting position (x coordinate, y coordinate, z coordinate) for photographing a specified inspection area using relative coordinates with a reference point set in advance according to a specific part of the structure as the origin. This allows the shooting position expressed in relative coordinates to be converted into absolute coordinates expressed by latitude, longitude, and height, if the latitude, longitude, and height of the reference point of the common structure can be measured and identified at the site where the common structure exists. Once the shooting position expressed in absolute coordinates is known, a UAV can be flown to the shooting position to take the photograph.

[0010] More specifically, the UAV's built-in RTK (Real Time Kinematic)-GNSS (Global Navigation Satellite System) function can be used to measure the latitude and longitude of a reference point, which can then be used to normalize the relative coordinates of the photographed position to the absolute coordinates of the photographed position. The UAV's RTK-GNSS positioning function can also be used to measure the latitude and longitude of two points on a common structure, and the azimuth angle can be calculated from the relative positions of the two points. For example, using the azimuth angle, if the common structure is a gantry crane, it is possible to determine in which direction the reach extends, or if the common structure is a steel tower for overhead power lines, it is possible to determine in which direction the power lines extend. This makes it possible to precisely identify areas where UAVs cannot fly.

[0011] However, with the above-described conventional technology, each time a UAV flies to inspect a common structure, the UAV pilot must go to the site and actually measure and identify the latitude, longitude, and altitude (height) of the reference point relative to the common structure to be inspected. This increases the man-hours required for inspecting the common structure and places a heavy burden on the UAV pilot and other workers involved in the inspection work. Furthermore, because the latitude, longitude, and altitude (height) of the reference point are measured by, for example, the UAV pilot (human), there are some concerns about the occurrence of measurement errors or measurement mistakes.

[0012] In view of the above, the present invention aims to enable UAV flight path data expressed in relative coordinates with a predetermined reference point as the origin to be normalized to flight path data in absolute coordinates without human intervention, for example, for the purpose of inspecting common structures, etc. [Means for solving the problem]

[0013] In order to solve the above problem, the flight path normalization device of the invention described in claim 1 comprises: a storage means for storing relative flight path data expressed in relative coordinates with a reference point set according to a predetermined portion of the target object as the origin; An acquisition means for acquiring from the UAV image data obtained by capturing images from the sky toward the ground using a camera mounted on the UAV, and image data including metadata including information on the image capturing position and orientation at the time of capturing the images; an image recognition means for recognizing the predetermined portion from the image data; a conversion means for converting pixel positions constituting the image of the predetermined portion into latitude and longitude in consideration of the height to the predetermined portion, the photographing position, and the orientation information; a reference point specifying means for specifying the latitude and longitude of the reference point based on the latitude and longitude obtained by the conversion means; a normalization means for converting the relative flight path data into automatic flight data in absolute coordinates by a normalization process consisting of rotational transformation and parallel translation based on a reference direction, with the latitude and longitude of the reference point as the origin; The present invention is characterized by comprising: [Effects of the Invention]

[0014] According to this invention, UAV flight path data expressed in relative coordinates can be quickly and accurately normalized to flight path data in absolute coordinates without manual intervention, thereby reducing the man-hours required for inspection work using UAVs and eliminating manual measurement errors and mistakes. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a diagram for explaining an example of the overall configuration of a UAV inspection system according to an embodiment. FIG. [Figure 2] 10A and 10B are diagrams for explaining the correction process of the relative altitude of a UAV according to an embodiment. [Figure 3] FIG. 10 is a diagram showing an example of a photographed image of a ground area including a gantry crane, which is a mobile common structure according to an embodiment. [Figure 4] 10A and 10B are diagrams for explaining a method of converting the position of a pixel in a captured image into latitude and longitude according to an embodiment. [Figure 5] 1 is a diagram for explaining an overview of a UAV and a flight path normalization device according to an embodiment. FIG. [Figure 6] 1 is a block diagram for explaining a configuration example of a UAV according to an embodiment. [Figure 7] FIG. 1 is a block diagram illustrating an example of the configuration of a flight path normalization device according to an embodiment. [Figure 8] FIG. 10 is a diagram for explaining an example of a flight path represented by relative coordinates of a gantry crane that is an inspection target in the embodiment. [Figure 9] 10 is a diagram for explaining an example of data stored in a relative flight path data file of the flight path normalization device according to the embodiment. FIG. [Figure 10] 10 is a diagram for explaining an example of data stored in an automatic flight path data file of the flight path normalization device according to the embodiment. FIG. [Figure 11]FIG. 2 is a diagram for explaining the normalization process of the flight path normalization device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, with reference to the drawings, embodiments of a flight path normalization device, a flight path normalization method, and a flight path normalization program according to the present invention will be described. The flight path normalization device, flight path normalization method, and flight path normalization program of the embodiments described below normalize the flight path of a UAV when inspecting a target object using the UAV. Furthermore, target objects inspected using a UAV include various tall structures such as gantry cranes, transfer cranes, bridges, steel towers, wind turbine towers for wind power generation, and Ferris wheels. In the embodiments described below, the target object will be described as an example, using a gantry crane installed in a port area.

[0017] [Example of overall configuration of UAV inspection system] FIG. 1 is a diagram illustrating an example of the overall configuration of a UAV inspection system according to an embodiment. As shown in FIG. 1, a gantry crane 4 is installed near a quay in a port area where ships (container ships) can dock. The gantry crane 4 loads and unloads containers onto the container ship. Gantry cranes come in various types, such as 17-row and 22-row gantry cranes, based on the number of containers that can be loaded widthwise onto the container ship. A 22-row gantry crane is a large steel structure with a total height of approximately 80 m, a total length of approximately 148 m, and an outreach portion extending out toward the sea that is approximately 63 m long. In this embodiment, the gantry crane 4 is assumed to be, for example, a large 22-row gantry crane. Because the gantry crane 4 faces the sea, it is susceptible to salt damage. Therefore, periodic inspections are important to properly monitor the state of rust and aging.

[0018] For simplicity's sake, only gantry crane 4 is shown in Figure 1. However, multiple gantry cranes with the same structure as gantry crane 4 are installed in the port area, allowing them to simultaneously load and unload containers onto multiple container ships. The gantry cranes installed in the port area can also move along rails installed along the quay. In other words, the installation positions of the gantry cranes installed in the port area, including gantry crane 4, are not fixed, but can be moved along the rails as needed.

[0019] Furthermore, as shown in Figure 1, container groups 5 formed by stacking containers that are about to be loaded onto a container ship or containers that have been unloaded from a container ship may exist near the gantry crane 4. The location and size of container groups 5 are fluid, but since they exist near the gantry crane 4, they may become obstacles to UAV flight and therefore must be kept in mind.

[0020] In FIG. 1, UAV 2 is used to inspect a gantry crane 4. In this embodiment, UAV 2 basically performs inspections by flying automatically (autonomously) based on flight path data created in advance. UAV 2 receives necessary information, such as flight path data, from devices installed in port office 1B and inspection management company 1A via IoT platform 3. UAV 2 can also be remotely controlled from a communication device in port office 1B via IoT platform 3, or from a communication device in the office of inspection management company 1A, which is located further away, via IoT platform 3. Of course, UAV 2 can also be controlled using a controller while a pilot visually checks its position, orientation, etc.

[0021] In this embodiment, the port office 1B is located, for example, several hundred meters or more, and in some cases, several kilometers or more, away from the port area where the gantry crane 4 is installed. Also, in this embodiment, the office of the inspection management company 1A is located several tens of kilometers or more, and in some cases, several hundred kilometers or more, away from the port area.

[0022] The IoT platform 3 includes the Internet, mobile phone networks, public telephone networks, wireless LANs (Local Area Networks), etc., and provides an environment in which devices connected to it can communicate with each other. Communication can be performed through the IoT platform 3 using the Internet Protocol (IP), the network protocol that realizes the Internet. Therefore, through the IoT platform 3, each device can use its IP address to notify the other party of its location on the IoT platform 3 and identify the other party of communication on the IoT platform 3. In Figure 1, the base station BS(1) is a base station for a mobile phone network, and as is well known, there are actually many of them.

[0023] In this embodiment of the UAV inspection system, as shown in FIG. 1, UAV 2 takes off from, for example, near port office 1B. It then flies to the port area where gantry crane 4 is located and automatically flies around gantry crane 4 to take photographs for inspection. For this purpose, flight path normalization device 1 is installed in port office 1B and the office of inspection management company 1A. UAV 2 and flight path normalization device 1 communicate with each other to cooperate and automatically perform pre-processing for automatic inspection flight. Of course, if necessary, it is also possible to switch to manual flight via remote control from a distance. Furthermore, flight can be performed by automatically setting a sequential route for inspection of multiple gantry cranes in the port area.

[0024] Pre-processing for automated inspection flight includes a relative altitude correction process and a process for normalizing relative flight path data to absolute flight path data. The relative altitude correction process corrects the relative altitude discrepancy that occurs when UAV2 flies in relative altitude mode due to differences in altitude between the takeoff point near Port Office 1B and the port area, as shown in Figure 1. The normalization process for relative flight path data to absolute flight path data normalizes the relative flight path data expressed in relative coordinates to absolute flight data expressed in absolute coordinates consisting of latitude, longitude, and altitude. Here, normalization refers to transforming data based on certain rules (regulations) to make it easier to use.

[0025] That is, the flight path normalization device 1 is provided with relative flight path data expressed in relative coordinates with a reference point preset according to a specific part of the gantry crane 4 as the origin so that it can be applied to gantry cranes of the same structure. This relative flight path data can be generated in advance using, for example, a computer-aided design (CAD) device. While viewing a three-dimensional computer graphics image of the gantry crane displayed on the CAD device's display, a user can operate a pointing device such as a mouse or a keyboard to set the intended shooting position, shooting direction, camera angle, and so on. Hereinafter, three-dimensional computer graphics images will be abbreviated as 3D images.

[0026] Therefore, in the UAV inspection system of this embodiment, the latitude, longitude, and altitude of a reference point that is preset according to a predetermined portion of the gantry crane 4 are identified from an image of the ground area including the gantry crane 4 captured by the camera of the UAV 2. The process of normalizing the relative flight path data to absolute flight path data based on the latitude, longitude, and altitude of this identified reference point is the normalization process of the relative flight path data to absolute flight path data. Below, we first describe the relative altitude correction process and the normalization process of the relative flight path data to absolute flight path data, which are pre-processing steps for automatic flight of the UAV 2 for inspection, and then describe configuration examples of the UAV 2 and the flight path normalization device 1.

[0027] In the embodiment described below, for example, inspection of a gantry crane 4 installed in a port area such as Kobe or Yokohama is performed by flying a UAV 2 from near a port office 1B. In this case, the UAV 2 communicates with a flight path normalization device installed in the office of an inspection management company 1A, for example, in Tokyo, to perform the above-mentioned pre-processing and receive automatic flight path data normalized to absolute coordinates. The UAV 2 is also assumed to have been transported to the port office 1B in advance and prepared for flight. The UAV 2 can be transported by land by vehicle, or by air, automatically flying the UAV 2 from a predetermined location.

[0028] Relative altitude correction FIG. 2 is a diagram for explaining the relative altitude correction process of UAV2 according to the embodiment. The altimeter included in UAV2 detects differences in air pressure due to height (altitude) to determine the altitude. Therefore, as shown in FIG. 2(A), if the state in which UAV2 is placed on the ground in takeoff / landing area Ar1 is set to an altitude of 0 m (meters), the altitude can be accurately measured in takeoff / landing area Ar1, which is at the same altitude, based on the air pressure that changes with altitude. In this way, the altitude measured in relative altitude mode is called relative altitude because it is measured, for example, with respect to the takeoff point.

[0029] However, as shown in Figure 2(A), suppose UAV2 flies to inspection flight area Ar2 on the sea side, which is at a lower altitude than takeoff / landing area Ar1. In this case, the relative altitude measured by the altimeter of UAV2 is determined based on takeoff / landing area Ar1, and therefore will be off by αm, the altitude difference (difference) between takeoff / landing area Ar1 and inspection flight area Ar2. In other words, suppose that the flight altitude of UAV2 is measured as 80m in takeoff / landing area Ar1. In this case, when UAV2 reaches inspection flight area Ar2, the flight altitude of UAV2 is actually 95m, but is erroneously measured as 80m. In this case, an error of 15m occurs in the relative altitude in inspection flight area Ar2.

[0030] Therefore, UAV2 acquires a measured altitude H and a relative altitude R near the gantry crane 4. The difference between the measured altitude H and the relative altitude R is made available as a correction value for the relative altitude R. That is, as shown in FIG. 2(B), UAV2 measures and acquires the measured altitude H using its own laser ranging function (RTG function) (step S1). At the same time, UAV2 measures and acquires the relative altitude R using its own altimeter (barometer) (step S2). After that, the difference α is calculated by subtracting the relative altitude R from the measured altitude H (step S3), and this difference α is made available as a correction value for the relative altitude R (step S4). As a result, even if the altitudes of the inspection flight area Ar2 and the takeoff / landing area Ar1 are different, the relative altitude measured by the altimeter can be corrected to an accurate value.

[0031] 2(B) has been described as being performed by UAV2, but this is not limited to this. The processes of steps S1 and S2 in FIG. 2(B) are performed by the UAV. However, the measured altitude H and relative altitude R acquired through steps S1 and S2 may be transmitted to flight path normalization device 1, and flight path normalization device 1 may determine the difference α between the measured altitude H and the relative altitude R and use this as a correction value for correcting the relative altitude R from UAV2.

[0032] [Normalization of relative flight path data to absolute flight path data] 3 and 4 are diagrams for explaining the process of normalizing relative flight path data to absolute flight path data performed by flight path normalization device 1. Specifically, FIG. 3 is a diagram showing an example of a captured image TP of a ground area including a gantry crane, which is a mobile common structure in the embodiment. FIG. 4 is a diagram for explaining a method of converting the positions of captured image pixels into latitude and longitude in the embodiment. As described above, reference point P, which is the origin of the relative flight path data, is set according to a predetermined part of the gantry crane to be inspected. In this embodiment, the center point of the top surface (roof portion) of the machine room of the gantry crane is used as reference point P.

[0033] As shown in FIG. 1, in the case of the UAV inspection system of this embodiment, when UAV 2 takes off from near port office 1B and arrives at the port area, UAV 2 captures images of the ground area including the gantry crane 4 to be inspected. In this case, UAV 2 points the gimbal of its camera mounted on the aircraft straight down (at -90 degrees) and captures images of the ground area including the entire gantry crane 4. The capturing altitude is determined based on the overall height of the gantry crane, and the ground area is captured from an altitude of, for example, 90 m to 130 m. UAV 2 transmits the captured image data to flight path normalization device 1 of inspection management company 1A via IoT platform 3.

[0034] The photographic data transmitted from UAV2 to the flight path normalization device 1 of inspection management company 1A includes image data obtained by photographing and Exif (Exchangeable Image File Format) information, which is metadata for the image data. In other words, the image data is information that forms the photographed photographic image TP. The Exif information also includes various information, such as location information and orientation information at the time of photographing, obtained from the GPS (Global Positioning System) unit installed in UAV2. Based on the Exif information of the photographic data provided by UAV2, flight path normalization device 1 obtains the azimuth angle of the upward direction of the photographic image, indicated by the arrow AZ in FIG. 3. The azimuth angle is measured clockwise, with north being 0 degrees. This allows each direction in the photographic image to be identified based on the upward direction of the photographic image.

[0035] Furthermore, the flight path normalization device 1 performs image recognition on the captured image TP formed from the image data of the shooting data provided by the UAV 2, and extracts (identifies) the machinery room 41 of the gantry crane 4. Next, the flight path normalization device 1 performs a process of converting the position of each pixel constituting the machinery room 41 of the gantry crane 4 extracted by image recognition into latitude and longitude. That is, the captured image TP is composed of a large number of pixels (picture elements) arranged at equal intervals, as shown in FIG. 4(A). Generally, images captured by a camera are captured and displayed in a perspective view. In an image captured in a perspective view, distant objects are displayed smaller than closer objects, thereby creating a sense of depth.

[0036] Therefore, since an image captured in perspective from the sky has depth, it is usually difficult to calculate the accurate latitude and longitude of, for example, the rooftop of a building. This is because the accurate latitude and longitude of the rooftop of the building cannot be calculated unless the exact height to the rooftop is known. Therefore, in the flight path normalization device 1 of this embodiment, as shown in Figure 4(B), a 3D space model is set in which the shooting position SP by the UAV 2 in the sky is set as the vertex, and the range of the angle of view indicated by the dotted line is photographed.

[0037] In this case, as shown in Figure 4(B), the height h to the top surface of the machine room 41 of the gantry crane 4 is taken into consideration. Furthermore, the angle θ between the perpendicular line PL drawn from the photographing position SP in the sky to the ground and the line segment SL connecting the photographing position SP in the sky to each pixel PX constituting the machine room of the gantry crane 4 in the photographed image is taken into consideration. The latitude and longitude of the perpendicular line PL are the same no matter which position it is taken from, except for the height. Therefore, by calculating the length N of the perpendicular line PP drawn from pixel PX to the perpendicular line PL, the latitude and longitude of pixel PX can be identified based on the latitude and longitude of the photographing position SP.

[0038] Specifically, as shown in FIG. 4(B), the length N of the perpendicular line PP is calculated using tan θ=N / Hh. In this case, the intersection of the perpendicular line PP and the perpendicular line dropped from the shooting position SP is used as a reference point, and the position of the pixel PX can be converted into latitude and longitude using the latitude and longitude of this intersection point, the orientation of the target pixel PX from the reference point, and the length of the perpendicular line PP. In other words, the latitude and longitude of the intersection point of the perpendicular line PP and the perpendicular line dropped from the shooting position SP are the same as those of the shooting position SP. Therefore, if the direction and distance of the target pixel PX from the intersection point are known, the position of the pixel PX can be converted into latitude and longitude.

[0039] In this way, the position of each pixel constituting the top surface of the machinery room 41 of the gantry crane 4 in the photographed image TP can be converted into latitude and longitude. Here, the letter "H" represents the altitude (height) of the photographing position SP measured by the UAV 2. The letter "h" represents the height to the top surface of the machinery room 41 of the gantry crane 4, which is information known from blueprints of the gantry crane 4, etc. Therefore, as described above, the height to the top surface of the machinery room 41 of the gantry crane 4 can be identified as known information in a 3D image of the gantry crane 4 when setting a flight path using relative coordinates.

[0040] Furthermore, as shown in FIG. 3, in a photographed image of the ground area including the gantry crane 4, the center point of the machine room 41 of the gantry crane 4, which is the reference point P set as the origin of the relative coordinates, is identified from the positions of the four corners of the machine room 41 recognized in the image. The latitude and longitude converted for the pixel corresponding to this identified center point can be used to identify the latitude and longitude of the reference point P, which is the center point of the machine room 41, and further, the altitude (height) h of the reference point P can be identified as known information. This makes it possible to identify the absolute coordinates (latitude, longitude, altitude (height)) of the reference point P, which is the origin (0,0,0) of the relative coordinates. Based on the absolute coordinates (latitude, longitude, altitude (height)) of the reference point P identified in this way, the flight path data expressed in relative coordinates is normalized to flight path data expressed in absolute coordinates.

[0041] In this way, if the absolute coordinates of reference point P can be identified for each gantry crane to be inspected, the relative flight path data, which has already been prepared and expressed in relative coordinates, can be normalized to automatic flight path data, which is expressed in absolute coordinates for each gantry crane to be inspected. Therefore, automatic flight path data in absolute coordinates can be prepared for each gantry crane to be inspected, allowing inspections to be performed automatically. Moreover, the latitude and longitude of reference point P can be identified without human intervention, eliminating the possibility of measurement errors or room for measurement mistakes.

[0042] In the photographed image TP shown in Figure 3, container groups CT located near the gantry crane 4 to be inspected are also extracted (identified) by image recognition and managed as obstacles to the flight of the UAV 2. In addition, in the photographed image TP shown in Figure 3, the rails R1 and R2 used for the movement of the gantry crane 4 to be inspected are also image recognized and used as information for identifying the extension direction of the reach of the gantry crane 4. In the photographed image TP shown in Figure 3, the straight line LN is the boundary between the sea and land.

[0043] [Overview of UAV2 and flight path normalizer 1] FIG. 5 is a diagram for explaining an overview of a UAV 2 and a flight path normalization device 1 according to an embodiment. The UAV 2, which will be described in detail later, includes a GNSS unit, various sensor units, and a camera unit 210. By including the GNSS unit and various sensor units, the UAV 2 is capable of automatic flight (autonomous flight). As shown enclosed by a dotted line in FIG. 5, the camera unit 210 is fixed to a support 210P, and is capable of tilting as indicated by arrow a and rolling as indicated by arrow b.

[0044] The tilt operation is an operation of changing the orientation of the camera unit 210 in the vertical (up and down) direction, and the camera unit 210 of the UAV 2 in this embodiment can tilt, for example, 45 degrees upward (+45 degrees) and 90 degrees downward (-90 degrees) from the 0 degree position (reference position). The roll operation is an operation of rotating the camera unit 210 around the optical axis of the camera unit 210, and the camera unit 210 of the UAV 2 in this embodiment can roll 90 degrees to the left and right (±90 degrees) from the 0 degree position (reference value).

[0045] As indicated by the arrow pointing from UAV 2 to flight path normalization device 1 in Fig. 5, UAV 2 acquires the measured altitude H and relative altitude R described with reference to Fig. 2 and provides them to flight path normalization device 1. UAV 2 also provides imaging data obtained by photographing the ground area including gantry crane 4 to be inspected to flight path normalization device 1. As described above, the imaging data includes image data obtained by photographing the inspection portion of gantry crane 4 to be inspected, and Exif information consisting of various metadata such as location information (latitude, longitude, altitude) at the time of imaging and orientation information at the time of imaging.

[0046] In this embodiment, the flight path normalization device 1 is installed in the office of the inspection management company 1A as described above, and includes an altitude correction value calculation unit 120 and an automatic flight path creation unit 130. As described with reference to FIG. 2, the altitude correction value calculation unit 120 realizes the function of subtracting the relative altitude R from the measured altitude H to calculate a correction value for the relative altitude R. Furthermore, the automatic flight path creation unit 130 generates necessary data while referencing the photographic data from the UAV 2 and other reference files, and stores the data in the generated file.

[0047] Furthermore, the automatic flight path creation unit 130 calculates the absolute coordinates (latitude, longitude, altitude) of the reference point P while referring to the photographic data from the UAV 2 and other reference files, and normalizes the flight path data expressed in relative coordinates to flight path data expressed in absolute coordinates. The normalized flight path data is stored in the automatic flight path data file 110, and then provided to the UAV 2 as shown by the arrow from the flight path normalization device 1 to the UAV 2 in Figure 5. As a result, the UAV 2 flies automatically (autonomously) according to the automatic flight path data and photographs the inspection location of the gantry crane 4 that is the inspection target.

[0048] In this way, the UAV 2 and flight path normalization device 1 work together to enable the UAV 2 to automatically fly around the gantry crane 4 to be inspected and perform the inspection. In particular, as pre-processing to enable the inspection of the gantry crane 4 to be performed by automatic flight, a series of processes are performed, including calculating a correction value for the relative altitude R and normalizing flight path data expressed in relative coordinates to flight path data expressed in absolute coordinates. Below, an example configuration of the UAV 2 and flight path normalization device 1 will be described.

[0049] [UAV2 configuration example] FIG. 6 is a block diagram illustrating an example configuration of a UAV2 according to an embodiment. There are various types of UAVs, broadly categorized into fixed-wing types with fixed blades like a typical airplane, helicopter types with one rotor, and multicopter types with three or more rotors. Helicopter and multicopter UAVs are capable of hovering, flying so as to remain in a fixed position in the air, making them suitable for inspection work at high altitudes. As shown in FIG. 5, the UAV2 according to this embodiment is a quadcopter type with four rotors.

[0050] As shown in Fig. 6, the UAV 2 comprises a flight mechanism 21 consisting of rotors and their drive units, and a drive control unit 22. The UAV 2 is capable of both remotely controlled flight and automatic flight (autonomous flight) according to set flight instruction data. In Fig. 2, the transmitting / receiving antenna 201A and wireless communication unit 201 transmit and receive information to and from the flight path normalization device 1, which will be described in detail later, for example, to receive automatic flight path data from the flight path normalization device 1. The transmitting / receiving antenna 201A and wireless communication unit 201 are also used when communicating with a remotely controlled device.

[0051] Although not shown, the control unit 202 is a microprocessor configured by connecting a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), non-volatile memory, etc. via a bus, and controls each part of the UAV 2. The storage device 203 is composed of a storage medium such as an SSD (Solid State Drive) and its driver, and stores various programs and various data required for processing, as well as being used as a working area for temporarily storing intermediate results of various processes. As described above, the storage device 203 also stores automatic flight path data from the flight path normalization device 1, etc., for use during automatic flight.

[0052] The shooting data file 204 is created in a storage medium of a predetermined storage device, and stores and holds shooting data including image data obtained by shooting using the camera unit 210 (described later) and Exif information consisting of various metadata for the image data. Specifically, the shooting data consists of image data and various metadata. The metadata includes the shooting position (latitude, longitude, and altitude at the time of shooting), movement information at the time of shooting (the orientation of the camera unit 210 at the time of shooting), and various information such as the tilt angle α at the time of shooting, the tilt pitch angle at the time of shooting, the roll angle β at the time of shooting, and the roll pitch angle at the time of shooting), and orientation information at the time of shooting (such as the azimuth angle of the direction directly above the captured image).

[0053] The image data forms a captured image obtained by capturing an image using the camera unit 210 of the UAV2. The capturing position is the latitude, longitude, and altitude indicating the current position of the UAV2 at the time of capturing an image, which is determined with high accuracy using a current position positioning means described below. The current position of the UAV2 can be considered the current position of the camera unit 210. The tilt angle α during capturing, which is the movement information of the camera unit 210 during capturing, is the angle when the orientation of the camera unit 210 is changed in the vertical (up and down) direction. Furthermore, the roll angle β during capturing, which is the movement information of the camera unit 210 during capturing, is the angle when the camera unit 210 is rotated around the optical axis of the camera unit 210. These pieces of movement information during capturing can be acquired from the camera unit 210.

[0054] Furthermore, the orientation of the camera unit 210 is information used when the UAV2 flies autonomously, and indicates the direction in which the camera unit 210 faces when the tilt angle α and roll angle β are both 0 degrees; in other words, the direction in which the front of the UAV2 faces. The tilt pitch angle is the tilt of the UAV2 forward and backward (tilt in the pitch direction), and the roll pitch angle is the tilt of the UAV2 left and right (tilt in the roll direction). The orientation, tilt pitch angle, and roll pitch angle of the camera unit 210 can be acquired through sensors mounted on the UAV2. Note that the tilt pitch angle and roll pitch angle can be used to correct the tilt angle α at the time of shooting and the roll angle β at the time of shooting, taking into account the tilt of the UAV2 itself. The orientation information at the time of shooting is the azimuth angle of the directly upward direction of the captured image (angle measured clockwise with north as 0 degrees), and using this as a reference, each direction in the captured image can be accurately determined. The orientation information at the time of shooting can also be acquired through sensors mounted on the UAV2.

[0055] The power supply unit 205 includes a battery and supplies the power required by each unit of the UAV2. The sensor unit 206 includes various sensors, such as a gyro sensor, a barometric pressure sensor, an acceleration sensor, an ultrasonic sensor, and a geomagnetic sensor. The gyro sensor is used for attitude control, and the barometric pressure sensor is used for altitude detection. The acceleration sensor is used for speed detection, and the ultrasonic sensor is used for distance detection to the target object. The geomagnetic sensor is used for direction detection. The gyro sensor and acceleration sensor form a six-axis sensor for forward / backward, left / right, and up / down, allowing for precise and appropriate attitude control of the UAV2.

[0056] The autonomous attitude control unit 207 controls the flight drive unit 209 so that the UAV 2 flies stably at an appropriate attitude, using the detection outputs from the various sensors mounted on the sensor unit 206 and the captured images from the camera unit 210. Note that since an ultrasonic sensor is used, it is not necessarily necessary to use the image information from the camera unit 210, but the image information from the camera unit 210 can also be used to check for obstacles. This information is particularly important during takeoff and landing.

[0057] The GNSS unit 208 and GNSS antenna 208A are components that realize the function of accurately detecting (locating) the current position (latitude and longitude) of the UAV by receiving and analyzing transmission signals (positioning information) from multiple artificial satellites. GNSS stands for Global Positioning Satellite System, and is a general term for satellite positioning systems such as GPS (Global Positioning System), GLONASS (Global Navigation Satellite System), Galileo, and Quasi-Zenith Satellite System (QZSS). The flight drive unit 209 supplies drive control signals to each of the four engine units of the four propeller mechanisms of the flight mechanism unit 21 under the control of the autonomous attitude control unit 207. This enables various movement operations, attitude control, and hovering position control of the UAV 2.

[0058] As described above, the camera unit 210 is attached to the support 210P so as to face one direction (forward) and is capable of tilting and rolling. It is also possible to configure the camera unit 210 so as to perform panning (movement to change orientation horizontally (left and right)). However, in the case of the UAV 2 of this embodiment, panning is not a function of the camera unit 210, but is realized by the UAV 2 itself changing its orientation. The camera unit 210 can change its tilt angle and roll angle and take images under the control of the control unit 202. The camera unit 210 also has a laser ranging function that emits laser light, receives the reflected light, and detects the phase difference to accurately measure the distance to an object that reflects the laser light. As a result, by pointing the camera unit 210 directly downward (tilt angle -90 degrees), emitting laser light, and receiving the reflected light, the distance (altitude) from the object that reflected the laser light to the UAV 2 can be accurately measured.

[0059] The flight control unit 211 cooperates with the autonomous attitude control unit 207 to control the flight drive unit 209 so that when the UAV 2 performs automatic flight, it receives data from the flight path normalization device 1 and flies along a flight route according to the automatic flight path data stored and held in the storage device 203. While the autonomous attitude control unit 207 mainly controls the attitude of the UAV 2, the flight control unit 211 controls the flight drive unit 209 so that the UAV 2 flies along a sequence of coordinate points including latitude, longitude, and altitude instructed by the automatic flight path data.

[0060] The current position correction unit 212 realizes positioning of the current position using the RTK method. The current position correction unit 212 transmits the current position of the aircraft, which has been determined via the GNSS unit 208, to a predetermined correction station on the IoT platform 3, for example, via the wireless communication unit 201 and the transmitting / receiving antenna 201A. Then, the current position correction unit 212 obtains correction information from the correction station and corrects the current position of the aircraft, thereby determining a more accurate current position of the aircraft. The correction station is provided with the current positions of other fixed stations determined by the fixed stations, and is capable of receiving information from a mobile station, such as the current position determined by the mobile station, and forming and providing correction information that corrects errors contained in the current position.

[0061] In addition, although the case where a predetermined correction station on the IoT platform 3 is used has been described here, the present invention is not limited to this. It is also possible to use a method in which a fixed station is placed near the UAV 2, the fixed station also measures the current position of the UAV, and information is exchanged between the UAV 2 and the fixed station via wireless communication, and errors contained in the measured current position are corrected to obtain a more accurate current position.

[0062] When an image is captured using the camera unit 210, the image capture log creation unit 213, under the control of the control unit 202, creates log data including the image capture position (current position at the time of capture), movement information of the camera unit 210, and orientation information. The image capture position includes the accurate current position (latitude, longitude) of the aircraft acquired by the current position correction unit 212 at the time of capture, and the altitude measured by the barometric pressure sensor of the sensor unit 206. The movement information of the camera unit 210 includes the tilt angle α at the time of capture and the roll angle β at the time of capture acquired from the camera unit 210, and the orientation of the camera unit 210 at the time of capture (orientation of the UAV 2), the tilt pitch angle, and the roll pitch angle acquired from the sensor unit 206. The orientation information includes the azimuth angle of the directly upward direction of the captured image, with north identified by the geomagnetic sensor of the sensor unit 206 as 0 degrees.

[0063] As a result, the control unit 202 can form shooting data by adding various other metadata, such as image data captured by the camera unit 210 and the above-mentioned shooting log information and status information of the camera unit 210, and record the data in the shooting data file 204. The UAV 2 having such a configuration can fly around the gantry crane, which is the inspection target, in accordance with the automatic flight path data described below, capture images of the inspection area, obtain the captured images, and record them in the shooting data file 204.

[0064] [Configuration example of flight path normalization device 1] FIG. 7 is a block diagram illustrating an example configuration of the flight path normalization device 1 according to an embodiment. The connection terminal 101T constitutes a connection terminal to the IoT platform 3. The communication unit 101 realizes the function of communicating through the IoT platform 3. The control unit 102, although not shown, is a microprocessor configured by connecting a CPU, ROM, RAM, nonvolatile memory, etc. via a bus, and controls each unit of the flight path normalization device 1. The storage device 103 is composed of a storage medium such as an SSD and its driver, and stores various programs and various data required for processing, as well as being used as a working area for temporarily storing intermediate results of various processes. The storage device 103 is also used to store and retain the measured altitude H, relative altitude R, and image data from the UAV 2 described above.

[0065] The inspection target 3D data file 104 stores and holds 3D image data of the gantry crane to be inspected. This 3D image data allows accurate determination of the size of various parts of the gantry crane, such as its overall height, overall length, and height to the top of the machinery room. The pixel latitude and longitude file 105 stores and holds, for each pixel, the converted latitude and longitude of each pixel constituting the top of the machinery room of the gantry crane 4, extracted by performing image recognition on the image data captured by the UAV 2, as described with reference to Figures 3 and 4.

[0066] The reference data file 106 stores and retains information indicating the latitude and longitude of linear objects such as the rails of the gantry crane 4 and the upward azimuth angle of the captured image, for example, to identify the orientation (direction of the extended reach) of the gantry crane 4 to be inspected. The linear object can be extracted by performing image recognition on the captured image formed from the image data included in the captured data from the UAV 2. The latitude and longitude of the linear object can be identified using a method similar to the method used to convert the latitude and longitude of each pixel constituting the top surface of the machine room of the gantry crane 4. Note that the position of a linear object can be determined by knowing the latitude and longitude of both ends of the object.

[0067] The obstacle management data file 107 stores and holds latitude, longitude, and height indicating the location of container groups and other structures, extracted by image recognition of the captured image formed from the image data included in the captured data from the UAV 2. The latitude and longitude indicating the location of container groups and other structures can be identified using a method similar to the method used to convert the latitude and longitude of each pixel that makes up the top surface of the machine room of the gantry crane 4. The location of container groups and other structures can be identified by the latitude and longitude of each vertex that can be recognized when viewed from above the container groups and other structures.

[0068] The height of a group of containers or other structures can be determined by subtracting the distance measured using the laser ranging function of the camera unit 210 of the UAV 2 from the altitude of the UAV 2. In the case of a group of containers, if the number of containers that can be stacked is determined, the height of the group of containers may be determined based on the maximum number that can be stacked. In the case of other structures that are other gantry cranes or structures installed on the ground, the height of the other structures may be determined based on 3D image data of the structures stored in the inspection target 3D data file, etc.

[0069] The no-fly area file 108 stores and holds data indicating no-fly areas in which the UAV 2 cannot fly when inspecting the gantry crane 4. As will be described later, the data indicating the no-fly areas takes into consideration data stored in the obstacle management data file 107, and stores and holds the latitude, longitude, and height of multiple points that can identify the outer edge of the three-dimensional area in which the UAV 2 cannot fly when inspecting the gantry crane 4.

[0070] The relative flight path data file 109 stores and holds relative flight path data expressed in relative coordinates. As described above, the relative flight path data can be created in advance using, for example, a 3D image of the gantry crane in the inspection target 3D data file 104. Figure 8 is a diagram for explaining an example of a flight path expressed in relative coordinates of the gantry crane that is the inspection target in this embodiment.

[0071] As shown in Figure 8, for example, a user can set the planned shooting position, shooting direction, camera angle, etc. by operating a pointing device such as a mouse or a keyboard while viewing a 3D image of the gantry crane 4 displayed on the display of the CAD device. In Figure 8, the center point of the top surface of the machine room 41 of the gantry crane 4 is set as the reference point P, and the shooting position is set in a relative coordinate system in which this reference point P is the origin (0,0,0).

[0072] In Figure 8, the positions indicated by the symbols RB, RS, FF, LS, LB, and BB are shooting positions, and are specified by relative coordinates with reference point P as the origin. That is, as shown in Figure 8, position RB is specified by relative coordinates (x1, y1, z1), and position RS is specified by relative coordinates (x2, y2, z2). Similarly, position FF is specified by relative coordinates (x3, y3, z3), and position LS is specified by relative coordinates (x4, y4, z4). Similarly, position LB is specified by relative coordinates (x5, y5, z5), and position BB is specified by relative coordinates (x6, y6, z6).

[0073] 8, dotted cubes are shown surrounding the imaging positions RB, RS, FF, LS, LB, and BB. This indicates that each of the imaging positions RB, RS, FF, LS, LB, and BB has a predetermined width in the positive and negative directions in the x, y, and z directions, and each of the imaging positions RB, RS, FF, LS, LB, and BB is identified as an area. The imaging positions are identified as areas with ranges in this way because UAV2 is usually susceptible to wind and has difficulty remaining in a fixed position by hovering, so the imaging positions are specified as ranges within a range that does not interfere with imaging.

[0074] In addition, in FIG. 8, the areas indicated by a1 to a8 are the inspection areas to be photographed. Therefore, it is shown that inspection areas a1, a2, a3, and a4 are photographed from photographing position RB, and inspection areas a5, a6, a7, and a8 are photographed from photographing position RS. In this way, the inspection area to be photographed for each photographing position, the camera direction (photographing direction), and the tilt angle α and roll angle β when photographing each inspection area are also set. Note that, as described above, photographing positions RB, RS, ... are specified as areas with ranges. Therefore, ranges are also set for the tilt angle α and roll angle β.

[0075] The relative flight path data set in this manner is stored in relative flight path data file 109. Fig. 9 is a diagram for explaining an example of data stored in the storage data file of relative flight path data file 109 of flight path normalization device 1 of the embodiment. As shown in Fig. 9, the relative flight path data stored in relative flight path data file 109 consists of "No.", "shooting position ID", "planned shooting position (relative coordinates)", "camera direction", "planned camera angle", and "inspection part ID".

[0076] "No." is a sequence number assigned to the relative flight path data, and in this embodiment, also indicates the shooting order. "Photographing position ID" is information for identifying the photographing position, and also serves as information that uniquely identifies each relative flight path data. "Planned photographing position (relative coordinates)" is information that indicates the planned photographing position in a relative coordinate system with reference point P as the origin (0,0,0), and consists of x-axis data, y-axis data, and z-axis data. As described above, each of the x-axis data, y-axis data, and z-axis data has a range in both the minus and plus directions.

[0077] The "camera orientation" refers to the direction in which the lens of the camera unit 210 is pointed when photographing an inspection site, and is represented as an azimuth angle in this embodiment. The "planned camera angle" consists of the tilt angle α and roll angle β of the camera unit 210 during photography. These tilt angle α and roll angle β are also expressed as ranges, for example, from -45 degrees to +45 degrees. The inspection site ID is information that identifies the inspection site to be photographed from the planned photography position. As shown in FIG. 9, multiple inspection sites may be photographed from a single planned photography position, and therefore multiple camera orientations and planned camera angles may be used for a single planned photography position, depending on the inspection site to be photographed. Although not shown in FIG. 9, other necessary information, such as the number of images to be photographed for each inspection site, can also be set.

[0078] Such relative flight path data is created in advance for the gantry crane to be inspected and stored in relative flight path data file 109. Note that, for simplicity of explanation, Figure 8 specifically illustrates six planned photography positions as part of the planned photography positions and eight inspection locations as part of the inspection locations. However, in reality, there are many inspection locations for a gantry crane, numbering in the hundreds, and therefore there will also be many planned photography positions depending on the inspection locations.

[0079] Automated flight path data expressed in absolute coordinates is stored in automated flight path data file 110. Fig. 10 is a diagram for explaining an example of data stored in automated flight path data file 110 of flight path normalization device 1 according to an embodiment. The automated flight path data stored in automated flight path data file 110 is basically the same data as the relative flight path data stored in relative flight path data file 109, but the planned shooting positions are normalized to absolute coordinates.

[0080] That is, the automatic flight path data stored in automatic flight path data file 110 consists of "No.", "Photographing position ID", "Planned photographing position (absolute coordinates)", "Camera direction", "Planned camera angle", and "Inspection part ID", as shown in Fig. 10. Each piece of information other than "Planned photographing position (absolute coordinates)" is the same as the corresponding information in the relative flight path data stored in relative flight path data file 109 described using Fig. 9.

[0081] However, as explained using Figures 3 and 4, the "planned photographing position (absolute coordinates)" is the relative flight path data expressed in relative coordinates by specifying the latitude, longitude, and altitude of reference point P, and normalizing the "planned photographing position (relative coordinates)" into absolute coordinates. In other words, the "planned photographing position (absolute coordinates)" is information indicating the planned photographing position in absolute coordinates based on the absolute coordinates (latitude, longitude, altitude) of reference point P, and is made up of latitude data (Lat), longitude data (Lon), and altitude data (AL). Note that each of the latitude data (Lat), longitude data (Lon), and altitude data (AL) has a range in both the minus and plus directions, just like the "planned photographing position (absolute coordinates)."

[0082] The automated flight path data in the automated flight path data file is not simply the "planned photographing position (relative coordinates)" of the relative flight path data normalized to absolute coordinates. Based on information in the no-fly area file 108, a "waypoint (absolute coordinates)" may be set in the same manner as the "planned photographing position (relative coordinates)" to prevent the flight path from entering an area that has become a no-fly area due to the influence of an obstacle or the like. Since no photographs will be taken at this "waypoint (absolute coordinates)," information such as "camera direction," "planned camera angle," and "inspection part ID" is not set.

[0083] The files 104 to 110 shown in Fig. 7 can be created in different storage areas in a single storage device such as an SSD, or can be created in different storage devices. Also, the files 104 to 110 shown in Fig. 7 can be created by distributing them across multiple storage devices.

[0084] The altitude correction value calculation unit 120 and the automatic flight path creation unit 130 are processing units that normalize relative flight path data expressed in relative coordinates to automatic flight path data expressed in absolute coordinates. Below, we will explain the components that make up the altitude correction value calculation unit 120 and the automatic flight path creation unit 130.

[0085] The altitude correction value calculation unit 120 performs a process to calculate a correction value for the relative altitude R. That is, as described with reference to FIG. 2, when the UAV2 arrives at the inspection flight area Ar2, it acquires the measured altitude H using the laser ranging function and the relative altitude R using the barometer of the sensor unit 206, and transmits this information to the flight path normalization device 1. The flight path normalization device 1 receives this information through the connection terminal 101T and the communication unit 101, and the control unit 102 records it in the storage device 103. Under the control of the control unit 102, the altitude correction value calculation unit 120 calculates the relative altitude R from the measured altitude H stored in the storage device 103 to obtain a difference α, and records this in the storage device 103 so that it can be used as a correction value for the relative altitude R.

[0086] Each unit of the automatic flight path creation unit 130 functions when image data obtained by photographing the ground area including the gantry crane 4 to be inspected from a predetermined altitude is transmitted from the UAV 2 that has arrived at the inspection flight area Ar2. Each unit of the automatic flight path creation unit 130 performs a series of processes for normalizing the relative flight path data into automatic flight path data in absolute coordinates. Here, each unit of the automatic flight path creation unit 130 will be described with reference to the flowchart of FIG. 11 for explaining the normalization process performed by the automatic flight path creation unit 130 of the flight path normalization device 1 of the embodiment.

[0087] The photographic data obtained by photographing the ground area including the gantry crane 4 to be inspected is received (acquired) through the connection terminal 101T and the communication unit 101 of the flight path normalization device 1, and is recorded in the storage device 103 by the control unit 102 (step S101). Thereafter, under the control of the control unit 102, the azimuth angle acquisition unit 131 functions to acquire the upward azimuth angle of the photographed image from the Exif information of the photographic data recorded in the storage device 103, and records the acquired azimuth angle in the reference data file 106 (step S102).

[0088] The image recognition unit 132 uses image recognition to extract (identify) the machine room 41 of the gantry crane 4 and other necessary objects in the captured image formed by the image data of the captured data recorded in the storage device 103 (step S103). The other necessary objects extracted in step S103 are, for example, containers and structures located around the gantry crane 4 that may obstruct the flight of the UAV 2.

[0089] Furthermore, the image recognition unit 132 uses image recognition to extract (identify) linear objects, such as rails for the movement of the gantry crane 4, in the captured image formed by the image data of the captured data recorded in the storage device 103 (step S104). This is to enable the extension direction of the reach of the gantry crane 4 to be identified based on the linear object and the upward azimuth angle of the image.

[0090] The pixel latitude and longitude conversion unit 133 converts the position of each pixel constituting the image of the top surface of the machine room 41 of the gantry crane 4 extracted (identified) in step S103 into latitude and longitude, and stores the converted values ​​for each pixel in the pixel latitude and longitude file 105 (step S105). The reference point identification unit 134 locates the center point of the machine room 41 of the gantry crane 4 based on the positions of the four corners of the machine room, extracts the latitude and longitude of the center point, identifies it as the latitude and longitude of the reference point P, and records it in the reference data file 106 (step S106).

[0091] That is, in this embodiment, as described above, the center point of the top surface of the machine room 41 of the gantry crane 4 is used as the reference point P. Therefore, in the image portion of the top surface of the machine room extracted (identified) in the captured image, that center point is identified, and further, the pixel corresponding to that center point is identified. Since the position of each pixel constituting the image of the top surface of the machine room is converted into latitude and longitude in step S105, it is possible to identify the center point of the top surface of the machine room 41, i.e., the latitude and longitude of reference point P, in step S106. Furthermore, since the height h to the top surface of the machine room 41 of the gantry crane 4 is also known information from the data in the inspection target 3D data file 104, it is possible to identify the latitude, longitude, and height of reference point P.

[0092] The reference orientation calculation unit 135 identifies the latitude and longitude of both ends of the line segment indicated by the straight object extracted (identified) by image recognition in step S104, and calculates the extension direction of the reach of the gantry crane 4 as the reference orientation using the orientation and azimuth angle indicated by the line segment (step S107). The calculated reference orientation is recorded, for example, in the reference data file 106 and made available for use. The latitude and longitude of both ends of the line segment indicated by the straight object can be identified in the same way as when the latitude and longitude of the pixels constituting the image of the top surface of the machine room 41 are identified using the method shown in FIG. 4.

[0093] This allows the extension direction of the straight object to be identified, and the orientation of this straight object can be found using the azimuth angle of the upward direction of the image as a reference. Using the orientation of this straight object as a reference, the orientation of the extension direction of the reach of the gantry crane 4 can be found. For example, if the straight object is a rail along which the gantry crane 4 moves, the direction perpendicular to the rail is the extension direction of the reach of the gantry crane 4. The reference orientation, which is the extension direction of the reach of the gantry crane 4 found in step S107, is recorded in the reference data file 106 and can be used as needed.

[0094] Additionally, in step S107, the obstacle management unit 136 also functions. The obstacle management unit 136 identifies the latitude, longitude, and height of the other necessary objects extracted (identified) in step S103, and stores these in the obstacle management data file 107 (step S107). As described above, the other necessary objects are objects that may obstruct the flight of the UAV 2. For each necessary object extracted (identified) by image recognition, the obstacle management unit 136 identifies the latitude and longitude of multiple points so that the outer edge (edge) of the object can be identified, and stores these in the obstacle management data file 107 for each necessary object.

[0095] The latitude and longitude of the required object can also be determined in the same manner as when determining the latitude and longitude of the pixels constituting the image of the top surface of the machine room 41 using the method shown in Figure 4. The height of the required object may be a fixed, predetermined height, or in the case of a structure whose height does not change, it may be set in advance in, for example, the reference data file 106 and used. Of course, the height of the required object may also be determined by using the laser ranging function of the UAV 2 to determine the distance from the sky to the required object and subtracting the distance to the required object from the flight altitude of the UAV 2 at that time.

[0096] The no-fly area identifying unit 137 uses information from the reference data file 106, the obstacle management data file 107, and the like to identify no-fly areas for the UAV 2 to photograph the inspection portion of the gantry crane 4 (step S108). In other words, the processing in step S108 is processing for creating areas where the UAV 2 can fly to inspect each part of the gantry crane 4. The no-fly area identifying unit 137 stores and retains the latitude, longitude, and altitude of multiple points as information for identifying the identified no-fly areas so that the outer edges of the three-dimensional area where the UAV 2 cannot fly when inspecting the gantry crane 4 can be identified. Note that the no-fly area identifying unit 137 identifies the no-fly areas taking into consideration the position, size, reach extension direction, etc. of the gantry crane 4 to be inspected.

[0097] Flight path normalization unit 138 performs a process of normalizing the relative flight path data in relative flight path data file 109 to movement flight path data expressed in absolute coordinates using the latitude, longitude, and altitude of reference point P identified in step S106 (step S109). In the process of step S109, flight path normalization unit 138 normalizes the relative flight path data to create movement flight path data expressed in absolute coordinates and records this in automatic flight path data file 110. In this case, flight path normalization unit 138 performs the normalization process by taking into account the information in no-fly area file 108, avoiding no-fly areas where flight is not possible due to the presence of obstacles, and using only flyable areas where safe flight is possible.

[0098] The automatic flight path data created in the automatic flight path data file 110 in this manner is provided to the UAV 2 via the IoT platform 3, as described with reference to Fig. 5. This allows the UAV 2 to fly automatically (autonomously) according to the automatic flight path data, automatically photograph predetermined inspection locations of the gantry crane 4, and accumulate photographic data of the predetermined inspection locations. In other words, by working in cooperation with the flight path normalization device 1, the UAV 2 can perform photographic processing for inspection of the gantry crane 4 without human intervention.

[0099] The process of converting the positions of obstacles such as container groups that have been image-recognized into latitude and longitude, and the process of converting the positions of both ends of linear objects such as rails for the movement of gantry cranes into latitude and longitude, can also be performed collectively in the pixel latitude and longitude conversion unit 133.

[0100] [Effects of the embodiment] According to the flight path normalization device 1 of the above-described embodiment, the absolute coordinates (latitude, longitude) of the reference point P, which serves as the origin (0,0,0) of the relative coordinate system, can be accurately identified without human intervention based on the image captured by the UAV 2. Based on the absolute coordinates (latitude, longitude) of the identified reference point P, the relative flight path data (flight path data expressed in relative coordinates), which is data prepared in advance, can be automatically normalized to automatic flight path data expressed in absolute coordinates.

[0101] In this way, since there is no need to manually measure the absolute coordinates (latitude, longitude) of the reference point P, the man-hours required for inspecting the gantry crane 4 using the UAV 2 can be reduced. Furthermore, there is no need for the operator of the UAV 2 to go to the site, and the UAV 2 cooperates with the flight path normalization device 1 located in a remote location, allowing the UAV 2 to fly automatically (autonomously) and perform inspection work essentially without human intervention.

[0102] [Variations] In the above-described embodiment, the center point of the top surface of the machine room 41 of the gantry crane 4 is set as the reference point P, but this is not limited to this. For example, the ground position where a perpendicular line is dropped from the center point of the top surface of the machine room 41 of the gantry crane 4 may be set as the reference point P. In this case, the latitude and longitude of the center point of the top surface of the machine room 41 can be used as the latitude and longitude of the reference point P, and the height of the reference point P can be set to 0 (zero) m.

[0103] In the above-described embodiment, linear objects such as rails for the movement of the gantry crane 4 are extracted (identified) by image recognition to identify the reference orientation, but the present invention is not limited to this. Various recognizable linear objects can be used as the linear object, such as the edge of a group of containers or the edge of a quay separating land from sea. Furthermore, the flight path normalization device 1 may attach a linear marker to a predetermined position on the captured image, and use this to identify the reference orientation.

[0104] Furthermore, the reference point P can be any location identifiable from an image of the ground area including the inspection object photographed from the air by the UAV2. For example, if the inspection object is a gantry crane, the reference point P can be any appropriate location, such as the seaward end of the crane's reach. Therefore, the inspection object is not limited to a gantry crane, but can also be any other tall structure, such as a bridge, a steel tower, a wind turbine tower for wind power generation, or a Ferris wheel. Even in this case, the reference point when creating the relative flight path data in advance can be any location identifiable from an image of the ground area including the inspection object photographed from the air by the UAV2.

[0105] Furthermore, port areas often have multiple gantry cranes, and in the case of overhead power line towers, multiple towers are located at a predetermined distance apart. Therefore, taking into account the flight time available for the UAV2, information indicating the direction of movement when the inspection of one inspection object is completed is prepared. After completing the inspection of one inspection object, the UAV moves along the boundary between the sea and land or along the power line according to the information indicating the direction of movement. Then, by performing image recognition on the images captured by the camera unit 210, once the next inspection object is identified, the ground area including the inspection object is photographed, and the photographed data is sent to the flight path normalization device 1 for inspection. In other words, it is possible to inspect multiple inspection objects consecutively.

[0106] As mentioned above, this invention can be applied not only to the inspection of gantry cranes, but also to various other tall structures such as bridges, steel towers, wind turbine towers for wind power generation, and Ferris wheels. In other words, even for fixed structures such as bridges and building sidings, the flight path data for inspection may be created using relative coordinates with a reference point determined according to a specific part of the inspection target as the origin. In such cases, this invention can also be used when normalizing this flight path data to absolute coordinates.

[0107] [others] As can be seen from the above description of the embodiment, the function of the claimed storage means is realized by relative flight path data file 109 of flight path normalization device 1 of the embodiment. The function of the claimed acquisition means is realized by cooperation between communication unit 101 and control unit 102 of flight path normalization device 1. Furthermore, the function of the claimed image recognition means is realized by image recognition unit 132 of flight path normalization device 1, and the function of the claimed conversion means is realized by pixel latitude / longitude conversion unit 133 of flight path normalization device 1. Furthermore, the function of the claimed reference point identification means is realized by reference point identification unit 134 of flight path normalization device 1, and the function of the claimed normalization means is realized by flight path normalization unit 138 of flight path normalization device 1.

[0108] Furthermore, the function of the claimed straight-line object identification means is realized by image recognition unit 132 of flight path normalization device 1, and the function of the claimed straight-line object position conversion means is realized by pixel latitude / longitude conversion unit 133 of flight path normalization device 1. Furthermore, the function of the claimed reference orientation calculation means is realized by reference orientation calculation unit 135 of flight path normalization device 1. Furthermore, the function of the claimed obstacle recognition means is realized by image recognition unit 132 of flight path normalization device 1, and the function of the claimed obstacle position conversion means is realized by pixel latitude / longitude conversion unit 133 of flight path normalization device 1.

[0109] 11 is one embodiment of a flight path normalization method executed by flight path normalization device 1. A program for executing the processing shown in the flowchart of Fig. 11 is one embodiment of a flight path normalization program executed by, for example, a computer. Therefore, the functions of the various units constituting altitude correction value calculation unit 120 and automatic flight path generation unit 130 of flight path normalization device 1 shown in Fig. 7 can be realized as functions of control unit 102 by a program executed in control unit 102. [Explanation of symbols]

[0110] 1A... inspection management company, 1B... port office, 1... flight path normalization device, 101T... connection terminal, 101... communication unit, 102... control unit, 103... storage device, 104... inspection target 3D data file, 105... pixel latitude and longitude file, 106... reference data file, 107... obstacle management data file, 108... no-fly area file, 109... relative flight path data file, 110... automatic flight path data file, 120... altitude correction value calculation unit, 130... automatic flight path creation unit, 131... azimuth angle acquisition unit, 132... image recognition unit, 133... pixel latitude and longitude conversion unit, 134... reference point identification unit, 135... reference orientation calculation unit, 136... obstacle management unit, 137... no-fly area identification unit, 138... flight path normalization unit, 2... UAV, 3... IoT platform, 4... gantry crane, 41... machine room

Claims

1. a storage means for storing relative flight path data expressed in relative coordinates with a reference point set according to a predetermined portion of the target object as the origin; an acquisition means for acquiring from the UAV image data obtained by imaging a subject from the sky toward the ground using a camera mounted on the UAV, and image data including metadata including information on the imaging position and orientation at the time of imaging; an image recognition means for recognizing the predetermined portion from the image data; a conversion means for converting pixel positions constituting the image of the predetermined portion into latitude and longitude in consideration of the height to the predetermined portion, the photographing position, and the orientation information; a reference point specifying means for specifying the latitude and longitude of the reference point based on the latitude and longitude obtained by the conversion means; a normalization means for converting the relative flight path data into automatic flight data in absolute coordinates by a normalization process consisting of rotational transformation and parallel translation based on a reference direction, with the latitude and longitude of the reference point as the origin; A flight path normalization device comprising:

2. 2. The flight path normalization device according to claim 1, wherein the conversion means calculates the latitude and longitude by geometric back projection based on intrinsic and extrinsic parameters of the camera.

3. 3. The flight path normalization device according to claim 1, wherein the height includes an altitude correction value based on at least one of the known height of the predetermined portion, the shooting altitude of the UAV, and elevation information of the earth's surface.

4. 3. The flight path normalization device according to claim 1, wherein the reference orientation is calculated based on at least one of a linear edge in an image, a facility layout line in map information, and orientation information from an inertial measurement unit or a magnetic sensor.

5. 3. The flight path normalization device according to claim 1, wherein the transformation performed by the transformation means is performed based on a single aerial image.

6. 3. The flight path normalization device according to claim 1, further comprising a function of converting pixel positions of obstacles detected by image recognition into latitude and longitude using the conversion means to identify the obstacle positions, and adjusting the automatic flight data in conjunction with no-fly areas based on geographic information.

7. 1. A flight path normalization method used in a flight path normalization device having a storage means for storing relative flight path data expressed in relative coordinates with a reference point set according to a predetermined portion of a target object as an origin, comprising: an acquisition step in which an acquisition means acquires from the UAV image data obtained by imaging from the sky toward the ground using a camera mounted on the UAV, and image data including metadata including information on the imaging position and orientation at the time of imaging; an image recognition step in which image recognition means recognizes the predetermined portion from the image data; a conversion step in which a conversion means converts pixel positions constituting the image of the predetermined portion into latitude and longitude in consideration of the height to the predetermined portion, the photographing position, and the orientation information; a reference point specifying step in which a reference point specifying means specifies the latitude and longitude of the reference point based on the latitude and longitude obtained by the conversion step; a normalization step in which a normalization means converts the relative flight path data into automatic flight data in absolute coordinates by a normalization process consisting of a rotational transformation and a parallel translation based on a reference orientation, with the latitude and longitude of the reference point as the origin; A flight path normalization method comprising:

8. A flight path normalization program executed by a computer mounted on a flight path normalization device having storage means for storing relative flight path data expressed in relative coordinates with a reference point set according to a predetermined portion of a target object as the origin, an acquisition step of acquiring, from the UAV, image data obtained by capturing an image from the sky toward the ground using a camera mounted on the UAV, and image data including metadata including information on the image capturing position and orientation at the time of capturing the image; an image recognition step of recognizing the predetermined portion from the image data; a conversion step of converting pixel positions constituting the image of the predetermined portion into latitude and longitude in consideration of the height to the predetermined portion, the photographing position, and the orientation information; a reference point specifying step of specifying the latitude and longitude of the reference point based on the latitude and longitude obtained by the conversion step; a normalization step of converting the relative flight path data into automatic flight data in absolute coordinates by a normalization process including rotational transformation and translation based on a reference orientation, with the latitude and longitude of the reference point as the origin; Run the flight path normalization program.

Images