Method for acquiring data for inspecting wind turbine, program for making computer execute the acquisition method, and unmanned aircraft

Abstract

To provide a technology for allowing an unmanned aircraft to acquire an image for appearance inspection of a blade of a wind turbine.SOLUTION: Processing executed by a controller of an unmanned aircraft includes the steps of: making a drone approach a wind turbine front while securing required separation (S1220); making the drone hovering at a photographing position (S1225); starting to photograph blades of a wind turbine while synchronizing each shutter timing of two cameras (S1230); executing steps S1215 to S1235 in the case of changing a photographing position of the drone (YES in S1240); and discriminating a hatch from a photographed image by using AI image analysis (S1250) when the whole photographing is completed (S1245).SELECTED DRAWING: Figure 12

Description

Technical Field

[0001] The present disclosure relates to a technique for inspecting blades of a windmill, and more particularly, to a technique for acquiring data for inspecting blades.

Background Art

[0002] Techniques for inspecting blades of a windmill used for wind power generation are known. For example, Japanese Patent Application Laid-Open No. 2017-090145 (Patent Document 1) discloses a "windmill blade deformation measurement device that can obtain blade rigidity simply and with high accuracy without requiring special synchronization for acquiring information on a sensor and wind speed."

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Since the length of the blade of a windmill can reach several tens of meters, when acquiring an image for inspecting the appearance of the blade, it is necessary to acquire a plurality of images because the necessary image cannot be acquired in a single shot. On the other hand, since a windmill has three blades, even if an image of a part of the blade is acquired, there may be cases where it is not clear which blade the image corresponds to. Therefore, a technique that can easily identify which blade an image is of is required.

[0005] The present disclosure has been made in view of the above background, and one of its objects is to disclose a technique that can identify which blade the blades included in a plurality of images correspond to.

Means for Solving the Problems

[0006] (1) According to one embodiment, a method for acquiring data for inspecting a wind turbine is provided. This method includes the steps of: using one or more cameras mounted on an unmanned aerial vehicle to take multiple simultaneous photographs of a predetermined part on the hub of the wind turbine and a part of one of a plurality of blades attached to the hub; repeating the step of taking simultaneous photographs of the remaining blades of the plurality of blades; and identifying the photographed blade based on the presence or absence of the predetermined part in the image based on the image data acquired by the photographs, the orientation of the predetermined part, and the positional relationship of the plurality of blades with respect to the predetermined part.

[0007] (2) In a given situation, the step of performing simultaneous shooting multiple times includes the steps of simultaneously shooting a predetermined part of the hub with a first camera from among one or more cameras, and shooting a part of one blade with a second camera from among one or more cameras, and in response to simultaneously shooting the predetermined part and the part, associating the time of shooting with the image data acquired by the first camera and the image data acquired by the second camera.

[0008] (3) The step of taking multiple simultaneous shots in a given situation includes acquiring still images. The step of associating time includes associating time with the image data acquired by the first camera and the image data acquired by the second camera.

[0009] (4) In a given situation, the step of taking multiple simultaneous shots includes acquiring video footage. The step of associating time includes associating time with frames included in video footage acquired by the first camera and frames included in video footage acquired by the second camera.

[0010] (5) In a given situation, the step of taking multiple simultaneous shots includes the step of directing the shooting direction of the first camera toward a predetermined part so that the part does not move out of the shooting range in accordance with the movement of the unmanned aerial vehicle.

[0011] (6) In a given scenario, cameras are provided on the upper and lower parts of the unmanned aerial vehicle's casing. The step of simultaneous shooting includes, when the blade to be photographed is above the hub, the camera provided on the upper part acting as a second camera to photograph a portion of one blade, and the camera provided on the lower part acting as a first camera to photograph a predetermined part.

[0012] (7) In a given scenario, cameras are provided on the upper and lower parts of the unmanned aerial vehicle's casing. The step of simultaneous shooting includes, when the blade to be photographed is below the hub, the camera provided on the lower part acting as a first camera to photograph a part of one blade, and the camera provided on the upper part acting as a second camera to photograph a predetermined part.

[0013] (8) In a given situation, the acquisition method further includes the step of identifying the blade to which an image of a portion of one blade acquired at the time an image including a predetermined part belongs.

[0014] (9) In a given situation, the identification step includes analyzing multiple images to extract pre-identified parts from each image, and labeling the blades included in the images of a portion of one blade acquired at the time each image was acquired, based on blade identification information that has been pre-identified based on the positional relationship between a predetermined part and multiple blades.

[0015] (10) In a given situation, the step of taking multiple simultaneous photographs includes taking simultaneous photographs while the unmanned aerial vehicle is hovering during blade operation.

[0016] (11) In a given situation, the step of taking multiple simultaneous shots includes taking simultaneous shots while moving the unmanned aerial vehicle while the blade is stationary.

[0017] (12) In a certain situation, moving the unmanned aircraft includes moving it from the tip of the blade to be photographed towards the hub, or moving the unmanned aircraft from the hub towards the tip of the blade to be photographed.

[0018] (13) According to other embodiments, a program is provided. This program causes a computer to execute the acquisition method described in any of the above.

[0019] (14) According to still other embodiments, an unmanned aircraft is provided. This unmanned aircraft includes a memory storing the above program, one or more cameras arranged on the unmanned aircraft, and a control device that executes the program to control the unmanned aircraft.

[0020] According to an embodiment, since it is possible to identify which blade the blades included in a plurality of images correspond to, the appearance inspection of the blades is facilitated.

[0021] The above and other objects, features, aspects and advantages of the present invention will become apparent from the following detailed description of the present invention understood in connection with the accompanying drawings.

Brief Description of the Drawings

[0022] [Figure 1] It is a diagram showing an outline of a method for inspecting the appearance of the windmill 10 during operation by the drone 100 as a flying object. [Figure 2] It is a diagram showing a part of the route along which the drone 100 flies when inspecting the appearance of the three blades 120, 121, and 122. [Figure 3] It is a functional block diagram illustrating the configuration of the drone 100. [Figure 4] It is a diagram illustrating the appearance of the drone 100. [Figure 5] It is a diagram illustrating a situation where the drone 100 performs multiple shootings to inspect a single blade. [Figure 6] It is a diagram of the windmill 10 seen from the side. [Figure 7] It is a diagram showing an example of an image 700 that photographs a part of the windmill 10 from below the windmill 10. [Figure 8] It is a diagram exemplifying one aspect of data stored in a recording medium 38 possessed by the drone 100. [Figure 9] It is a diagram exemplifying a state of photographing the nacelle 160 from above the windmill 10. [Figure 10] It is a diagram exemplifying the hardware configuration of a computer device 1000 that functions as an image analysis device. [Figure 11] It is a diagram exemplifying an image 1100 photographed from above the windmill 10. [Figure 12] It is a flowchart showing a part of the process performed to identify the blades from a still image of the windmill 10. [Figure 13] It is a flowchart showing a part of the process performed to identify the blades from a moving image of the windmill 10. [Figure 14] It is a flowchart showing a part of the process performed to identify the blades from both the moving image and the still image. [Figure 15] It is a flowchart showing a part of the process performed when photographing the hub 140 and the blades using one camera. [Figure 16] It represents an image obtained by the first camera 18 photographing the hub 140. [Figure 17] It represents an image 1700 in which the tip of the blade 120 identified in step S1225 is shown.

Embodiments for Carrying Out the Invention

[0023] Hereinafter, embodiments of the present invention will be described while referring to the drawings. In the following description, the same parts are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed descriptions thereof will not be repeated.

[0024] [Overview of Appearance Inspection Using a Drone] Referring to Figures 1 and 2, a method for inspecting a wind turbine 10 using a drone 100 according to one embodiment will be described. Figure 1 is a diagram illustrating a method by which a drone 100, as an aerial object, inspects the appearance of a wind turbine 10 while it is in operation.

[0025] The wind turbine 10 is installed on a foundation 180, for example, in an offshore or onshore location with relatively high wind speeds. The wind turbine 10 comprises blades 120, 121, and 122, a hub 140, and a tower 130. The hub 140 is provided with a hatch 150.

[0026] The drone 100 is used to inspect the appearance of the wind turbine 10. The drone 100 is equipped with a first camera 18 capable of taking pictures above or in front of the drone 100, a second camera 20 capable of taking pictures below or in front of the drone 100, and a LiDAR (Laser Imaging Detection and Ranging) sensor 24.

[0027] Drone 100 approaches the vicinity of the wind turbine 10 without contacting the blades 120, 121, and 122, and moves from bottom to top, top to bottom, left to right, or right to left, stopping at the shooting location and taking pictures in sync using the first camera 18 and the second camera 20. Drone 100 drives one of the two cameras to maintain an attitude that allows it to photograph a predetermined part (for example, the hatch 150, a marker placed for inspection, etc.). Using the other camera, Drone 100 photographs the appearance of each blade from the tip toward the center, or from the center toward the tip, in sync with the timing of the first camera 18 and the second camera 20. By having the two cameras take pictures simultaneously, the image of the hatch 150 and the image of the blade's appearance can be associated at each timing. Furthermore, since each captured image is associated with an image of hatch 150, the positional relationship of the blade to hatch 150 becomes clear, making it easy to identify which blade was photographed.

[0028] Once shooting is completed at one location, the drone moves to the next location and performs similar shooting. The image data obtained from the shooting is stored in the drone 100, and then image analysis is performed by a computer device. In other scenarios, the drone 100 can transmit image data to a computer device on the ground. In this case, the computer device can start image analysis before the drone 100 returns, so inspection results can be obtained earlier. The detailed configuration of the drone 100 will be described later.

[0029] Furthermore, photographing the blades 120, 121, and 122 using the drone 100 is possible both when the wind turbine 10 is stopped and when it is in operation. Alternatively, the blades may be photographed while the drone 100 is hovering without moving. If photography can be performed while the wind turbine 10 is in operation, image data for visual inspection can be acquired while preventing a decrease in power generation.

[0030] Figure 2 shows a portion of the flight path of the drone 100 when inspecting the appearance of the three blades 120, 121, and 122. The drone 100, having taken off from the ground, approaches the vicinity of the tip of the first inspection target, blade 120 (path 210, drone 100A). The drone 100 takes a photograph at its initial position near blade 120 and simultaneously photographs the hatch 150. The photograph may be a still image or a video.

[0031] For example, the drone 100 simultaneously takes images of the hatch 150 using the first camera 18 and a portion (the tip) of the blade 120 using the second camera 20. Each captured image is associated with a timestamp indicating the time of capture. This timestamp is obtained, for example, using the GPS function of the drone 100. The image data generated from the first camera 18 and the image data generated from the second camera 20 are linked by the timestamp. In addition, one of the two cameras is always capturing images of the hatch 150, and from the position of the hatch 150, it is possible to identify which of the three blades 120, 121, or 122 was captured at that time.

[0032] Subsequently, the drone 100 moves to the next position near the blade 120 and takes similar images. Each position where the drone 100 stops for shooting is set so that parts of the captured images overlap, taking into account the field of view of the camera that continues to photograph the blade 120 (the second camera 20 in the above example). On the other hand, the first camera 18 also photographs the area including the hatch 150 in the next shot after the drone 100 has moved. In this way, by ensuring that one of the two cameras always photographs the hatch 150, the blade can be identified based on the orientation of the hatch 150, making it easier to reconstruct an overall image of the appearance of the blade 120.

[0033] When Drone 100 moves along blade 120 to the vicinity of hub 140 (path 220), it begins photographing blade 121, the second object to be inspected (Drone 100B). Drone 100 moves from the base of blade 121 (near hub 140) towards the tip, repeatedly taking photographs. After finishing photographing blade 121, Drone 100 moves to the tip of blade 122, the third object to be inspected (path 230, Drone 100C). Drone 100 repeatedly moves and stops from the tip of blade 122 towards hub 140, repeatedly photographing the exterior of blade 122. After finishing photographing blade 122, Drone 100 returns to the ground (path 240).

[0034] In the case of photographing the blade 120 shown in Figure 2, the drone 100 may photograph the blade 120 while moving from above to below (i.e., from the hub 140 to the tip of the blade 120). In other situations, the drone 100 may photograph the blade while moving from below to above. If sunlight may be backlighting the drone 100 depending on the time of day, the drone 100 can obtain a good image of the blade surface by moving from above to below or by photographing from an oblique angle.

[0035] On the ground, image data acquired by the drone 100 is read out and input into a computer device that performs image analysis. The computer device arranges the image data in chronological order based on timestamps, displaying images of adjacent parts in sequence. Furthermore, since the blades are identified based on the orientation of the hatch 150 in the images, the computer device can indicate which blade each image represents. Therefore, an inspector viewing the monitor of the computer device can visually inspect each blade by examining the images arranged in chronological order. Visual inspection may be performed using AI (Artificial Intelligence).

[0036] In other scenarios, each image data may be transmitted wirelessly to a ground-based computer device while the drone 100 is in flight. In this case, a visual inspection of any of the blades whose entire image has been captured can be performed in advance while the drone 100 is in flight.

[0037] [Drone configuration] The configuration of the drone 100 will be described with reference to Figures 3 and 4. Figure 3 is a functional block diagram illustrating the configuration of the drone 100. Figure 4 is a diagram illustrating the external appearance of the drone 100.

[0038] As shown in Figure 3, the drone 100 comprises a main body 11, four propeller motors 13, a motor drive circuit 15, an engine 16, a generator 17, a first camera 18, a gimbal 19, a second camera 20, a gimbal 21, and a LiDAR sensor 24. The main body 11 includes a controller 30, a storage device 31, an inertial measurement unit 32, a transceiver 33, a GPS (Global Positioning System) receiver 34, a power supply circuit 35, and a storage battery 36. The storage device 31 has a reader / writer 37 and a recording medium 38. If the drone 100 is electric and does not have an engine 16 and a generator 17, the storage battery 36 drives the propeller motors 13.

[0039] The generator 17, driven by the engine 16, generates the power necessary for the operation of the drone 100. This enables the drone 100 to fly for extended periods and in low temperatures.

[0040] The first camera 18 is mounted on the top surface of the main body 11 so that it can capture images in front of or above the drone 100. The first camera 18 can change its shooting direction from horizontal to upward by an actuator driven by an external signal. The actuator may also be rotatable about a vertical axis of rotation. The actuator is optional.

[0041] The second camera 20 is mounted on the underside of the main body 11 so as to be able to photograph the area in front of or below the drone 100. In one configuration, the second camera 20 is positioned to photograph the direction in which the drone 100 is moving. In other configurations, the second camera 20 may be mounted on the main body 11 so as to be able to rotate about a vertical axis of rotation.

[0042] The first camera 18 may include a high-resolution camera 18A for capturing inspection images and a low-resolution camera 18B for capturing video for FPV (First Person View). The second camera 20 may include a high-resolution camera 20A for capturing inspection images and a low-resolution camera 20B for capturing video for FPV. Video captured by the low-resolution cameras 18B and 20B is transmitted to a wireless communication device (not shown) at the base facility. This allows inspectors at the base facility to view the video captured by cameras 18B and 20B in real time. The shooting direction of the first camera 18 and the second camera 20 is adjusted by the actuators constituting the gimbals 19 and 21, respectively. The actuators constituting the gimbals 19 and 21 can also be adjusted by signals from the wireless communication device at the base facility.

[0043] The gimbals 19 and 21 are electric actuators that can adjust the orientation of the first camera 18 and the second camera 20 in three dimensions, respectively. Based on the detection values ​​of the acceleration sensor and the angular velocity sensor, the gimbals 19 and 21 can maintain a constant imaging direction for the first camera 18 and the second camera 20, respectively, even when the attitude of the drone 100 changes.

[0044] The LiDAR sensor 24 detects the distance and angle to an object based on the reflected light from the object by scanning with pulsed laser light. Other distance sensors may be used instead of the LiDAR sensor 24.

[0045] The controller 30 controls the operation of the motor drive circuit 15, the first camera 18, the second camera 20, the LiDAR sensor 24, and other components. The controller 30 may be composed of a microcomputer including a CPU (Central Processing Unit), RAM (Random Access Memory), and non-volatile memory, or it may be composed of an FPGA (Field Programmable Gate Array), or it may be composed of a dedicated circuit such as an ASIC (Application Specific Integrated Circuit). Alternatively, the controller 30 may be composed of a combination of at least two of these components.

[0046] The storage device 31 includes, for example, a removable non-volatile recording medium 38 such as an SD memory card, and a reader / writer 37 for writing data to and reading data from the recording medium 38. The reader / writer 37 stores still images for inspection taken by the first camera 18, control contents of the controller 30, and flight information in the recording medium 38 according to the commands of the controller 30.

[0047] The inertial measurement unit 32 is a sensor unit that integrates an acceleration sensor, an angular velocity sensor (gyro sensor), a geomagnetic sensor, a barometric pressure sensor, a temperature sensor, and other sensors into a single package. The controller 30 performs autonomous flight and attitude control of the drone 100 based on the detection values ​​of the various sensors in the inertial measurement unit 32.

[0048] The transceiver 33 transmits and receives signals, data, and other information to and from the transceiver installed on the wind turbine under inspection and the transceiver on the terminal equipment of the onshore base facility. For example, the transceiver 33 receives information on the operating status of the wind turbine and commands from the terminal equipment of the base facility. In addition, the transceiver 33 transmits monitoring video images captured by the low-resolution cameras 18B and 20B to the base facility in accordance with the commands of the controller 30.

[0049] The GPS receiver 34 receives signals transmitted from each of multiple GPS satellites. The controller 30 detects the current position of the drone 100 based on the signals received by the GPS receiver 34.

[0050] The power supply circuit 35 generates power supply voltages to drive the various parts of the drone 100 based on the power generated by the generator 17.

[0051] The battery 36 is used as an auxiliary power source for the drone 100. For example, the battery 36 supplies power to drive the controller 30, storage device 31, transceiver 33, etc., when the engine 16 and generator 17 are stopped.

[0052] Figure 4(A) is a front view of the drone 100. Figure 4(B) is a side view of the drone 100.

[0053] As shown in Figure 4(A), the drone 100 has four sets of propeller motors 13 and propellers 14. Each propeller motor 13 is connected to the body 11 via arms 12. Legs 22 are connected to the bottom of the body 11.

[0054] Figure 4 illustrates a drone 100 equipped with four sets of propeller motors 13 and propellers 14, but the flight mechanism of the drone 100 is not limited to the illustrated configuration. In other aspects, by controlling the rotational speed of each propeller motor 13, the drone 100 can ascend and descend vertically, ascend and descend in any diagonal direction, hover in mid-air, move forward, move backward, move right, move left, rotate right, rotate left, etc. That is, the propeller motors 13 and propellers 14 can constitute a propulsion mechanism 23 for propelling and hovering the drone 100.

[0055] [Overview of the shoot] Referring to Figures 5 and 6, a method of taking photographs using the drone 100 according to this embodiment will be described. Figure 5 illustrates a situation in which the drone 100 takes multiple photographs to inspect a single blade.

[0056] Figure 5 shows the wind turbine 10 viewed from the front. An example is shown in which a drone 100 photographs a single blade from 10 locations. The number of photographs is calculated in advance by the user of the drone 100, for example, based on the shooting range and resolution of the first camera 18 and the second camera 20 mounted on the drone 100. The number of photographs is calculated as the number of times the edges of the images of the surface of a single blade are adjacent, so that an image of the appearance of a single blade is reconstructed from each image.

[0057] Drone 100, having taken off from the ground, approaches the tip of the first inspection target, blade 120, and at position 100-1, takes photographs of a portion of blade 120 and the hatch 150. Drone 100 then moves to position 100-2. At position 100-2, Drone 100 simultaneously takes photographs of a portion of blade 120 and the hatch 150, similar to the photography at position 100-1. Drone 100 then moves to the next position, as indicated by the arrow, and again simultaneously photographs a portion of blade 120 and the hatch 150. Drone 100 then moves again to the final position 100-10, where it takes simultaneous photographs. Once Drone 100 has finished photographing blade 120, it moves to the tip of the next blade and begins similar simultaneous photography. Furthermore, the drone 100 can perform simultaneous shooting while moving from the blade attachment point toward the tip of the next blade without moving to the tip of the next blade. After completing multiple simultaneous shootings for each of the three blades, the drone 100 returns to the ground. The shooting may be performed using both the first camera 18 and the second camera 20, or using only one of them.

[0058] Depending on the drone 100's position, hatch 150 may not be visible from the drone 100. Therefore, the image data obtained by photographing each blade may include image data of hatch 150 and image data of a portion of the blade, or it may not include image data of hatch 150. Since the relative positions of hatch 150 and each blade are known in advance, it is possible to identify which blade a portion of a blade photographed simultaneously belongs to. Therefore, image data obtained by photographing a single blade multiple times can be reconstructed by image processing using a computer.

[0059] Figure 6 is a side view of the wind turbine 10. At position 610, the drone 100 measures the distance to the front of the tower 130. In this case, the drone 100 determines position 610 such that this distance is greater than the distance between the blades 120 and the tower 130. This can prevent a collision between the drone 100 and the blades 120 during filming at position 610.

[0060] Drone 100 measures the distance to hub 140 (position 620) when it is in front of hub 140. Near hub 140, since drone 100 and its blades do not interfere with each other, drone 100 can get closer to the front of hub 140. For example, the distance between drone 100 and hub 140 will be shorter than the distance between drone 100 and tower 130. The camera used to photograph the front of hub 140 may be either the first camera 18 or the second camera 20.

[0061] The drone 100 can also photograph the back of the wind turbine 10. For example, the drone 100 measures the distance to the nacelle 160, approaches the nacelle 160, and photographs its exterior (position 630). The camera used to photograph the nacelle 160 may be either the first camera 18 or the second camera 20.

[0062] Furthermore, the drone 100 can also photograph the back of the tower 130. In this case as well, the drone 100 measures the distance to the tower 130, approaches the tower 130, and photographs the exterior of the tower 130 (position 640). The camera used to photograph the tower 130 may be either the first camera 18 or the second camera 20.

[0063] [Example of a photograph] An example of a captured image will be explained with reference to Figure 7. Figure 7 shows an example of an image 700 taken from below the wind turbine 10, showing a part of the wind turbine 10. Image 700 includes the blades 120, 121, 122, the tower 130, the hub 140, the hatch 150, and the nacelle 160.

[0064] Since image 700 includes hatch 150, considering the position of the blade relative to hatch 150, it is easy to identify which blade corresponds to which blade in the simultaneously captured images.

[0065] [Data structure] Therefore, with reference to Figure 8, the data structure acquired by the drone 100 will be explained. Figure 8 is a diagram illustrating one aspect of the data stored in the recording medium 38 of the drone 100. The drone 100 associates two image data acquired by simultaneous shooting using the first camera 18 and the second camera 20 with time information and writes them to the recording medium 38.

[0066] For example, image data 810 includes a photograph of the hatch 150 taken by the first camera 18. The time when image data 810 was taken is March 15, 2022, at 11:22:00. From the orientation of the hatch 150, the blade visible below the hub 140 is identified as blade 121. On the other hand, image data 811 includes a photograph of blade 121 taken by the second camera 20. The time when image data 811 was generated is also March 15, 2022, at 11:22:00. As an example, photography by the drone 100 is performed at 2-second intervals, and the timestamps of each image data are also matched in seconds.

[0067] Similarly, since image data 820 and image data 821 are associated with the same time information, image data 820 and image data 821 are data acquired from images taken at the same time.

[0068] Since the timestamps of image data 810 and image data 811 are the same, and the timestamps of image data 820 and image data 821 are the same, the computer device performing image analysis can associate image data 810 with image data 811, and image data 820 with image data 821, and display them on the monitor. This allows the inspector to easily recognize that the blade 121 has been photographed, taking into account the orientation of the hatch 150 and its positional relationship with each blade as seen in image data 810 or image data 820.

[0069] [Shot from above] Refer to Figure 9 to explain other shooting methods. Figure 9 is an example of a situation in which the nacelle 160 is photographed from above the wind turbine 10.

[0070] Depending on the location and time of day of the wind turbine 10, the drone 100 may be subjected to backlighting when photographing the wind turbine 10 from below. Therefore, to avoid backlighting, the drone 100 may simultaneously photograph the wind turbine 10 from above. In this case, the drone 100 will photograph at least the hub 140 so that the hatch 150 is included in the captured image. The drone 100 will then photograph the blades at the same time as photographing the hub 140.

[0071] In the example shown in Figure 9, the drone 100 moves from above to below the wind turbine 10, simultaneously taking photographs of the blades and the hub 140. Since the drone 100 can simultaneously take photographs both upwards (Figure 5) and downwards (Figure 9), it is possible to reliably take simultaneous photographs in a direction that is not exposed to backlight.

[0072] The bearing of Nacelle 160 can be calculated, for example, as follows: [First calculation method] (1) The position, direction, and angle of the drone 10 used to determine the bearing of the nacelle 160 shall be constant relative to the wind turbine 10. (2) Images of Nacelle 160 from various angles are taken in advance, and a deep learning model is constructed using each of these images as training data. The training data is, for example, images obtained by taking pictures of Nacelle 160 from the same altitude and angle. The deep learning model is constructed using, for example, a Convolutional Neural Network. (3) The deep learning model is installed on the drone 10 or on a computing device in the flight management base (for example, the computer device 1000 in Figure 10). (4) After the drone 10 takes a photograph at the shooting position for calculating the bearing of the nacelle 160, the image data acquired by the photograph is input to the above-mentioned computing device. (5) The inspector executes the nacelle heading calculation program from the user interface screen of the flight management program of the computing device and obtains the estimated result of the nacelle heading.

[0073] [Second calculation method] (1) The position, direction, and angle of the drone 10 used to determine the bearing of the nacelle 160 shall be constant relative to the wind turbine 10. (2) The rotational (yaw) plane of the nacelle 160 is constant on the captured image (two-dimensional projection plane). (3) From the image of the nacelle 160 taken by the drone 10 from a predetermined position, the structurally unique characteristic shape of the nacelle 160 (for example, the direction of the main body that houses the rotation axis of the hub 140, or the direction of the bottom surface of the main body) is captured as a line segment, and the actual angle (direction) is geometrically calculated from the angle of that line segment on the captured image, and then the nacelle 160 calculation is performed. The direction of the characteristic shape may be specified by the inspector.

[0074] [Third calculation method] (1) The drone 100 flies in a circular motion around the wind turbine 10, maintaining a constant distance at the same altitude as the nacelle 160, while facing the wind turbine 10. (2) While monitoring the images transmitted from the camera of the drone 100, the inspector changes the drone 100 to a hovering state when the drone 100 reaches a position that is in the same orientation as the nacelle 160 or directly facing it. (3) The computer used by the inspector displays the drone's orientation at that time on the flight management user interface screen, and the inspector reads the bearing of the nacelle 160 from that orientation.

[0075] [An example of an image analysis device] Next, with reference to Figure 10, an image analysis device for analyzing the appearance of a wind turbine according to this embodiment will be described. Figure 10 is a diagram illustrating the hardware configuration of a computer device 1000 that functions as an image analysis device.

[0076] The computer device 1000 includes, as its main components, a CPU (Central Processing Unit) 1 for executing programs, a mouse 2 and a keyboard 3 for receiving input instructions from the user of the computer device 1000, RAM 4 for volatilely storing data generated by the execution of programs by the CPU 1 or data input via the mouse 2 or keyboard 3, a hard disk 5 for non-volatilely storing data, an optical disc drive 6, a communication interface (I / F) 7, and a monitor 8. Each component is connected to the others by a data bus. A CD-ROM 9 or other optical disc is mounted in the optical disc drive 6.

[0077] Processing in the computer device 1000 is realized by each piece of hardware and software executed by the CPU 1. Such software may be pre-stored on the hard disk 5. Alternatively, the software may be stored on a CD-ROM 9 or other recording medium and distributed as a computer program. Or, the software may be provided as a downloadable application program by an information provider connected to the so-called Internet. Such software is read from the recording medium by an optical disc drive 6 or other reading device, or downloaded via a communication interface 7, and then temporarily stored on the hard disk 5. The CPU 1 reads the software from the hard disk 5 and stores it in RAM 4 in the form of an executable program. The CPU 1 then executes the program.

[0078] The components of the computer device 1000 shown in Figure 10 are common. Therefore, one of the essential parts of the technical concept relating to this disclosure can be said to be the software stored in the RAM 4, hard disk 5, CD-ROM 9, and other recording media, or software that can be downloaded via a network. The recording media may include non-temporary, computer-readable data recording media. Since the operation of each hardware component of the computer device 1000 is well known, a detailed explanation will not be repeated.

[0079] Furthermore, the recording medium is not limited to CD-ROMs, FDs (Flexible Disks), and hard disks, but may also include magnetic tapes, cassette tapes, optical discs (MO (Magnetic Optical Disc) / MD (Mini Disc) / DVD (Digital Versatile Disc)), IC (Integrated Circuit) cards (including memory cards), SD cards, optical cards, mask ROMs, EPROMs (Electronically Programmable Read-Only Memory), EEPROMs (Electronically Erasable Programmable Read-Only Memory), flash ROMs, and other semiconductor memory media that permanently store programs.

[0080] The term "program" as used here includes not only programs that can be directly executed by the CPU, but also programs in source code format, compressed programs, encrypted programs, and so on.

[0081] The computer device 1000 receives image data (Figure 8) stored in the recording medium 38 of the drone 100. The computer device 1000 performs analysis processing using the image data. For example, the CPU 1 extracts an image of the hatch 150 from each image data and, based on the orientation of the hatch 150 and its positional relationship with each blade, can identify which blade the image of was acquired when the image of the hatch 150 was acquired. In this way, the inspector, who is the user of the computer device 1000, can easily perform a visual inspection of the blades using the images and can easily identify which blade has been found to be damaged.

[0082] [Example image data] Referring to Figure 11, another example of image data obtained by the drone 100 will be described. Figure 11 illustrates an image 1100 taken from above the wind turbine 10. This image data is read from the drone 100 to the computer device 1000, and an image based on the image data is displayed on the monitor 8.

[0083] Image 1100, shown in Figure 11, was taken by drone 100 from behind the nacelle 160. Since drone 100 is equipped with an inertial measurement unit 32 and a GPS receiver 34, the position information of drone 100 at the time of shooting is acquired. Assuming that the shooting by drone 100 is performed with the first camera 18 or the second camera 20 aligned with the direction of drone 100's movement, the direction that drone 100 is facing at the time of shooting can be estimated. For example, if image 1100 is a view downwards from in front of drone 100, the bearing of nacelle 160 in image 1100 can be estimated. The computer device 1000 can calculate the actual bearing of nacelle 160 based on the bearing of drone 100 and the bearing of nacelle 160 in image 1100.

[0084] [Processing steps for acquiring still images] Referring to Figure 12, the process of identifying blades from video data acquired according to this embodiment will be described. Figure 12 is a flowchart showing part of the process performed to identify blades from a still image of the wind turbine 10.

[0085] In step S1210, the controller 30 estimates the current bearing of the nacelle 160 from the image obtained by the drone 100. This image is acquired by the drone 100 during the preparation phase before it begins to photograph the blades.

[0086] In step S1215, the controller 30 sets the angle of the first camera 18 located on the top of the drone 100 or the second camera 20 located on the bottom to match the shooting position of the drone 100. This setting is done, for example, based on the shooting angles of the first camera 18 and the second camera 20 and the distance to the wind turbine 10.

[0087] In step S1220, the controller 30 uses control with the GPS receiver 34 and LiDAR sensor 24 to move the drone 100 closer to the front or side of the wind turbine 10 while ensuring the necessary distance (necessary separation) between the drone 100 and the wind turbine 10.

[0088] In step S1225, the controller 30 causes the drone 100 to hover (stop in the air) at the shooting position.

[0089] In step S1230, the controller 30 starts taking pictures of the wind turbine 10 (its blades) while synchronizing the shutter timings of the two cameras (first camera 18 and second camera 20).

[0090] In step S1235, the controller 30 terminates the partial imaging of the wind turbine 10.

[0091] In step S1240, the controller 30 determines whether or not to change the shooting position of the drone 100. This determination is made, for example, based on whether or not the required number of shots, which are predetermined according to the blade length, have been taken. If the controller 30 determines that the shooting position of the drone 100 should be changed (YES in step S1240), it returns control to step S1215. Otherwise (NO in step S1240), the controller 30 switches control to step S1245.

[0092] In step S1245, the controller 30 determines that it has completed taking all the images of the object to be inspected. The drone 100 is then returned to the ground. The still image data stored in the recording medium 38 of the drone 100 is input to the computer device 1000.

[0093] In step S1250, the CPU 1 of the computer device 1000 analyzes the image data obtained by the capture and identifies the hatch 150. For example, the CPU 1 uses AI (Artificial Intelligence) to analyze the image containing the hub 140 and identifies the hatch 150 from the image.

[0094] In step S1255, CPU1 labels the image containing the hatch image and the image captured simultaneously as the first blade (for example, blade 120). For example, if blade 120 is located approximately vertically below hub 140, one direction of capture is when drone 100 ascends from the tip of blade 120 (below wind turbine 10) toward hub 140. In this case, the first camera 18 positioned above drone 100 constantly captures images of hub 140, and the second camera 20 positioned below drone 100 captures images of a portion of blade 120. Therefore, in this case, the image containing the hatch image corresponds to the image captured by the first camera 18. Also, the image captured simultaneously with the image containing the hatch image corresponds to the image captured by the second camera 20. The completion of labeling the first blade is determined, for example, based on the drone 100 rising to a position in front of the hub 140 and the extraction of an image of the hub 140 from the image acquired by the second camera 20. Alternatively, the CPU 1 determines that labeling of the first blade is complete when analysis is completed using a number of images corresponding to a number of shots calculated in advance based on the blade length, the shooting range, and the distance between the drone 100 and the blade.

[0095] In step S1260, after the labeling of the first blade is completed, CPU1 starts labeling the second blade (for example, blade 121). Labeling of the second blade is performed in the same manner as labeling of the first blade. The determination of completion of labeling of the second blade is performed in the same manner as the determination of completion of labeling of the first blade.

[0096] In step S1265, after the labeling of the second blade is completed, CPU1 begins labeling the third blade (for example, blade 122). Labeling of the third blade is performed in the same manner as labeling of the first or second blade. The determination of completion of labeling of the third blade is performed in the same manner as the determination of completion of labeling of the first blade.

[0097] Note that the labeling of the first, second, and third blades may be performed simultaneously by CPU1 without separating the steps.

[0098] In step S1270, CPU1 reconstructs an overall image of each blade using each labeled image. For example, CPU1 creates an overall image by arranging the images labeled as belonging to the same blade in chronological order of capture time. In this case, the same part may be included in multiple images. Note that since each image is labeled and the captured blade is identified, an overall image does not necessarily need to be reconstructed.

[0099] In step S1275, CPU1 displays an image of each blade on monitor 8. The displayed image may be either a labeled individual image or a reconstructed overall image. The inspector can check for damage by looking at the image displayed on monitor 8 and confirming the condition of the surface of each blade.

[0100] [Processing steps for acquiring video footage] Referring to Figure 13, the process of identifying blades from video data acquired according to this embodiment will be described. Figure 13 is a flowchart showing part of the process performed to identify blades from video data of the wind turbine 10. Processes identical to those shown in Figure 12 are given the same numbers. Therefore, the explanation of the same process will not be repeated.

[0101] In step S1330, the drone 100 simultaneously starts recording video using two cameras, namely the first camera 18 and the second camera 20. The video data acquired by the first camera 18 and the video data acquired by the second camera 20 are associated with the time obtained from the GPS receiver 34.

[0102] In step S1345, the controller 30 determines that it has completed recording all video of the object to be inspected. The drone 100 is then returned to the ground. The video data stored in the drone 100's recording medium 38 is input to the computer device 1000.

[0103] In step S1350, the CPU 1 of the computer device 1000 analyzes the motion image data obtained by the capture and identifies the hatch 150. For example, the CPU 1 uses AI (Artificial Intelligence) to analyze each frame image that makes up the motion image data containing the hub 140, and identifies the hatch 150 from the frame image.

[0104] In step S1355, CPU1 labels the image frame containing the hatch image and the image frame acquired simultaneously as the first blade (for example, blade 120). For example, if blade 120 is located approximately vertically below hub 140, as one direction of shooting, drone 100 shoots video while ascending from the tip of blade 120 (below wind turbine 10) toward hub 140. In this case, the first camera 18 positioned above drone 100 constantly shoots hub 140, and the second camera 20 positioned below drone 100 shoots a portion of blade 120. Therefore, in this case, the image frame containing the hatch image corresponds to the frame that makes up the video acquired by the first camera 18. Also, the image frame acquired simultaneously with the image frame containing the hatch image corresponds to the frame that makes up the video acquired by the second camera 20. The completion of labeling the first blade is determined, for example, based on the drone 100 ascending to a position in front of the hub 140 and the extraction of an image of the hub 140 from the video footage captured by the second camera 20.

[0105] [Processing procedure for acquiring still and moving images] Furthermore, in other situations, both moving and still images may be acquired simultaneously. In this case, if the timestamp of a frame included in the moving image and the timestamp of the still image are the same, simultaneous capture of the hub 140 and the blade can be achieved using the frame and the mounted still image.

[0106] Therefore, with reference to Figure 14, the shooting process according to the other phases will be explained. Figure 14 is a flowchart showing part of the process performed to identify the blade from moving and still images. Note that the same process as described above is given the same number. Therefore, the explanation of the same process will not be repeated.

[0107] In step S1430, the drone 100 captures a still image using one of its two cameras and a moving image using the other. For example, the drone 100 captures a video of the hub 140 and a still image of the blade using the other camera. The camera used to capture the moving image is determined according to the shooting position and direction of the drone 100 (for example, the direction from the tip of the blade toward the hub 140, or the opposite direction). In other situations, the drone 100 may capture a still image of the hub 140 and a moving image of the blade.

[0108] In step S1435, the controller 30 terminates partial shooting. That is, the drone 100 terminates the capture of still and moving images at the initial position. In this case, the drone 100 may hold the camera used for capturing moving images in a paused state so that shooting at the next position can be quickly realized.

[0109] In step S1240, the controller 30 determines whether or not to change the shooting position. If the drone 100 determines to change the shooting position (YES in step S1240), it switches control to step S1225. Otherwise (NO in step S1240), the controller 30 switches control to step S1445.

[0110] In step S1445, the controller 30 determines that it has completed capturing still and moving images of all the objects to be inspected. The drone 100 is then returned to the ground. The still image data stored in the drone 100's recording medium 38 is input to the computer device 1000.

[0111] In step S1450, the CPU 1 of the computer device 1000 analyzes the motion and still image data obtained by the capture to identify the hatch 150. For example, the CPU 1 identifies an image frame from the motion image data that contains the same timestamp as the data in the still image, and determines whether a portion corresponding to all or part of the hatch 150 is included in either that image frame or the still image. Once the hatch 150 is identified, the CPU 1 makes a decision based on the images that do not contain the hatch 150 and the images that capture the blade, and begins identifying the blade (step S1355).

[0112] Thus, the drone 100 may use one of the two cameras to capture still images and the other to capture moving images, thereby capturing images of the hub 140 and the blades.

[0113] [Processing procedure when photographing the Hub 140 and blades with a single camera] Furthermore, in other situations where a wide field of view can be achieved with a single camera, the drone 100 may capture images of the hub 140 and the blades with a single camera. In this case, since each image data will include the hub 140 and the blades, processing to associate multiple image data becomes unnecessary.

[0114] Therefore, the process in this phase will be explained with reference to Figure 15. Figure 15 is a flowchart showing some of the processes performed when photographing the hub 140 and blade using a single camera. Note that the same processes as described above are given the same numbers. Therefore, the explanation of the same process will not be repeated.

[0115] In step S1515, the controller 30 adjusts the angle of one camera for photographing the hub 140 and blades to match the position of the drone 100. The camera used for photography may be either the first camera 18 or the second camera 20. The camera may be selected, for example, by the position on the drone 100 where the first camera 18 and the second camera 20 are mounted.

[0116] In step S1530, the controller 30 starts recording video using either camera 18 or the second camera 20.

[0117] In step S1535, the controller 30 terminates the one-way imaging of the wind turbine 10.

[0118] In step S1540, the controller 30 determines whether or not to change the drone's shooting position. This determination is made in the same way as in step S1240. If the controller 30 determines to change the shooting position of the drone 100 (YES in step S1540), it returns control to step S1515. Otherwise (NO in step S1540), the controller 30 switches control to step S1345.

[0119] In step S1555, the controller 30 labels the blade visible on the left side of the screen as the first blade, among the blades visible in the same frame as the frame showing the hatch.

[0120] As described above, if the field of view of one camera is wide enough to capture the hub 140 and part of the blade and the hub 140 in the same frame, the drone 100 may use one camera to capture images of the blade and the hub 140 to obtain image data of the blade surface. Although the flowchart illustrated in Figure 15 shows the case of capturing moving images, the drone 100 can also acquire image data in which part of the blade and the hub 140 are included in the same frame if one camera continuously captures still images or if one camera simultaneously captures continuous still images and moving images.

[0121] [Example image] The images acquired by the drone 100 will be described with reference to Figures 16 and 17. Figure 16 shows an image obtained when the first camera 18 photographs the hub 140.

[0122] As shown in Figure 16, since the hatch 150 is provided on the hub 140, the blade that was photographed can be identified based on the positional relationship between the hatch 150 and the blade in the image.

[0123] Figure 17 shows image 1700 of the tip of blade 120 identified in step S1225. The tip contains scratches 1710 and 1720. The inspector can examine blade 120 and report if necessary to determine the need for repair.

[0124] As described above, the drone 100 according to one embodiment has two cameras, and while synchronizing the timing of shooting by each camera, one camera photographs a part of the blade, and the other camera photographs a predetermined part of the wind turbine 10, such as the hatch 150. Since the hatch 150 and a part of the blade are always photographed, in post-image analysis, the blade to which the photographed blade corresponds is identified based on the presence or direction of the part and the positional relationship between each blade and the hatch 150. As a result, even if only a part of the blade is photographed, it is possible to identify which blade's image was acquired, so if scratches or other damage are found during an external inspection using the image, it is easy to determine which blade is damaged.

[0125] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of the present invention is indicated by the claims rather than by the foregoing description, and all modifications within the meaning and scope equivalent to the claims are intended to be included. [Industrial applicability]

[0126] The disclosed technical concept can be used to acquire data necessary for visual inspection using drones or other unmanned aerial vehicles equipped with devices capable of simultaneously photographing two locations. [Explanation of Symbols]

[0127] 1 CPU, 2 Mouse, 3 Keyboard, 4 RAM, 5 Hard disk, 6 Optical disc drive, 7 Communication interface, 8 Monitor, 9 CD-ROM, 10 Wind turbine, 11 Main body, 12 Arms, 13 Propeller motor, 14 Propeller, 15 Motor drive circuit, 16 Engine, 17 Generator, 18 First camera, 18A, 18B, 20A, 20B Cameras, 19, 21 Gimbal, 20 Second camera, 22 Legs, 23 Propulsion mechanism, 24 Sensors, 30 Controller, 31 Storage device, 32 Inertial measurement unit, 33 Transmitter / receiver, 34 GPS receiver, 35 Power circuit, 36 Battery, 37 Reader / writer, 38 Recording medium, 100, 100A, 100B, 100C Drone, 120, 121, 122 Blades, 130 Tower, 140 Hubs, 150; hatches, 160; nacelles, 180; foundations, 210, 220, 230, 240; paths, 700, 1100, 1500; images, 810, 811, 820, 821; image data, 1000; computer equipment.

Claims

1. using one or more cameras mounted on an unmanned aerial vehicle to simultaneously photograph a pre-specified portion of a hub of the wind turbine and a portion of one of a plurality of blades attached to the hub multiple times; repeating the step of performing simultaneous imaging a plurality of times for the remaining blades of the plurality of blades; A method for acquiring data for inspecting a wind turbine, comprising: identifying the photographed blade based on the presence or absence of the pre-identified part or the orientation of the part in an image based on image data acquired by simultaneous shooting, and the positional relationship of the multiple blades to the pre-identified part.

2. The step of performing simultaneous imaging a plurality of times includes: Simultaneously capturing an image of a pre-specified portion of the hub with a first camera of the one or more cameras and capturing an image of a portion of one of the blades with a second camera of the one or more cameras; 2. The method of acquiring data for inspecting a wind turbine according to claim 1, further comprising a step of associating the image data acquired by the first camera and the image data acquired by the second camera with the time at which the images were captured in response to simultaneously capturing images of the pre-specified portion and the part.

3. The step of performing simultaneous photographing a plurality of times includes acquiring still images; 3. The method for acquiring data for inspecting a wind turbine according to claim 2, wherein the step of associating the time includes associating the time with the image data acquired by the first camera and the image data acquired by the second camera.

4. The step of performing simultaneous photographing a plurality of times includes acquiring a moving image; 3. The method for acquiring data for inspecting a wind turbine according to claim 2, wherein the step of associating the time includes associating the time with a frame included in video data acquired by the first camera and a frame included in video data acquired by the second camera.

5. The method for acquiring data for inspecting a wind turbine as described in claim 2, wherein the step of performing simultaneous photographing multiple times includes a step of directing the photographing direction of the first camera toward the pre-identified part so that the pre-identified part does not leave the photographing range in accordance with the movement of the unmanned aerial vehicle.

6. The unmanned aerial vehicle has cameras provided at the top and bottom of the housing, The method for acquiring data for inspecting a wind turbine described in any one of claims 2 to 5, wherein the step of simultaneously photographing includes, when the blade to be photographed is above the hub, the camera provided at the upper part serves as the second camera to photograph a part of one of the blades, and the camera provided at the lower part serves as the first camera to photograph the pre-specified part.

7. The unmanned aerial vehicle has cameras provided at the top and bottom of the housing, The method for acquiring data for inspecting a wind turbine described in any one of claims 2 to 5, wherein the step of simultaneously photographing includes, when the blade to be photographed is below the hub, a camera provided at the lower part serves as the first camera to photograph a part of one of the blades, and a camera provided at the upper part serves as the second camera to photograph the pre-specified part.

8. The method for acquiring data for inspecting a wind turbine according to any one of claims 1 to 7, further comprising a step of identifying a blade to which an image of a portion of one blade acquired at the timing when an image including the pre-identified portion is acquired belongs.

9. The step of identifying includes: analyzing a plurality of said images to extract said pre-identified region from each of said images; and labeling blades included in an image of a portion of one blade acquired at the timing when each of the images was acquired, based on blade identification information that is pre-specified based on a positional relationship between the pre-specified portion and the plurality of blades.

10. A method for acquiring data for inspecting a wind turbine as described in any one of claims 1 to 9, wherein the step of performing simultaneous photographing multiple times includes performing the simultaneous photographing while hovering the unmanned aerial vehicle while the blades are rotating.

11. A method for acquiring data for inspecting a wind turbine as described in any one of claims 1 to 9, wherein the step of performing simultaneous photographing multiple times includes performing the simultaneous photographing while moving the unmanned aerial vehicle while the blades are stationary.

12. Moving the unmanned aerial vehicle includes: moving the blade tip toward the hub; or The method of acquiring data for inspecting a wind turbine according to claim 11 , comprising moving the unmanned aerial vehicle from the hub towards a tip of a blade to be photographed.

13. A program for causing a computer to execute the acquisition method according to any one of claims 1 to 12.

14. An unmanned aerial vehicle, A memory storing the program according to claim 13; one or more cameras disposed on the unmanned aerial vehicle; An unmanned aerial vehicle comprising a control device that executes the program and controls the unmanned aerial vehicle.

Images