Method and device for calculating position of blade defect

Abstract

A method for calculating the position of a defect, according to one aspect of the present invention, comprises the steps of: acquiring images of the nose cone and a blade of a wind power generator by means of a camera of an aerial vehicle; analyzing the images to determine a root point of the blade; analyzing the images to determine a tip point of the blade; detecting a defect included in the images; stitching the images together to generate a merged image; calculating the length per pixel by comparing the number of pixels of the blade included in the merged image and the actual length of the blade; and calculating the position of the defect in the merged image.

Description

Method and device for calculating the location of a defect in a blade

[0001] Relating to a method and device for calculating the location of a defect in a blade.

[0002]

[0003] Aerial vehicles (AVs) fly to approach a target, and various information about the target can be collected through the vehicle's camera. Aerial photography technology is being integrated into various fields and is even being used for the inspection of industrial structures. When the target is an industrial structure, the aerial vehicle's camera can capture images of the exterior or interior, allowing for the identification of any malfunctions or damage to the structure.

[0004] Meanwhile, demand is increasing for technology that merges and outputs images captured by aircraft to make it easier for users to check and manage them.

[0005] When a vehicle captures a subject, it is necessary to capture images with overlapping layers to avoid missing any portion of the subject. Furthermore, stitching multiple images requires a high overlap rate. When a vehicle captures a subject with a high overlap rate, multiple images are generated. These images may contain unnecessary information. Furthermore, storing multiple images requires more storage space, and processing them requires more resources (GPU, CPU, etc.).

[0006] An aircraft can photograph wind turbines and analyze the captured images to identify any defects. Wind turbines have three blades, which are very long structures. Therefore, if a blade is defective, it is necessary to pinpoint the exact location of the defect. Using GPS information to pinpoint the location of the defect can be difficult due to potential inaccuracies in the GPS data.

[0007]

[0008] The present invention provides a method and device for calculating the location of a blade defect. Furthermore, the present invention provides a computer-readable recording medium containing a program for executing the method on a computer. The technical problems to be solved are not limited to the technical problems described above, and other technical problems may exist.

[0009]

[0010] A method for calculating a location of a defect in a blade according to one aspect comprises the steps of: acquiring images of a nose and blade of a wind turbine through a camera of an aircraft; analyzing the images to determine a root point of the blade; analyzing the images to determine a tip point of the blade; detecting a defect included in the images; stitching the images to generate a merged image; calculating a length per pixel by comparing an actual length of the blade with the number of pixels from the root point to the tip point in the merged image; and calculating a location of the defect using the length per pixel.

[0011] A computer-readable recording medium according to another aspect includes a recording medium having recorded thereon a program for executing the above-described method on a computer.

[0012] A device for calculating a location of a defect in a blade according to another aspect comprises: at least one memory; and at least one processor; wherein the at least one processor comprises: acquiring images of a nose and blades of a wind turbine captured through a camera of an aircraft; analyzing the images to determine a root point of the blade; analyzing the images to determine a tip point of the blade; detecting a defect included in the images; stitching the images to generate a merged image; comparing an actual length of the blade with the number of pixels from the root point to the tip point in the merged image to calculate a length per pixel, and calculating a location of the defect using the length per pixel.

[0013]

[0014] FIG. 1 is a drawing for explaining an example of photographing an object using an aircraft according to one embodiment.

[0015] FIG. 2 is a drawing for explaining the relationship between an aircraft, a controller, a server, and a station according to one embodiment.

[0016] Figure 3 is a schematic diagram illustrating an example of an aircraft according to one embodiment.

[0017] FIG. 4 is a flowchart illustrating an example of a method for calculating the location of a defect in a blade according to one embodiment.

[0018] FIG. 5 is a drawing for explaining a method for calculating the location of a defect according to one embodiment.

[0019] FIG. 6 is a drawing illustrating an aircraft moving and photographing a blade according to one embodiment.

[0020] FIG. 7 is a diagram illustrating a processor according to an embodiment of the present invention stitching images by matching common feature points in the images.

[0021] FIG. 8 is a diagram illustrating a method for determining a root point according to one embodiment.

[0022] FIG. 9 is a diagram illustrating a method for determining a root point according to one embodiment.

[0023] FIG. 10 is a drawing for explaining a method for determining a tip point according to one embodiment.

[0024] FIG. 11 is a drawing for explaining a method for determining a tip point according to one embodiment.

[0025] FIG. 12 is a schematic diagram illustrating an example of a device for removing duplicate images according to one embodiment.

[0026] FIG. 13 is a flowchart illustrating a method for indicating a defect according to one embodiment.

[0027] FIG. 14 is a drawing for explaining a method for obtaining an inspection image according to one embodiment.

[0028] FIG. 15 is a drawing for explaining a method for obtaining an inspection image according to one embodiment.

[0029] FIG. 16 is a drawing for explaining a method for detecting a defect in an inspection image according to one embodiment.

[0030] FIG. 17 is a drawing for explaining a method of indicating a defect according to one embodiment.

[0031] FIG. 18 is a drawing for explaining a method of grouping shapes according to one embodiment.

[0032] FIG. 19 is a drawing for explaining a method of grouping and displaying shapes according to one embodiment.

[0033] FIG. 20 is a drawing for explaining a method of grouping shapes by changing their scale according to one embodiment.

[0034] FIG. 21 is a drawing for explaining a method of grouping according to a defect according to one embodiment.

[0035] FIG. 22 is a diagram illustrating a method of grouping using the distance between defects according to one embodiment.

[0036] FIG. 23 is a schematic diagram illustrating an example of a device that exhibits a defect according to one embodiment.

[0037] FIG. 24 is a flowchart illustrating an example of a method for merging images according to one embodiment.

[0038] FIG. 25 is a drawing illustrating an aircraft hopping and photographing an object according to one embodiment.

[0039] FIG. 26 is a drawing illustrating an aircraft moving and photographing a blade according to one embodiment.

[0040] Figure 27 is a drawing for explaining identification of an outline according to one embodiment.

[0041] FIG. 28 is a drawing for explaining a method of placing an image in an initial position according to one embodiment.

[0042] FIGS. 29 and 30 are diagrams illustrating a method for determining an overlapping area according to one embodiment.

[0043] FIG. 31 is a drawing for explaining a method of merging images according to one embodiment.

[0044] FIG. 32 is a drawing for explaining merging images taken of a blade according to one embodiment.

[0045] FIG. 33 is a schematic diagram illustrating an example of a device for merging images according to one embodiment.

[0046]

[0047] The terms used in the examples are selected from widely used, current terms, as much as possible. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, the applicant may arbitrarily select terms, in which case their meanings will be described in detail in the relevant description. Therefore, the terms used in the specification should be defined based on their meaning and the overall content of the specification, rather than simply their names.

[0048] When a part of a specification is said to "include" a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise stated.

[0049] Additionally, terms including ordinal numbers, such as "first" or "second," used in the specification may be used to describe various components, but the components should not be limited by the terms. The terms may be used to distinguish one component from another.

[0050] Below, embodiments are described in detail with reference to the attached drawings. However, the embodiments may be implemented in various different forms and are not limited to the examples described herein.

[0051]

[0052] FIG. 1 is a drawing for explaining an example of photographing an object using an aircraft according to one embodiment.

[0053] The aircraft (10) may include any aircraft capable of flight, including a drone, an Unmanned Aerial Vehicle (UAV), an Unmanned Aerial Mobility (UAM), an aircraft, a helicopter, etc.

[0054] The aircraft (10) can fly alone or with multiple aircraft capable of collaborating. Furthermore, the aircraft (10) can also collaborate with other types of devices, such as vehicles and robots. Furthermore, the aircraft (10) can automatically fly around the target object (20) along a predetermined flight path, or can manually fly around the target object (20) under user control.

[0055] The aircraft (10) can capture images of the object (20) while flying around the object (20). For example, the captured images of the object (20) can be used to determine whether the object (20) has a defect (e.g., breakdown, damage, etc.). The user can detect, recognize, and / or identify the object (20) through the aircraft (10).

[0056] As an example, the aircraft (10) can photograph the body or blades of a wind turbine. Specifically, the aircraft (10) can fly around the wind turbine to photograph the body or blades. For example, the aircraft (10) can take off from a station at a starting point and fly to the nose of the wind turbine. Then, the aircraft (10) can start photographing from the nose of the wind turbine and can photograph the blades while flying autonomously (or manually) along the blades. Typically, a wind turbine can include three blades, and the aircraft (10) can photograph the three blades to acquire images. At this time, the images photographed by the aircraft (10) can be utilized for inspection of the blades.

[0057] As another example, the aircraft (10) can photograph the surface of a large building. For example, the aircraft (10) can acquire images by photographing the surface while flying around the perimeter of the large building. In this case, the images captured by the aircraft (10) can be utilized for inspection of the surface of the large building.

[0058] As another example, the aircraft (10) can photograph various structures within a military facility. For example, a military facility may include barbed wire fences, buildings, and exterior walls, and the aircraft (10) can fly over various points within the military facility and photograph the surfaces of the structures. In this case, the images captured by the aircraft (10) can be utilized for inspection of the military facility.

[0059] Although wind turbines, bridges, large buildings, and military facilities are depicted as objects (20) in FIG. 1, the objects are not limited thereto. In other words, any structure having a shape may be applied to the objects (20) without limitation. For example, the objects (20) may be structures in the industrial field. The objects (20) may be structures for power generation (e.g., wind turbines, thermal power plants, hydroelectric power plants, nuclear power plants, solar power plants, etc.), large buildings (e.g., factories, exhibition halls, stadiums, etc.), bridges, dams, power lines, roads, etc., but are not limited thereto. As another example, the objects (20) may be structures that must be detected or information collected in the security and military fields (e.g., barbed wire fences, ammunition depots, exterior walls, etc.). As another example, anything that is difficult for a user to inspect entirely with the naked eye, dangerous, or requires a lot of manpower and cost for inspection may be applied to the objects (20).

[0060] FIG. 2 is a drawing for explaining the relationship between an aircraft, a controller, a server, and a station according to one embodiment.

[0061] Referring to FIG. 2, the server (30), the controller (40), and the station (50) can independently or jointly control the aircraft (10). For example, the server (30), the controller (40), and the station (50) can control the operation (e.g., movement, rotation, etc.) of the aircraft (10) or control the filming of the aircraft (10).

[0062] The aircraft (10) includes at least one camera, and can capture images of an object (20) using the camera. For example, the camera can be installed at a location advantageous for capturing images during flight of the aircraft (10) (e.g., an area not obscured by a propeller, etc., at the front or lower front of the aircraft).

[0063] For example, the aircraft (10) can fly using a global navigation satellite system (GNSS) and / or an inertial navigation system (INS).

[0064] For example, the aircraft (10) can transmit and receive data with a server (30), a controller (40), and / or a station (50). In addition, the controller (40) and the server (30), the server (30) and the station (50), and the station (50) and the controller (40) can transmit and receive data with each other.

[0065] Here, the data may include data required to control the flight of the aircraft (10), data on a flight image of the aircraft (10), data on an image taken of an object (20) by the aircraft (10), etc.

[0066] A flight image represents the field of view of an aircraft (10) when the aircraft (10) is flying. For example, the flight image may be a dynamic image acquired in real time, but is not limited thereto.

[0067] The image captured by the aircraft (10) of the target object (20) refers to an image captured by the aircraft (10) while flying around the target object (20). At this time, the image captured by the aircraft (10) of the target object (20) can be used as an image to check for defects in the target object (20). In this case, the image captured by the aircraft (10) of the target object (20) can be referred to as an inspection image.

[0068] For example, the flight image may be an image with a relatively low resolution (or a low Ground Sampling Distance (GSD)) compared to the inspection image, and the inspection image may be an image with a relatively high resolution (or a high GSD) compared to the flight image. Meanwhile, depending on the type of aircraft (10), the cameras that generate the flight image and the inspection image may be the same or different.

[0069] A user can control the aircraft (10) using the interface of the controller (40). For example, the controller (40) can generate a control signal based on user input received through the interface and transmit the control signal to the aircraft (10). The controller (40) can transmit the control signal to the aircraft (10) via wireless communication. The control signal may be a signal that controls the flight, attitude, navigation, etc. of the aircraft.

[0070] The aircraft (10) can control the motor to rotate the propeller according to the control signal received from the controller (40). The aircraft (10) can move, rotate, etc. by changing the speed and / or attitude, etc. by the rotation of the propeller. Here, the attitude of the aircraft can be expressed as pitch (Y), roll (X), yaw (Z), etc. In addition, the aircraft can perform photographing of the target object (20), etc. according to the control signal received from the controller (40).

[0071] The controller (40) may further include a display device, and the user may check the flight image and / or inspection image of the aircraft (10) through the display device.

[0072] The controller (40) may be a device on which an application for controlling an aircraft (10) is installed. For example, the device on which the application is installed may be a variety of portable devices such as a smartphone, tablet, smart pad, laptop, or wearable device.

[0073] The server (30) or station (50) can control the aircraft (10) by directly transmitting a control signal to the aircraft. In addition, the aircraft (10) can transmit a flight image and / or an inspection image to the server (30), controller (40), or station (50).

[0074] The aircraft (10), server (30), controller (40), and station (50) can each analyze the inspection image. For example, the analysis of the inspection image may be to determine whether there is a defect in the target object (20) in the inspection image through an algorithm such as machine learning or deep learning. The aircraft (10) may directly determine whether there is a defect in the target object (20), or may transmit the inspection image to the server (30), controller (40), or station (50). The server (30), controller (40), or station (50) may analyze the inspection image received from the aircraft (10) to determine whether there is a defect in the target object (20).

[0075] Meanwhile, in the case of large structures, there may be areas that are difficult for humans to inspect. Furthermore, inspecting the entire structure by humans can be time-consuming or dangerous. Therefore, using an aircraft (10) to inspect a large structure offers the advantages of shortening inspection time and eliminating the risk of casualties.

[0076] Meanwhile, for easy user confirmation and management, the images captured by the aircraft (10) need to be merged and output. However, in the case of large structures, the merged results of the images captured by the aircraft (10) may be distorted due to various variables. For example, when the aircraft (10) captures a large building with a lot of glass, if an object reflected in the glass is used as a feature point, distortion may occur in the merged image. Merging images may mean creating a single image by matching the common parts included in two or more images. Merging may be expressed as stitching.

[0077] The user can check and manage the target object (20) through the merged image. The user can visually check the target object (20) at a glance through the merged image. In addition, the target object (20) can be photographed periodically (e.g., once a week, once a month, once a year, etc.) using the aircraft (10), and the merged images of the target object (20) can be stored. Accordingly, the user can check the change in defects of the target object (20) over time through the merged images, and repair / manage the defects.

[0078] Figure 3 is a schematic diagram illustrating an example of an aircraft according to one embodiment.

[0079] Referring to FIG. 3, the aircraft (10) may include a sensor (110), a camera (120), a memory (130), a driving device (140), a communication device (150), and a processor (160). However, the components of the aircraft (10) are not limited to those illustrated in FIG. 3. In other words, the aircraft (10) may include at least one more component in addition to the components illustrated in FIG. 3, or at least one of the components illustrated in FIG. 3 may be excluded.

[0080] The sensor (110) detects various information necessary for the operation of the aircraft (10) (e.g., flight, photography, etc.), such as the aircraft (10) itself, the surrounding environment of the aircraft (10), identification of the target (20), and confirmation of the distance between the aircraft (10) and the target (20). The sensor (110) may include, but is not limited to, a gyro sensor, a barometer, an ultrasonic sensor, a magnetic sensor, an acceleration sensor, a proximity sensor, a lidar, a radar, and / or a GPS sensor.

[0081] For example, a gyro sensor and / or an acceleration sensor can measure the three-axis angular velocity of the aircraft (10). A barometer can measure pressure changes and / or air pressure in the atmosphere around the aircraft (10). Since air pressure varies with altitude, the aircraft (10) can also measure the altitude of the aircraft (10) using the barometer. An ultrasonic sensor can measure the distance between the aircraft (10) and the ground or an object (20). A magnetic sensor is a type of terrestrial magnetism sensor (compass sensor) and can detect geomagnetic information.

[0082] For example, a proximity sensor can measure the proximity state of an object (20) to an aircraft (10), the distance between the aircraft (10) and the object (20), and can include an ultrasonic sensor that can measure the distance to the object (20) from a signal reflected from the object (20) by outputting ultrasonic waves. A GPS sensor can calculate the current coordinates (x, y, z) of the aircraft (10) using GPS signals.

[0083] The sensor (110) may include an attitude and heading reference system (AHSR). For example, the attitude and heading reference system may include an inertial sensor or an inertial measurement unit (IMU). For example, the attitude and heading reference system may include a gyro sensor, an acceleration sensor, and a magnetic sensor, and fuse the sensor values ​​to obtain the attitude value of the aircraft (10). ) can be output. Here, the detailed value ( ) can be an angle based on three-dimensional coordinates (x-axis coordinate, y-axis coordinate, z-axis coordinate) according to GPS coordinates.

[0084] The camera (120) can capture images of the object (20) according to instructions from the processor (160). For example, the aircraft (10) can include at least one camera, and can include a low-resolution camera and / or a high-resolution camera.

[0085] The camera (120) can be combined with a gimbal whose angle can be adjusted. Accordingly, the shooting angle of the camera (120) can be adjusted by the gimbal.

[0086] The memory (130) may include any non-transitory computer-readable recording medium. As an example, the memory (130) may include a non-permanent mass storage device such as a random access memory (RAM), a read only memory (ROM), a disk drive, a solid state drive (SSD), a flash memory, etc. As another example, the non-permanent mass storage device such as a ROM, an SSD, a flash memory, a disk drive, etc. may be a separate permanent storage device distinct from the memory. In addition, the memory (130) may store an operating system (OS) and at least one program code (e.g., a code for the processor (160) to perform an operation to be described later with reference to FIGS. 4 to 12).

[0087] These software components may be loaded from a computer-readable recording medium separate from the memory (130). This separate computer-readable recording medium may be a recording medium that can be directly connected to a computer, and may include, for example, a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. Alternatively, the software components may be loaded into the memory (130) via a communication device (150) other than a computer-readable recording medium. For example, at least one program may be loaded into the memory (160) based on a computer program that is installed by files provided by developers or a file distribution system that distributes installation files of applications via the communication device (150).

[0088] The driving device (140) controls the driving of the motor at a speed and direction according to instructions from the processor (160), and accordingly, the rotational speed and direction of the propeller connected to the motor can be controlled. For example, the driving device (140) may include a motor and a propeller.

[0089] The communication device (150) performs data communication between the aircraft (10) and an external device. For example, the communication device (150) may communicate with the controller (40), server (30), and / or station (50) using various communication methods such as infrared communication, RF (Radio Frequency) communication, Wi-Fi communication, ZigBee communication, Bluetooth communication, laser communication, UWB (Ultra-Wideband) communication, LTE, 5G, 6G, and wireless LAN. However, the communication method employed in the communication device (150) is not limited to the above-described method.

[0090] The processor (160) can process commands of a computer program by performing basic arithmetic, logic, and input / output operations. Here, the commands can be provided from memory (130) or an external device (e.g., a server (30), a controller (40), a station (50), etc.). In addition, the processor (160) can generally control the operations of other components included in the aircraft (10).

[0091] For example, the functions performed by each module included in the processor (160) may be performed by one processor or by separate processors. The processor (160) may perform calculations or data processing related to control and / or communication of at least one other component of the aircraft (10).

[0092] The processor (160) acquires static images by capturing at least a portion of the target object (20) through the camera (120) of the aircraft (10). The processor (160) can merge the static images. Since the static images are captured while the aircraft (10) is flying, the captured information may be different from each other. The captured information may be information about the aircraft (10) when the aircraft (10) generates the static image. For example, the captured information includes the position of the aircraft (10), the attitude of the aircraft (10), the distance between the aircraft (10) and the target object (20), the angle of view of the camera (120), and the angle of the camera (120). The processor (160) can merge the static images by using feature points included in the static images. The captured information may be stored in conjunction with the captured static image when capturing the static image. However, since the stored shooting information may differ from the actual shooting information, the processor (160) may estimate the actual shooting information by merging the static images. In other words, when the processor (160) merges the static images based on the stored shooting information, the feature points included in the static images may not match. In this case, the processor (160) merges the static images so that the feature points match, and during the merging process, the actual shooting information may be estimated by moving, rotating, and adjusting the size of the static images.

[0093] In another embodiment, the processor (160) may control the camera (120) to change the angle of the camera (120) to capture the object (20) while the aircraft (10) is positioned at an n-th point (where n is a natural number greater than or equal to 1). In other words, the aircraft (10) may not generate static images while moving, but may stop at a point and generate two or more static images. For example, the aircraft (10) may generate two or more static images at one point and two or more static images at another point.

[0094] Then, the processor (160) generates a partial image by merging at least two or more of the static images using at least one feature point included in each of the static images. For example, the processor (160) can identify at least one common feature point in a first static image and a second static image selected from among the static images. Thereafter, the processor (160) can determine at least one candidate solution for matching the at least one common feature point. Thereafter, the processor (160) can merge the first static image and the second static image based on the optimal solution selected from the at least one candidate solution. In one example, the processor (160) can generate a partial image by matching at least four common features in the first static image and the second static image. The first static image and the second static image are images taken by the aircraft (10) at the same point.

[0095] For example, the processor (160) can use a machine learning model to identify a predetermined object (e.g., glass, background, etc.) in each of the static images, and identify common features in the remaining portions of each of the static images excluding the predetermined object.

[0096] Here, the candidate solution includes a method for matching common feature points by at least one of translation, rotation, and scaling of at least one of the first and second static images. Furthermore, the optimal solution includes a method for selecting at least one solution based on at least one pre-determined criterion.

[0097] Then, the processor (160) generates a block image by merging at least two or more of the partial images using at least one feature point included in each of the partial images. For example, the processor (160) can identify at least one common feature point in a first partial image and a second partial image selected from among the partial images. Thereafter, the processor (160) can determine at least one candidate solution for matching the at least one common feature point. Thereafter, the processor (160) can merge the first partial image and the second partial image based on an optimal solution selected from among the at least one candidate solution.

[0098] The processor (160) may be implemented as an array of a plurality of logic gates, or may be implemented as a combination of a general-purpose microprocessor and a memory storing a program that can be executed on the microprocessor. For example, the processor (160) may include a general-purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, and the like. In some environments, the processor (160) may include an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), and the like. For example, the processor (160) may also refer to a combination of processing devices, such as a combination of a digital signal processor (DSP) and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors coupled with a digital signal processor (DSP) core, or any other such combination of configurations.

[0099] Meanwhile, the operation of the processor (160) described above with reference to FIG. 3 may be implemented by a separate device. In this case, the separate device (hereinafter referred to as a “device for removing duplicate images”) may be included in at least one of the aircraft (10), the server (30), the controller (40), the station (50), and another external device. When the device for removing duplicate images is included in at least one of the server (30), the controller (40), the station (50), and another external device, the aircraft (10) may capture static images and dynamic images, and transmit the captured images to the device for removing duplicate images.

[0100] FIG. 4 is a flowchart illustrating an example of a method for calculating the location of a defect in a blade according to one embodiment.

[0101] The method illustrated in FIG. 4 is comprised of steps that are processed sequentially in the aircraft (10) or processor (160) illustrated in FIGS. 1 to 3. Therefore, even if omitted below, the content described above regarding the aircraft (10) or processor (160) can also be applied to the method illustrated in FIG. 4.

[0102] At step 410, the processor (160) acquires images of the nose and blades of the wind turbine through the camera (120) of the aircraft (10). The acquired images are high-definition static images. The wind turbine has a nose and three blades connected to the nose. The aircraft (10) can capture images by moving from the nose toward the tips of the blades, or by moving from the tips of the blades toward the nose. The aircraft (10) can capture images by capturing the nose or a portion of the blades. The aircraft (10) can capture images by capturing the blades in various ways.

[0103] At step 420, the processor (160) analyzes the images to determine the root point of the blade. The processor (160) determines an image including the nose of the wind turbine from among the acquired images. The processor (160) determines the root point in the image including the nose. Determining the root point may mean that the processor (160) determines a point in the image as the starting point of the blade. In other words, the processor (160) may designate the point where the blade starts as the root point.

[0104] The processor (160) can determine a root point from an image that includes both a nose and a blade. The processor (160) determines an image that includes both a first bounding box representing the nose and a second bounding box representing the blade. The processor (160) can determine the center points of the first bounding box and the second bounding box from the determined image as root points. If the first bounding box and the second bounding box overlap, the processor (160) can determine the center of the overlapping area as the root point (center point). If the first bounding box and the second bounding box do not overlap, the processor (160) can determine the center of the area between the first bounding box and the second bounding box as the root point (center point).

[0105] When the processor (160) indicates the location of a defect in the blade, it indicates the distance from the root point as the location of the defect. For example, if the defect is 20 meters away from the root point, the processor (160) indicates the location of the defect as 20 meters.

[0106] The length of a blade can be expressed as the distance from the root point to the tip point. Accordingly, after determining the root point and the tip point, the processor (160) can generate a centerline connecting the root point and the tip point. The processor (160) can display the distance traveled along the centerline from the root point as the location of the defect.

[0107] At step 430, the processor (160) analyzes the images to determine the tip point of the blade. The processor (160) determines an image containing the tip of the blade from among the acquired images. The processor (160) determines the tip point in the image containing the tip. Determining the tip point may mean that the processor (160) determines a point in the image as the end point of the blade.

[0108] When the aircraft (10) moves horizontally and takes a picture of the blade, the processor (160) compares the horizontal length of the bounding box representing the blade with the horizontal length of the image to determine an image that includes a tip point. For example, if the ratio of the horizontal length of the bounding box to the horizontal length of the image is less than or equal to a threshold value, the processor (160) may determine that the image includes a tip point.

[0109] When the aircraft (10) moves vertically and captures the blade, the processor (160) compares the vertical length of the bounding box representing the blade with the vertical length of the image to determine an image that includes a tip point. For example, if the ratio of the vertical length of the bounding box to the vertical length of the image is less than or equal to a threshold value, the processor (160) may determine that the image includes a tip point.

[0110] The processor (160) may determine the pixel furthest from the pixels of the blade located on the edge of the image as the tip point. Alternatively, the processor (160) may determine the vertex of a bounding box representing the blade in the image including the tip point as the tip point.

[0111] At step 440, the processor (160) detects defects contained in the images. The processor (160) can analyze the images. For example, the processor (160) can determine whether there is a defect in the images through an algorithm such as machine learning or deep learning. The aircraft (10) can directly identify a defect in the blade, or can transmit the image to a server (30), a controller (40), or a station (50). The server (30), a controller (40), or a station (50) can analyze the image received from the aircraft (10) to identify a defect in the blade.

[0112] The processor (160) divides one image into multiple patches. The image captured by the aircraft (10) is a high-quality image. Analyzing a high-quality image requires a lot of resources. Therefore, the processor (160) can divide one image into multiple patches and then analyze each patch. A patch represents a portion of the image. A patch can be expressed as a unit, piece, tile, etc. The processor (160) identifies a defect contained in each patch. The processor (160) analyzes each patch to identify the presence of a defect, the size or shape of the defect, the type of defect, the severity of the defect, etc. The processor (160) can also use artificial intelligence to identify the defect contained in the patches.

[0113] The processor (160) determines a shape representing a defect. "Representing a defect" may mean displaying a shape on the image to identify the defect. In other words, the processor (160) may display a shape at the location where the defect exists on the image. The shape may be displayed overlapping the defect on the image. Accordingly, the processor (160) can identify the location and size of the defect on the image.

[0114] Additionally, the shape may be represented as a straight line, curve, or the like containing a defect. The processor (160) may use the shape to represent the defect. For example, the shape may be a square, circle, non-linear curve, or the like. The shape may be a mask having the same shape as the defect. The shape may be displayed in black and white or color, and only the border of the shape may be displayed, or the border may be displayed as a dotted line, a solid line, or the like.

[0115] In step 450, the processor (160) stitches the images to generate a merged image. The processor (160) may stitch the images using at least one feature point included in each of the images. In one example, the processor (160) may stitch the images using at least four feature points included in each of the images. The merged image may be generated by rearranging the images on a two-dimensional plane. In other words, the processor (160) may arrange the images on the two-dimensional plane by moving, rotating, resizing, etc. The resizing may be zooming in or out.

[0116] At step 460, the processor (160) calculates the length per pixel by comparing the actual length of the blade with the number of pixels from the root point to the tip point in the merged image. The processor (160) can calculate the length per pixel by dividing the actual length of the blade by the number of pixels.

[0117] At step 470, the processor (160) calculates the location of the defect using the length per pixel. The location of the defect can also be expressed as an R value. The processor (160) calculates the number of pixels from the root point to the location of the defect. The processor (160) determines the location of the defect on the centerline of the blade connecting the root point and the tip point. In other words, the processor (160) determines any point on the centerline as the location of the defect. Once the location of the defect on the centerline is determined, the processor (160) counts the number of pixels from the root point to the location of the defect. The processor (160) calculates the location of the defect by multiplying the number of pixels to the location of the defect by the length per pixel.

[0118] As another example, the processor (160) can calculate the location of a defect using a straight line directly connecting the defect from a root point.

[0119] If the defect is not located on the center line, the processor (160) determines the location of the defect using a straight line passing through the defect and perpendicular to the center line. The processor (160) determines the point where the straight line intersects the center line as the location of the defect. The processor (160) may use a straight line passing through the center of the defect or a straight line passing through the top or bottom of the defect. The location of the defect where the straight line passes through can be determined in various ways. The defect may be represented as a bounding box, and the processor (160) may determine the point where a straight line passing through the center of the bounding box intersects the center line perpendicularly as the location of the defect.

[0120] FIG. 5 is a drawing for explaining a method for calculating the location of a defect according to one embodiment.

[0121] Referring to FIG. 5, the processor (160) can detect a defect (500) and calculate the location (R) of the defect. As shown in FIG. 5, the location (R) of the defect is the distance from the root point (540) to the defect (500). The processor (160) can calculate the R value and indicate the location of the defect.

[0122] The processor (160) can identify the nose (510) and the blade (520), and determine the root point (540) using the bounding boxes of the nose (510) and the blade (520). The root point (540) may be an end point of the blade (520). In addition, the processor (160) can determine the tip point (550) using the bounding box of the blade (520). The center line (530) is a straight line connecting the root point (540) and the tip point (550). The number of pixels of the center line (530) may represent the length of the blade (520). The number of pixels may be a unit representing the length in the image.

[0123] FIG. 6 is a drawing illustrating an aircraft moving and photographing a blade according to one embodiment.

[0124] The aircraft (10) can photograph the target object (20) while maintaining the shooting direction and changing the shooting position. In FIG. 6, the target object (20) may be a wind turbine. More specifically, the aircraft (10) can photograph the blades (21, 22, 23) of the wind turbine. The moving direction of the aircraft (10) may be perpendicular to the shooting direction, but the moving direction may vary depending on the direction of the blades, the angle of the camera (120), the attitude of the aircraft (10), etc.

[0125] In FIG. 6, only the shooting positions (610, 620) are shown, but the aircraft (10) can shoot all sides of the blades (21, 22, 23) at more shooting positions.

[0126] The aircraft (10) can photograph the first blade (21) at the first photographing position (610) and move to the second photographing position (620) to photograph the first blade (21). In this manner, the aircraft (10) can photograph the blades (21, 22, 23) while moving up and down. The aircraft (10) can photograph the blades (21, 22, 23) at multiple photographing positions while moving from the root to the tip of the blade or from the tip to the root.

[0127] In FIG. 6, the areas (611, 621) are shown spaced apart for convenience of explanation, but the areas (611, 621) may overlap, and although only two shooting positions (610, 620) are shown, the aircraft (10) may shoot the blades (21, 22, 23) at a greater number of shooting positions.

[0128] The processor (160) can control the aircraft (10) to move up and down while maintaining the same distance (D) from the target (20).

[0129] A merged image may be generated for each blade. For example, a first merged image may be generated for the first blade (21), a second merged image may be generated for the second blade (22), and a third merged image may be generated for the third blade (23).

[0130] The aircraft (10) can hop and photograph the blades (21, 22, 23). Hopping of the aircraft (10) means moving from one point to another. The aircraft (10) can photograph the blades (21, 22, 23) while changing the angle of the camera (120) while being located at the n-th point (where n is a natural number greater than or equal to 1). The first to n-th points may be locations where at least two of latitude, longitude, or altitude are the same. For example, the aircraft (10) can move by maintaining the same latitude and longitude and changing the altitude.

[0131] In one example, the direction of movement of the aircraft (10) and the direction in which the angle of the camera (120) changes may be perpendicular to each other. For example, if the aircraft (10) hops in a horizontal direction, the angle of the camera (120) may change in a vertical direction. Conversely, if the aircraft (10) hops in a vertical direction, the angle of the camera (120) may change in a horizontal direction.

[0132] In another example, the direction of movement of the aircraft (10) and the direction in which the angle of the camera (120) changes may be the same. For example, if the aircraft (10) hops horizontally, the angle of the camera (120) may change horizontally. Conversely, if the aircraft (10) hops vertically, the angle of the camera (10) may change vertically.

[0133] The aircraft (10) is positioned at a first shooting position (610) and can capture a first blade (21) while changing the angle of the camera (120) at the first point (610) to generate multiple images. The angle of the camera (120) can be changed up and down or left and right. When the aircraft (10) includes multiple cameras (120), the aircraft (10) can acquire multiple images at one point without changing the angle of the camera (120).

[0134] After that, the aircraft (10) moves from the first shooting position (610) to the second shooting position (620), and the blade (21) can be shot while changing the angle of the camera (120) at the second shooting position (620).

[0135] The processor (160) may stitch images acquired at a single shooting location to generate a partial image, and may also stitch the partial images to generate a merged image. For example, the processor (160) may stitch three images at a first shooting location (610) to generate a single partial image. The processor (160) may stitch n partial images generated at n shooting locations to generate a single merged image representing the first blade (21).

[0136] The processor (160) can determine the direction in which the images are stitched depending on the direction in which the angle of the camera (120) changes. If the aircraft (10) is in the same shooting position, and the processor (160) adjusts the angle of the camera (120) in the vertical direction to capture the blade, the processor (160) can stitch the images in the vertical direction. In other words, the processor (160) can stitch the images captured while changing the angle of the camera (120) in the vertical direction to create a partial image. In the same manner, if the processor (160) adjusts the angle of the camera (120) in the horizontal direction to capture the blade, the processor (160) can stitch the images in the horizontal direction. In other words, the processor (160) can stitch the images captured while changing the angle of the camera (120) in the horizontal direction to create a partial image.

[0137] The processor (160) can obtain information about the position of the aircraft (10) at the time when the images are captured, information about the distance between the aircraft (10) and the target object (20), information about the angle of view of the camera (120), and information about the direction in which the camera (120) captures the images (i.e., the capturing angle of the camera (120). The information obtained at the time when the images are captured can be displayed as capturing information, and the capturing information is matched with the images and stored. The processor (160) can stitch the images using the capturing information. In one example, the processor (160) can use the capturing information when stitching the images captured at the first capturing location (610).

[0138] This is a diagram illustrating a processor (160) stitching images by matching common feature points in the images according to an embodiment of the present invention. The processor (160) can stitch the images to create a merged image. The processor (160) can determine a root point and a tip point in the merged image, and can represent the length of the blade in the merged image as a number of pixels.

[0139] The first static image (710) and the second static image (720) may be continuously captured images. The first static image (710) and the second static image (720) may commonly include the same portion of the object (20). The first and second static images (710, 720) may be partial images as described in FIG. 6. In other words, the method of stitching the first and second static images (710, 720) in FIG. 7 may be equally applied to the method of stitching partial images to create a merged image.

[0140] The processor (160) identifies at least one feature point (square, star, triangle) in the first static image (710). In Fig. 7, three feature points are illustrated as an example for convenience of explanation, but at least four feature points may be required for stitching. For example, the processor (160) may identify at least one feature point by considering the type of the object (20), the external features of the object (20), the external environment (weather, time, etc.) at the time the object (20) was photographed, etc.

[0141] In the same manner as described above, the processor (160) identifies at least one feature point (circle, star, triangle) in the second static image (720).

[0142] The processor (160) identifies common features in the first and second static images (710, 720). In FIG. 7, the processor (160) can identify stars and triangles as common features. Accordingly, the processor (160) matches the stars and triangles by moving, rotating, and resizing the first and second static images (710, 720).

[0143] The processor (160) can perform stitching using feature points included in the object (20). For example, a static image can include the object (20) and a background. The background can be an object other than the object (20). The feature points can be included in the object (20) or can be included in the background. Therefore, the processor (160) can use only feature points included in the object (20) for stitching. For example, if the object (20) is a blade, the static image can include the blade and the background. Therefore, the processor (160) can stitch the static images using only specific points included in the blade.

[0144] The processor (160) determines at least one candidate solution for matching at least one common feature point. For example, the candidate solution includes a method for matching the common feature point by at least one of translation, rotation, and scaling of at least one of the first and second static images (710, 720).

[0145] The processor (160) determines a candidate solution to which common feature points (stars, triangles) included in the first and second static images (710, 720) can be matched. For example, the processor (160) may determine a candidate solution that manipulates at least one of the first and second static images (710, 720) so that the positions and directions of the feature points (stars, triangles) can be matched identically. Here, the manipulation includes moving, rotating, and / or resizing at least one of the first and second static images (710, 720). In addition, the resizing includes enlarging or reducing the size of the image.

[0146] The processor (160) stitches the first and second static images (710, 720) based on an optimal solution selected from at least one candidate solution.

[0147] The processor (160) can determine at least one candidate solution that can match common features. However, considering the conditions (or method) under which the aircraft (10) captured the static images (710, 720), some of the candidate solutions may not be suitable for image stitching. Therefore, the processor (160) selects the optimal solution among the candidate solutions, taking into account the shooting method.

[0148] For example, the optimal solution can be selected from among candidate solutions based on at least one predefined criterion. The criterion can be predefined in relation to the translation, rotation, and / or scaling of the image. Furthermore, the criterion can be set differently depending on whether the image is to be merged horizontally or vertically. Furthermore, the criterion can be set differently depending on how the image is captured.

[0149] As an example, the processor (160) may exclude from the candidate solutions a solution whose image rotation angle exceeds a threshold value. If the processor (160) acquires the first and second static images (710, 720) by only changing the angle of the camera (120), the processor (160) may exclude (or assign a low weight to) a candidate solution that includes a method of matching with a rotation angle exceeding the threshold value from the candidate solutions.

[0150] As another example, the processor (160) may exclude from among candidate solutions a solution whose degree of image resizing exceeds a threshold.

[0151] As another example, the processor (160) may exclude from among candidate solutions a solution whose image movement degree exceeds a threshold value.

[0152] To generate a merged image, the processor (160) performs translation, rotation, and / or resizing of static images. Once the merged image is generated, the translation, rotation, and / or resizing of the static images included in the merged image is completed.

[0153] The processor (160) can obtain two-dimensional coordinates of static images using stitching information. The stitching information includes information on movement, rotation, and size adjustment of the static images during the process of generating a merged image. The two-dimensional coordinates of the static images may refer to the relative positions and sizes of the static images. The positions and sizes of the static images may be changed during the process of generating the merged image, and the processor (160) obtains two-dimensional coordinates of the static images through the changed positions and sizes of the static images after generating the merged image.

[0154] Fig. 8 is a diagram for explaining a method for determining a root point according to one embodiment. Fig. 8 is a diagram for explaining a method for determining a root point (830) when a nose bounding box (810) and a blade bounding box (820) intersect.

[0155] The processor (160) can determine a root point (830) using a nose bounding box (810) and a blade bounding box (820). The nose bounding box (810) is a shape that identifies the nose of a wind turbine and is represented in the image (800). The nose bounding box (810) may be a rectangle that includes the nose. The blade bounding box (820) is a shape that identifies the blade of a wind turbine and is represented in the image (800). The blade bounding box (820) may be a rectangle that includes the blade. The processor (160) can determine a root point (830) using an image (800) that includes both the nose bounding box (810) and the blade bounding box (820).

[0156] The processor (160) may determine any point in the area where the nose bounding box (810) and the blade bounding box (820) intersect as the root point (830). For example, the area where the nose bounding box (810) and the blade bounding box (820) intersect may be a rectangle, and the processor (160) may determine the center of the rectangle as the root point (830). The center of the rectangle may be the intersection of the diagonals of the rectangle.

[0157] In another example, the processor (160) may determine the vertex of the blade bounding box (820) located within the nose bounding box (810) as the root point.

[0158] In another example, the processor (160) may determine the vertex of the nose bounding box (810) located within the blade bounding box (820) as the root point.

[0159] FIG. 9 is a diagram for explaining a method for determining a root point according to one embodiment. FIG. 9 is a diagram for explaining a method for determining a root point (930) when a nose bounding box (910) and a blade bounding box (920) do not intersect.

[0160] The processor (160) may determine any point between the nose bounding box (910) and the blade bounding box (920) as the root point (930). For example, the processor (160) may determine the point where the straight lines connecting the vertices of the nose bounding box (910) and the blade bounding box (920) intersect as the root point (930).

[0161] In another example, the processor (160) may determine the center point of the side of the nose bounding box (910) as the root point. At this time, the side of the nose bounding box (910) may be a side adjacent to the blade bounding box (930). In FIG. 9, the processor (160) may determine the center point of the side of the upper side of the nose bounding box (910) as the root point.

[0162] In another example, the processor (160) may determine the center point of a side of the blade bounding box (920) as a root point. The side of the blade bounding box (920) may be a side adjacent to the nose bounding box (910). In FIG. 9, the processor (160) may determine the center point of the side at the bottom of the blade bounding box (920) as a root point.

[0163] FIG. 10 is a diagram illustrating a method for determining a tip point according to one embodiment. The processor (160) can determine an image (1010) including a tip point (1050) and the tip point (1050) among the images.

[0164] Figure 10 is an image (1010) obtained by photographing a blade (1030) in a horizontal direction of an aircraft (10). If the ratio of the horizontal length of the bounding box (1020) to the horizontal length of the image (1010) is less than or equal to a threshold value, the processor (160) may determine that the image (1010) includes a tip point.

[0165] In another example, if the identified blade (1030) intersects only one side of the image (1010), the processor (160) may determine that the image (1010) includes a tip point. The blade (1030) intersecting the image (1010) may mean that pixels of the blade (1030) are located on the edges of the image (1010). The image (1010) may be divided into left and right sides with respect to the center. The blade (1030) intersecting only one side of the image (1010) may mean that edges on the left side of the image (1010) intersect the blade (1030) or edges on the right side of the image (1010) intersect the blade (1030). Region (1040) of FIG. 10 represents pixels of the blade (1030) intersecting the edges on the left side of the image (1010).

[0166] The processor (160) can determine the pixel that is farthest from the pixels located on the edge of the image (1010) as the tip point (1050). The processor (160) can identify the blade (1030) in the image (1010) and determine which pixels of the image (1010) are included in the blade (1030) through segmentation. The processor (160) can determine the pixel that is farthest from the pixels located on the edge of the image (1010) among the pixels included in the blade (1030) as the tip point (1050). Accordingly, the processor (160) determines the pixel that is farthest from the pixels located in the area (1040) as the tip point (1050).

[0167] In another example, the processor (160) may determine the point where the bounding box (1020) and the blade (1030) intersect as the tip point (1050).

[0168] FIG. 11 is a diagram illustrating a method for determining a tip point according to one embodiment. The processor (160) can determine an image (1110) including a tip point (1150) and a tip point (1150) among images.

[0169] Fig. 11 is an image (1110) obtained by photographing a blade (1130) in a vertical direction by an aircraft (10). If the ratio of the vertical length of the image (1110) to the vertical length of the bounding box (1120) is less than or equal to a threshold value, the processor (160) may determine that the image (1110) includes a tip point (1150).

[0170] In another example, if the identified blade (1130) intersects only one side of the image (1110), the processor (160) may determine that the image (1110) includes the tip point (1150). The blade (1130) intersecting the image (1110) may mean that a pixel of the blade (1130) is located on an edge of the image (1110). The image (1110) may be divided into an upper side and a lower side based on the center. The blade (1130) intersecting only one side of the image (1110) may mean that the edges on the upper side of the image (1110) intersect the blade (1130), or that the edges on the lower side of the image (1010) intersect the blade (1130). Area (1140) of FIG. 11 represents where pixels of the blade (1130) intersect the edge on the upper side of the image (1110).

[0171] The processor (160) can determine the pixel that is farthest from the pixels located on the edge of the image (1110) as the tip point (1150). The processor (160) can identify the blade (1130) in the image (1110) and determine which pixels of the image (1110) are included in the blade (1130) through segmentation. The processor (160) can determine the pixel that is farthest from the pixels located on the edge of the image (1110) among the pixels included in the blade (1130) as the tip point (1150). Accordingly, the processor (160) determines the pixel that is farthest from the pixels located in the area (1140) as the tip point (1150).

[0172] In another example, the processor (160) may determine the point where the bounding box (1120) and the blade (1130) intersect as the tip point (1150).

[0173] Fig. 12 is a schematic diagram illustrating an example of a device for calculating the location of a defect according to one embodiment.

[0174] Referring to FIG. 12, a device (1200) for calculating a defect location includes a processor (1210), a memory (1220), and a communication device (1230). However, the components of the device (1200) are not limited to those illustrated in FIG. 12. In other words, the device (1200) may include at least one more component in addition to the components illustrated in FIG. 12, or at least one of the components illustrated in FIG. 12 may be excluded.

[0175] As described above with reference to FIG. 3, the device (1200) may be included in at least one of the aircraft (10), the server (30), the controller (40), the station (50), and other external devices. Accordingly, the processor (160), the memory (130), and the communication device (150) described above with reference to FIG. 3 may correspond to the processor (1210), the memory (1220), and the communication device (1230) of the device (1200), respectively. Therefore, a detailed description of the processor (1210), the memory (1220), and the communication device (1230) is omitted.

[0176]

[0177] FIG. 13 is a flowchart illustrating a method for indicating a defect according to one embodiment.

[0178] The method illustrated in FIG. 13 consists of steps that are processed sequentially in the aircraft (10) or processor (160) illustrated in FIGS. 1 to 3. Therefore, even if omitted below, the content described above regarding the aircraft (10) or processor (160) can also be applied to the method illustrated in FIG. 13.

[0179] In step 410, the processor (160) acquires an inspection image in which a portion of the target object (20) is captured through the camera (120) of the aircraft (10). The aircraft (10) can capture the target object (20) in various ways to acquire inspection images. The method by which the aircraft (10) acquires the inspection image is described in detail with reference to FIGS. 14 and 15. The inspection image is an image used to determine whether the target object (20) has a defect.

[0180] In step 420, the processor (160) divides the inspection image into multiple patches. The inspection image is a high-quality image. Analyzing a high-quality inspection image requires a lot of resources. Therefore, the processor (160) can analyze each patch after dividing the inspection image into patches. A patch represents a portion of the inspection image. A patch can be expressed as a unit, piece, tile, etc.

[0181] At step 430, the processor (160) identifies defects contained in each patch. The processor (160) analyzes each patch to identify the presence of a defect, the size or shape of the defect, the type of defect, the severity of the defect, etc. The processor (160) may also use artificial intelligence to identify defects contained in the patches.

[0182] At step 440, the processor (160) determines a shape representing a defect. "Representing a defect" may refer to displaying a shape on the image to identify the defect. In other words, the processor (160) may display a shape at the location where the defect exists on the image. The shape may be displayed overlapping the defect on the inspection image. Additionally, the shape may be displayed as a straight line, curve, or the like that includes the defect.

[0183] The processor (160) may use a shape to represent a defect. For example, the shape may be a square, a circle, a non-linear curve, etc. The shape may be a mask having the same shape as the defect. The shape may be displayed in black and white or in color, and only the shape's border may be displayed, or the border may be displayed as a dotted line, a solid line, etc.

[0184] In step 450, the processor (160) groups two or more shapes among the shapes. Grouping two or more shapes may mean expressing two or more shapes as one shape.

[0185] The processor (160) may group two or more shapes when two or more shapes overlap. Alternatively, the processor (160) may determine shapes to be grouped by considering the type and severity of the defect, the size of the defect, the distance between the defects, etc.

[0186] The processor (160) can change the scale of a shape representing a defect, and group two or more shapes when two or more shapes overlap according to the changed scale. The scale of the shape can represent the distance between the defect and the shape, or the ratio of the size of the defect to the size of the shape. For example, changing to a small scale means shortening the distance between the defect and the shape, or a small ratio of the size of the shape to the size of the defect. Conversely, changing to a large scale means widening the distance between the defect and the shape, or a large ratio of the size of the shape to the size of the defect.

[0187] The processor (160) can identify the type of defect and, if two or more shapes representing the same defect overlap, group the two or more shapes. The types of defects may include contamination, damage, erosion, hole, open, crack, etc.

[0188] The processor (160) identifies the severity of the defect. When there are defects with the same severity (or a specified severity), and two or more shapes representing the defects overlap, the processor (160) may group the two or more shapes. The severity may be expressed as a grade. For example, the severity may be expressed as 1 to 5. The severity may be determined based on the degree of contamination, damage, cracking, etc. For example, a severity of 1 may indicate simple contamination, and a severity of 14 may indicate that the defect must be repaired immediately or that operation must be stopped. For example, when there are two defects with a severity of 3, and the two shapes representing the defects overlap, the processor (160) may group the two shapes. Alternatively, when there are two defects with low severities of 1 and 2, and the two shapes representing the defects overlap, the processor (160) may group the two shapes.

[0189] The processor (160) may apply a smaller scale to shapes representing low-severity defects. Conversely, the processor (160) may apply a larger scale to shapes representing low-severity defects. The user can adjust the scale to determine whether low-severity defects are displayed as a single shape or as multiple shapes.

[0190] The processor (160) may apply a smaller scale to a shape representing a high-severity defect. Conversely, the processor (160) may apply a larger scale to a shape representing a high-severity defect. The user can adjust the scale to determine whether high-severity defects are displayed as a single shape or as multiple shapes.

[0191] The processor (160) may determine whether to group defects based on the distance between defects and / or the size of the defects. The distance between defects may be the distance between the center points of two defects or the shortest distance between two defects. The size of the defect may be the area of ​​the defect, the maximum length of the defect, etc. The processor (160) may group defects when the distance between defects is less than or equal to a threshold value. The threshold value may be determined based on the size of the defect. If the size of the defect is large, the threshold value may be increased, and if the size of the defect is small, the threshold value may be decreased.

[0192] In step 460, the processor (160) displays the grouping result on the inspection image. The grouping result may mean displaying two or more shapes as a single shape. The processor (160) displays a shape representing a defect on the inspection image. The processor (160) may overlap the shape on the inspection image. In other words, the processor (160) may overlap the defect and the shape. The processor (160) may display the shape on the inspection image so that the user can easily check the location and size of the defect.

[0193] Fig. 14 is a diagram illustrating a method for acquiring an inspection image according to one embodiment. An aircraft (10) can acquire an inspection image by hopping and photographing a target object (20).

[0194] Referring to FIG. 14, the processor (160) can photograph the target object (20) while changing the angle of the camera (120) while the aircraft (10) is located at the n-th point (where n is a natural number greater than or equal to 1).

[0195] For example, the first to nth points may be positions having the same altitude, and the aircraft (10) may photograph the object (20) at the first to nth points while moving in the horizontal direction. Conversely, the first to nth points may be positions having different altitudes, and the aircraft (10) may photograph the object (20) at the first to nth points while moving in the vertical direction.

[0196] In one example, the direction of movement of the aircraft (10) and the direction in which the angle of the camera (120) changes may be perpendicular to each other. For example, if the aircraft (10) hops horizontally, the angle of the camera (120) may change vertically. Conversely, if the aircraft (10) hops vertically, the angle of the camera (120) may change horizontally. Hopping of the aircraft (10) means moving from one point to another.

[0197] In another example, the direction of movement of the aircraft (10) and the direction in which the angle of the camera (120) changes may be the same. For example, if the aircraft (10) hops horizontally, the angle of the camera (120) may change horizontally. Conversely, if the aircraft (10) hops vertically, the angle of the camera (10) may change vertically.

[0198] In another example, the aircraft (10) may include a plurality of cameras (120). The cameras (120) may be arranged vertically or horizontally. The cameras (120) may be attached to the aircraft (10) so that their angles of view overlap by a predetermined ratio. When the aircraft (10) photographs an object (20) using a plurality of cameras (120), there is no need to change the angles of the cameras (120) while photographing the object (20). Therefore, the time required for the aircraft (10) to photograph the object (20) can be shortened.

[0199] First, the aircraft (10) is positioned at a first point, and can capture a portion (510) of the target object (20) by changing the angle of the camera (120) at the first point to generate multiple inspection images. Accordingly, at least one image representing the portion (510) can be acquired. The angle of the camera (120) can be changed up and down or left and right. When the aircraft (10) includes multiple cameras (120), the aircraft (10) can acquire multiple inspection images at one point without changing the angle of the camera (120).

[0200] Thereafter, the aircraft (10) can move from the first point to the second point, and change the angle of the camera (120) at the second point to capture a portion (520) of the target object (20). Accordingly, at least one image representing the portion (520) can be acquired. In Fig. 14, for convenience of explanation, the areas captured by the aircraft (10) at the first point and the second point are depicted as being spaced apart, but the areas captured by the aircraft (10) at the first point and the second point may overlap each other.

[0201] In the same manner, the aircraft (10) may move from a second point to a third point and photograph a portion (530), thereby obtaining at least one image representing the portion (530). In FIG. 14, for convenience of explanation, the areas photographed by the aircraft (10) at the second point and the third point are depicted as being spaced apart, but the areas photographed by the aircraft (10) at the second point and the third point may overlap each other.

[0202] The inspection images can be acquired through a single flight of the aircraft (10). Specifically, the images generated by photographing part (510) can all be acquired during the same flight. Similarly, the images generated by photographing part (520) and the images generated by photographing part (530) can all be acquired during the same flight.

[0203] Of course, some of the images generated as portions (510) to (530) are captured may be acquired in other flights. However, capturing a portion is accomplished in a single flight (i.e., one flight). One flight may refer to the period from when the aircraft (10) takes off to begin capturing images until it lands.

[0204] The processor (160) may determine the number of images acquired for each section (510, 520, 530) by considering the overlap rate between images, the angle of view of the camera (120), the required resolution of the images, etc. For example, the required value for the GSD of the images may be determined by the distance between the aircraft (10) and the target (20) and the number of pixels of the camera, but is not limited thereto. The number of images acquired for each section (510, 520, 530) may be 1 to 7, but is not limited thereto. In addition, the processor (160) may determine the distance to the target (20) by considering the GSD of the images, the specifications of the camera, and / or the overlap rate between the images. For example, in order to easily perform merging of images, the aircraft (10) may generate an odd number of images at one point.

[0205] In addition, although FIG. 14 illustrates an aircraft (10) photographing an object (20) while moving in a horizontal direction, this is not limited thereto. For example, the aircraft (10) may also photograph an object (20) while moving in a vertical direction.

[0206] FIG. 15 is a drawing for explaining a method for obtaining an inspection image according to one embodiment.

[0207] The aircraft (10) can photograph the target object (20) while maintaining the shooting direction and changing the shooting position. The moving direction of the aircraft (10) may be perpendicular to the shooting direction, but may vary depending on the angle of the camera (120) or the attitude of the aircraft (10).

[0208] FIG. 15 illustrates examples of photographing positions (610, 620) of an object (20) and an aircraft (10) when viewed from the side. Although only the photographing positions (610, 620) are illustrated in FIG. 15, the aircraft (10) can photograph the object (20) from more photographing positions.

[0209] Referring to FIG. 15, the aircraft (10) can photograph the object (20) at the photographing position (610) and move to the photographing position (620) to photograph the object (20). In this manner, the aircraft (10) can photograph the object (20) or elements (21, 22, 23) while moving up and down. For example, when the object (20) is a wind power generator, the aircraft (10) can photograph the wind power generator or blades at multiple photographing positions while moving from the root to the tip of the blade or from the tip to the root.

[0210] Meanwhile, the number of shooting positions can be set in various ways so that the front of the object (20) or elements (21, 22, 23) is captured without omission. In other words, the processor (160) can set the number of shooting positions so that the front of the elements (21, 22, 23) is captured without omission by considering the areas (611, 621) captured at the shooting positions (610, 620).

[0211] In FIG. 15, areas (611, 621) are shown spaced apart for convenience of explanation, but areas (611, 621) may overlap, and although only two shooting positions (610, 620) are shown, the aircraft (10) may shoot elements (21, 22, 23) at a greater number of shooting positions.

[0212] At this time, the processor (160) can control the aircraft (10) to move up and down while maintaining the same distance (D) as the target (20).

[0213] FIG. 16 is a drawing for explaining a method for detecting a defect in an inspection image according to one embodiment.

[0214] The processor (160) divides the inspection image (700) into a plurality of patches (710). Since the inspection image (700) is a high-quality image, the processor (160) requires a lot of resources to perform image processing on the inspection image (700). Therefore, the processor (160) divides the inspection image (700) into small units, called patches (710), and then performs image processing on each of the patches (710). The processor (160) performing image processing may mean identifying a defect included in the image, and may mean distinguishing between an object (20) included in the image and the background.

[0215] Referring to FIG. 16, the inspection image (700) includes a target object (20). For example, the target object (20) may be a part of a blade. In the inspection image (700), the target object (20) includes two defects. The two defects are included in the first patch (721) and the second patch (722), respectively.

[0216] FIG. 17 is a drawing for explaining a method of indicating a defect according to one embodiment.

[0217] Indicating a defect can mean graphically indicating the location and / or size of the defect. Indicating a defect can also be expressed as marking the defect.

[0218] Referring to FIG. 17, the inspection image (800) includes two defects. When the processor (160) identifies a defect, it displays shapes (821, 822) to indicate the location and size of the defect. The processor (160) may display the shapes (821, 822) at the same location as the defect. Additionally, the processor (160) may display the shapes (821, 822) at a size equal to or larger than the defect. Additionally, the processor (160) may display the shapes (821, 822) so as to overlap with the defect.

[0219] In FIG. 17, the processor (160) displays defects as rectangular shapes (821, 822). The processor (160) can display defects in various shapes. The processor (160) can display the shapes (821, 822) by overlapping them on the inspection image (800).

[0220] The processor (160) displays defects as shapes (821, 822), so that the user can quickly identify the location of the defect. The processor (160) may display different shapes depending on the type of defect, the severity of the defect, etc. For example, the processor (160) may display shapes such as squares and circles depending on the type (or severity) of the defect. In addition, the processor (160) may display different colors depending on the type (or severity) of the defect. For example, the processor (160) may display shapes such as yellow and red depending on the type (or severity) of the defect.

[0221] FIG. 18 is a diagram illustrating a method of grouping shapes according to one embodiment. Referring to FIG. 18, the processor (160) can group adjacent shapes (911, 912, 913) and display them as a single shape (910). Grouping shapes (911, 912, 913) may mean displaying two or more shapes (911, 912, 913) as a single shape (910).

[0222] The inspection image (900) includes one defect. However, as described in FIG. 16, since the processor (160) identifies a defect by dividing the inspection image (900) into multiple patches, as in FIG. 18, one defect may be detected in multiple patches. Since a defect is detected in each of the patches, the processor (160) displays the defect with shapes (911, 912, 913). However, since the shapes (911, 912, 913) represent different parts of one defect, the processor (160) may display the shapes (911, 912, 913) as one shape (910). In other words, the processor (160) may group the shapes (911, 912, 913) and display them as one shape (910).

[0223] The processor (160) may display shapes (911, 912, 913) on the inspection image (900) to indicate a defect, but may display one shape (910) on the inspection image (900) instead of the shapes (911, 912, 913).

[0224] FIG. 19 is a diagram illustrating a method for grouping and displaying shapes according to one embodiment. Referring to FIG. 19, the processor (160) groups masks (1011, 1012) into one and displays them as a single mask (1020). The masks (1011, 1012) each represent a defect. The processor (160) can generate the masks (1011, 1012) by displaying the edges of areas identified as defects with solid or dotted lines.

[0225] In Fig. 19, (a) illustrates two defects represented by two masks (1011, 1012). The mask (1011) is represented by a solid line, and the mask (1012) is represented by a dotted line. Since the two masks (1011, 1012) overlap, the processor (160) can group the two masks (1011, 1012).

[0226] (b) shows the result of the processor (160) grouping the masks (1011, 1012). All masks (1011, 1012) grouped into the same group are indicated by solid lines.

[0227] (c) shows the result of the processor (160) displaying the masks (1011, 1012) as one mask (1020).

[0228] FIG. 20 is a diagram illustrating a method for grouping shapes by changing the scale of the shapes according to one embodiment. Referring to FIG. 20, the processor (160) can change the scale of the shapes (1111, 1112) and determine whether to group the shapes (1111, 1112) according to the changed scale. The processor (160) displays the shapes (1111, 1112) as a single shape (1120). The shapes (1111, 1112) each represent a defect.

[0229] In Fig. 20, (a) illustrates two defects represented by two shapes (1111, 1112). In (a), the processor (160) does not group the shapes (1111, 1112) because the shapes (1111, 1112) do not overlap.

[0230] (b) shows the result of the processor (160) changing the scale of the shapes (1111, 1112). The processor (160) can change the size of the shapes (1111, 1112) by changing the scale of the shapes (1111, 1112). (b) shows that the shapes (1111, 1112) overlap as the scale of the shapes (1111, 1112) increases.

[0231] (c) shows the result of the processor (160) displaying the shapes (1111, 1112) as a single shape (1120). Although the two defects do not overlap, since the shapes (1111, 1112) representing the defects overlap, the processor (160) can group the shapes (1111, 1112) to display a single shape (1120).

[0232] The user can control the degree of grouping desired by specifying the scale of the shapes (1111, 1112). In other words, if the user wants to display a small number of shapes, he can specify a large scale, and the processor (160) can apply a large scale to the shapes (1111, 1112) to increase the probability that the shapes (1111, 1112) will be grouped.

[0233] Conversely, if a user wants to display a large number of shapes, he or she can specify a small scale, and the processor (160) can apply the small scale to the shapes (1111, 1112) to reduce the probability that the shapes (1111, 1112) will be grouped.

[0234] FIG. 21 is a diagram illustrating a method of grouping according to defects according to one embodiment. Referring to FIG. 21, the processor (160) may change the scales of figures (1211, 1212, 1213) and determine whether to group the figures (1211, 1212, 1213) according to the changed scales. Additionally, the processor (160) may determine whether to group the figures (1211, 1212, 1213) according to the types (or severities) of defects indicated by the figures (1211, 1212, 1213).

[0235] In Figure 21, defects are indicated by lines, two shapes are indicated by solid lines, and one defect is indicated by a dotted line. The defects indicated by solid lines and the defects indicated by dotted lines are of different types (or severities).

[0236] In Fig. 21, (a) illustrates three defects represented by three shapes (1211, 1212, 1213). In (a), the processor (160) does not group the shapes (1211, 1212, 1213) because the shapes (1211, 1212, 1213) do not overlap.

[0237] (b) shows the result of the processor (160) changing the scale of the shapes (1211, 1212, 1213). The processor (160) can change the size of the shapes (1211, 1212, 1213) by changing the scale of the shapes (1211, 1212, 1213). (b) shows that the shapes (1211, 1212, 1213) overlap as the scale of the shapes (1211, 1212, 1213) increases.

[0238] (c) shows the result of the processor (160) displaying two shapes (1211, 1212) as one shape (1220). Although the two defects do not overlap, since the shapes (1211, 1212) representing the defects overlap, the processor (160) can group the shapes (1211, 1212) to display one shape (1220). In the case of the shape (1213), although it overlaps with the adjacent shape (1211), the type (or severity) of the defect represented by the shape (1213) is different from the type (or severity) of the defect represented by the shape (1211), and therefore, the processor (160) may not group the shape (1213) and the shape (1211).

[0239] If all three defects are of the same type (or severity), the processor (160) may group the three shapes (1211, 1212, 1213) to display one shape.

[0240] FIG. 22 is a diagram illustrating a method of grouping using the distance between defects according to one embodiment. In FIG. 22, a case in which two defects (1310, 1320) are identified in an inspection image (1300) is described as an example.

[0241] The processor (160) can determine whether to group the shapes representing the defects (1310, 1320) based on the distance between the defects (1310, 1320). The distance between the defects (1310, 1320) can be determined in two ways. In the first way, the distance can be determined as the distance (D1) between the center points of the defects (1310, 1320). In the second way, the distance can be determined as the shortest distance (D2) between the defects (1310, 1320).

[0242] The processor (160) may compare the distance (D1 or D2) between the defects (1310, 1320) with a threshold value, and if the distance (D1 or D2) between the defects (1310, 1320) is less than or equal to the threshold value, the two shapes representing the defects (1310, 1320) may be grouped and displayed as a single shape. The threshold value may be determined according to the size of the defect. If the size of the defect is large, the threshold value may also be large, and if the size of the defect is small, the threshold value may also be small.

[0243] FIG. 23 is a schematic diagram illustrating an example of a device that exhibits a defect according to one embodiment.

[0244] Referring to FIG. 23, a device (1400) indicating a defect includes a processor (1410), a memory (1420), and a communication device (1430). However, the components of the device (1400) are not limited to those illustrated in FIG. 14. In other words, the device (1400) may include at least one more component in addition to the components illustrated in FIG. 23, or at least one of the components illustrated in FIG. 23 may be excluded.

[0245] As described above with reference to FIG. 3, the device (1400) may be included in at least one of the aircraft (10), the server (30), the controller (40), the station (50), and other external devices. Accordingly, the processor (160), the memory (130), and the communication device (150) described above with reference to FIG. 3 may correspond to the processor (1410), the memory (1420), and the communication device (1430) of the device (1400), respectively. Therefore, a detailed description of the processor (1410), the memory (1420), and the communication device (1430) is omitted.

[0246]

[0247] FIG. 24 is a flowchart illustrating an example of a method for merging images according to one embodiment.

[0248] The method illustrated in FIG. 24 consists of steps that are processed sequentially in the aircraft (10) or processor (160) illustrated in FIGS. 1 to 3. Therefore, even if omitted below, the content described above regarding the aircraft (10) or processor (160) can also be applied to the method illustrated in FIG. 24.

[0249] In step 410, the processor (160) acquires a first image and a second image of a portion of the object (20) through the camera (120) of the aircraft (10). The aircraft (10) can acquire images by capturing the object (20) in various ways. The first image and the second image can overlap each other. The aircraft (10) can determine the shooting position of the first image and the shooting position of the second image so that the first image and the second image overlap. In addition, the aircraft (10) can also adjust the angle of the camera (120) so that the first image and the second image overlap.

[0250] In one embodiment, the first image may be generated by merging two or more static images taken by the aircraft (10) at a first point, and the second image may be generated by merging two or more static images taken by the aircraft (10) at a second point. Accordingly, the first image may be a first partial image, and the second image may be a second partial image.

[0251] At step 420, the processor (160) identifies the outline of the object included in the first image and the second image. The first image and the second image may be low-resolution images. The processor (160) may have difficulty identifying the feature points of the object (20) in the low-resolution images. Therefore, the processor (160) may use the outline of the object (20) rather than the feature points of the object (20) to merge the first image and the second image.

[0252] The processor (160) can identify the outline of the object (20) by performing segmentation (or edge detection) on the first image and the second image. The processor (160) can display the outline using points. The processor (160) can display only the outline identified in the image.

[0253] In step 430, the processor (160) determines an overlapping area where the outlines included in the first image and the second image overlap. In other words, the processor (160) determines an overlapping area where the outline of the object (20) included in the first image and the outline of the object (20) included in the second image overlap.

[0254] When capturing the first image and the second image, the processor (160) positions the first image and the second image at initial positions based on the position and attitude of the aircraft (10), the angle and field of view of the camera (120), etc. The initial positions are positions where the images are positioned based on the capturing information when acquiring the images, regardless of the object (20) included in the images. In other words, the processor (160) can determine the initial positions of the first image and the second image based on the capturing information acquired when capturing the first image and the second image, rather than the object (20) included in the first image and the second image.

[0255] In one example, the processor (160) may determine an overlapping area using a central axis. The processor (160) may determine an area including straight lines passing through the outlines of both the first image and the second image among straight lines perpendicular to the central axis as the overlapping area.

[0256] The processor (160) can determine the central axis of the outlines of the first image and the second image. The processor (160) calculates the average position of the points of the outline. The points can have coordinates on a two-dimensional plane. The average position can be the center position of the points of the outline. The processor (160) determines the inclination of the central axis according to the positions where the points are arranged. The processor (160) can determine the inclination according to the tendency of the points to be arranged. For example, the processor (160) can determine the inclination using PCA (Principal Component Analysis). The processor (160) determines the central axis that passes through the average position and has the determined inclination.

[0257] In another example, the processor (160) may determine a set of points in different images whose distance between them is less than or equal to a threshold value as an overlapping area. The processor (160) may calculate the distance between points in a first image and points in a second image, and determine a set of points whose calculated distance is less than or equal to the threshold value as an overlapping area. The distance between points may be the distance from a point in the first image to a point in the second image that is located at the closest distance.

[0258] In step 440, the processor (160) merges the first image and the second image by aligning the outlines included in the overlapping area. The processor (160) may move, rotate, enlarge / reduce, rotate, etc. the first image and / or the second image to align the points included in the overlapping area. In other words, the processor (160) may move, rotate, enlarge / reduce, or rotate the images according to the movement of the points included in the overlapping area.

[0259] In one example, the processor (160) may align contours using an Iterative Point Cloud Algorithm. The processor (160) may perform alignment by moving points using the algorithm until the error between points becomes less than a threshold value.

[0260] FIG. 25 is a drawing illustrating an aircraft hopping and photographing an object according to one embodiment.

[0261] Referring to FIG. 25, the processor (160) can photograph the target object (20) while changing the angle of the camera (120) while the aircraft (10) is located at the n-th point (where n is a natural number greater than or equal to 1).

[0262] For example, the first to nth points may be positions having the same altitude, and the aircraft (10) may photograph the object (20) at the first to nth points while moving in the horizontal direction. Conversely, the first to nth points may be positions having different altitudes, and the aircraft (10) may photograph the object (20) at the first to nth points while moving in the vertical direction.

[0263] In one example, the direction of movement of the aircraft (10) and the direction in which the angle of the camera (120) changes may be perpendicular to each other. For example, if the aircraft (10) hops horizontally, the angle of the camera (120) may change vertically. Conversely, if the aircraft (10) hops vertically, the angle of the camera (120) may change horizontally. Hopping of the aircraft (10) means moving from one point to another.

[0264] In another example, the direction of movement of the aircraft (10) and the direction in which the angle of the camera (120) changes may be the same. For example, if the aircraft (10) hops horizontally, the angle of the camera (120) may change horizontally. Conversely, if the aircraft (10) hops vertically, the angle of the camera (10) may change vertically.

[0265] In another example, the aircraft (10) may include a plurality of cameras (120). The cameras (120) may be arranged vertically or horizontally. The cameras (120) may be attached to the aircraft (10) so that their angles of view overlap by a predetermined ratio. When the aircraft (10) captures static images using the plurality of cameras (120), there is no need to change the angle of the cameras (120) while capturing the target (20). Therefore, the time required for the aircraft (10) to capture the target (20) can be shortened.

[0266] First, the aircraft (10) is positioned at a first point, and can capture a portion (510) of the target object (20) by changing the angle of the camera (120) at the first point to generate multiple static images. Accordingly, at least one image representing the portion (510) can be acquired. The angle of the camera (120) can be changed up and down or left and right. When the aircraft (10) includes multiple cameras (120), the aircraft (10) can acquire multiple static images at one point without changing the angle of the camera (120).

[0267] Thereafter, the aircraft (10) can move from the first point to the second point, and change the angle of the camera (120) at the second point to capture a portion (520) of the target object (20). Accordingly, at least one image representing the portion (520) can be acquired. In FIG. 25, for convenience of explanation, the areas captured by the aircraft (10) at the first point and the second point are depicted as being spaced apart, but the areas captured by the aircraft (10) at the first point and the second point may overlap each other.

[0268] In the same manner, the aircraft (10) may move from a second point to a third point and photograph a portion (530), thereby obtaining at least one image representing the portion (530). In FIG. 25, for convenience of explanation, the areas photographed by the aircraft (10) at the second point and the third point are depicted as being spaced apart, but the areas photographed by the aircraft (10) at the second point and the third point may overlap each other.

[0269] Static images can be acquired during a single flight of the aircraft (10). Specifically, the images generated by photographing portion (510) can all be acquired during the same flight. Similarly, the images generated by photographing portion (520) and the images generated by photographing portion (530) can all be acquired during the same flight.

[0270] Of course, some of the images generated as portions (510) to (530) are captured may be acquired in other flights. However, capturing a portion is accomplished in a single flight (i.e., one flight). One flight may refer to the period from when the aircraft (10) takes off to begin capturing images until it lands.

[0271] The processor (160) may determine the number of images acquired for each section (510, 520, 530) by considering the overlap rate between images, the angle of view of the camera (120), the required resolution of the images, etc. For example, the required value for the GSD of the images may be determined by the distance between the aircraft (10) and the target (20) and the number of pixels of the camera, but is not limited thereto. The number of images acquired for each section (510, 520, 530) may be 1 to 7, but is not limited thereto. In addition, the processor (160) may determine the distance to the target (20) by considering the GSD of the images, the specifications of the camera, and / or the overlap rate between the images. For example, in order to easily perform merging of images, the aircraft (10) may generate an odd number of images at one point.

[0272] In addition, although FIG. 25 illustrates an aircraft (10) photographing an object (20) while moving in a horizontal direction, this is not limited thereto. For example, the aircraft (10) may also photograph an object (20) while moving in a vertical direction.

[0273] FIG. 26 is a drawing illustrating an aircraft moving and photographing a blade according to one embodiment.

[0274] The aircraft (10) can photograph a target object (20) while maintaining the shooting direction and changing the shooting position. The target object (20) may be a wind turbine. The moving direction of the aircraft (10) may be perpendicular to the shooting direction, but may vary depending on the angle of the camera (120) or the attitude of the aircraft (10).

[0275] FIG. 26 illustrates examples of photographing positions (610, 620) of an object (20) and an aircraft (10) when viewed from the side. In FIG. 26, only photographing positions (610, 620) are illustrated, but the aircraft (10) can photograph the object (20) from more photographing positions.

[0276] Referring to FIG. 26, the aircraft (10) can photograph the blade (21) at the photographing position (610) and move to the photographing position (620) to photograph the blade (21). In this manner, the aircraft (10) can photograph the blades (21, 22, 23) while moving up and down. For example, the aircraft (10) can photograph the wind generator or blades (21, 22, 23) at multiple photographing positions while moving from the root to the tip of the blade or from the tip to the root.

[0277] Meanwhile, the number of shooting positions can be set in various ways so that the front surfaces of the blades (21, 22, 23) are captured without omission. In other words, the processor (160) can set the number of shooting positions so that the front surfaces of the blades (21, 22, 23) are captured without omission by considering the areas (611, 621) captured at the shooting positions (610, 620).

[0278] In FIG. 26, the areas (611, 621) are shown spaced apart for convenience of explanation, but the areas (611, 621) may overlap, and although only two shooting positions (610, 620) are shown, the aircraft (10) may shoot the blades (21, 22, 23) at a greater number of shooting positions.

[0279] At this time, the processor (160) can control the aircraft (10) to move up and down while maintaining the same distance (D) as the target (20).

[0280] When photographing a wind turbine (20), a merged image can be generated for each blade. For example, a first merged image can be generated for the first blade (21), a second merged image can be generated for the second blade (22), and a third merged image can be generated for the third blade (23).

[0281] When the object (20) is photographed in the same manner as in Fig. 25 and static images are generated, the processor (160) may stitch the static images to generate a partial image and may also stitch the partial images to generate a merged image.

[0282] The processor (160) can determine the direction in which the static images are stitched according to the direction in which the angle of the camera (120) is changed. If it is assumed that the processor (160) captures the object (20) while adjusting the angle of the camera (120) in the vertical direction while the aircraft (10) is positioned at the same point, the processor (160) can stitch the static images in the vertical direction. In other words, the processor (160) can stitch the static images captured while changing the angle of the camera (120) in the vertical direction to generate a partial image. In the same manner, if the processor (160) captures the object while adjusting the angle of the camera (120) in the horizontal direction, the processor (160) can stitch the static images in the horizontal direction. In other words, the processor (160) can stitch the static images captured while changing the angle of the camera (120) in the horizontal direction to generate a partial image.

[0283] Since a partial image is created by stitching together multiple static images, the area of ​​the object (20) included in the partial image is wider than the area of ​​the object (20) included in the static image. In other words, by connecting parts of the object (20) included in the static image, the partial image can include an area wider than the static image.

[0284] In the same vein, the area of ​​the object (20) included in the merged image is wider than the area of ​​the object (20) included in the partial unknown.

[0285] When the object (20) is photographed in the same manner as in Fig. 26 and static images are generated, the processor (160) can stitch static images of a portion of the object (20) to generate a merged image.

[0286] The processor (160) can obtain information about the position of the aircraft (10) at the time when the static images are captured, information about the distance between the aircraft (10) and the target (20), information about the angle of view of the camera (120), and information about the direction in which the camera (120) captures the image (i.e., the shooting angle of the camera (120). The information obtained at the time when the image is captured can be displayed as shooting information, and the shooting information is stored in a manner matched with the static image.

[0287] FIG. 27 is a diagram illustrating identification of an outline according to one embodiment. Referring to FIG. 27, the processor (160) can identify an outline (711, 721) of an object (20) included in a first image (710) and a second image (720).

[0288] The first image (710) and the second image (720) include an object (20). In one example, the object (20) may be a blade. The processor (160) may perform segmentation (or edge detection) or the like on the first image (710) and the second image (720) to identify an outline (711, 721) of the object (20). The outline (711, 721) may represent a border of the object (20). The processor (160) may identify the outline (711) in the first image (710) and identify the outline (721) in the second image (720).

[0289] When it is difficult to extract feature points from the object (20) included in the first image (710) and the second image (720), the processor (160) can identify the outlines (711, 721). When the resolution of the first image (710) and the second image (720) is low and it is difficult to identify feature points other than the outlines (711, 721) of the object (20), the processor (160) can identify the outlines (711, 721) of the object (20) and display only the outlines (711, 721).

[0290] FIG. 28 is a drawing for explaining a method of placing an image in an initial position according to one embodiment.

[0291] When capturing the first image (710) and the second image (720), the processor (160) places the first image (710) and the second image (720) at initial positions based on shooting information such as the position and attitude of the aircraft (10), the angle and field of view of the camera (120), etc. The initial positions are positions where the first image (710) and the second image (720) are placed based on the shooting information when acquiring the first image (710) and the second image (720). The initial positions of the first image (710) and the second image (720) can be determined regardless of the target object (20). In other words, the processor (160) can place the first image (710) and the second image (720) at the initial positions regardless of the outline (711) and the outline (721). The initial positions are positions before matching the first image (710) and the second image (720). The initial position may be a two-dimensional plane. The processor (160) may project the first image (710) and the second image (720) onto the two-dimensional plane and place them at the initial position.

[0292] The overlapping area (800) represents an area where the first image (710) and the second image (720) overlap. The overlapping area (800) is determined based on the initial positions of the first image (710) and the second image (720). In addition, the overlapping area (800) may be determined by contours (711, 721) included in the first image (710) and the second image (720).

[0293] FIGS. 29 and 30 are diagrams illustrating a method for determining an overlapping area according to one embodiment. FIGS. 29 and 30 are diagrams illustrating the overlapping area (800) in more detail. For convenience of explanation, FIGS. 29 and 30 illustrate only the outlines included in the first image (710) and the second image (720).

[0294] The processor (160) can determine a central axis (910) and straight lines (921, 922) perpendicular to the central axis (910). The overlapping region (800) can be a region including the straight lines (921, 922).

[0295] The processor (160) can determine the central axis (910) of the contours (711, 721) of the first image (710) and the second image (720). The processor (160) calculates the average position of the points of the contours (711, 721). The points of the contours (711, 721) are points arranged on a two-dimensional plane constituting the contours (711, 721). The points can have coordinates on the two-dimensional plane. The average position can be the center position of the points. The processor (160) determines the slope of the central axis (910) according to the positions where the points are arranged. The processor (160) can determine the slope according to the tendency of the points to be arranged. For example, the processor (160) can determine the slope using PCA (Principal Component Analysis). The processor (160) determines the central axis (910) that passes the average position and has the determined slope.

[0296] The overlapping area (800) may be an area including straight lines (921, 922). In other words, the overlapping area (800) may be an area including contours (or points) located between the straight lines (921, 922). The boundary line of the overlapping area (800) may be the straight lines (921, 922). The straight line (921) is a straight line located at the uppermost position among the straight lines that pass through the point of the contour (721) and are perpendicular to the central axis (910). In other words, the straight line (921) is a straight line that passes through the point located at the end of the contour (721) included in the second image (720) and is perpendicular to the central axis (910).

[0297] The straight line (922) is the straight line located at the lowest position among the straight lines that pass through the point of the outline (711) and are perpendicular to the central axis (910). In other words, the straight line (922) is a straight line that passes through the point located at the end of the outline (711) included in the first image (710) and is perpendicular to the central axis (910).

[0298] In Fig. 30, the straight line (922) is described in more detail. As illustrated in Fig. 30, the straight line (922) is the straight line located at the lowest position among the straight lines perpendicular to the central axis (910). Since the outline (711) may have different lengths on the left and right sides with respect to the central axis (910), the processor (160) may determine the straight line (922) located at the lowest position as the boundary line of the overlapping area (800).

[0299] FIG. 31 is a diagram illustrating a method for merging images according to one embodiment. Referring to FIG. 31, the processor (160) can merge a first image (710) and a second image (720) by aligning points located within an overlapping area (800).

[0300] The processor (160) can move, rotate, enlarge / reduce, rotate, etc. the first image (710) and / or the second image (720) to align the points included in the overlapping area (800). In other words, the processor (160) can determine the distance and / or direction in which the points included in the overlapping area (800) move, and the first image (710) and the second image (720) are merged according to the determined distance and / or direction.

[0301] FIG. 32 is a diagram illustrating merging images of blades according to one embodiment. Referring to FIG. 32, the processor (160) can merge images (1210 to 1230) of blades (1200) of a wind turbine.

[0302] The processor (160) can identify the outline of the blade (1200) included in the first to third partial images (1210 to 1230) and align the outlines. The processor (160) can merge the first to third partial images (1210 to 1230) using only the outlines included in the first to third partial images (1210 to 1230).

[0303] In FIG. 32, the processor (160) can merge first to third partial images (1210 to 1230). Each of the first to third partial images (1210 to 1230) can include a plurality of static images. The first partial image (1210) can be generated by aligning static images of the blade (1200) captured while changing the angle of the camera (120) at a first point, the second partial image (1220) can be generated by aligning static images of the blade (1200) captured while changing the angle of the camera (120) at a second point, and the third partial image (1230) can be generated by aligning static images of the blade (1200) captured while changing the angle of the camera (120) at a third point.

[0304] In one example, the first partial image (1210) may be generated using only the shooting information. The processor (160) may generate the first partial image (1210) using the shooting information, such as the position and attitude of the aircraft (10) at the first point and the angle of the camera (120). The processor (160) may generate the first partial image (1210) by merging two or more static images acquired at the first point based on the shooting information. The second and third partial images (1220, 1230) may also be generated in the same manner.

[0305] The processor (160) can sequentially merge the first to third partial images (1210 to 1230). For example, the processor (160) can merge the first partial image (1210) and the second partial image (1220) to generate a first merged image, and can merge the first merged image with the third partial image (1230) to generate a second merged image. In addition, the processor (160) can merge the second partial image (1220) and the third partial image (1230) to generate a first merged image, and can merge the first merged image with the first partial image (1210) to generate a second merged image.

[0306] FIG. 33 is a schematic diagram illustrating an example of a device for merging images according to one embodiment.

[0307] Referring to FIG. 33, a device (1300) for merging images includes a processor (1310), a memory (1320), and a communication device (1330). However, the components of the device (1300) are not limited to those illustrated in FIG. 33. In other words, the device (1300) may include at least one additional component in addition to the components illustrated in FIG. 33, or at least one of the components illustrated in FIG. 33 may be excluded.

[0308] As described above with reference to FIG. 3, the device (1300) may be included in at least one of the aircraft (10), the server (30), the controller (40), the station (50), and other external devices. Accordingly, the processor (160), the memory (130), and the communication device (150) described above with reference to FIG. 3 may correspond to the processor (1310), the memory (1320), and the communication device (1330) of the device (1300), respectively. Therefore, a detailed description of the processor (1310), the memory (1320), and the communication device (1330) is omitted.

[0309] Meanwhile, the above-described method can be written as a program that can be executed on a computer, and can be implemented on a general-purpose digital computer that runs the program using a computer-readable recording medium. In addition, the structure of the data used in the above-described method can be recorded on a computer-readable recording medium through various means. The computer-readable recording medium includes storage media such as magnetic storage media (e.g., ROM, RAM, USB, floppy disk, hard disk, etc.) and optical reading media (e.g., CD-ROM, DVD, etc.).

[0310] Those skilled in the art will appreciate that the present invention can be implemented in modified forms without departing from the essential characteristics of the above-described invention. Therefore, the disclosed methods should be considered illustrative rather than restrictive. The scope of the claims, not the foregoing description, is defined by the scope of the patent, and should be interpreted to encompass all differences within the scope equivalent thereto.

[0311]

[0312] [Explanation of symbols]

[0313] 10: Aircraft

[0314] 20: Object

Claims

1. A step of acquiring images of the nose and blades of a wind turbine through a camera of an aircraft; A step of analyzing the above images to determine the root point of the blade; A step of analyzing the above images to determine the tip point of the blade; A step of detecting defects contained in the above images; A step of stitching the above images to create a merged image; A step of calculating the length per pixel by comparing the actual length of the blade with the number of pixels from the root point to the tip point in the merged image; and A method comprising: calculating the location of the defect using the length per pixel; 2. In paragraph 1, The step of determining the above root point is: A method of determining an image including both a first bounding box representing the nose and a second bounding box representing the nose and the blade, and determining the center points of the first bounding box and the second bounding box as the root point.

3. In paragraph 1, The step of determining the above root point is: A method for determining the center point of the overlapping area of ​​the first bounding box and the second bounding box as the root point when the first bounding box representing the nose and the second bounding box representing the nose and the blade overlap.

4. In paragraph 1, The step of determining the above root point is: A method for determining the center point of the area between the first bounding box and the second bounding box as the root point when the first bounding box representing the nose and the second bounding box representing the nose and the blade do not overlap.

5. In paragraph 1, The steps to determine the above tip points are: A method of determining, in an image including the tip of the blade, the pixel of the blade that is furthest from a pixel of the blade located on the edge of the image as the tip point.

6. In paragraph 1, The steps to determine the above tip points are: A method for determining an image including the tip point by comparing the horizontal or vertical length of a bounding box representing the blade among the images with the horizontal or vertical length of the image.

7. In paragraph 1, A method in which the location of the defect is the distance from the root point to the bounding box representing the defect.

8. In paragraph 7, The step of calculating the location of the above defect is: A method for calculating the position of the above combination as an actual distance using the length per pixel.

9. In paragraph 1, The steps for generating the above merged image are: A method comprising the step of stitching the above images and arranging them on a two-dimensional plane.

10. A computer-readable recording medium recording a program for executing the method of Article 1 on a computer.

11. At least one memory; and comprising at least one processor; At least one processor, A device for acquiring images of a nose and blades of a wind turbine through a camera of an aircraft, analyzing the images to determine a root point of the blade, analyzing the images to determine a tip point of the blade, detecting a defect included in the images, stitching the images to create a merged image, calculating a length per pixel by comparing the actual length of the blade with the number of pixels from the root point to the tip point in the merged image, and calculating a location of the defect using the length per pixel.

12. In paragraph 11, At least one processor, A device for determining an image including both a first bounding box representing the nose and a second bounding box representing the nose and the blade, and determining the center points of the first bounding box and the second bounding box as the root point.

13. In paragraph 11, At least one processor, A device for determining the center point of the overlapping area of ​​the first bounding box and the second bounding box as the root point when the first bounding box representing the nose and the second bounding box representing the nose and the blade overlap.

14. In paragraph 11, At least one processor, A device for determining the center point of the area between the first bounding box and the second bounding box as the root point when the first bounding box representing the nose and the second bounding box representing the nose and the blade do not overlap.

15. In paragraph 11, At least one processor, A device for determining, in an image including the tip of the blade, the pixel of the blade that is furthest from a pixel of the blade located on the edge of the image as the tip point.

16. In paragraph 11, At least one processor, A device for determining an image including the tip point by comparing the horizontal or vertical length of a bounding box representing the blade among the images with the horizontal or vertical length of the image.

17. In paragraph 11, A device wherein the location of the defect is the distance from the root point to the bounding box representing the defect.

18. In paragraph 11, At least one processor, A device for calculating the position of the above combination using the length per pixel, and the actual distance expressed as the actual distance using the length per pixel.

19. In paragraph 11, At least one processor, A device that stitches the above images and places them on a two-dimensional plane.

Images