Method and apparatus for outputting image

Abstract

A method for outputting an image, according to an aspect, comprises the steps of: capturing, through a camera of an aerial vehicle, at least one static image representing at least a portion of an object; generating at least one virtual image corresponding to the at least one static image; and superimposing and outputting the at least one virtual image on a dynamic image captured through the camera during flight of the aerial vehicle.

Description

Method and device for outputting images

[0001] It relates to methods and devices for outputting images.

[0002] Aerial vehicles (AVs) fly to approach a target, and various information about the target can be collected through the vehicle's camera. Aerial photography technology is being integrated into various fields and is even being used for the inspection of industrial structures. When the target is an industrial structure, the aerial vehicle's camera can capture images of the exterior or interior, allowing for the identification of any malfunctions or damage to the structure.

[0003] When a user inspects a target using images captured by an aircraft, the aircraft must capture all parts of the target without missing anything. However, depending on the shape, size, and other factors, it can be difficult to determine whether the aircraft has captured all parts of the target. Consequently, demand is growing for technologies that can verify whether any parts of the target are missing from images captured by the aircraft.

[0004] The present invention provides a method and device for outputting an image. Furthermore, the present invention provides a computer-readable recording medium containing a program for executing the method on a computer. The technical problems to be solved are not limited to the technical problems described above, and other technical problems may exist.

[0005] According to one aspect, a method for outputting an image includes: capturing at least one static image representing at least a portion of an object through a camera of an aircraft; generating at least one virtual image corresponding to the at least one static image; and superimposing the at least one virtual image on a dynamic image captured through the camera during flight of the aircraft and outputting the image.

[0006] In another aspect, a computer-readable recording medium includes a recording medium having recorded thereon a program for executing the above-described method on a computer.

[0007] According to another aspect, a device for outputting an image includes at least one memory; and at least one processor; wherein the at least one processor obtains at least one static image representing at least a part of an object captured through a camera of an aircraft, generates at least one virtual image corresponding to the at least one static image, and generates a control signal so that the at least one virtual image is superimposed on a dynamic image captured through the camera during flight of the aircraft and outputted.

[0008] FIG. 1 is a drawing for explaining an example of photographing an object using an aircraft according to one embodiment.

[0009] FIG. 2 is a drawing for explaining the relationship between an aircraft, a controller, a server, and a station according to one embodiment.

[0010] Figure 3 is a schematic diagram illustrating an example of an aircraft according to one embodiment.

[0011] FIG. 4 is a flowchart illustrating an example of a method for outputting an image according to one embodiment.

[0012] FIG. 5 is a drawing for explaining an example of a static image according to one embodiment.

[0013] FIG. 6 is a drawing for explaining an example of information on capturing a static image according to one embodiment.

[0014] FIG. 7 is a drawing for explaining an example of a virtual image according to one embodiment.

[0015] FIGS. 8A to 8C are drawings for explaining examples of changes in the shape of a virtual image superimposed on a dynamic image according to one embodiment.

[0016] FIG. 9 is a drawing for explaining an example in which the position of an aircraft is output on a dynamic image according to one embodiment.

[0017] FIG. 10 is a drawing for explaining an example of an indicator being output on a dynamic image according to one embodiment.

[0018] FIG. 11 is a diagram illustrating an example in which information indicating an area where shooting is missing according to one embodiment is output.

[0019] Figure 12 is a drawing for explaining an example of outputting overlapping and non-overlapping areas between virtual images.

[0020] Figure 13 is a drawing for explaining an example of outputting virtual images by classifying them according to the order of the time at which the virtual images were captured.

[0021] Fig. 14 is a schematic diagram illustrating an example of a device for outputting an image according to one embodiment.

[0022] The terms used in the examples are selected from widely used, current terms, as much as possible. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, the applicant may arbitrarily select terms, in which case their meanings will be described in detail in the relevant description. Therefore, the terms used in the specification should be defined based on their intended meaning and the overall content of the specification, rather than simply their names.

[0023] When a part of a specification is said to "include" a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise stated.

[0024] Additionally, terms including ordinal numbers, such as "first" or "second," used in the specification may be used to describe various components, but the components should not be limited by the terms. The terms may be used to distinguish one component from another.

[0025] Below, embodiments are described in detail with reference to the attached drawings. However, the embodiments may be implemented in various different forms and are not limited to the examples described herein.

[0026] FIG. 1 is a drawing for explaining an example of photographing an object using an aircraft according to one embodiment.

[0027] The aircraft (10) may include any aircraft capable of flight, including a drone, an Unmanned Aerial Vehicle (UAV), an Unmanned Aerial Mobility (UAM), an aircraft, a helicopter, etc.

[0028] The aircraft (10) can fly alone or with multiple aircraft capable of collaborating. Furthermore, the aircraft (10) can also collaborate with other types of devices, such as vehicles and robots. Furthermore, the aircraft (10) can automatically fly around the target object (20) along a predetermined flight path, or can manually fly around the target object (20) under user control.

[0029] The aircraft (10) can capture images of the object (20) while flying around the object (20). For example, the captured images of the object (20) can be used to determine whether the object (20) has a defect (e.g., breakdown, damage, etc.). The user can detect, recognize, and / or identify the object (20) through the aircraft (10).

[0030] As an example, the aircraft (10) can photograph the body or blades of a wind turbine. Specifically, the aircraft (10) can fly around the wind turbine to photograph the body or blades. For example, the aircraft (10) can take off from a station at a starting point and fly to the nose of the wind turbine. Then, the aircraft (10) can start photographing from the nose of the wind turbine and can photograph the blades while flying autonomously (or manually) along the blades. Typically, a wind turbine can include three blades, and the aircraft (10) can photograph the three blades to acquire images. At this time, the images photographed by the aircraft (10) can be utilized for inspection of the blades.

[0031] As another example, the aircraft (10) can photograph the surface of a large building. For example, the aircraft (10) can acquire images by photographing the surface while flying around the perimeter of the large building. In this case, the images captured by the aircraft (10) can be utilized for inspection of the surface of the large building.

[0032] Although wind turbines, bridges, and large buildings are illustrated as objects (20) in FIG. 1, the object is not limited thereto. In other words, any structure having a shape may be applied to the object (20) without limitation. For example, the object (20) may be a structure in the industrial field. The object (20) may be a structure for power generation (e.g., wind turbines, thermal power plants, hydroelectric power plants, nuclear power plants, solar power plants, etc.), a large building (e.g., factories, exhibition halls, stadiums, etc.), a bridge, a dam, a power distribution line, a road, etc., but is not limited thereto. As another example, the object (20) may be a structure that must be detected or information must be collected in the security and military fields (e.g., barbed wire fences, ammunition depots, exterior walls, etc.). As another example, anything that is difficult for a user to inspect entirely with the naked eye, dangerous, or requires a lot of manpower and cost for inspection may be applied to the object (20).

[0033] FIG. 2 is a drawing for explaining the relationship between an aircraft, a controller, a server, and a station according to one embodiment.

[0034] Referring to FIG. 2, the server (30), the controller (40), and the station (50) can independently or jointly control the aircraft (10). For example, the server (30), the controller (40), and the station (50) can control the operation (e.g., movement, rotation, etc.) of the aircraft (10) or control the filming of the aircraft (10).

[0035] The aircraft (10) includes at least one camera, and can capture images of an object (20) using the camera. For example, the camera can be installed at a location advantageous for capturing images during flight of the aircraft (10) (e.g., an area not obscured by a propeller, etc., at the front or lower front of the aircraft).

[0036] For example, the aircraft (10) can fly using a global navigation satellite system (GNSS) and / or an inertial navigation system (INS).

[0037] For example, the aircraft (10) can transmit and receive data with a server (30), a controller (40), and / or a station (50). In addition, the controller (40) and the server (30), the server (30) and the station (50), and the station (50) and the controller (40) can transmit and receive data with each other.

[0038] Here, the data may include data required to control the flight of the aircraft (10), data on a flight image of the aircraft (10), data on an image of the aircraft (10) capturing an object (20), etc.

[0039] The flight video represents the field of view of the aircraft (10) when the aircraft (10) is flying. For example, the flight video may be a dynamic image acquired in real time, but is not limited thereto.

[0040] The image captured by the aircraft (10) of the target object (20) refers to an image captured by the aircraft (10) while flying around the target object (20). At this time, the image captured by the aircraft (10) of the target object (20) can be used as an image to check for defects in the target object (20). In this case, the image captured by the aircraft (10) of the target object (20) can be referred to as an inspection image.

[0041] For example, the flight footage may be a relatively low-resolution image compared to the inspection image, and the inspection image may be a relatively high-resolution image compared to the flight footage. Meanwhile, depending on the type of aircraft (10), the cameras that generate the flight footage and the inspection image may be the same or different.

[0042] A user can control the aircraft (10) using the interface of the controller (40). For example, the controller (40) can generate a control signal based on user input received through the interface and transmit the control signal to the aircraft (10). The controller (40) can transmit the control signal to the aircraft (10) via wireless communication. The control signal may be a signal that controls the flight, attitude, navigation, etc. of the aircraft.

[0043] The aircraft (10) can control the motor to rotate the propeller according to the control signal received from the controller (40). The aircraft (10) can move, rotate, etc. by changing the speed and / or attitude, etc. by the rotation of the propeller. Here, the attitude of the aircraft can be expressed as pitch (Y), roll (X), yaw (Z), etc. In addition, the aircraft can perform photographing of the target object (20), etc. according to the control signal received from the controller (40).

[0044] The controller (40) may further include a display device, and the user may check the flight image and / or inspection image of the aircraft (10) through the display device.

[0045] The controller (40) may be a device on which an application for controlling an aircraft (10) is installed. For example, the device on which the application is installed may be a variety of portable devices such as a smartphone, tablet, smart pad, laptop, or wearable device.

[0046] The server (30) or station (50) can control the aircraft by directly transmitting a control signal to the aircraft (10). In addition, the aircraft (10) can transmit flight images and / or inspection images to the server (30), controller (40), or station (50).

[0047] The aircraft (10), server (30), controller (40), and station (50) can each analyze the inspection image. For example, the analysis of the inspection image may be to determine whether there is a defect in the target object (20) in the inspection image through an algorithm such as machine learning or deep learning. The aircraft (10) may directly determine whether there is a defect in the target object (20), or may transmit the inspection image to the server (30), controller (40), or station (50). The server (30), controller (40), or station (50) may analyze the inspection image received from the aircraft (10) to determine whether there is a defect in the target object (20).

[0048] Meanwhile, when photographing a target object (20) using an aircraft (10), it is necessary to confirm whether there is any part of the target object (20) that has been missed from the photograph. In particular, when determining whether the target object (20) has a defect using an inspection image, an inspection image of the entire target object (20) must be secured to accurately determine whether the target object (20) has a defect. Accordingly, after photographing the target object (20) is completed, it must be confirmed which part of the target object (20) each of the images acquired through the photographing represents.

[0049] Previously, in order to check whether or not the subject (20) was missing from the shot, the user directly reviewed the images with the naked eye after the flight of the aircraft (10) was completed, or reviewed the images through a solution that included an image stitching function.

[0050] However, additional development resources are required to provide a stitching function applicable to images of various objects (20), and it may be difficult for users to immediately check whether images of objects (20) with similar patterns repeated in appearance are missing.

[0051] In addition, during the filming of the object (20), it must be confirmed which part of the object (20) has been filmed. For example, if filming of the object (20) is interrupted midway, it is necessary to confirm which part of the object (20) should be filmed again when resuming filming. In this case, if there is no peculiarity in the appearance of the object (20), it may be difficult for the user to intuitively confirm which part of the object (20) has been filmed in the previously filmed images.

[0052] In addition, if a missing image of the object (20) is discovered after the shooting of the object (20) is completed, it may be difficult to accurately determine where the missing image occurred, and even if the object (20) is a large structure, it may be difficult to determine whether all desired inspection images have been acquired.

[0053] According to one embodiment, the processor may generate a control signal so that a part of the object (20) that has been previously captured is output on the dynamic image. For example, the control signal of the processor may cause the display device to superimpose a virtual image on the dynamic image corresponding to the flight image and output it. Here, the virtual image is generated in correspondence with the previously captured static image. The virtual image may correspond one-to-one with the static image. Accordingly, the user can intuitively confirm the previously captured part of the object (20) simply by checking the dynamic image. In addition, the user can confirm which part of the object (20) is missing in the inspection image through the virtual image output on the dynamic image, and can easily determine the location where the next inspection image should be captured.

[0054] Figure 3 is a schematic diagram illustrating an example of an aircraft according to one embodiment.

[0055] Referring to FIG. 3, the aircraft (10) may include a sensor (110), a camera (120), a memory (130), a driving device (140), a communication device (150), and a processor (160). However, the components of the aircraft (10) are not limited to those illustrated in FIG. 3. In other words, the aircraft (10) may include at least one more component in addition to the components illustrated in FIG. 3, or at least one of the components illustrated in FIG. 3 may be excluded.

[0056] The sensor (110) detects various information necessary for the operation of the aircraft (10) (e.g., flight, photography, etc.), such as the aircraft (10) itself, the surrounding environment of the aircraft (10), identification of the target (20), and confirmation of the distance between the aircraft (10) and the target (20). The sensor (110) may include, but is not limited to, a gyro sensor, a barometer, an ultrasonic sensor, a magnetic sensor, an acceleration sensor, a proximity sensor, a lidar, a radar, and / or a GPS sensor.

[0057] For example, a gyro sensor and / or an acceleration sensor can measure the three-axis angular velocity of the aircraft (10). A barometer can measure pressure changes and / or air pressure in the atmosphere surrounding the aircraft (10). An ultrasonic sensor can measure the distance between the aircraft (10) and the ground or an object (20). A magnetic sensor is a type of terrestrial magnetism sensor (compass sensor) and can detect geomagnetic information.

[0058] For example, a proximity sensor can measure the proximity state of an object (20) to an aircraft (10), the distance between the aircraft (10) and the object (20), and can include an ultrasonic sensor that can measure the distance to the object (20) from a signal reflected from the object (20) by outputting ultrasonic waves. A GPS sensor can calculate the current coordinates (x, y, z) of the aircraft (10) using GPS signals.

[0059] The sensor (110) may include an attitude and heading reference system (AHSR). For example, the attitude and heading reference system may include an inertial sensor or an inertial measurement unit (IMU). For example, the attitude and heading reference system may include a gyro sensor, an acceleration sensor, and a magnetic sensor, and may fuse sensor values ​​to obtain an attitude value (φ, ) of the aircraft (10). , ø) can be output. Here, the detail value (φ, , ψ) can be an angle based on three-dimensional coordinates (x-axis coordinate, y-axis coordinate, z-axis coordinate) according to GPS coordinates.

[0060] The camera (120) can photograph the object (20) according to instructions from the processor (160). For example, the aircraft (10) can include at least one camera, and can include a low-resolution camera and / or a high-resolution camera.

[0061] The camera (120) can be combined with a gimbal whose angle can be adjusted. Accordingly, the shooting angle of the camera (120) can be adjusted by the gimbal.

[0062] The memory (130) may include any non-transitory computer-readable recording medium. As an example, the memory (130) may include a non-permanent mass storage device such as a random access memory (RAM), a read only memory (ROM), a disk drive, a solid state drive (SSD), a flash memory, etc. As another example, the non-permanent mass storage device such as a ROM, an SSD, a flash memory, a disk drive, etc. may be a separate permanent storage device distinct from the memory. In addition, the memory (130) may store an operating system (OS) and at least one program code (e.g., code for the processor (160) to perform an operation to be described later with reference to FIGS. 4 to 14).

[0063] These software components may be loaded from a computer-readable recording medium separate from the memory (130). This separate computer-readable recording medium may be a recording medium that can be directly connected to a computer, and may include, for example, a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. Alternatively, the software components may be loaded into the memory (130) via a communication device (150) other than a computer-readable recording medium. For example, at least one program may be loaded into the memory (160) based on a computer program (e.g., a computer program for the processor (160) to perform the operations described below with reference to FIGS. 3 to 14) that is installed by files provided by developers or a file distribution system that distributes installation files of applications via the communication device (150).

[0064] The driving device (140) controls the driving of the motor at a speed and direction according to instructions from the processor (160), and accordingly, the rotational speed and direction of the propeller connected to the motor can be controlled. For example, the driving device (140) may include a motor and a propeller.

[0065] The communication device (150) performs data communication between the aircraft (10) and an external device. For example, the communication device (150) may communicate with the controller (40), server (30), and / or station (50) using various communication methods such as infrared communication, RF (Radio Frequency) communication, Wi-Fi communication, ZigBee communication, Bluetooth communication, laser communication, UWB (Ultra-Wideband) communication, LTE, 5G, 6G, and wireless LAN. However, the communication method employed in the communication device (150) is not limited to the above-described method.

[0066] The processor (160) can process commands of a computer program by performing basic arithmetic, logic, and input / output operations. Here, the commands can be provided from memory (130) or an external device (e.g., a server (30), a controller (40), a station (50), etc.). In addition, the processor (160) can generally control the operations of other components included in the aircraft (10).

[0067] For example, the functions performed by each module included in the processor (160) may be performed by one processor or by separate processors. The processor (160) may perform calculations or data processing related to control and / or communication of at least one other component of the aircraft (10).

[0068] For example, the processor (160) can obtain at least one static image representing at least a portion of the object (20) captured by the camera (120) of the aircraft (10). Here, the static image means an image captured of the exterior and / or interior of at least a portion of the object (20).

[0069] Additionally, the processor (160) may generate at least one of information about the position of the aircraft (10) at the time when at least one static image is captured, information about the distance between the aircraft (10) and the target (20), information about the angle of view of the camera (120), and information about the direction in which the camera (120) captures the at least one static image (i.e., the capturing angle of the camera (120)).

[0070] The processor (160) generates at least one virtual image corresponding to at least one static image. Then, the processor (160) generates a control signal so that at least one virtual image is superimposed on a dynamic image captured by the camera (120) during the flight of the aircraft (10) and output to a display device. The dynamic image represents an image generated by light measured by the camera (120). The virtual image represents an artificial image whose size, color, shape, etc. are determined by the processor (160). Since the dynamic image changes in real time due to the movement of the position of the aircraft (10), the processor (160) can generate a virtual image in real time according to the changing dynamic image. Here, the virtual image is generated to match the static image, and serves to display a portion of the object (20) on the static image on the dynamic image in real time. The virtual image can be generated together with the static image.

[0071] At this time, the dynamic image with the virtual image superimposed can be output to at least one of the display device of the aircraft (10), the display device of the server (30), the display device of the controller (40), the display device of the station (50), and the display device of another external device.

[0072] For example, the processor (160) may generate a control signal so that the shape of at least one virtual image superimposed on the dynamic image changes in response to at least one of a change in an object (20) appearing on the dynamic image, a change in the position of the aircraft (10), and a change in the angle of view of the camera (120). In addition, the processor (160) may generate a control signal so that a virtual image representing the position of the aircraft (10) at the time the static image was captured is further superimposed on the dynamic image and output.

[0073] In addition, the processor (160) may generate a control signal so that an indicator is output that indicates the ratio of the area of ​​the object (20) shown in the at least one static image to the entire area of ​​the object (20). In addition, the processor (160) may generate a control signal so that information indicating an area of ​​the entire area of ​​the object (20) that has been missed from being photographed is output based on the area of ​​the object (20) shown in the at least one static image. In addition, the processor (160) may generate a control signal so that a static image corresponding to a virtual image is further superimposed on a dynamic image and output. Here, the entire area of ​​the object (20) may represent an area planned to be photographed using the aircraft (10). For example, if the object (20) is a building and the planned area is the front of the building, the entire area of ​​the object (20) may be the front of the building. Additionally, if the target (20) is a wind turbine and the planned area is three blades, the entire area of ​​the target (20) may be the surface of the three blades.

[0074] In addition, when there are multiple virtual images, the processor (160) can generate a control signal so that overlapping and non-overlapping areas between the virtual images are distinguished and output. In addition, when there are multiple virtual images, the processor (160) can generate a control signal so that the virtual images are distinguished and output according to the order in which the static images were captured.

[0075]

[0076] *Specific descriptions of the operations of the aircraft (10) described above with reference to FIG. 3 will be described later with reference to FIGS. 4 to 13.

[0077] For example, the processor (160) may be implemented as an array of a plurality of logic gates, or it may be implemented as a combination of a general-purpose microprocessor and a memory storing a program that can be executed on the microprocessor. For example, the processor (160) may include a general-purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, etc. In some environments, the processor (160) may include an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc. For example, the processor (160) may also refer to a combination of processing devices, such as a combination of a digital signal processor (DSP) and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors coupled with a digital signal processor (DSP) core, or any other such combination of configurations.

[0078] Meanwhile, the operation of the processor (160) described above with reference to FIG. 3 may be implemented by a separate device. In this case, the separate device (hereinafter referred to as an “image output device”) may be included in at least one of the aircraft (10), the server (30), the controller (40), the station (50), and another external device. When the image output device is included in at least one of the server (30), the controller (40), the station (50), and another external device, the aircraft (10) may capture static images and dynamic images, and transmit the captured images to the image output device. An example of the image output device will be described below with reference to FIG. 14.

[0079] FIG. 4 is a flowchart illustrating an example of a method for outputting an image according to one embodiment.

[0080] The method illustrated in FIG. 4 is comprised of steps that are processed sequentially in the aircraft (10) or processor (160) illustrated in FIGS. 1 to 3. Therefore, even if omitted below, the content described above regarding the aircraft (10) or processor (160) may also be applied to the method illustrated in FIG. 4. In addition, the operations of the processor (160) below may also be performed by a device that outputs an image illustrated in FIG. 14.

[0081] At step 410, the aircraft (10) captures at least one static image representing at least a portion of the target object through the camera (120). In other words, the processor (160) acquires at least one static image captured through the camera (120).

[0082] Additionally, the processor (160) can obtain at least one of information about the position of the aircraft (10) at the time when at least one static image is captured, information about the distance between the aircraft (10) and the target (20), information about the angle of view of the camera (120), and information about the direction in which the camera (120) captures at least one static image (i.e., the capturing angle of the camera (120)).

[0083] Hereinafter, with reference to FIGS. 5 and 6, an example in which an aircraft (10) captures a static image and at least one piece of information about the capture of the static image is described.

[0084] FIG. 5 is a drawing for explaining an example of a static image according to one embodiment.

[0085] Referring to FIG. 5, the aircraft (10) can capture a target object (20) using a camera (120) to generate static images (510, 520). Specifically, the aircraft (10) can fly around the target object (20) and generate static images (510, 520) according to a user's capture signal via the controller (40) and / or a predetermined standard (e.g., capture at predetermined time intervals).

[0086] Depending on the size of the target object (20), the distance between the aircraft (10) and the target object (20), the shooting angle of the camera (120), the angle of view of the camera (120), etc., the static image (510, 520) may only include a part of the target object (20).

[0087] FIG. 6 is a drawing for explaining an example of information on capturing a static image according to one embodiment.

[0088] Referring to FIG. 6, when the aircraft (10) captures a static image (610), the processor (160) can generate information regarding the capture of the static image (610). In addition, the generated information can be stored in the memory (130) together with the static image (610).

[0089] The processor (160) can generate information about the position of the aircraft (10) at the time the static image (610) was captured. For example, the processor (160) can generate three-dimensional spatial information about the aircraft (10) through the sensor (110). The processor (160) can calculate coordinates (x, y, z) for the current position of the aircraft (10) using a GPS sensor, but is not limited thereto.

[0090] In addition, the processor (160) can generate information about the distance (d) between the aircraft (10) and the target (20) at the time the static image (610) was captured. For example, the processor (160) can measure the distance (d) between the aircraft (10) and the target (20) using a proximity sensor, an ultrasonic sensor, a lidar, a radar, etc. included in the sensor, but is not limited thereto.

[0091] Additionally, the processor (160) can generate information about the angle of view (θ) of the camera (120) at the time the static image (610) was captured. Here, the angle of view (θ) refers to a parameter indicating the range within which the camera (120) can capture images. For example, the angle of view (θ) may be a value preset in the camera (120), or may be a value adjusted by the server (30), the controller (40), and / or the station (50).

[0092] In addition, the processor (160) can generate information about the shooting direction of the camera (120) at the time the static image (610) was captured. Here, the shooting direction means the shooting angle. For example, the processor (160) can determine the inclination of the aircraft (10) by measuring the three-axis angular velocity of the aircraft (10) using a gyro sensor, an acceleration sensor, etc. If the camera (120) is installed on a gimbal, the processor (160) can determine the inclination of the camera (120) through the inclination of the gimbal. Accordingly, the processor (160) can determine the shooting direction of the camera (120) at the time the static image (610) was captured.

[0093] Referring again to FIG. 4, at step 420, the processor (160) generates at least one virtual image corresponding to at least one static image. For example, the processor (160) may generate two virtual images for one static image.

[0094] A virtual image refers to an image that shows which part of a target object (20) was photographed from which direction and at which angle by a static image. Accordingly, the virtual image can be determined in various ways depending on changes in the position of the aircraft (10), the distance (d) between the aircraft (10) and the target object (20), the angle of view (θ) of the camera (120), the shooting direction (i.e., the shooting angle) of the camera (120), etc.

[0095] For example, a virtual image can be created in the form of a rectangular border, a translucent rectangle, a rectangle filled with hatching, etc., but is not limited thereto. In other words, the form of a virtual image can be created in various forms that can express a portion shown in a static image, or can be created in a form that can indicate the direction or area captured by the static image.

[0096] In another embodiment, the processor (160) may identify an object (20) included in a static image and generate a virtual image representing the identified portion. The object (20) included in the static image may be identified using deep learning, etc. In this case, the virtual image may be generated in a different color from the object (20) and may be displayed on the surface of the object (20) in the same shape as the identified portion. For example, the processor (160) may generate a virtual image having the same shape as a portion of the object (20) indicated in black in the static image (610) of FIG. 6, or may generate a virtual image with only the border of a portion of the object (20) indicated in black.

[0097] When a static image is generated, the processor (160) can determine the location of the virtual image. Assuming that the camera (120) captures an arbitrary plane in three-dimensional space and a static image of the plane is generated, the processor (160) can determine the location of the plane included in the static image as the location of the virtual image. For example, when the static image is a square, the processor (160) can determine the GPS coordinates of the four corners of the square as the location of the virtual image.

[0098] As an example, the arbitrary plane may be a plane parallel to the object (20). The processor (160) may determine the location of the virtual image using coordinates on the plane parallel to the object (20) included in the static image. The processor (160) may calculate the plane parallel to the object (20) using information about the object (20).

[0099] As another example, the arbitrary plane may be a plane perpendicular to the shooting angle. The processor (160) may determine the position of the virtual image using coordinates on the plane perpendicular to the shooting angle. The processor (160) may calculate the shooting angle of the static image using the inclination of the aircraft (10) and the shooting direction of the camera (120). The plane perpendicular to the shooting angle of the static image may be a plane located between the aircraft (10) and the object (20) or on the surface of the object (20). The processor (160) may determine the position of the virtual image using the shooting angle of the static image and the distance (d) between the aircraft (10) and the object (20). For example, the arbitrary plane may be a plane perpendicular to the shooting angle of the static image at a location that is a distance (d).

[0100] The processor (160) can determine a plane and determine the size of a virtual image through the angle of view and distance (d) of the camera (120). The wider the angle of view of the camera (120) and the longer the distance (d), the larger the size of the virtual image can be.

[0101] Hereinafter, with reference to Fig. 7, an example of creating a virtual image is described.

[0102] FIG. 7 is a drawing for explaining an example of a virtual image according to one embodiment.

[0103] Referring to FIG. 7, when a static image (710) is generated by the camera (120) capturing an object (20), the processor (160) generates a virtual image (730) corresponding to the static image (710).

[0104] For example, the processor (160) can check the position of the aircraft (10) at the time of shooting, the distance (d) between the aircraft (10) and the target (20), the angle of view (θ) of the camera (120), and the shooting direction (i.e., shooting angle) of the camera (120). In addition, the processor (160) can check the position of the static image (710). For example, the processor (160) can check the position of the static image (710) using GPS coordinates corresponding to the center point or corner of the static image (710).

[0105] Accordingly, the processor (160) can detect a portion of the object (20) corresponding to the static image (710) (i.e., a portion of the object (20) appearing in the static image (710)) and generate a virtual image (730) including the detected portion.

[0106] Since the virtual image (730) is generated in response to the static image (710), the size, shape, etc. of the virtual image (730) can be adaptively changed depending on the position, direction, size, etc. of the object (20) appearing in the flight image (i.e., dynamic image) (720) of the aircraft (10).

[0107] Referring again to FIG. 4, at step 430, the processor (160) superimposes at least one virtual image on a dynamic image captured by the camera (120) during the flight of the aircraft (10) and outputs it.

[0108] For example, the dynamic image may be a flight image generated based on the current flight of the aircraft (10), or may be a flight image previously generated based on a past flight of the aircraft (10).

[0109] As the aircraft (10) flies, the GPS coordinates corresponding to the current location of the aircraft (10) change, and accordingly, the difference between the GPS coordinates of the static image and the GPS coordinates corresponding to the current location of the aircraft (10) can change in real time. Accordingly, the processor (160) can adaptively change the size, shape, etc. of the virtual image superimposed on the dynamic image according to the flight of the aircraft (10) by using the difference between the GPS coordinates of the aircraft (10) and the GPS coordinates of the static image.

[0110] Hereinafter, with reference to FIGS. 8a to 13, examples of a processor (160) superimposing a virtual image on a dynamic image and outputting it will be described.

[0111] FIGS. 8A to 8C are drawings for explaining examples of changes in the shape of a virtual image superimposed on a dynamic image according to one embodiment.

[0112] Referring to FIGS. 8A to 8C, the processor (160) generates a control signal so that the shape of a virtual image superimposed on the dynamic image changes in response to at least one of a change in an object appearing on the dynamic image, a change in the position of the aircraft (10), and a change in the angle of view of the camera (120). In response to the control signal of the processor (160), the shape of the virtual image in the dynamic image output to the display device changes in real time.

[0113] Meanwhile, in FIGS. 8a to 8c, it is assumed that the static image captured is the static image (710) shown in FIG. 7.

[0114] Fig. 8a illustrates an example of a change in an object (20) appearing on a dynamic image. For example, it is assumed that an aircraft (10) flies from the bottom to the top of the object (20). Accordingly, the object (20) is output from the bottom to the top in the dynamic image as well. The aircraft (10) captures a static image of a portion of the upper portion of the object (20), and the virtual image (831) can be output by overlapping the dynamic image when the upper portion of the object (20) appears in the dynamic image.

[0115] For convenience of explanation, FIG. 8a illustrates a still cut (810) representing the lower part of a dynamic image, a still cut (820) representing the middle part, and a still cut (830) representing the upper part. Accordingly, as described above, the virtual image (831) can only be expressed on the still cut (830) representing the upper part.

[0116] FIG. 8b illustrates an example of a situation where the position of an aircraft (10) capturing a dynamic image changes. For example, it is assumed that the aircraft (10) is flying in a direction that gradually moves away from the target object (20). Accordingly, a larger area of ​​the target object (20) is gradually output on the dynamic image. Accordingly, the size of the virtual images (841, 851) superimposed on the dynamic image is also output so as to gradually decrease.

[0117] For convenience of explanation, FIG. 8b illustrates a still cut (840) at a point in time when the dynamic image is close to the target object (20) and a still cut (850) at a point in time when the dynamic image is far from the target object (20). Accordingly, as described above, the size of the virtual image (841) can be expressed larger than the size of the virtual image (851).

[0118] Figure 8c illustrates an example of a camera (120) changing its angle of view. For example, it is assumed that the camera (120) gradually increases its angle of view while the position of the aircraft (10) remains fixed. Accordingly, a larger area of ​​the target object (20) is gradually output on the dynamic image. Accordingly, the size of the virtual images (861, 871) superimposed on the dynamic image is output so as to become gradually smaller.

[0119] For convenience of explanation, FIG. 8c illustrates a still cut (850) at a point in time when the angle of view increases and a still cut (840) at a point in time when the angle of view decreases among the dynamic images. Accordingly, as described above, the size of the virtual image (861) can be expressed as larger than the size of the virtual image (871).

[0120] Meanwhile, although not illustrated in FIGS. 8A to 8C, the aircraft (10) can fly while rotating 360° around the target object (20). Furthermore, the examples illustrated in FIGS. 8A to 8C can be continuously generated. Even in this case, the virtual image can be output in real time on the dynamic image, or its size or shape can be changed in response to changes in the target object appearing on the dynamic image.

[0121] FIG. 9 is a drawing for explaining an example in which the position of an aircraft is output on a dynamic image according to one embodiment.

[0122] Referring to FIG. 9, the processor (160) generates a control signal so that a virtual image (930) indicating the position of the aircraft (10) at the time the static image was captured is output so as to be further overlapped on the dynamic image. In FIG. 9, the virtual image (930) indicating the position of the aircraft (10) is displayed as a dot, but the virtual image (930) may be displayed as a square or an arrow, etc. When the virtual image (930) is displayed as an arrow, the direction of the arrow may indicate the movement direction of the aircraft (10) or the shooting direction. According to the control signal of the processor (160), the position of the aircraft (10) at the time the static image was captured can be confirmed in the dynamic image output to the display device.

[0123] For example, the processor (160) may generate a virtual image (930) representing the position of the aircraft (10) differently from the virtual image (920) corresponding to the static image. In addition, the virtual image (920) and the virtual image (930) may be output together on the dynamic image. Accordingly, the user may simultaneously check the portion where the static image was captured and the position of the aircraft (10) at the time the static image was captured by observing the dynamic image.

[0124] For convenience of explanation, a still cut (910) of a dynamic image is illustrated in FIG. 9. Accordingly, the number of virtual images (920) and the number of virtual images (930) can be expressed equally in the still cut (910).

[0125] Although the virtual image (920) is displayed at a distance from the object (20) in FIGS. 9 and 10 , the virtual image (920) may also be displayed on the surface of the object (20). The still cut (910) of the dynamic image is an image taken from the side of the wind turbine, and the virtual image (920) was generated in response to a static image taken from the front of the wind turbine. Accordingly, in FIG. 9 , the virtual image (920) is expressed as if a square were viewed from the side.

[0126] FIG. 10 is a drawing for explaining an example of an indicator being output on a dynamic image according to one embodiment.

[0127] Referring to FIG. 10, the processor (160) generates a control signal to output an indicator (1020) indicating a ratio of an area of ​​the object (20) shown (or included) in at least one static image to the entire area of ​​the object (20). According to the control signal of the processor (160), information on the area of ​​the object (20) captured as a static image can be confirmed in a dynamic image output to a display device.

[0128] In FIG. 10, the indicator (1020) indicates to what extent the target object (20) has been photographed, but the indicator (1020) may also indicate flight time, mission completion rate, etc.

[0129] For convenience of explanation, a still cut (1010) of a dynamic image is illustrated in FIG. 10. Accordingly, in the dynamic image, the ratio within the indicator (1020) can be changed in real time.

[0130] The processor (160) can identify a portion corresponding to the object (20) in a previously captured static image. In addition, it can also identify overlapping portions of the object (20) among a plurality of static images. For example, the processor (160) can identify a portion corresponding to the object (20) and overlapping portions in a previously captured static image through information about the position of the aircraft (10) at the time the static image was captured, information about the distance between the aircraft (10) and the object (20), information about the angle of view of the camera (120), and information about the direction in which the camera (120) captures at least one static image (i.e., the capturing angle of the camera (120). Accordingly, the processor (160) can calculate a ratio of an area captured as a static image among the entire area of ​​the object (20), and generate a control signal so that the calculated ratio is output to the indicator (1020).

[0131] Meanwhile, although a bar is illustrated as an example of an indicator (1020) in FIG. 10, the present invention is not limited thereto. In other words, any form that allows for the ratio of the area captured as a static image to the entire area of ​​the target object (20) can be used as an indicator without limitation.

[0132] FIG. 11 is a diagram illustrating an example in which information indicating an area where shooting is missing according to one embodiment is output.

[0133] Referring to FIG. 11, the processor (160) generates a control signal to output information indicating an area (1110) of the entire area of ​​the object (20) where shooting is missed, based on an area of ​​the object (20) shown in at least one static image. Depending on the control signal of the processor (160), a user can intuitively check whether shooting is missed.

[0134] FIG. 11 illustrates an example in which a portion (1110) of an object (20) is missing from a static image captured by a camera (120). The processor (160) can analyze an area of ​​the object (20) within a previously captured static image to determine whether there is a portion (1110) missing from the capture.

[0135] For example, if there is a part (1110) where the shooting is missed, the processor (160) may output a message regarding the missing shooting (1121). For example, a voice may be output to notify of the missing shooting.

[0136] As another example, if there is a portion (1110) where the shooting is missing, the processor (160) may output an image (1122) of the area where the shooting is missing. For example, only the portion of the object (20) where the shooting is missing may be highlighted, or an image that separately represents only the portion where the shooting is missing may be output.

[0137] As another example, if there is a portion (1110) where a shot is missed, the processor (160) may output an alarm signal (1123) notifying of the missed shot. For example, the alarm signal may be a preset melody or a vibration signal.

[0138] Additionally, in addition to the examples described above with reference to FIG. 11, the processor (160) may generate various signals or information that may notify the user of a missed shot.

[0139] Meanwhile, although not shown in FIGS. 7 to 11, the processor (160) may generate a control signal so that a static image corresponding to a virtual image is output so as to be further overlapped on the dynamic image. For example, referring to FIG. 7, a static image (710) corresponding to a virtual image (730) may be output so as to be overlapped on the inside of a virtual image (730). Generally, a virtual image (730) is displayed with a rectangular border on a dynamic image, but a static image (710) may be output together with the inside of the virtual image (730).

[0140] As another example, only a static image (710) may be output instead of a virtual image (730), or only an object (20) included in the static image (710) may be output.

[0141] Figure 12 is a drawing for explaining an example of outputting overlapping and non-overlapping areas between virtual images.

[0142] Referring to FIG. 12, when there are multiple virtual images, the processor (160) can generate a control signal so that overlapping and non-overlapping areas are distinguished and output between the virtual images. Accordingly, in the dynamic image output to the display device, the portion where the object (20) is captured overlapping among the static images can be identified.

[0143] For convenience of explanation, FIG. 12 illustrates a still cut (1210) of a dynamic image. For example, assume that a first static image and a second static image are captured, and that there is an overlapping area between them. In this case, a first virtual image (1220) matched to the first static image and a second virtual image (1230) matched to the second static image may be output on the still cut (1210).

[0144] In addition, an overlapping area (1241, 1242) may be included between the virtual images (1220, 1230). At this time, the processor (160) may display the overlapping area (1242) using a hatching, color, etc. to distinguish it from a non-overlapping area.

[0145] Figure 13 is a drawing for explaining an example of outputting virtual images by classifying them according to the order of the time at which the virtual images were captured.

[0146] Referring to FIG. 13, when there are multiple virtual images, the processor (160) can generate a control signal so that the virtual images are output in a differentiated manner according to the order in which the static images were captured. Accordingly, the order in which the static images were captured can be confirmed in the dynamic images output to the display device.

[0147] For convenience of explanation, a still cut (1310) of a dynamic image is illustrated in FIG. 13. For example, it is assumed that a total of three static images are captured while an aircraft (10) flies from the bottom to the top of an object (20). At this time, virtual images (1310, 1320, 1330) matching each of the static images can be output so as to be distinguished from each other on the still cut (1310). The thickness of the virtual image (1330) is output thicker than the thickness of the virtual image (1320), and the thickness of the virtual image (1320) is output thicker than the thickness of the virtual image (1310).

[0148] Figure 13 illustrates an example in which a virtual image matched to a static image captured later is displayed with a thicker outline. However, this is not limited to the example illustrated in Figure 13, and any form that allows the capture order of the static images to be confirmed may be applied without limitation.

[0149] Fig. 14 is a schematic diagram illustrating an example of a device for outputting an image according to one embodiment.

[0150] Referring to FIG. 14, a device (1400) for outputting an image includes a processor (1410), a memory (1420), and a communication device (1430). However, the components of the device (1400) are not limited to those illustrated in FIG. 14. In other words, the device (1400) may include at least one more component in addition to the components illustrated in FIG. 14, or at least one of the components illustrated in FIG. 14 may be excluded.

[0151] As described above with reference to FIG. 3, the device (1400) may be included in at least one of the aircraft (10), the server (30), the controller (40), the station (50), and other external devices. Accordingly, the processor (160), the memory (130), and the communication device (150) described above with reference to FIG. 3 may correspond to the processor (1410), the memory (1420), and the communication device (1430) of the device (1400), respectively. Therefore, a detailed description of the processor (1410), the memory (1420), and the communication device (1430) is omitted.

[0152] As described above, the processor (160, 1410) can generate a control signal so that which part of the object (20) has been previously captured is output on the dynamic image. For example, the control signal of the processor (160, 1410) can cause the display device to superimpose a virtual image on the dynamic image corresponding to the flight image and output it. Here, the virtual image is generated corresponding to the previously captured static image. Accordingly, the user can intuitively confirm the previously captured part of the object (20) simply by checking the dynamic image. In addition, the user can confirm which part of the object (20) has been missed in the inspection image through the virtual image output on the dynamic image, and can easily determine the location where the next inspection image should be captured.

[0153] Meanwhile, the above-described method can be written as a program that can be executed on a computer, and can be implemented on a general-purpose digital computer that runs the program using a computer-readable recording medium. In addition, the structure of the data used in the above-described method can be recorded on a computer-readable recording medium through various means. The computer-readable recording medium includes storage media such as magnetic storage media (e.g., ROM, RAM, USB, floppy disk, hard disk, etc.) and optical reading media (e.g., CD-ROM, DVD, etc.).

[0154] Those skilled in the art will appreciate that the present invention can be implemented in modified forms without departing from the essential characteristics of the above-described invention. Therefore, the disclosed methods should be considered illustrative rather than restrictive. The scope of the claims, not the foregoing description, is defined by the scope of the patent, and should be interpreted to encompass all differences within the scope equivalent thereto.

[0155] [Explanation of symbols]

[0156] 10: Aircraft

[0157] 20: Object

Claims

1. At least one image showing at least a portion of the object through the camera of the aircraft. Steps to capture a still image; At least one virtual image corresponding to at least one static image above; Step of generating; and On the dynamic image captured by the camera during the flight of the above aircraft A method for outputting an image, comprising: a step of outputting at least one virtual image by overlapping it; 2. In paragraph 1, A method further comprising the step of generating at least one of information about a position of the aircraft at a time when the at least one static image is captured, information about a distance between the aircraft and the object, information about an angle of view of the camera, and information about a direction in which the camera captures the at least one static image.

3. In paragraph 1, The above output step is, A method in which the shape of at least one virtual image superimposed on the dynamic image changes in response to at least one of a change in the target object appearing on the dynamic image, a change in the position of the aircraft, and a change in the angle of view of the camera.

4. In paragraph 1, The above output step is, A method of further superimposing a virtual image representing the position of the aircraft at the time the static image was captured onto the dynamic image and outputting the same.

5. In paragraph 1, A method further comprising the step of outputting an indicator indicating a ratio of an area of the object shown in the at least one static image to the entire area of the object.

6. In paragraph 1, A method further comprising the step of outputting information indicating an area of the entire area of the object from which imaging was missed based on an area of the object shown in at least one static image.

7. In paragraph 1, The above output step is, A method for outputting the static image corresponding to the virtual image by further overlaying it on the dynamic image.

8. In paragraph 1, If there are multiple virtual images above, The above output step is, A method for outputting overlapping and non-overlapping areas between the above virtual images.

9. In paragraph 1, If there are multiple virtual images above, The above output step is, A method for outputting virtual images by distinguishing them according to the order in which the static images were captured.

10. A computer-readable recording medium having recorded thereon a program for executing the method of Article 1 on a computer.

11. At least one memory; and comprising at least one processor; At least one processor of the above, At least a portion of an object photographed through a camera of an aircraft; Obtaining one static image and writing at least one corresponding static image It also creates a virtual image and passes it through the camera during the flight of the aircraft. A device for outputting an image that generates a control signal so that at least one virtual image is superimposed and output on a captured dynamic image.

12. In paragraph 11, At least one processor of the above, A device that generates at least one of information about a position of the aircraft at a time when the at least one static image is captured, information about a distance between the aircraft and the target, information about an angle of view of the camera, and information about a direction in which the camera captures the at least one static image.

13. In paragraph 11, At least one processor of the above, A device that generates a control signal so that the shape of at least one virtual image superimposed on the dynamic image changes in response to at least one of a change in the target object appearing on the dynamic image, a change in the position of the aircraft, and a change in the angle of view of the camera.

14. In paragraph 11, At least one processor of the above, A device that generates a control signal so that a virtual image representing the position of the aircraft at the time the static image was captured is output so as to be further superimposed on the dynamic image.

15. In paragraph 11, At least one processor of the above, A device that generates a control signal so as to output an indicator indicating the ratio of an area of the object shown in the at least one static image to the entire area of the object.

16. In paragraph 11, At least one processor of the above, A device for generating a control signal so as to output information indicating an area of the entire area of the object that was missed from being photographed based on an area of the object that appears in at least one static image.

17. In paragraph 11, At least one processor of the above, A device that generates a control signal so that the static image corresponding to the virtual image is output so as to be further overlapped on the dynamic image.

18. In paragraph 11, If there are multiple virtual images above, At least one processor of the above, A device that generates a control signal so that overlapping and non-overlapping areas between the above virtual images are output separately.

19. In paragraph 11, If there are multiple virtual images above, At least one processor of the above, A device that generates a control signal so that the virtual images are output separately according to the order in which the static images were captured.

20. At least one camera; at least one memory; and comprising at least one processor; At least one processor of the above, An aircraft that acquires at least one static image representing at least a portion of a target object captured through the camera, generates at least one virtual image corresponding to the at least one static image, and outputs an image that generates a control signal so that the at least one virtual image is superimposed on a dynamic image captured through the camera during the flight of the aircraft.

Images