Method and device for identifying location information of area

Abstract

A method for identifying location information of an area, according to one aspect of the present invention, comprises the steps of: acquiring static images by photographing a predetermined area by means of a camera of an aerial vehicle; generating a point cloud corresponding to the predetermined area by using feature points extracted from the static images; generating a merged image corresponding to the predetermined area by using the feature points to stitch the static images together; and identifying location information of at least some spots shown on the merged image by using the point cloud and the merged image.

Description

Method and device for identifying location information of an area

[0001] Relating to a method and device for identifying location information of an area.

[0002]

[0003] Aerial vehicles can collect various information about an object through their cameras. If the object is the ground, a map can be generated based on images of the ground captured by the camera. At this time, location information (e.g., GPS coordinates) for each spot on the map must be identified.

[0004] When satellite images, etc. are used to identify location information, there are cases where the GPS coordinates are inaccurate and the resolution is low, making it difficult to confirm the exact location information of the desired point.

[0005] Furthermore, even when images captured by an aircraft from different locations contain the same point, the GPS coordinates for that point may be calculated differently. While GPS coordinates are calculated based on factors such as the aircraft's position and the camera's angle, errors in the calculated GPS coordinates can result in different coordinates for the same point. If different GPS coordinates are derived for the same point, stitching the images can result in distorted images.

[0006]

[0007] The present invention provides a method and device for identifying location information in a region. Furthermore, the present invention provides a computer-readable recording medium containing a program for executing the method on a computer. The technical challenges to be addressed are not limited to the technical challenges described above, and other technical challenges may exist.

[0008]

[0009] A method for identifying location information of an area according to one aspect includes: acquiring static images by photographing a predetermined area through a camera of an aircraft; generating a point cloud corresponding to the predetermined area using feature points extracted from the static images; generating a merged image corresponding to the predetermined area by stitching the static images using the feature points; and identifying location information of at least some spots appearing on the merged image using the point cloud and the merged image.

[0010] A computer-readable recording medium according to another aspect includes a recording medium having recorded thereon a program for executing the above-described method on a computer.

[0011] A device for identifying location information of an area according to another aspect includes: at least one memory storing at least one program; and at least one processor performing a calculation by executing the at least one program; wherein the at least one processor acquires static images by photographing a predetermined area through a camera of an aircraft, generates a point cloud corresponding to the predetermined area using feature points extracted from the static images, generates a merged image corresponding to the predetermined area by stitching the static images using the feature points, and identifies location information of at least some spots appearing on the merged image using the point cloud and the merged image.

[0012] According to another aspect, an aircraft comprises: at least one camera; at least one memory storing at least one program; and at least one processor performing a calculation by executing the at least one program; wherein the at least one processor acquires static images by photographing a predetermined area through the camera, generates a point cloud corresponding to the predetermined area using feature points extracted from the static images, generates a merged image corresponding to the predetermined area by stitching the static images using the feature points, and identifies location information of at least some spots appearing on the merged image using the point cloud and the merged image.

[0013]

[0014] FIG. 1 is a drawing for explaining an example of photographing an object using an aircraft according to one embodiment.

[0015] FIG. 2 is a drawing for explaining the relationship between an aircraft, a controller, a server, and a station according to one embodiment.

[0016] Figure 3 is a schematic diagram illustrating an example of an aircraft according to one embodiment.

[0017] FIG. 4 is a flowchart illustrating an example of a method for identifying location information of an area according to one embodiment.

[0018] FIG. 5 is a diagram illustrating an example of a processor obtaining static images according to one embodiment.

[0019] FIG. 6 is a flowchart illustrating an example of a processor generating a point cloud according to one embodiment.

[0020] FIG. 7 is a diagram illustrating an example of a processor identifying common features in images according to one embodiment.

[0021] FIG. 8 is a diagram illustrating an example of a processor determining a candidate solution and an optimal solution according to one embodiment.

[0022] FIG. 9 is a diagram illustrating an example of a processor generating a point cloud based on an optimal solution according to one embodiment.

[0023] FIG. 10 is a diagram illustrating an example of a processor stitching static images according to one embodiment.

[0024] FIG. 11 is a diagram illustrating an example of a processor segmenting an object from a merged image and updating the boundary of the segmented object using user input according to one embodiment.

[0025] FIG. 12 is a diagram illustrating an example of a processor matching a point cloud and a merged image and identifying location information of a point corresponding to a feature point of the merged image according to one embodiment.

[0026] FIG. 13 is a diagram illustrating an example of a processor calculating location information of a point using interpolation according to one embodiment.

[0027] FIG. 14 is a schematic diagram illustrating an example of a device for identifying location information of an area according to one embodiment.

[0028]

[0029] The terms used in the examples are selected from widely used, current terms, as much as possible. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, the applicant may arbitrarily select terms, in which case their meanings will be described in detail in the relevant description. Therefore, the terms used in the specification should be defined based on their intended meaning and the overall content of the specification, rather than simply their names.

[0030] When a part of a specification is said to "include" a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise stated.

[0031] Additionally, terms including ordinal numbers, such as "first" or "second," used in the specification may be used to describe various components, but the components should not be limited by the terms. The terms may be used to distinguish one component from another.

[0032] Below, embodiments are described in detail with reference to the attached drawings. However, the embodiments may be implemented in various different forms and are not limited to the examples described herein.

[0033]

[0034] FIG. 1 is a drawing for explaining an example of photographing an object using an aircraft according to one embodiment.

[0035] The aircraft (10) may include any aircraft capable of flight, including a drone, UAV (Unmanned Aerial Vehicle), UAM (Unmanned Aerial Mobility), aircraft, helicopter, etc.

[0036] The aircraft (10) can fly alone or with multiple aircraft capable of collaborating. Furthermore, the aircraft (10) can also collaborate with other types of devices, such as vehicles and robots. Furthermore, the aircraft (10) can automatically fly around the target object (20) along a predetermined flight path, or can manually fly around the target object (20) under user control.

[0037] The aircraft (10) can capture images of the object (20) while flying around the object (20). For example, the captured images of the object (20) can be used to determine whether the object (20) has a defect (e.g., a breakdown, damage, etc.). The user can detect, recognize, and / or identify the object (20) through the aircraft (10).

[0038]

[0039] As an example, an aircraft (10) can photograph the ground surface. For example, the aircraft (10) can generate images of the ground surface while flying over the area to be photographed. The images captured by the aircraft (10) can be stitched together to create a map representing the photographed area.

[0040] As another example, the aircraft (10) can photograph the body or blades of a wind turbine. Specifically, the aircraft (10) can fly around the wind turbine to photograph the body or blades. For example, the aircraft (10) can take off from a station at a starting point and fly to the nose of the wind turbine. Then, the aircraft (10) can start photographing from the nose of the wind turbine and can photograph the blades while flying autonomously (or manually) along the blades. Typically, a wind turbine can include three blades, and the aircraft (10) can photograph the three blades to acquire images. In this case, the images photographed by the aircraft (10) can be utilized for inspection of the blades.

[0041] As another example, the aircraft (10) can photograph the surface of a large building. For example, the aircraft (10) can acquire images by photographing the surface while flying around the perimeter of the large building. In this case, the images captured by the aircraft (10) can be utilized for inspection of the surface of the large building.

[0042] As another example, the aircraft (10) can photograph various structures within a military facility. For example, a military facility may include barbed wire fences, buildings, and exterior walls, and the aircraft (10) can fly over various points within the military facility and photograph the surfaces of the structures. In this case, the images captured by the aircraft (10) can be utilized for inspection of the military facility.

[0043] In Fig. 1, the object (20) is illustrated as the ground surface, wind turbines, bridges, large buildings, and military facilities, but is not limited thereto. In other words, any object that can be captured by a camera can be the object (20) without limitation. As an example, the object (20) may be the ground surface including a predetermined area (e.g., farmland, industrial complex, residential area, commercial complex, etc.). As another example, the object (20) may be a structure in the industrial field. The object (20) may be a structure for power generation (e.g., wind turbines, thermal power plants, hydroelectric power plants, nuclear power plants, solar power plants, etc.), large buildings (e.g., factories, exhibition halls, stadiums, etc.), bridges, dams, power lines, roads, etc., but is not limited thereto. As another example, the object (20) may be a structure (e.g., barbed wire fences, ammunition depots, exterior walls, etc.) that must be detected or information collected in the security and military fields. As another example, anything that is difficult or dangerous for the user to visually inspect in its entirety, or that requires a lot of manpower and cost to inspect, may be considered a target (20).

[0044] FIG. 2 is a drawing for explaining the relationship between an aircraft, a controller, a server, and a station according to one embodiment.

[0045] Referring to FIG. 2, the server (30), the controller (40), and the station (50) can independently or jointly control the aircraft (10). For example, the server (30), the controller (40), and the station (50) can control the operation (e.g., movement, rotation, etc.) of the aircraft (10) or control the filming of the aircraft (10).

[0046] The aircraft (10) includes at least one camera, and can capture images of an object (20) using the camera. For example, the camera can be installed at a location advantageous for capturing images during flight of the aircraft (10) (e.g., an area not obscured by a propeller, etc., at the front or lower front of the aircraft).

[0047] For example, the aircraft (10) can fly using a global navigation satellite system (GNSS) and / or an inertial navigation system (INS).

[0048] For example, the aircraft (10) can transmit and receive data with a server (30), a controller (40), and / or a station (50). In addition, the controller (40) and the server (30), the server (30) and the station (50), and the station (50) and the controller (40) can transmit and receive data with each other.

[0049] Here, the data may include data required to control the flight of the aircraft (10), data on a flight image of the aircraft (10), data on an image taken of an object (20) by the aircraft (10), etc.

[0050] A flight image represents the field of view of an aircraft (10) when the aircraft (10) is flying. For example, the flight image may be a dynamic image acquired in real time, but is not limited thereto.

[0051] The image captured by the aircraft (10) of the target object (20) refers to an image captured by the aircraft (10) while flying around the target object (20). At this time, the image captured by the aircraft (10) of the target object (20) can be used as an image to check for defects in the target object (20). In this case, the image captured by the aircraft (10) of the target object (20) can be referred to as an inspection image.

[0052] Alternatively, if the target object (20) is a ground surface, an image of the target object (20) can be used as a map.

[0053] For example, the flight image may be an image with a relatively low resolution (or a low GSD (Ground Sampling Distance)) compared to the inspection image, and the inspection image may be an image with a relatively high resolution (or a high GSD) compared to the flight image. Meanwhile, depending on the type of aircraft (10), the cameras that generate the flight image and the inspection image may be the same or different.

[0054] A user can control the aircraft (10) using the interface of the controller (40). For example, the controller (40) can generate a control signal based on user input received through the interface and transmit the control signal to the aircraft (10). The controller (40) can transmit the control signal to the aircraft (10) via wireless communication. The control signal may be a signal that controls the flight, attitude, navigation, etc. of the aircraft.

[0055] The aircraft (10) can control the motor to rotate the propeller according to the control signal received from the controller (40). The aircraft (10) can move, rotate, etc. by changing the speed and / or attitude, etc. by the rotation of the propeller. Here, the attitude of the aircraft can be expressed as pitch (Y), roll (X), yaw (Z), etc. In addition, the aircraft can perform photographing of the target object (20), etc. according to the control signal received from the controller (40).

[0056] The controller (40) may further include a display device, and the user may check the flight image and / or inspection image of the aircraft (10) through the display device.

[0057] The controller (40) may be a device on which an application for controlling an aircraft (10) is installed. For example, the device on which the application is installed may be a variety of portable devices such as a smartphone, tablet, smart pad, laptop, or wearable device.

[0058] The server (30) or station (50) can control the aircraft (10) by directly transmitting a control signal to the aircraft. In addition, the aircraft (10) can transmit a flight image and / or an inspection image to the server (30), controller (40), or station (50).

[0059] The aircraft (10), server (30), controller (40), and station (50) can each analyze the inspection image. For example, the analysis of the inspection image may be to determine whether there is a defect in the target object (20) in the inspection image through an algorithm such as machine learning or deep learning. The aircraft (10) may directly determine whether there is a defect in the target object (20), or may transmit the inspection image to the server (30), controller (40), or station (50). The server (30), controller (40), or station (50) may analyze the inspection image received from the aircraft (10) to determine whether there is a defect in the target object (20).

[0060] Meanwhile, when satellite images, etc. are used to identify location information, there are cases where the GPS coordinates are inaccurate and the resolution is low, making it difficult to confirm the exact location information of the desired point.

[0061] Even when images captured by an aircraft from different locations contain the same point, the GPS coordinates for that point may be calculated differently. While GPS coordinates are calculated based on factors such as the aircraft's position and the camera's angle, errors in the calculated GPS coordinates can result in different coordinates for the same point. If different GPS coordinates are derived for the same point, stitching the images together can result in distorted images.

[0062] Meanwhile, an aircraft (10) can photograph the ground surface, and the captured images can be stitched together to create a map (or navigation image). At this time, GPS coordinates for each point shown on the map need to be calculated.

[0063] When a map is generated using images captured by an aircraft (10), the map may be distorted. Therefore, the exact GPS coordinates for each point shown on the map may not be confirmed.

[0064] When a vehicle moves using GPS coordinates, knowing the exact GPS coordinates is very important. In particular, unmanned vehicles may have difficulty in stable operation if accurate GPS coordinates are not provided. For example, when a military vehicle moves, the GPS coordinates of the road are needed to calculate the width of the road and determine whether the road is suitable for the military vehicle. If the aircraft (10) photographs the area in which the vehicle will move in advance and provides the photographed data to the vehicle, the vehicle can use the photographed data to identify the location or road it can move on. The photographed data includes images captured by the aircraft (10) and metadata of the images, and the metadata includes the GPS coordinates of the aircraft (10), the attitude of the aircraft (10), the angle of the gimbal, etc. when the aircraft (10) takes the image.

[0065] Furthermore, for unmanned tractors to accurately perform cultivation within the confines of the farmland, precise GPS coordinates of the farmland's boundaries are required. Conventionally, GPS receivers were installed on tractors to acquire these coordinates as they moved along the farmland's boundaries. However, this method made accurate GPS coordinate measurement difficult and required considerable time for the tractor to move.

[0066] In addition, conventional methods required precise GPS measurements of multiple points, such as obtaining GPS coordinates at specific points or corners using GPS coordinates, such as RTK (Real Time Kinematic) GPS. This not only takes a lot of time and effort, but can also lower overall accuracy due to errors between measurement points. In other words, conventional methods required directly measuring GPS coordinates of all key points within an area (e.g., corners of farmland, intersections of roads, vertices of buildings, etc.). To do this, it was necessary to move to each point and measure the coordinates one by one while carrying high-precision GPS equipment such as RTK GPS. This method was very time-consuming and manpower-intensive, and there were problems such as the possibility of lowering the accuracy of overall location information due to GPS errors between measurement points.

[0067] On the other hand, in the present invention, only the exact GPS coordinates of an aircraft (10), such as a drone, need to be measured, and the GPS coordinates of feature points can be calculated based on the GPS coordinates of the drone, thereby enabling more efficient and accurate acquisition of location information. That is, in the case of the present invention, since the GPS coordinates of all points within an area can be automatically calculated using the images captured by the aircraft (10) and the GPS coordinates of the aircraft, not only can the work time and cost be significantly reduced, but overall, more accurate and consistent location information can be obtained.

[0068] According to one embodiment, a processor accurately identifies location information of points within a captured area using images of the Earth's surface. Hereinafter, identifying location information of a point (or area) includes calculating the GPS coordinates of the point (or points within the area).

[0069] Additionally, if the captured area includes a road, the processor can accurately identify the road's location. For example, the processor can calculate the width and / or height of the road to determine whether a vehicle can move. Alternatively, the processor can also determine whether the road is blocked or has been washed out.

[0070] Additionally, if the photographed area includes farmland, the processor can accurately identify the location information of the farmland.

[0071] Figure 3 is a schematic diagram illustrating an example of an aircraft according to one embodiment.

[0072] Referring to FIG. 3, the aircraft (10) may include a sensor (110), a camera (120), a memory (130), a driving device (140), a communication device (150), and a processor (160). However, the components of the aircraft (10) are not limited to those illustrated in FIG. 3. In other words, the aircraft (10) may include at least one more component in addition to the components illustrated in FIG. 3, or at least one of the components illustrated in FIG. 3 may be excluded.

[0073] The sensor (110) detects various information necessary for the operation of the aircraft (10) (e.g., flight, photography, etc.), such as the aircraft (10) itself, the surrounding environment of the aircraft (10), identification of the target (20), and confirmation of the distance between the aircraft (10) and the target (20). The sensor (110) may include, but is not limited to, a gyro sensor, a barometer, an ultrasonic sensor, a magnetic sensor, an acceleration sensor, a proximity sensor, a lidar, a radar, and / or a GPS sensor.

[0074] For example, a gyro sensor and / or an acceleration sensor can measure the three-axis angular velocity of the aircraft (10). A barometer can measure pressure changes and / or air pressure in the atmosphere around the aircraft (10). Since air pressure varies with altitude, the aircraft (10) can also measure the altitude of the aircraft (10) using the barometer. An ultrasonic sensor can measure the distance between the aircraft (10) and the ground or an object (20). A magnetic sensor is a type of terrestrial magnetism sensor (compass sensor) and can detect geomagnetic information.

[0075] For example, a proximity sensor can measure the proximity state of an object (20) to an aircraft (10), the distance between the aircraft (10) and the object (20), and can include an ultrasonic sensor that can measure the distance to the object (20) from a signal reflected from the object (20) by outputting ultrasonic waves. A GPS sensor can calculate the current coordinates (x, y, z) of the aircraft (10) using GPS signals.

[0076] The sensor (110) may include an attitude and heading reference system (AHSR). For example, the attitude and heading reference system may include an inertial sensor or an inertial measurement unit (IMU). For example, the attitude and heading reference system may include a gyro sensor, an acceleration sensor, and a magnetic sensor, and fuse the sensor values ​​to obtain the attitude value of the aircraft (10). ) can be output. Here, the detailed value ( ) can be an angle based on three-dimensional coordinates (x-axis coordinate, y-axis coordinate, z-axis coordinate) according to GPS coordinates.

[0077] The camera (120) can photograph the object (20) according to instructions from the processor (160). For example, the aircraft (10) can include at least one camera, and can include a low-resolution camera and / or a high-resolution camera.

[0078] The camera (120) can be combined with a gimbal whose angle can be adjusted. Accordingly, the shooting angle of the camera (120) can be adjusted by the gimbal.

[0079] The memory (130) may include any non-transitory computer-readable recording medium. As an example, the memory (130) may include a non-permanent mass storage device such as a random access memory (RAM), a read only memory (ROM), a disk drive, a solid state drive (SSD), a flash memory, etc. As another example, the non-permanent mass storage device such as a ROM, an SSD, a flash memory, a disk drive, etc. may be a separate permanent storage device distinct from the memory. In addition, the memory (130) may store an operating system (OS) and at least one program code (e.g., code for the processor (160) to perform an operation to be described later with reference to FIGS. 4 to 14).

[0080] These software components may be loaded from a computer-readable recording medium separate from the memory (130). This separate computer-readable recording medium may be a recording medium that can be directly connected to a computer, and may include, for example, a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. Alternatively, the software components may be loaded into the memory (130) via a communication device (150) other than a computer-readable recording medium. For example, at least one program may be loaded into the memory (160) based on a computer program (e.g., a computer program for the processor (160) to perform the operations described below with reference to FIGS. 3 to 14) that is installed by files provided by developers or a file distribution system that distributes installation files of applications via the communication device (150).

[0081] The driving device (140) controls the driving of the motor at a speed and direction according to instructions from the processor (160), and accordingly, the rotational speed and direction of the propeller connected to the motor can be controlled. For example, the driving device (140) may include a motor and a propeller.

[0082] The communication device (150) performs data communication between the aircraft (10) and an external device. For example, the communication device (150) may communicate with the controller (40), server (30), and / or station (50) using various communication methods such as infrared communication, RF (Radio Frequency) communication, Wi-Fi communication, ZigBee communication, Bluetooth communication, laser communication, UWB (Ultra-Wideband) communication, LTE, 5G, 6G, and wireless LAN. However, the communication method employed in the communication device (150) is not limited to the above-described method.

[0083] The processor (160) can process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Here, the instructions can be provided from memory (130) or an external device (e.g., a server (30), a controller (40), a station (50), etc.). In addition, the processor (160) can generally control the operations of other components included in the aircraft (10).

[0084] For example, the functions performed by each module included in the processor (160) may be performed by one processor or by separate processors. The processor (160) may perform calculations or data processing related to control and / or communication of at least one other component of the aircraft (10).

[0085] The processor (160) acquires static images by photographing a predetermined area through the camera (120) of the aircraft (10). For example, the processor (160) can control the operation of the camera (120) so that an overlapping area is included between at least two or more of the static images.

[0086] The processor (160) generates a point cloud corresponding to a predetermined area using feature points extracted from static images. For example, the processor (160) may identify at least one common feature point in a first static image and a second static image that are adjacent to each other among the static images. Then, the processor (160) may determine at least one candidate solution for matching the at least one common feature point. Then, the processor (160) may generate a point cloud based on an optimal solution selected from the at least one candidate solution. For example, the point cloud may include location information of each feature point extracted from the static images.

[0087] The processor (160) stitches static images using feature points to generate a merged image corresponding to a predetermined area. For example, the processor (160) may stitch static images based on an optimal solution used to generate a point cloud.

[0088] The processor (160) uses the point cloud and the merged image to identify location information of at least some spots appearing on the merged image. For example, the processor (160) can match the point cloud and the merged image. Then, the processor (160) can identify location information of each point corresponding to the feature points included in the merged image based on the matching result. For example, the processor (160) can use interpolation to calculate location information of the remaining points excluding the points corresponding to the feature points among the points appearing on the merged image.

[0089] The processor (160) segments at least one object included in the merged image identified using a machine learning model. For example, the processor (160) may update the boundary of the segmented object using user input regarding the boundary of the at least one object.

[0090] A detailed description of the operations of the aircraft (10) described above with reference to FIG. 3 will be described later with reference to FIGS. 4 to 14.

[0091] For example, the processor (160) may be implemented as an array of a plurality of logic gates, or may be implemented as a combination of a general-purpose microprocessor and a memory storing a program that can be executed on the microprocessor. For example, the processor (160) may include a general-purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), a neural network processing unit (NPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, and the like.

[0092] In some environments, the processor (160) may include an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), or the like. For example, the processor (160) may refer to a combination of processing devices, such as a combination of a digital signal processor (DSP) and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors in conjunction with a digital signal processor (DSP) core, or any other such combination of configurations.

[0093] Meanwhile, the operation of the processor (160) described above with reference to FIG. 3 may be implemented by a separate device. In this case, the separate device (hereinafter referred to as a “device for identifying location information of an area”) may be included in at least one of the aircraft (10), the server (30), the controller (40), the station (50), and another external device. When the device for identifying location information of an area is included in at least one of the server (30), the controller (40), the station (50), and another external device, the aircraft (10) may capture static images or dynamic images and transmit the captured images to the device for identifying location information of an area. An example of a device for identifying location information of an area will be described below with reference to FIG. 14.

[0094] FIG. 4 is a flowchart illustrating an example of a method for identifying location information of an area according to one embodiment.

[0095] The method illustrated in FIG. 4 is comprised of steps that are processed sequentially in the aircraft (10) or processor (160) illustrated in FIGS. 1 to 3. Therefore, even if omitted below, the content described above regarding the aircraft (10) or processor (160) may also be applied to the method illustrated in FIG. 4. In addition, the operations of the processor (160) below may also be performed by a device that identifies location information of the area illustrated in FIG. 14.

[0096] At step 410, the processor (160) acquires static images by photographing a predetermined area through the camera (120) of the aircraft (10).

[0097] While the aircraft (10) is in flight, the processor (160) controls the camera (120) to capture a predetermined area. For example, as the aircraft (10) moves within the predetermined area and captures the image, multiple static images may be acquired. Here, the predetermined area refers to an area desired for analysis and is included in a map or navigation image.

[0098] The processor (160) may store metadata corresponding to each of the static images in the memory (130). Alternatively, the processor (160) may transmit the metadata to an external device (e.g., a server (30), a controller (40), a station (50), etc.) via a communication device (150). For example, the metadata may include GPS coordinates of the aircraft (10) at the time the static image is captured, the attitude of the aircraft (10) (e.g., a 3-axis angle), an angle of the gimbal, the capture time, the resolution of the camera (120), wide-angle / zoom camera information, focal length, etc. For example, the GPS coordinates may be acquired using RTK (Real Time Kinematic) GPS, but are not limited thereto.

[0099] The processor (160) can control the operation of the camera (120) so that an overlapping area is included between at least two or more static images. For example, the processor (160) can control the operation of the camera (120) so that the overlapping rate of the static images is the same. Hereinafter, an example of obtaining static images will be described with reference to FIG. 5.

[0100] FIG. 5 is a diagram illustrating an example of a processor obtaining static images according to one embodiment.

[0101] Referring to FIG. 5, an aircraft (10) can photograph an area (510) while flying over the ground surface. Here, the area (510) refers to an area where analysis is desired. Although FIG. 5 illustrates the aircraft (10) moving from right to left while maintaining the same altitude and performing photography, the direction of movement of the aircraft is not limited to that illustrated in FIG. 5.

[0102] In addition, although FIG. 5 illustrates that there are four or more static images (521, 522, 523, …, 52n) (where n is a natural number greater than or equal to 4), the number of static images is not limited to that illustrated in FIG. 5.

[0103] The processor (160) can control the operation of the camera (120) so that an overlapping area (531, 532) exists between adjacent static images among the static images (521, 522, 523, …, 52n). In other words, the processor (160) can set the shooting area of ​​the camera (120) so that an overlapping area (531, 532) exists between adjacent static images. Here, the shooting area means the range of the area (510) included in each of the static images (521, 522, 523, …, 52n).

[0104] For example, the overlapping area (531) between the static image (521) and the static image (522) and the overlapping area (532) between the static image (522) and the static image (523) may have the same area. In other words, the static images (521, 522, 523, …, 52n) may be acquired so that the overlapping ratio is the same.

[0105] The higher the overlap ratio, the more advantageous it may be for stitching static images (521, 522, 523, …, 52n). However, the higher the overlap ratio, the more time it takes for the camera (120) to capture static images (521, 522, 523, …, 52n) and the more time it takes for stitching static images (521, 522, 523, …, 52n).

[0106] As the altitude of the aircraft (10) at the time of shooting increases, the shooting area included in one static image becomes wider. Accordingly, the number of static images (521, 522, 523, …, 52n) covering the entire area (510) and the time required to shoot the entire area (510) can be reduced. However, the GSD (Ground Sample Distance) of each of the static images (521, 522, 523, …, 52n) increases. In other words, the resolution of each of the static images (521, 522, 523, …, 52n) decreases. Therefore, the altitude of the aircraft (10) can be determined according to the GSD required for the static images (521, 522, 523, …, 52n). Here, GSD means the distance between the center points of two adjacent pixels included in a single image, and is a value representing the resolution of the image.

[0107] Although FIG. 5 illustrates a single aircraft (10) moving around and photographing an area (510), this is not a limitation. For example, two or more aircraft may photograph an area (510). In this case, the areas to be photographed by the two or more aircraft may be distributed.

[0108] In addition, a plurality of cameras having different shooting areas are installed on the aircraft (10), and the processor (160) can control the plurality of cameras to shoot an area (510). In addition, a single camera with a changeable shooting angle is installed on the aircraft (10), and the processor (160) can control the single camera to shoot an area (510) while changing the shooting angle at the same location.

[0109] Referring again to FIG. 4, at step 420, the processor (160) generates a point cloud corresponding to a predetermined area using feature points extracted from static images. The point cloud is generated by indicating the locations of the feature points on a virtual three-dimensional coordinate system. The processor (160) can generate the point cloud by indicating the relative locations of the feature points.

[0110] A feature point can be a point included in a static image and distinguished from its surroundings, and serves as a reference point when matching static images. For example, the processor (160) may extract feature points using a pre-trained machine learning model.

[0111] The processor (160) generates a point cloud based on common feature points among the feature points of static images. For example, the point cloud may include location information for each feature point extracted from the static images. Below, an example of the processor (160) generating a point cloud will be described with reference to FIGS. 6 to 9 .

[0112] FIG. 6 is a flowchart illustrating an example of a processor generating a point cloud according to one embodiment.

[0113] At step 610, the processor (160) identifies at least one common feature point in a first static image and a second static image that are adjacent to each other among the static images.

[0114] The first static image and the second static image may be sequentially captured images, or images with overlapping areas. For example, assume that five images are sequentially acquired while the aircraft (10) changes position or while positioned at the same point and changing the angle of the camera (120). In this case, if the first static image is the third image captured, the second static image may be the fourth image captured.

[0115] For example, the processor (160) can extract feature points from each of the static images. In addition, the processor (160) can analyze the feature points of the first static image and the feature points of the second static images to extract at least one common feature point.

[0116] At step 620, the processor (160) determines at least one candidate solution for matching at least one common feature point.

[0117] Here, the candidate solution includes a method for matching common features by at least one of translation, rotation, and scaling of at least one of the first and second static images. In other words, determining the candidate solution may mean determining how the processor (160) will match adjacent static images. The processor (160) may match the static images in various ways, and the matching method may be a candidate solution.

[0118] At step 630, the processor (160) generates a point cloud based on an optimal solution selected from at least one candidate solution.

[0119] Here, the optimal solution includes a method of selecting at least one candidate solution based on at least one preset criterion. For example, the processor (160) may determine one candidate solution satisfying the criterion among multiple candidate solutions as the optimal solution, or may determine the candidate solution with the highest score among multiple candidate solutions as the optimal solution. The processor (160) may assign weights to translation, rotation, and scaling, and may assign scores based on how much the candidate solution has translated, rotated, and scaled static images. FIG. 7 is a diagram illustrating an example of a processor identifying common features in images according to one embodiment.

[0120] Referring to FIG. 7, the processor (160) selects a first static image (720) and a second static image (730) from among the static images (710). At this time, the first static image (720) and the second static image (730) may be images captured continuously. Accordingly, the first static image (720) and the second static image (730) may include an overlapping area.

[0121] The processor (160) extracts feature points (721, 722, 723) from the first static image (720). For example, the processor (160) may extract feature points (721, 722, 723) from the first static image (720) by considering the type of object included in the first static image (720), the external features of the object, the external environment (weather, time, etc.) at the time the first static image (720) was captured, etc.

[0122] In the same manner as described above, the processor (160) extracts feature points (731, 732, 733) from the second static image (730).

[0123] The method by which the processor (160) extracts feature points from the first static image (720) and the second static image (730) is the same as the conventional method by which feature points are extracted within an image, so a detailed description thereof is omitted.

[0124] The processor (160) identifies at least one common feature point among the feature points (721, 722, 723) and the feature points (731, 732, 733). In FIG. 7, the processor (160) can identify that the feature points (722, 723) of the first static image (720) and the feature points (731, 732) of the second static image (730) are common feature points.

[0125] Meanwhile, the processor (160) may identify feature points by checking only a portion of the static images (720, 730). For example, the processor (160) may identify at least one common feature point by checking only feature points included in overlapping areas of the static images (720, 730).

[0126] FIG. 8 is a diagram illustrating an example of a processor determining a candidate solution and an optimal solution according to one embodiment.

[0127] Referring to FIG. 8, feature points (811, 812) of the first static image (810) and feature points (821, 822) of the second static image (820) correspond to common feature points. As the feature points (811, 812) and feature points (821, 822) are aligned, the first static image (810) and the second static image (820) can be stitched.

[0128] The processor (160) determines candidate solutions (830, 83k) (wherein, k is a natural number greater than or equal to 2) with which the feature points (811, 812) and the feature points (821, 822) can be aligned. For example, the processor (160) may determine candidate solutions (830, 83k) that manipulate at least one of the first static image (810) and the second static image (820) so that the positions and directions of the feature points (811, 812) and the feature points (821, 822) can be aligned in the same manner.

[0129] Here, the manipulation includes moving, rotating, and / or resizing at least one of the first static image (810) and the second static image (820). Resizing also includes enlarging or reducing the size of the static image.

[0130] For example, the processor (160) can use a machine learning model to determine candidate solutions (830, 83k).

[0131] As described above with reference to FIG. 8, the processor (160) can determine candidate solutions (830, 83k) with which common features can be aligned. However, considering the conditions (or method) under which the aircraft (10) captured the static images, some of the candidate solutions (830, 83k) may not be suitable for merging images. Therefore, the processor (160) selects an optimal solution (840) from among the candidate solutions (830, 83k).

[0132] For example, the optimal solution (840) may be selected from among candidate solutions based on at least one preset criterion. Here, the criterion may be preset in relation to the translation, rotation, and / or scaling of the image. Furthermore, the criterion may be set differently depending on whether the image is to be merged horizontally or vertically. Furthermore, the criterion may be set differently depending on the method by which the image is captured.

[0133] As an example, the processor (160) may exclude candidate solutions (830, 83k) whose static image rotation angle exceeds a threshold value. If the processor (160) acquires the first static image (810) and the second static image (820) while only changing the shooting angle of the camera (120) at the same location, the processor (160) may exclude (or assign a low weight to) a candidate solution that includes a method of matching the rotation angle exceeding the threshold value among the candidate solutions.

[0134] As another example, the processor (160) may exclude from among the candidate solutions (830, 83k) a solution whose degree of resizing of the static image exceeds a threshold.

[0135] As another example, the processor (160) may exclude from among the candidate solutions (830, 83k) a solution in which the degree to which the static image moves exceeds a threshold value.

[0136] FIG. 9 is a diagram illustrating an example of a processor generating a point cloud based on an optimal solution according to one embodiment.

[0137] Referring to FIG. 9, the processor (160) matches common feature points of static images (911, 912, 913) using an optimal solution. Here, the static images (911, 912, 913) are images that capture at least a portion of an area (920). Therefore, the common feature points refer to feature points included in the same portion (931, 932) on the area (920). In other words, the common feature points of the static image (911) and the static image (912) are included in the portion (931). In addition, the common feature points of the static image (912) and the static image (913) are included in the portion (932).

[0138] The processor (160) aligns common feature points so that the common feature points are located at the same location of the parts (931, 932) based on the optimal solution. Then, the processor (160) places each of the remaining feature points, excluding the common feature points, among the static images (911, 912, 913), at a corresponding location in the area (920). In this manner, the processor (160) generates a point cloud (940) in which feature points are located in a three-dimensional space.

[0139] A point cloud (940) can be created on a virtual three-dimensional coordinate system. For example, the reference point (origin) of the virtual three-dimensional coordinate system can be designated as a location with precisely known GPS coordinates (e.g., the point where the aircraft (10) begins its flight).

[0140] Accordingly, each of the points included in the point cloud (940) corresponds to a feature point of the static images (911, 912, 913). Therefore, the point cloud (940) includes location information (e.g., GPS coordinates) of each of the feature points of the static images (911, 912, 913).

[0141] Meanwhile, since the point cloud (940) is generated using static images captured at different locations for the same feature point, the depth of the feature point can be estimated from the point cloud (940). The processor (160) can calculate the GPS coordinates of the feature point included in at least two or more static images. For example, the processor (160) can calculate the GPS coordinates of the feature point using static images captured at at least two or more different locations. Alternatively, the processor (160) can calculate the GPS coordinates of the feature point using static images captured at at least two or more different angles at the same location. The processor (160) can calculate the GPS coordinates of the feature point using the GPS coordinates of the aircraft (10) when the aircraft (10) generates the static image, the angle of the gimbal, the attitude of the aircraft (10), etc.

[0142] Referring again to FIG. 4, at step 430, the processor (160) stitches static images using feature points to generate a merged image corresponding to a given area.

[0143] For example, the processor (160) can stitch static images based on the optimal solution used to generate the point cloud. An example of the processor (160) stitching static images will be described below with reference to FIG. 10 .

[0144] FIG. 10 is a diagram illustrating an example of a processor stitching static images according to one embodiment.

[0145] For convenience of explanation, FIG. 10 shows the number of static images (1010) as three, but the number of static images (1010) is not limited thereto.

[0146] Referring to FIG. 10, the processor (160) identifies common feature points (1021) in the static image (1011) and the static image (1012) among the static images (1010). Here, the static image (1011) and the static image (1012) are images captured in adjacent areas. In addition, an example of identifying common feature points (1021) is as described above with reference to FIG. 7.

[0147] Then, the processor (160) stitches the static image (1011) and the static image (1012) based on the optimal solution. For example, the processor (160) can align common feature points (1021) based on the optimal solution and correct the static image (1011) and the static image (1012) accordingly. Thereafter, the processor (160) can orthogonally project the corrected images onto a two-dimensional space and stitch the static image (1011) and the static image (1012) through texturing.

[0148] In the same manner as described above, the processor (160) identifies common features (1022) in the static image (1012) and the static image (1013). Then, the processor (160) stitches the static image (1012) and the static image (1013) based on the optimal solution.

[0149] Accordingly, the processor (160) can generate a merged image (1030) using static images (1010).

[0150]

[0151] Meanwhile, the processor (160) may segment the merged image (1030). Segmenting the image may mean identifying which area each pixel of the image belongs to. Alternatively, segmenting the image may mean assigning a label to each pixel of the image to indicate which object it corresponds to. For example, the object may be a distinguishable area on the ground surface, such as a road, a river, a rice paddy, a field, a building, or a mountain.

[0152] The processor (160) can segment at least one object included in the merged image (1030). In other words, the processor (160) can extract objects on a pixel-by-pixel basis. For example, the processor (160) can identify at least one object in the merged image (1030) using a pre-trained machine learning model. Then, the processor (160) can segment the identified object from the merged image (1030).

[0153] Additionally, the processor (160) may update the boundary of a segmented object using user input regarding the boundary of at least one object. Referring now to FIG. 11, an example of the processor (160) segmenting an object from a merged image (1030) and updating the boundary of the segmented object using user input will be described.

[0154] FIG. 11 is a diagram illustrating an example of a processor segmenting an object from a merged image and updating the boundary of the segmented object using user input according to one embodiment.

[0155] Referring to FIG. 11, the processor (160) can segment an object (1120) from the merged image (1110). For example, the processor (160) can identify an object (1120) (e.g., a road, farmland, building, mountain, etc.) included in the merged image (1110) using a machine learning model.

[0156] Meanwhile, the processor (160) can update the boundary of the object (1120) using user input. For example, the processor (160) can obtain a user input that distinguishes a location (1111) included in the object (1120) and a location (1112) not included in the object (1120) in the merged image (1110). Then, the processor (160) can update the previously identified boundary according to the user input.

[0157] For example, the processor (160) may obtain user input regarding a portion where the previously identified boundary is unclear. Specifically, the processor (160) may inform the user of the portion where the previously identified boundary is unclear, and may update the previously identified boundary when the user specifies which object the location (1111) and location (1112) belong to. In other words, the processor (160) may perform prompt-based segmentation on the merged image (1110).

[0158] Accordingly, the boundary of the object (1120) becomes clear, and the processor (160) can accurately segment the object (1140) from the merged image (1130).

[0159] Referring again to FIG. 4, at step 440, the processor (160) uses the point cloud and the merged image to identify location information of at least some points appearing on the merged image.

[0160] For example, the processor (160) can match a point cloud with a merged image. Furthermore, the processor (160) can identify location information for each point corresponding to a feature point included in the merged image based on the matching result. Hereinafter, with reference to FIG. 12, an example of the processor (160) matching a point cloud with a merged image and identifying location information for a point corresponding to a feature point in the merged image will be described.

[0161] Additionally, the processor (160) can use interpolation to calculate the location information of points appearing on the merged image, excluding points corresponding to feature points. Hereinafter, with reference to FIG. 13, an example of the processor (160) calculating the location information of points using interpolation will be described.

[0162] FIG. 12 is a diagram illustrating an example of a processor matching a point cloud and a merged image and identifying location information of a point corresponding to a feature point of the merged image according to one embodiment.

[0163] Figure 12 illustrates a point cloud (1210) and a merged image (1220). The processor (160) can match the point cloud and the merged image.

[0164] For example, the processor (160) can match the point cloud (1210) and the merged image (1220) by projecting the three-dimensional point cloud (1210) onto the two-dimensional merged image (1220). Specifically, the three-dimensional coordinates of the point cloud (1210) can be projected into the two-dimensional coordinates of the same point through scaling and translation. As described above with reference to FIG. 9, each of the points included in the point cloud (1210) corresponds to a feature point and has GPS coordinates. Accordingly, the feature point (1221) on the merged image (1220) matched with the point cloud (1210) also has GPS coordinates.

[0165] FIG. 13 is a diagram illustrating an example of a processor calculating location information of a point using interpolation according to one embodiment.

[0166] Referring to Fig. 13, points (1311, 1312, 1313) corresponding to feature points included in the merged image (1310) each have GPS coordinates. On the other hand, if point (1314) does not correspond to a feature point, point (1314) does not have GPS coordinates.

[0167] At this time, the processor (160) can calculate the GPS coordinates of the point (1314) using interpolation. For example, the processor (160) can create a triangle by connecting three points (1311, 1312, 1313) whose GPS coordinates are known, and the processor (160) can calculate the coordinates of the point (1314) inside the triangle using the coordinates on the plane of the triangle.

[0168] Accordingly, the processor (160) can identify the location information of all points on the merged image (1320). In other words, the processor (160) can confirm the GPS coordinates of points (1311, 1312, 1313) corresponding to feature points through the GPS coordinates included in the point cloud (1210). In addition, the processor (160) can calculate the GPS coordinates of points (1324) that do not correspond to feature points using interpolation.

[0169] For example, the processor (160) may identify a specific object in the merged image (1320) and calculate GPS coordinates only for the boundaries of the identified object. For example, the processor (160) may identify farmland or a road in the merged image (1320) and calculate GPS coordinates only for the boundaries of the farmland or road.

[0170] Accordingly, the processor (160) can accurately identify the location information of the road in the merged image (1320). For example, the processor (160) can calculate the width and / or height of the road using GPS coordinates of the road boundary. Accordingly, the processor (160) can determine whether a vehicle can move on the road. Alternatively, if there is a sudden change in the GPS coordinates of the road boundary, the processor (160) can determine that the road is blocked or a loss has occurred on the road.

[0171] FIG. 14 is a schematic diagram illustrating an example of a device for identifying location information of an area according to one embodiment.

[0172] Referring to FIG. 14, a device (1400) for identifying location information of an area includes a processor (1410), a memory (1420), and a communication device (1430). However, the components of the device (1400) are not limited to those illustrated in FIG. 14. In other words, the device (1400) may include at least one more component in addition to the components illustrated in FIG. 14, or at least one of the components illustrated in FIG. 14 may be excluded.

[0173] As described above with reference to FIG. 3, the device (1400) may be included in at least one of the aircraft (10), the server (30), the controller (40), the station (50), and other external devices. Accordingly, the processor (160), the memory (130), and the communication device (150) described above with reference to FIG. 3 may correspond to the processor (1410), the memory (1420), and the communication device (1430) of the device (1400), respectively. Therefore, a detailed description of the processor (1410), the memory (1420), and the communication device (1430) is omitted.

[0174] According to the above, the processor (160, 1410) can use the aircraft (10) to photograph a predetermined area and accurately confirm the location information (e.g., GPS coordinates) of points included in the photographed area.

[0175] Meanwhile, the above-described method can be written as a program that can be executed on a computer, and can be implemented on a general-purpose digital computer that runs the program using a computer-readable recording medium. In addition, the structure of the data used in the above-described method can be recorded on a computer-readable recording medium through various means. The computer-readable recording medium includes storage media such as magnetic storage media (e.g., ROM, RAM, USB, floppy disk, hard disk, etc.) and optical reading media (e.g., CD-ROM, DVD, etc.).

[0176] Those skilled in the art will appreciate that the present invention can be implemented in modified forms without departing from the essential characteristics of the above-described invention. Therefore, the disclosed methods should be considered illustrative rather than restrictive. The scope of the claims, not the foregoing description, is defined by the scope of the patent, and should be interpreted to encompass all differences within the scope equivalent thereto.

[0177]

[0178] [Explanation of symbols]

[0179] 10: Aircraft

[0180] 20: Object

Claims

1. A step of obtaining static images by photographing a predetermined area through a camera of an aircraft; A step of generating a point cloud corresponding to the predetermined area using feature points extracted from the static images; A step of creating a merged image corresponding to the predetermined area by stitching the static images using the above feature points; and Identifying location information of an area including a step of identifying location information of at least some spots appearing on the merged image using the point cloud and the merged image. method.

2. In paragraph 1, The above acquisition steps are: Controlling the operation of the camera so that an overlapping area is included between at least two of the static images. method.

3. In paragraph 1, The steps for generating the above point cloud are: A step of identifying at least one common feature point in a first static image and a second static image that are adjacent to each other among the above static images; determining at least one candidate solution for matching at least one common feature point; and A step of generating the point cloud based on an optimal solution selected from at least one candidate solution. method.

4. In paragraph 1, The above point cloud is, Contains location information of each feature point extracted from the above static images. method.

5. In paragraph 1, The steps for generating the above merged image are: Stitching the static images based on the optimal solution used to generate the above point cloud. method.

6. In paragraph 1, Further comprising a step of segmenting at least one object included in the merged image identified using a machine learning model. method.

7. In paragraph 6, The above dividing step is, Updating the boundary of the segmented object using user input regarding the boundary of at least one object. method.

8. In paragraph 1, The above identifying step is, A step of matching the point cloud and the merged image; and A step of identifying location information of each point corresponding to the feature points included in the merged image based on the result of the above matching. method.

9. In paragraph 8, The above identifying step is, It further includes a step of calculating the location information of the remaining points, excluding the points corresponding to the feature points, among the points appearing on the merged image using interpolation. method.

10. A computer-readable program for executing the method of paragraph 1 on a computer. Recording medium.

11. At least one memory in which at least one program is stored; and At least one processor that performs an operation by executing at least one program; At least one processor, By photographing a predetermined area through a camera of an aircraft, static images are obtained, a point cloud corresponding to the predetermined area is generated using feature points extracted from the static images, a merged image corresponding to the predetermined area is generated by stitching the static images using the feature points, and location information of an area is identified by identifying location information of at least some spots appearing on the merged image using the point cloud and the merged image. device.

12. In paragraph 11, At least one processor, Controlling the operation of the camera so that an overlapping area is included between at least two of the static images. device.

13. In paragraph 11, At least one processor, Identifying at least one common feature point in adjacent first and second static images among the static images, determining at least one candidate solution for matching the at least one common feature point, and generating the point cloud based on an optimal solution selected from the at least one candidate solution. device.

14. In paragraph 11, The above point cloud is, Contains location information of each feature point extracted from the above static images. device.

15. In paragraph 11, At least one processor, Stitching the static images based on the optimal solution used to generate the above point cloud. device.

16. In paragraph 11, At least one processor, Segmenting at least one object included in the merged image identified using a machine learning model. device.

17. In paragraph 16, At least one processor, Updating the boundary of the segmented object using user input regarding the boundary of at least one object. device.

18. In paragraph 11, At least one processor, Matching the point cloud and the merged image, and identifying the location information of each point corresponding to the feature points included in the merged image based on the result of the matching. device.

19. In paragraph 18, At least one processor, Using interpolation, the location information of the remaining points, excluding the points corresponding to the feature points, among the points appearing on the merged image are calculated. device.

20. At least one camera; At least one memory having at least one program stored therein; and At least one processor that performs an operation by executing at least one program; At least one processor, By photographing a predetermined area through the camera, static images are obtained, a point cloud corresponding to the predetermined area is generated using feature points extracted from the static images, a merged image corresponding to the predetermined area is generated by stitching the static images using the feature points, and location information of at least some spots appearing on the merged image is identified using the point cloud and the merged image. Aircraft.

Images