Apparatus and method for determining position of aerial vehicle

Abstract

An aerial vehicle of the present invention comprises: a camera for generating a captured image by photographing the surroundings of the aerial vehicle; a memory for storing at least one instruction, and a meta map in which objects included in a map are marked as figures; and at least one processor for executing the at least one instruction, wherein the processor receives the captured image from the camera, identifies an object included in the captured image, generates a meta image in which the object is marked as a figure, and determines the position of the aerial vehicle by matching the meta map with the pattern of figures included in the meta image.

Description

Device and method for determining the position of an aircraft

[0001] The present invention relates to a device and method for determining the location of an aircraft using deep learning technology and pattern matching techniques.

[0002] The content described in this section merely provides background information for the present embodiment and does not constitute prior art.

[0003] Typically, an aircraft's location is determined based on GPS (Global Positioning System) signals. However, due to GPS spoofing and other threats, the GPS signals received by the aircraft may be false, leading the aircraft to misjudge its location based on these false signals. Therefore, relying on GPS signals to determine an aircraft's location presents a challenge, making it difficult to accurately determine its location.

[0004] Accordingly, research and development are steadily underway on technologies that determine an aircraft's location without relying on GPS signals. However, methods that determine an aircraft's location by comparing images captured by the aircraft with maps suffer from the disadvantages of excessive map size and the significant computational and time required to compare images. Furthermore, even if an aircraft captures the same area, the images can be perceived differently depending on factors such as the direction of light and the season, making the method of comparing the images themselves problematic.

[0005] Therefore, there has been a significant need for a technology that can accurately determine the position of an aircraft with relatively low computational complexity, without relying on GPS signals.

[0006] An object of the present invention is to provide a device and method capable of accurately determining the position of an aircraft without relying on GPS signals.

[0007] In addition, an object of the present invention is to provide a device and method capable of determining the location of an aircraft using an image captured by the aircraft through a camera.

[0008] Another object of the present invention is to provide a device and method capable of determining the location of an aircraft by analyzing a pattern in which objects are arranged in an image.

[0009] In addition, an object of the present invention is to provide a device and method for identifying objects included in an image using deep learning, expressing the objects in a predefined form, and then determining the location of an aircraft using the pattern of the objects.

[0010] An aircraft according to an embodiment of the present invention comprises: a camera for photographing the surroundings of the aircraft to generate a photographed image; a memory for storing at least one instruction and a meta map that graphically displays objects included in the map; and at least one processor for executing the at least one instruction, wherein the processor receives the photographed image from the camera, identifies an object included in the photographed image, generates a meta image that graphically displays the object, and matches patterns of figures included in the meta map and the meta image to determine a location.

[0011] Additionally, the processor matches the shapes of the meta image with the patterns of the shapes of the meta map using the relative distances and relative angles of the shapes.

[0012] Additionally, the processor calculates a positional relationship between the position of the aircraft and two reference figures in the meta image, selects two candidate figures from the meta map, determines center candidates in which the candidate figures have the same positional relationship with the reference figures, and determines the position of the aircraft among the center candidates.

[0013] In addition, the processor determines one of the intersections of two circles with different sizes of radii of each of the candidate shapes as the center candidate, and the ratio of the radii of the two circles is the same as the relative distance of the reference shapes, and the relative angle of the candidate shapes is the same as the relative angle of the reference shapes.

[0014] Additionally, the processor sets a region of interest in the meta map, determines center candidates for a combination of shapes located within the region of interest, calculates the number of matching shapes and / or a matching error for the center candidates, and determines the location of the aircraft among the center candidates.

[0015] Additionally, the processor determines the region of interest using one or more of the most recent position, direction of movement, and distance of movement of the aircraft.

[0016] Additionally, the objects included in the captured image and the objects included in the map include at least one of a building, a playground, a park, a green space, an intersection, a river, farmland, and a mountain.

[0017] Additionally, the processor generates the meta image by displaying an object included in the captured image as a dot at the center position of the object.

[0018] An aircraft according to an embodiment of the present invention can determine the location of the aircraft by using an image captured by a camera of the aircraft without relying on a GPS signal.

[0019] Additionally, the aircraft according to an embodiment of the present invention can determine the location of the aircraft by using patterns of objects identified through deep learning.

[0020] An aircraft according to an embodiment of the present invention can reduce the capacity of data stored in the aircraft and quickly determine the location of the aircraft through pattern matching.

[0021] In addition to the above-described contents, specific effects according to embodiments of the present invention are described together with specific matters for carrying out the invention below.

[0022] FIG. 1 is a drawing for explaining an example of photographing an object using an aircraft according to an embodiment of the present invention.

[0023] FIG. 2 is a drawing for explaining the relationship between an aircraft, a controller, a server, and a station according to an embodiment of the present invention.

[0024] Figure 3 is a block diagram of an aircraft according to an embodiment of the present invention.

[0025] Figure 4 is a flowchart of a method for determining the position of an aircraft according to an embodiment of the present invention.

[0026] FIG. 5 is a drawing for explaining a process for determining the position of an aircraft according to an embodiment of the present invention.

[0027] FIG. 6 is a diagram for explaining a process of generating a meta image according to an embodiment of the present invention.

[0028] FIG. 7 is a diagram for explaining the structure of an object recognition model according to an embodiment of the present invention.

[0029] Figure 8 is a drawing for explaining calculating relative distances and relative angles for reference points in a meta image.

[0030] Figure 9 is a drawing for explaining setting a region of interest and selecting candidate points in the region of interest.

[0031] Figure 10 is a drawing for explaining a method for determining the location of a sensor candidate using candidate points.

[0032] FIG. 11 and FIG. 12 are drawings for explaining a method for determining the position of an aircraft in a meta map according to an embodiment of the present invention.

[0033] The terms used in the examples are selected from widely used, current terms, as much as possible. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, the applicant may arbitrarily select terms, in which case their meanings will be described in detail in the relevant description. Therefore, the terms used in the specification should be defined based on their meaning and the overall content of the specification, rather than simply their names.

[0034] When a part of a specification is said to “include” a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise stated.

[0035] Additionally, terms including ordinal numbers, such as “first” or “second,” used in the specification may be used to describe various components, but the components should not be limited by the terms. The terms may be used to distinguish one component from another.

[0036] Below, embodiments are described in detail with reference to the attached drawings. However, the embodiments may be implemented in various different forms and are not limited to the examples described herein.

[0037] FIG. 1 is a drawing for explaining an example of photographing an object using an aircraft according to one embodiment.

[0038] The aircraft (10) may include any aircraft capable of flight, including a drone, an Unmanned Aerial Vehicle (UAV), an Unmanned Aerial Mobility (UAM), an aircraft, a helicopter, etc.

[0039] The aircraft (10) can fly alone or with multiple aircraft capable of collaborating. Furthermore, the aircraft (10) can also collaborate with other types of devices, such as vehicles and robots. Furthermore, the aircraft (10) can automatically fly around the target object (20) along a predetermined flight path, or can manually fly around the target object (20) under user control.

[0040] The aircraft (10) can capture images of the object (20) while flying around the object (20). For example, the captured images of the object (20) can be used to determine whether the object (20) has a defect (e.g., breakdown, damage, etc.). The user can detect, recognize, and / or identify the object (20) through the aircraft (10).

[0041] As an example, the aircraft (10) can photograph the body or blades of a wind turbine. Specifically, the aircraft (10) can fly around the wind turbine to photograph the body or blades. For example, the aircraft (10) can take off from a station at a starting point and fly to the nose of the wind turbine. Then, the aircraft (10) can start photographing from the nose of the wind turbine and can photograph the blades while flying autonomously (or manually) along the blades. Typically, a wind turbine can include three blades, and the aircraft (10) can photograph the three blades to acquire images. At this time, the images photographed by the aircraft (10) can be utilized for inspection of the blades.

[0042] As another example, the aircraft (10) can photograph the surface of a large building. For example, the aircraft (10) can acquire images by photographing the surface while flying around the perimeter of the large building. In this case, the images captured by the aircraft (10) can be utilized for inspection of the surface of the large building.

[0043] As another example, the aircraft (10) can photograph various structures within a military facility. For example, a military facility may include barbed wire fences, buildings, and exterior walls, and the aircraft (10) can fly over various points within the military facility and photograph the surfaces of the structures. In this case, the images captured by the aircraft (10) can be utilized for inspection of the military facility.

[0044] Although wind turbines, bridges, large buildings, and military facilities are depicted as objects (20) in FIG. 1, the objects are not limited thereto. In other words, any structure having a shape may be applied to the objects (20) without limitation. For example, the objects (20) may be structures in the industrial field. The objects (20) may be structures for power generation (e.g., wind turbines, thermal power plants, hydroelectric power plants, nuclear power plants, solar power plants, etc.), large buildings (e.g., factories, exhibition halls, stadiums, etc.), bridges, dams, power lines, roads, etc., but are not limited thereto. As another example, the objects (20) may be structures that must be detected or information collected in the security and military fields (e.g., barbed wire fences, ammunition depots, exterior walls, etc.). As another example, anything that is difficult for a user to inspect entirely with the naked eye, dangerous, or requires a lot of manpower and cost for inspection may be applied to the objects (20).

[0045] FIG. 2 is a drawing for explaining the relationship between an aircraft, a controller, a server, and a station according to one embodiment.

[0046] Referring to FIG. 2, the server (30), the controller (40), and the station (50) can independently or jointly control the aircraft (10). For example, the server (30), the controller (40), and the station (50) can control the operation (e.g., movement, rotation, etc.) of the aircraft (10) or control the filming of the aircraft (10).

[0047] The aircraft (10) includes at least one camera, and can capture images of an object (20) using the camera. For example, the camera can be installed at a location advantageous for capturing images during flight of the aircraft (10) (e.g., an area not obscured by a propeller, etc., at the front or lower front of the aircraft).

[0048] For example, the aircraft (10) can fly using a global navigation satellite system (GNSS) and / or an inertial navigation system (INS).

[0049] For example, the aircraft (10) can transmit and receive data with a server (30), a controller (40), and / or a station (50). In addition, the controller (40) and the server (30), the server (30) and the station (50), and the station (50) and the controller (40) can transmit and receive data with each other.

[0050] Here, the data may include data required to control the flight of the aircraft (10), data on a flight image of the aircraft (10), data on an image taken of an object (20) by the aircraft (10), etc.

[0051] A flight image represents the field of view of an aircraft (10) when the aircraft (10) is flying. For example, the flight image may be a dynamic image acquired in real time, but is not limited thereto.

[0052] The image captured by the aircraft (10) of the target object (20) refers to an image captured by the aircraft (10) while flying around the target object (20). At this time, the image captured by the aircraft (10) of the target object (20) can be used as an image to check for defects in the target object (20). In this case, the image captured by the aircraft (10) of the target object (20) can be referred to as an inspection image.

[0053] For example, the flight image may be an image with a relatively low resolution (or a low Ground Sampling Distance (GSD)) compared to the inspection image, and the inspection image may be an image with a relatively high resolution (or a high GSD) compared to the flight image. Meanwhile, depending on the type of aircraft (10), the cameras that generate the flight image and the inspection image may be the same or different.

[0054] A user can control the aircraft (10) using the interface of the controller (40). For example, the controller (40) can generate a control signal based on user input received through the interface and transmit the control signal to the aircraft (10). The controller (40) can transmit the control signal to the aircraft (10) via wireless communication. The control signal may be a signal that controls the flight, attitude, navigation, etc. of the aircraft.

[0055] The aircraft (10) can control the motor to rotate the propeller according to the control signal received from the controller (40). The aircraft (10) can move, rotate, etc. by changing the speed and / or attitude, etc. by the rotation of the propeller. Here, the attitude of the aircraft can be expressed as pitch (Y), roll (X), yaw (Z), etc. In addition, the aircraft can perform photographing of the target object (20), etc. according to the control signal received from the controller (40).

[0056] The controller (40) may further include a display device, and the user may check the flight image and / or inspection image of the aircraft (10) through the display device.

[0057] The controller (40) may be a device on which an application for controlling an aircraft (10) is installed. For example, the device on which the application is installed may be a variety of portable devices such as a smartphone, tablet, smart pad, laptop, or wearable device.

[0058] The server (30) or station (50) can control the aircraft (10) by directly transmitting a control signal to the aircraft. In addition, the aircraft (10) can transmit a flight image and / or an inspection image to the server (30), controller (40), or station (50).

[0059] The aircraft (10), server (30), controller (40), and station (50) can each analyze the inspection image. For example, the analysis of the inspection image may be to determine whether there is a defect in the target object (20) in the inspection image through an algorithm such as machine learning or deep learning. The aircraft (10) may directly determine whether there is a defect in the target object (20), or may transmit the inspection image to the server (30), controller (40), or station (50). The server (30), controller (40), or station (50) may analyze the inspection image received from the aircraft (10) to determine whether there is a defect in the target object (20).

[0060]

[0061] Figure 3 is a schematic diagram illustrating an example of an aircraft according to one embodiment.

[0062] Referring to FIG. 3, the aircraft (10) may include a sensor (110), a camera (120), a memory (130), a driving device (140), a communication device (150), and a processor (160). However, the components of the aircraft (10) are not limited to those illustrated in FIG. 3. In other words, the aircraft (10) may include at least one more component in addition to the components illustrated in FIG. 3, or at least one of the components illustrated in FIG. 3 may be excluded.

[0063] The sensor (110) detects various information necessary for the operation of the aircraft (10) (e.g., flight, photography, etc.), such as the aircraft (10) itself, the surrounding environment of the aircraft (10), identification of the target (20), and confirmation of the distance between the aircraft (10) and the target (20). The sensor (110) may include, but is not limited to, a gyro sensor, a barometer, an ultrasonic sensor, a magnetic sensor, an acceleration sensor, a proximity sensor, a lidar, a radar, and / or a GPS sensor.

[0064] For example, a gyro sensor and / or an acceleration sensor can measure the three-axis angular velocity of the aircraft (10). A barometer can measure pressure changes and / or air pressure in the atmosphere surrounding the aircraft (10). An ultrasonic sensor can measure the distance between the aircraft (10) and the ground or an object (20). A magnetic sensor is a type of terrestrial magnetism sensor (compass sensor) and can detect geomagnetic information.

[0065] The proximity sensor can measure the proximity state of the object (20) to the aircraft (10) and the distance between the aircraft (10) and the object (20), and can include an ultrasonic sensor that can measure the distance to the object (20) from a signal reflected from the object (20) by outputting ultrasonic waves. The GPS sensor can calculate the current coordinates (x, y, z) of the aircraft (10) using GPS signals.

[0066] The sensor (110) may include an attitude and heading reference system (AHSR). For example, the attitude and heading reference system may include an inertial sensor or an inertial measurement unit (IMU). For example, the attitude and heading reference system may include a gyro sensor, an acceleration sensor, and a magnetic sensor, and fuse the sensor values ​​to obtain the attitude value of the aircraft (10). can be printed. Here, the detailed value can be an angle based on three-dimensional coordinates (x-axis coordinate, y-axis coordinate, z-axis coordinate) according to GPS coordinates.

[0067] The camera (120) can capture images of the object (20) according to instructions from the processor (160). For example, the aircraft (10) may include at least one camera, and may include a low-resolution camera and / or a high-resolution camera. The camera (120) may be coupled to a gimbal capable of adjusting the angle. Accordingly, the camera (120) may have its shooting angle adjusted by the gimbal.

[0068] The memory (130) may include any non-transitory computer-readable recording medium. As an example, the memory (130) may include a non-permanent mass storage device such as a random access memory (RAM), a read only memory (ROM), a disk drive, a solid state drive (SSD), a flash memory, etc. As another example, the non-permanent mass storage device such as a ROM, an SSD, a flash memory, a disk drive, etc. may be a separate permanent storage device distinct from the memory. In addition, the memory (130) may store an operating system (OS) and at least one program code.

[0069] These software components may be loaded from a computer-readable recording medium separate from the memory (130). This separate computer-readable recording medium may be a recording medium that can be directly connected to a computer, and may include, for example, a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. Alternatively, the software components may be loaded into the memory (130) via a communication device (150) other than a computer-readable recording medium. For example, at least one program may be loaded into the memory (130) based on a computer program that is installed by files provided by developers or a file distribution system that distributes installation files of applications via the communication device (150).

[0070] The memory (130) may store commands, information, and / or data related to the operations of each component included in the aircraft (10). For example, the memory (130) may store instructions that, when executed, enable the processor (160) to perform various operations described in this document. As another example, the memory (130) may store various algorithms or models that may be used when the aircraft (10) performs flight, inspection, etc.

[0071] The driving device (140) controls the driving of the motor at a speed and direction according to instructions from the processor (160), and accordingly, the rotational speed and direction of the propeller connected to the motor can be controlled. For example, the driving device (140) may include a motor and a propeller.

[0072] The communication device (150) performs data communication between the aircraft (10) and an external device. For example, the communication device (150) may communicate with the controller (40), server (30), and / or station (50) using various communication methods such as infrared communication, RF (Radio Frequency) communication, Wi-Fi communication, ZigBee communication, Bluetooth communication, laser communication, UWB (Ultra-Wideband) communication, LTE, 5G, 6G, and wireless LAN. However, the communication method employed in the communication device (150) is not limited to the above-described method.

[0073] The processor (160) can process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Here, the instructions can be provided from memory (130) or an external device (e.g., a server (30), a controller (40), a station (50), etc.).

[0074] At this time, the processor (160) may be operatively connected to the memory (130) to perform the overall function of the aircraft (10). In addition, the processor (160) may generally control the operation of other components included in the aircraft (10).

[0075] The processor (160) can obtain at least one static image representing at least a portion of the object (20) captured by the camera (120) of the aircraft (10). Here, the static image means an image captured of the exterior and / or interior of the object (20). In addition, the processor (160) can generate at least one of information about the position of the aircraft (10) at the time when the at least one static image is captured, information about the distance between the aircraft (10) and the object (20), information about the angle of view of the camera (120), and information about the direction in which the camera (120) captures the at least one static image (i.e., the capturing angle of the camera (120).

[0076] At this time, the functions performed by each module included in the processor (160) may be performed by one processor or may be performed by separate processors. The processor (160) may execute calculations or data processing related to control and / or communication of at least one other component of the aircraft (10). In addition, the processor (160) may be implemented as an array of a plurality of logic gates, or may be implemented as a combination of a general-purpose microprocessor and a memory storing a program that can be executed in the microprocessor. For example, the processor (160) may include a general-purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), an artificial intelligence processing unit (NPU) microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, etc. In some environments, the processor (160) may include an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc. For example, the processor (160) may refer to a combination of processing devices, such as a combination of a digital signal processor (DSP) and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors coupled with a digital signal processor (DSP) core, or any other such combination of configurations.

[0077] The processor (160) can determine the position of the aircraft (10).

[0078] The processor (160) can create a meta image using a captured image, and then determine the location of the aircraft (10) through pattern matching of the meta map and objects included in the meta image. The meta map can be created using a map on a server (30) or the like, and stored in the memory (130) of the aircraft (10). The captured image can be an image captured using a camera (120).

[0079] For example, the processor (160) can determine the location of the meta map corresponding to the center point of the meta image as the location of the aircraft (10). The specific process by which the processor (160) determines the location of the aircraft (10) will be described later.

[0080] Meanwhile, all or part of the operations of the processor (160) described above with reference to FIG. 3 may be implemented by a separate device. For example, the separate device may be an aircraft (10), a server (30), a controller (40), a station (50), or another external device. The aircraft (10) may capture static and dynamic images, transmit the captured images to a separate device, and the results of the operations performed by the separate device may be transmitted to the aircraft (10).

[0081] Figure 4 is a flowchart of a method for determining the position of an aircraft according to one embodiment of the present invention.

[0082] In step 410, the camera (120) captures the surroundings of the aircraft (10) to generate a captured image. While the aircraft (10) is flying, the camera (120) captures the surroundings of the aircraft (10). The captured image includes objects. In one example, the captured image used to determine the position of the aircraft (10) may be an image captured directly below the aircraft (10), and the position of the aircraft (10) may be the center point of the captured image.

[0083] In addition, the captured image may be an image captured from an angle other than directly downward. In this case, when the processor (160) determines the position of the aircraft (10) using an image captured from an angle other than directly downward, the processor (160) can calculate the relative position of the captured image and the aircraft (10) based on the angle of the camera (120) and the attitude of the aircraft (10), and the position of the aircraft (10) can be determined as the position calculated from the captured image.

[0084] In step 420, the processor (160) identifies an object included in the captured image and generates a meta-image. The processor (160) can identify an object included in the captured image through deep learning and can express the object as various shapes. For example, the processor (160) can display the object as a dot. The dot representing the object can be displayed at the center of the object in the meta-image. Therefore, the meta-image is an image in which the object is displayed as a dot. The following description will be based on a meta-image and meta-map in which the object is displayed as a dot, but the object can be displayed in various shapes.

[0085] At step 430, the processor (160) determines the location of the aircraft (10) by matching the pattern of shapes included in the meta map and the meta image. Since the processor (160) can calculate the location of the aircraft (10) from the meta image, if an area where the meta image matches the meta map is found, the location of the aircraft (10) can be determined from the meta map.

[0086] A metamap is created using a map. Objects included in the map are displayed as shapes on the metamap. Objects included in the metamap can also be displayed as points, just like objects in the metaimage. The metamap can be created in advance before the aircraft (10) takes flight and stored in the aircraft (10). Storing the map in the memory (130) requires a large amount of storage space. However, converting the map into a metamap and storing the metamap in the memory (130) allows the metamap to be stored in a small amount of storage space.

[0087] The meta map contains latitude and longitude information. Therefore, by determining the location of the aircraft (10) in the meta map, the latitude and longitude of the aircraft (10) can be known.

[0088] The processor (160) can calculate the altitude of the aircraft (10) using the relative distances of the points and the information of the camera (120). The meta map contains accumulated information. Therefore, the processor (160) can calculate the altitude of the aircraft (10) by comparing the relative distance between two points based on the position of the aircraft (10) in the meta image and the relative distance between two points based on the position of the aircraft (10) in the meta map.

[0089] The processor (160) determines the location of the aircraft (10) in the meta-image, and determines where the location of the aircraft (10) in the meta-image is in the meta-map. For example, if the meta-image is an image taken directly below the aircraft (10), the location of the aircraft (10) in the meta-image may be the center point of the meta-image. The processor (160) may determine the location in the meta-map corresponding to the center point of the meta-image as the location of the aircraft (10).

[0090] The processor (160) can determine the location of the aircraft (10) in the meta map by pattern matching of points included in the meta image and the meta map. The meta image and the meta map include objects represented by points, and the processor (160) can perform pattern matching of the meta image and the meta map by utilizing the relative positional relationship of the points.

[0091] The processor (160) can perform pattern matching using the relative distance and relative angle between points. When the processor (160) uses the relative distance and relative angle, the position of the aircraft (10) can be determined without being affected by the size and / or rotation of the meta-image and meta-map. Pattern matching may mean finding points in the meta-map that have a positional relationship most similar to the positional relationship between points calculated based on the center point of the meta-image.

[0092] Relative distance represents the ratio of the distance from the origin to the points, and relative angle represents the angular difference between points with respect to the origin. In the meta-image, the origin can be the location of the aircraft (10), and in the meta-map, the origin can be a center candidate. Relative distance and relative angle are described in more detail in Fig. 9.

[0093] The processor (160) calculates the relative distance and relative angle of points based on the position of the aircraft (10) in the meta-image. If the meta-image is generated from an image taken directly below the aircraft (10), the processor (160) can calculate the relative distance and relative angle of points based on the center point of the meta-image. In other words, the processor (160) can set the center point as the origin of the polar coordinate system and calculate the polar coordinates of the surrounding points.

[0094] The processor (160) calculates the relative distances and relative angles of points based on the center candidate in the meta map. Since the location of the aircraft (10) has not yet been determined in the meta map, the processor (160) first determines the center candidate and calculates the relative distances and relative angles of points based on the center candidate as the origin. The method for determining the center candidate is described in detail with reference to FIG. 10.

[0095] The processor (160) determines whether points included in the meta-image and meta-map match. Matching points may mean that the error in the positional relationship between two points (points in the meta-image and points in the meta-map) is less than a threshold value. The processor (160) may calculate the positional relationship of other points based on a reference point. The processor (160) may set one point in the meta-image (or meta-map) as a reference point and calculate the relative distance and relative angle between the reference point and other points.

[0096] The processor (160) counts the number of points matched to each center candidate in the metamap. For example, if there are 10 center candidates in the metamap, the processor (160) can count the number of points matched to each center candidate. The more points matched to a center candidate, the higher the probability that it will be determined as the location of the aircraft (10).

[0097] The processor (160) calculates the matching error for each center candidate. The processor (160) may calculate the standard deviation of the errors of the matched points as the matching error. For example, if the number of matched points for the first center candidate is five, the matching error for the first center candidate may be the standard deviation of the errors of the five points. The fact that there are five matched points indicates that the errors of each point are below a threshold value.

[0098] The error of a single point can be calculated as in Equation 1, but the error can be calculated using various mathematical formulas.

[0099]

[0100] is the error of the kth point

[0101] is the distance from the center point to the kth point in the meta image

[0102] is the distance from the center point to the i-th point (reference point) in the meta image

[0103] is the distance from the center candidate to the kth point in the meta map.

[0104] is the distance from the center candidate to the i-th point (reference point) in the meta map.

[0105] is the angle in polar coordinates of the kth point in the meta-image

[0106] is the angle in polar coordinates of the ith point (reference point) in the meta image

[0107] is the angle in polar coordinates of the kth point in the meta map

[0108] is the angle in polar coordinates of the ith point (reference point) in the meta map

[0109]

[0110] The processor (160) can calculate an error between one point of the meta map and k points of the meta image, and determine whether the error is smaller than a threshold value to determine whether the point of the meta map matches the point of the meta image. The processor (160) can repeat the process of calculating an error between each of the k points of the meta image and all points (or some points) of the region of interest of the meta map.

[0111] The processor (160) may determine the location of the aircraft (10) using the number of matched points and / or the matching error. For example, the processor (160) may determine a center candidate whose number of matched points exceeds a threshold as the location of the aircraft (10). Alternatively, the processor (160) may determine a center candidate with the largest number of matched points within the region of interest as the location of the aircraft (10).

[0112] For example, the processor (160) may determine the center candidate with the largest number of matched points within a given time period as the location of the aircraft (10). Alternatively, the processor (160) may determine the center candidate with the smallest matching error within a given time period as the location of the aircraft (10). Alternatively, the processor (160) may determine the center candidate with a number of matched points greater than a threshold value and a matching error less than a threshold value as the location of the aircraft (10).

[0113]

[0114] FIG. 5 is a drawing for explaining a process for determining the position of an aircraft according to an embodiment of the present invention.

[0115] The processor (160) identifies objects included in the captured image and generates a meta-image by marking the identified objects with dots. Seven objects were identified in the captured image, and the processor (160) generates a meta-image by marking the seven objects with seven dots. The processor (160) can generate a meta-image by marking a dot at the center of the object. In the meta-image, the center point (MI_C) indicates the location of the aircraft (10). The center point is marked with a cross.

[0116] The metamap (MM) is generated from the map image (MAP). Since the map image (MAP) contains 15 objects, the metamap also displays 15 points.

[0117] The processor (160) can determine the location of the aircraft (10) by matching the points of the meta image with the points of the meta map, and the location of the aircraft (10) determined in the meta map is indicated by a cross (MM_C).

[0118]

[0119] FIG. 6 is a diagram for explaining a process of generating a meta image according to an embodiment of the present invention.

[0120] Referring to FIGS. 5 and 6, the processor (160) can generate a meta-image from a captured image using an object recognition model.

[0121] At this time, the object recognition model can be stored in advance in the memory (130) within the aircraft (10) as illustrated in Fig. 6. When a captured image is input, the object recognition model can output a meta image based on the captured image.

[0122] For example, an object recognition model can identify objects contained in a captured image and generate a meta-image by representing each identified object in the form of a predetermined shape. For example, the object recognition model can generate a meta-image such that the shape is positioned at the center point of the identified object.

[0123] An object may refer to an object whose shape does not change significantly over time. For example, objects may include buildings, playgrounds, parks, green spaces, intersections, rivers, farmland, mountains, etc., but embodiments of the present invention are not limited thereto. In this case, shapes may include circles, polygons (e.g., triangles, squares, pentagons, hexagons, octagons, etc.), dots, etc., but embodiments of the present invention are not limited thereto. For the convenience of the following description, it will be explained assuming that the object recognition model expresses objects in the form of dots, as illustrated in FIG. 5.

[0124] At this time, the object recognition model can perform labeling for each dot when expressing each object included in the captured image in the form of a dot. That is, the object recognition model can determine the type of each object (e.g., building, playground, park, green space, intersection, river, farmland, mountain) and label the determined type of dot for the converted object.

[0125] Meanwhile, object recognition models can perform the aforementioned operations using deep learning technology. In this case, object recognition models utilizing deep learning technology can be trained based on machine learning. To explain in more detail, deep learning, a type of machine learning, involves learning at multiple levels, deepening the data base. In other words, deep learning refers to a set of machine learning algorithms that extract core data from multiple data sets by increasing the level.

[0126] At this time, object recognition models can utilize various well-known deep learning structures. For example, object recognition models can utilize structures such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), deep belief networks (DBNs), graph neural networks (GNNs), generative adversarial networks (GANs), transformers, autoencoders, and long short-term memory (LSTMs).

[0127] Specifically, a Convolutional Neural Network (CNN) is a model that mimics the function of the human brain, based on the assumption that when recognizing an object, humans extract its basic features, then perform complex computations in the brain to recognize the object based on the results. CNNs can include, but are not limited to, well-known structures such as LeNet, AlexNet, VGGNet, GoogleNet, and ResNet.

[0128] RNN (Recurrent Neural Network) is widely used in natural language processing, etc., and is an effective structure for processing time-series data that changes over time. It can construct an artificial neural network structure by stacking layers at each moment.

[0129] A DBN (Deep Belief Network) is a deep learning structure constructed by stacking multiple layers of Restricted Boltzman Machines (RBMs), a deep learning technique. By repeatedly training RBMs (Restricted Boltzman Machines), a certain number of layers can be created, creating a DBN (Deep Belief Network) with that number of layers.

[0130] GNN (Graphic Neural Network, hereinafter referred to as GNN) represents an artificial neural network structure implemented in a way that derives similarity and feature points between modeling data by using modeling data modeled based on data mapped between specific parameters.

[0131] A Generative Adversarial Network (GAN) is an artificial neural network structure that uses a generative neural network and a discriminative neural network to generate new data in a similar form to the input data. GANs may include the well-known DCGAN (Deep Convolutional GAN), CGAN (Conditional GAN), WGAN (Wasserstein GAN), StyleGAN (Style-Based GAN), CycleGAN, etc., but embodiments of the present invention are not limited thereto.

[0132] Transformer is an artificial neural network with an attention-based encoder-decoder structure that can understand the overall meaning between input and output sequences. Transformer uses the attention mechanism to ensure that all elements of the input sequence influence the output sequence, allowing both the encoder and decoder to consider the entire sequence. Transformer can use natural language, time-series data, and even patched images as input.

[0133] An autoencoder is a deep learning architecture that extracts and reconstructs data features. Typically, an autoencoder comprises an encoder, which compresses input values, and a decoder, which restores the compressed data. The encoder transforms the input values ​​into a low-dimensional latent representation, and the decoder reconstructs the latent representation to the same dimensionality as the input values. Each encoder and decoder can be configured as a multilayer perceptron (MLP). When training an autoencoder, input data is input, and weights and biases are trained to minimize the difference between the output and the input values. This trained autoencoder can effectively extract input data features and reconstruct noisy input data. Autoencoders are primarily used in fields such as data compression, dimensionality reduction, noise removal, and data generation, and can also be utilized in areas such as image recognition, natural language processing, and speech recognition.

[0134] Meanwhile, the artificial neural network training of the object recognition model can be achieved by adjusting the weights of the connections between nodes (and adjusting the bias value if necessary) to produce the desired output for a given input. Furthermore, the artificial neural network can continuously update the weight values ​​through learning. Furthermore, methods such as backpropagation can be used for the training of the artificial neural network. At this time, machine learning methods for the artificial neural network can include unsupervised learning, semi-supervised learning, and supervised learning. Furthermore, the object recognition model (ORM) can be controlled to automatically update the artificial neural network structure for outputting post-training analysis data according to settings.

[0135] Hereinafter, the structure of an object recognition model (ORM) according to an embodiment of the present invention will be described with reference to FIG. 7.

[0136] FIG. 7 is a diagram for explaining the structure of an object recognition model according to an embodiment of the present invention.

[0137] Referring to FIGS. 6 and 7, the object recognition model includes an input layer (input) that uses a photographed image as an input node, an output layer (Output) that uses a meta image as an output node, and M hidden layers positioned between the input layer and the output layer.

[0138] Here, weights can be assigned to the edges connecting the nodes of each layer. These weights or the presence or absence of edges can be added, removed, or updated during the learning process. Therefore, the weights of the nodes and edges between the k input nodes and i output nodes can be updated during the learning process.

[0139] Before an object recognition model begins training, all nodes and edges can be set to initial values. However, as information accumulates, the weights of nodes and edges change. This process allows for a matching between the parameters input as learning factors (maps and captured images) and the values ​​assigned to output nodes (metamaps and metaimages).

[0140] Additionally, when using a cloud server, the object recognition model (ORM) can receive and process a large number of parameters. Therefore, the ORM can learn based on massive amounts of data.

[0141] The weights of the nodes and edges between the input and output nodes that constitute an object recognition model (ORM) can be updated through the ORM's learning process. Furthermore, the parameters input or output from an ORM can be expanded to include various data, including the user's virtual face, time intervals between viewpoints, information on performed procedures, and facial change prediction data.

[0142]

[0143] Figures 8 to 10 are diagrams illustrating a method for determining a center candidate in a meta map. The processor (160) can select any two points in the meta image and determine the location of the center candidate on the meta map using the positional relationship between the two selected points.

[0144] FIG. 8 is a diagram illustrating calculating relative distances and relative angles for reference points (RP1, RP2) in a meta-image. The processor (160) can select any two points from among the points included in the meta-image, and the two selected points can be indicated as reference points (RP1, RP2).

[0145] The processor (160) can determine the relative distance and relative angle of the reference points (RP1, RP2) with the center point (MI_C) as the origin. The center point is the center of the meta-image, and in the case of FIG. 8, it indicates the position of the aircraft (10) in the meta-image. If the position of the aircraft (10) is not the center, the processor (160) calculates the relative distance and relative angle of the reference points (RP1, RP2) with the position of the aircraft (10) as the origin. ri is the distance from the center point to the reference point (RP1), and rj is the distance from the center point to the reference point (RP2). Hereinafter, a case where the position of the aircraft (10) is the center point will be described as an example.

[0146] Relative distance ( ) can be expressed as the ratio of the distance (rj) from the center point to the reference point (RP2) to the distance (ri) from the center point to the reference point (RP1). When calculating the relative distance in a meta-image, the distance (ri) can be the standard. That is, the relative distance for the k-th point can be calculated as the distance (rk) from the center point to the k-th point to the distance (ri) from the center point to the reference point (RP1).

[0147] Relative angle ( ) is the angle formed by the center point, the reference point (RP1) and the reference point (RP2), and the relative angle can be determined in a clockwise or counterclockwise direction. When the processor (160) determines the relative angle in the clockwise or counterclockwise direction in the meta image, it also determines the relative angle in the same direction when determining the relative angle in the meta map. In FIG. 8, the relative angle ( )Is It is indicated as . When calculating the relative angle in the meta image, the reference point (RP1) can be used. That is, the relative angle for the k-th point can be the angle formed by the center point, the reference point (RP1), and the k-th point.

[0148]

[0149] Figure 9 is a diagram for explaining how a processor sets a region of interest and selects candidate points in the region of interest.

[0150] The meta map illustrated in FIG. 9 represents a portion of the entire meta map, and the entire meta map may include the entire operational area or all areas in which the aircraft (10) can fly.

[0151] Candidate points (CP1, CP2) are used to determine center candidates in the metamap. The processor (160) can select candidate points as combinations (2-permutations) of points located within the region of interest (ROI). For example, if a total of 7 points are located within the region of interest in FIG. 9, the processor (160) can select candidate points with 7x6=42 combinations.

[0152] The processor (160) can determine candidate points (CP1, CP2) based on the reference points (RP1, RP2) selected from the meta-image. The processor (160) can select candidate points (CP1, CP2) having the same labeling result as the labeling result of the reference points (RP1, RP2). Having the same labeling result means that the types of objects are the same. If the labeling results of the reference point (RP1) and the reference point (RP2) are different, the processor (160) selects the candidate point (CP1) having the same labeling result as the reference point (RP1), and selects the candidate point (CP2) having the same labeling result as the reference point (RP2).

[0153] The processor (160) determines an area of ​​interest (ROI) in the meta map. For example, the processor (160) may determine the area of ​​interest (ROI) based on the most recent location of the aircraft (10). In other words, the processor (160) may determine an area within a predetermined distance from the center of the most recent location of the aircraft (10) as the area of ​​interest. The area of ​​interest is set as a portion of the meta map and is an area where the aircraft (10) is expected to be located.

[0154] In Fig. 9, the region of interest is indicated by a circle, but the region of interest can be determined as a circle, a polygon, etc. Additionally, it can be determined as an area including a fixed number of points. For example, the processor (160) can determine the region of interest as a circle with the recent position of the aircraft (10) as its origin, and determine the region of interest so that at least N points are included within the region of interest. N is a natural number.

[0155] After determining the region of interest, the processor (160) may change or enlarge the region of interest if it fails to determine the location of the aircraft (10) within the region of interest. For example, the processor (160) may enlarge the region of interest to include more points. Alternatively, the processor (160) may change the region of interest in the direction of movement of the aircraft (10).

[0156] The processor (160) can determine a region of interest (ROI) based on the previous GPS signal measured by the GPS sensor and the moving direction and moving distance of the aircraft (10).

[0157]

[0158] Figure 10 is a drawing for explaining a method for determining the location of a center candidate using candidate points (CP1, CP2).

[0159] The processor (160) determines center candidates and determines one of the center candidates as the location of the aircraft (10). The center candidate is determined based on which candidate points are selected in the area of ​​interest. FIG. 10 illustrates a method for determining a center candidate when candidate points (CP1, CP2) are selected.

[0160] The processor (160) determines the center candidate so that the positional relationship between the center candidate and candidate points (CP1, CP2) and the positional relationship between the center point (MI_C) and reference points (RP1, RP2) are the same.

[0161] The processor (160) performs circling so that the ratio of the relative distance between the center point (MI_C) and the reference points (RP1, RP2) and the radii of the circles (C1, C2) with the candidate points (CP1, CP2) as their origins is the same. Performing circling may mean drawing a circle with each of the candidate points (CP1, CP2) as its origin and changing the lengths of the radii of the two circles. The processor (160) changes the lengths of the radii (Ri, Rj) while maintaining the ratio of the radii (Ri, Rj) of the two circles (C1, C2) to the relative distance between the center point (MI_C) and the reference points (RP1, RP2). The processor (160) changes the lengths of the radii (Ri, Rj) while changing the angle ( , ) is calculated. The angle between the candidate points (CP1, CP2) is calculated. , ) represents the angle formed by the radii (Ri, Rj) with the intersection points (IP1, IP2) as the origin.

[0162] The processor (160) calculates the angle between the candidate points (CP1, CP2) , ) becomes equal to the relative angle between the reference points (RP1, RP2), the circling stops. The relative angle of the candidate points (CP1, CP2) is two angles with the same absolute value but different signs ( , ) is included. The processor (160) includes two relative angles ( , ) among the reference points (RP1, RP2) is determined as a center candidate by having the same sign as the relative angle. If, If the relative angles of these reference points (RP1, RP2) have the same sign, the processor (160) determines the intersection point (IP1) as a center candidate for the candidate points (CP1, CP2).

[0163]

[0164] Figures 11 and 12 are diagrams illustrating a method for determining the location of an aircraft in a metamap according to an embodiment of the present invention. The processor (160) determines whether points in the metaimage and points in the metamap match. The processor (160) can determine the location of the aircraft (10) using the number of matched points and / or the matching error.

[0165] FIG. 11 is a diagram for explaining calculating the relative distance and relative angle of surrounding points with the center point (MI_C) as the origin in a meta-image. As in the example illustrated in FIG. 11, the processor (160) can calculate the polar coordinates of seven surrounding points with the center point (MI_C) as the origin. At this time, the processor (160) can calculate the relative distance and relative angle for the remaining points with respect to the first point. For example, the relative distance between the third point and the center point (MI_C) can be expressed as a relative length with respect to the distance between the first point and the center point (MI_C). In addition, the relative angle of the third point can be expressed as the angle formed by the first point and the third point with the center point (MI_C) as the origin.

[0166]

[0167] Figure 12 is a diagram illustrating the calculation of relative distances and relative angles between the first and second center candidates in the metamap and the surrounding points, with the origin as the center point. While Figure 12 illustrates only two center candidates as an example, there may be as many center candidates as there are combinations of points.

[0168] The processor (160) determines whether the points of the meta map of FIG. 12 match the points of the meta image of FIG. 11. The first, second, eighth, and ninth points of the meta map are points used to determine the first and second center candidates, and therefore can be excluded when calculating the error.

[0169] The processor (160) determines whether the third to seventh points of the meta map for the first center candidate match the third to seventh points of the meta image. In addition, the processor (160) determines whether the tenth to twelfth points of the meta map for the second center candidate match the third to seventh points of the meta image.

[0170] The processor (160) can determine multiple center candidates by varying the combination of candidate points (CP1, CP2) as described above, and can determine one of the multiple center candidates as the location of the aircraft (10). If there is no center candidate that satisfies the conditions in the area of ​​interest, the processor (160) can change the area of ​​interest and repeat the same process.

[0171] The processor (160) calculates a matching error for the first center candidate. The processor (160) calculates an error for each of the third to seventh points of the meta map, and calculates a standard deviation of the errors to calculate the matching error for the first center candidate.

[0172] Hereinafter, a method for calculating an error for a third point of a meta map will be described. The processor (160) calculates a relative distance and a relative angle from a first center candidate to the third point. The processor (160) determines whether a point matching the relative distance and the relative angle to the third point is included in the meta image. The processor (160) determines whether the third point of the meta map matches each of the third to seventh points of the meta image of FIG. 11. For example, the processor (160) can calculate an error using mathematical expression 1 and determines whether each error is smaller than a threshold value. If a point having an error smaller than the threshold value is included in the meta image, the processor (160) determines the point having an error smaller than the threshold value as a point matching the third point.

[0173] The processor (160) determines whether the fourth to seventh points of the meta map match the third to seventh points of the meta image in the same manner. The processor (160) counts the number of points among the third to seventh points of the meta map that match the third to seventh points of the meta image.

[0174] The processor (160) calculates the standard deviation of the errors for the matched points. For example, if the sixth and seventh points of the meta map are respectively matched with points of the meta image, the processor (160) calculates the standard deviation of the errors for the sixth and seventh points.

[0175] The processor (160) can count the number of matching points for the second to Nth center candidates and calculate the matching error in the same manner as for the first center candidate. The processor (160) determines the position of the aircraft (10) using the number of matching points and the matching error for the first and Nth center candidates.

[0176]

[0177] The above description is merely an example of the technical idea of ​​the present embodiment, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of ​​the present embodiment, but rather to explain it, and the scope of the technical idea of ​​the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present embodiment.

Claims

1. A camera that captures images by taking pictures of the surroundings of the aircraft; Metadata that represents objects contained in at least one instruction and map as shapes Memory to store the map; and At least one processor comprising at least one instruction for executing the above But, The above processor, Receive the captured image from the camera, Identifying an object included in the above-mentioned captured image and displaying the object as a shape Create a meta image, An aircraft that determines its location by matching the patterns of shapes included in the above meta map and the above meta image.

2. In paragraph 1, The above processor, An aircraft characterized in that it matches the patterns of shapes of the meta image and shapes of the meta map using the relative distance and relative angle of the shapes.

3. In paragraph 1, The above processor, Calculate the position of the aircraft and the positional relationship between the two reference shapes in the above meta-image, Select any two candidate shapes from the above meta map, and Determine the center candidates that have the same positional relationship as the above reference figures, An aircraft characterized in that the position of the aircraft is determined among the above center candidates.

4. In paragraph 3, The above processor, The size of the radius of two circles with each of the above candidate shapes as the origin While changing the intersection of the two circles, one of the intersections is determined as the center candidate. The ratio of the radii of the above two circles is equal to the relative distance of the above reference figures. go, An aircraft characterized in that the relative angles of the above candidate shapes are the same as the relative angles of the above reference shapes.

5. In paragraph 4, The above processor, Set the area of ​​interest in the above meta map, Determine the center candidates for the combination of shapes located within the above region of interest, An aircraft characterized in that the position of the aircraft is determined among the center candidates by calculating the number of matching shapes and / or matching errors for the center candidates.

6. In paragraph 5, The above processor, An aircraft characterized in that the area of ​​interest is determined using at least one of the most recent position, direction of movement, and distance of movement of the aircraft.

7. In paragraph 1, The objects included in the above photographed image and the objects included in the above map are, An aircraft characterized by including at least one of a building, a playground, a park, a green space, an intersection, a river, farmland, and a mountain.

8. In paragraph 1, The above processor, An aircraft characterized in that the meta-image is generated by displaying an object included in the above-described photographed image as a dot at the center position of the object.

9. In a method for determining the position of an aircraft, A step of photographing the surroundings of the aircraft to create a photographed image; A step of identifying an object included in the above-described captured image and generating a meta-image representing the object as a shape; A step of determining the location of the aircraft by matching the patterns of shapes included in the meta image and meta map, The above meta map is characterized by displaying objects included in the map as shapes. How to.

10. In paragraph 9, The step of determining the position of the above aircraft is: A method characterized in that the shapes of the meta image and the patterns of the shapes of the meta map are matched using the relative distance and relative angle of the shapes.

11. In paragraph 10, The step of determining the position of the above aircraft is: The position of the aircraft and the positional relationship between the two reference figures in the above meta-image Calculate, Select any two candidate shapes from the above meta map, and Determine the center candidates that have the same positional relationship as the above reference figures, A method characterized in that the location of the aircraft is determined among the above center candidates.

12. In paragraph 11, The step of determining the position of the above aircraft is: The size of the radius of the two circles that have each of the above candidate shapes as the origin is changed, and one of the intersection points of the two circles is determined as the center candidate. The ratio of the radii of the above two circles is equal to the relative distance of the above reference figures, A method characterized in that the relative angles of the above candidate shapes are the same as the relative angles of the above reference shapes.

13. In paragraph 12, The step of determining the position of the above aircraft is: Set the area of ​​interest in the above meta map, Determine the center candidates for the combination of shapes located within the above region of interest, A method characterized in that the position of the aircraft is determined among the center candidates by calculating the number of matching shapes and / or matching errors for the center candidates.

14. In paragraph 13, The step of determining the position of the above aircraft is: A method characterized in that the area of ​​interest is determined by using at least one of the most recent position, direction of movement, and distance of movement of the aircraft.

15. In paragraph 9, The step of determining the position of the above aircraft is: A method characterized in that the meta image is generated by marking an object included in the above-mentioned photographed image as a dot at the center position of the object.

16. A computer-readable recording medium recording a program for executing the method of Article 9 on a computer.

Images