Estimation system, estimation method, and estimation program

Abstract

To accurately estimate the position of an aircraft using aerial images and map images. [Solution] The estimation system comprises at least one processor. The at least one processor acquires ground images taken from above by a visual sensor mounted on the aircraft, acquires a target map image from a map database that includes the 3D position of each location within a geographical area used to estimate the position of the aircraft, converts the ground altitude of the aircraft into a virtual focal length expressed in pixels based on the ground sample distance of the target map image, normalizes the ground image to a reference scale which is the scale of the target map image based on the virtual focal length to generate a normalized image, and estimates the position of the aircraft based on the normalized image and the target map image.

Description

[Technical Field]

[0001] One aspect of this disclosure relates to estimation systems, estimation methods, and estimation programs. [Background technology]

[0002] Techniques for estimating the geographical location of an aircraft are known. For example, Patent Document 1 describes an aircraft control system comprising a storage unit that stores structural information including location information indicating the location of structures that intersect with bodies of water, and a location identification unit that identifies the location of an aircraft based on the structural information. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2021-18080 [Overview of the project] [Problems that the invention aims to solve]

[0004] A system is needed to accurately estimate the position of an aircraft using aerial images and map images. [Means for solving the problem]

[0005] An estimation system relating to one aspect of this disclosure comprises at least one processor. The at least one processor acquires ground images taken from above by a vision sensor mounted on the aircraft, acquires a target map image from a map database that includes the 3D position of each location within a geographical area used to estimate the position of the aircraft, converts the ground altitude of the aircraft into a virtual focal length expressed in pixels based on the ground altitude of the aircraft and the ground sample distance of the target map image, normalizes the ground image to a reference scale which is the scale of the target map image based on the virtual focal length to generate a normalized image, and estimates the position of the aircraft based on the normalized image and the target map image.

[0006] In such an aspect, the altitude above ground level of the aircraft that captured the ground image is converted to a virtual focal length expressed in pixel units based on the ground sample distance of the target map image. This conversion enables the ground image, the target map image, and the altitude above ground level to be processed using a common index of pixels, so the process of normalizing the ground image to the scale of the target map image can be executed accurately. By using the normalized image and the target map image whose scales match each other, the position of the aircraft can be accurately estimated. [Effects of the Invention]

[0007] According to one aspect of the present disclosure, the position of the aircraft can be accurately estimated using the image from the sky and the map image. [Brief Description of the Drawings]

[0008] [Figure 1] It is a diagram showing an example of application of the estimation system. [Figure 2] It is a diagram showing an example of the hardware configuration of the aircraft. [Figure 3] It is a flowchart showing an example of processing executed by the aircraft. [Figure 4] It is a diagram showing an example of processing for correcting a ground image. [Figure 5] It is a diagram for explaining processing of converting the altitude above ground level into a virtual focal length. [Figure 6] It is a flowchart showing an example of processing for normalizing a ground image. [Figure 7] It is a diagram for explaining processing of calculating an incident angle. [Figure 8] It is a diagram for explaining processing of normalizing a pixel distance. [Figure 9] It is a diagram for explaining processing of normalizing a pixel position. [Figure 10] It is a diagram for explaining normalization of a ground image and estimation of the position of an aircraft. [Figure 11] It is a diagram showing another example of application of the estimation system. [Figure 12]This is a diagram illustrating an example of a hardware configuration of a server. [Description of Embodiments]

[0009] Hereinafter, various examples of the present disclosure will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same or equivalent elements are denoted by the same reference numerals, and duplicate descriptions are omitted.

[0010] [System Overview] The estimation system according to the present disclosure is a computer system that estimates the position of a flying object based on a ground image captured from above by a visual sensor mounted on the flying object and a pre-prepared map image. The position of the flying object refers to the geographical location where the flying object is present. The flying object refers to an artificial object capable of moving through the air. The flying object may be a manned aircraft such as an airplane or a helicopter, or an unmanned aircraft such as a drone.

[0011] In order to estimate the position of the flying object based on the ground image and the map image, it is necessary to match the scales between the ground image and the map image. However, the scale of the ground image often differs from the scale of the map image due to various factors such as changes in the flying object's altitude above ground (the vertical distance from the ground surface to the flying object) or attitude, and the characteristics of the visual sensor. The estimation system converts the altitude above ground into a virtual focal length in pixel units based on the flying object's altitude above ground and the Ground Sampling Distance (GSD) of the map image. GSD is an index representing how many meters on the ground surface correspond to one pixel of the image. The estimation system normalizes the ground image to the scale of the map image based on the virtual focal length to generate a normalized image. In the present disclosure, the scale of the map image is also referred to as the reference scale. The normalized image is a ground image normalized to the reference scale. The estimation system estimates the position of the flying object based on the normalized image and the map image.

[0012] By converting ground altitude to virtual focal length, ground images, map images, and ground altitude can be processed using a common metric—pixels—allowing for accurate normalization of ground images to the map image scale. Using normalized images and map images with matching scales, the position of an aircraft can be accurately estimated.

[0013] Thus, the estimation system estimates the position of the aircraft using a method different from that of the Global Navigation Satellite System (GNSS).

[0014] In one example, the estimation system is used as an alternative when the position of an object cannot be obtained by GNSS. For example, the estimation system estimates the position of an object when GNSS is unavailable due to various factors such as GNSS equipment failure, decreased GNSS reliability, the effects of disturbances, attacks such as GNSS jamming and GNSS spoofing, or interference or tampering by malicious third parties. A decreased GNSS reliability can occur when the number of satellites whose signals are received by the GNSS receiver is lower than usual. Since the estimation system estimates the position of an object based on ground imagery and map imagery, it can determine the position of an object without using GNSS. In another example, the estimation system may be used independently of GNSS. For example, an object may fly based on the position estimated by the estimation system without using GNSS.

[0015] [System Configuration] Figure 1 shows an example of the application of the estimation system. In this example, the estimation system is applied to an aircraft 10 that flies automatically or autonomously. The aircraft 10 can also be described as an example of applying the estimation system to an edge device. The aircraft 10 comprises an image acquisition unit 11, a map database 12, a map acquisition unit 13, an altitude calculation unit 14, an altitude conversion unit 15, a normalization unit 16, a position estimation unit 17, and a flight control unit 18 as functional components.

[0016] The image acquisition unit 11 is a functional module that acquires ground images taken from above by a visual sensor. The map database 12 is a functional module that stores map images used to estimate the position of the aircraft 10. The map acquisition unit 13 is a functional module that acquires a target map image from the map database 12 for comparison with the ground image. The target map image is a part of the map image. The altitude calculation unit 14 is a functional module that calculates the altitude of the aircraft 10 above ground. The altitude conversion unit 15 is a functional module that converts that altitude above ground into a virtual focal length. The normalization unit 16 is a functional module that normalizes the ground image to the scale of the target map image, i.e., the reference scale, based on the virtual focal length, and generates a normalized image. The position estimation unit 17 is a functional module that estimates the position of the aircraft 10 based on the normalized image and the target map image. The flight control unit 18 is a functional module that controls the flight of the aircraft 10 based on the estimated position of the aircraft 10.

[0017] The normalization unit 16 includes an incident angle calculation unit 161, a pixel normalization unit 162, and an image normalization unit 163. The incident angle calculation unit 161 is a functional module that calculates the incident angle for each of the multiple pixels that make up the ground image. The pixel normalization unit 162 is a functional module that normalizes the pixel distance of each pixel to a reference scale based on the incident angle. The image normalization unit 163 is a functional module that normalizes the ground image to a reference scale based on the normalized pixel distance. Details of these processes related to the normalization unit 16 will be described later.

[0018] The map image stored in the map database 12 is image data that includes the three-dimensional position of each location within a predetermined area. In one example, the map image is transmitted from a ground computer to the aircraft 10 before takeoff and stored in the map database 12. The three-dimensional position of each location shown in the map image is location information including the latitude, longitude, and elevation of the location. For example, the map image shows the location information of each location by associating a geographic location as location information with each pixel of the map image. The GSD of the map image is set in advance, and therefore the GSD of the target map image is known. The map image may be implemented using a digital elevation model (DEM), which is data that shows the elevation of the ground surface in relation to geographic coordinates (latitude and longitude). The map database 12 pre-stores map images of a geographic area that includes the planned flight area of ​​the aircraft 10. In one example, the map image is composed of a collection of multiple map tiles. A map tile is a unit image having the size of a predetermined geographic area.

[0019] As mentioned above, in this example, the target map image is a part of the map image. The target map image shows a limited geographical area used to estimate the position of the aircraft 10, whereas the map image stored in the map database 12 shows a wider area than that geographical area. The reference scale of the map image is also the reference scale of the target map image.

[0020] Figure 2 shows an example of the hardware configuration of the aircraft 10. For example, the aircraft 10 comprises a processor 101, main memory 102, auxiliary memory 103, communication control device 104, input device 105, output device 106, GNSS device 107, vision sensor 108, inertial measurement unit (IMU) 109, and altimeter 110. The processor 101 is a computing unit that executes the operating system and application programs, and is, for example, a CPU or GPU. The main memory 102 is composed of, for example, ROM and RAM. The auxiliary memory 103 is composed of, for example, a hard disk or flash memory, and generally stores more data than the main memory 102. The communication control device 104 is composed of, for example, a wireless communication module. The input device 105 is composed of, for example, a touch panel or various switches. The output device 106 is composed of, for example, a light, speaker, or touch panel. The GNSS device 107 is a device that receives signals from each of multiple artificial satellites to determine the position of the aircraft 10. As mentioned above, the GNSS device 107 may be omitted. The visual sensor 108 is, for example, a monocular camera, an infrared camera, or a stereo camera. In one example, the visual sensor 108 is mounted on the aircraft 10 so as to face downwards along the vertical direction. The IMU 109 is a device that detects three-dimensional inertial motion (translational and rotational motion) and is composed of an accelerometer and a gyroscope. The IMU 109 outputs the attitude angles of the aircraft 10, including the roll angle, pitch angle, and yaw angle. The altimeter 110 is a device that measures the current altitude of the aircraft 10 above sea level and is, for example, a barometric altimeter, a radio altimeter, or a laser altimeter.

[0021] Each functional module of the aircraft 10 is realized by an estimated program 120 pre-stored in the auxiliary memory 103. Each functional module is realized by loading the estimated program 120 into the processor 101 or main memory 102 and having the processor 101 execute the estimated program 120. The processor 101 operates the communication control device 104, input device 105, output device 106, GNSS device 107, vision sensor 108, IMU 109, and altimeter 110 according to the estimated program 120, and reads and writes data to the main memory 102 or auxiliary memory 103.

[0022] The estimation program 120 may be provided on a non-temporary recording medium such as a CD-ROM, DVD-ROM, or semiconductor memory. Alternatively, the estimation program 120 may be provided via a communication network as a data signal superimposed on a carrier wave.

[0023] [System operation] Referring to Figure 3, we will now describe the process performed by the aircraft 10 as an example of the estimation method relating to this disclosure. Figure 3 is a flowchart showing an example of that process as the process flow S1.

[0024] In step S11, the flight control unit 18 initiates flight control. For example, the flight control unit 18 causes the aircraft 10 to take off to head towards its destination along a flight route that follows a predetermined number of waypoints. In one example, the flight control unit 18 refers to a flight route pre-stored in the auxiliary memory device 103 and flies the aircraft 10 based on its position and the flight route. The flight control unit 18 controls the flight while communicating with a ground station such as an air traffic control room as needed.

[0025] In step S12, the image acquisition unit 11 acquires a ground image. The image acquisition unit 11 controls the visual sensor 108 to take an image of the ground from above. The image acquisition unit 11 acquires the image obtained from the visual sensor 108 as a ground image.

[0026] In step S13, the image acquisition unit 11 corrects the ground image. This correction includes at least one of the following: correction of distortion in the ground image that may occur depending on the type of lens of the visual sensor 108, and correction to match the orientation of the ground image to the orientation of the map image.

[0027] If the visual sensor 108 is equipped with a fisheye lens, a ground image will be obtained in which the subject appears distorted. In this case, the image acquisition unit 11 performs distortion correction on the ground image by calibration to correct the distortion of the ground image. Depending on the type of visual sensor 108 (for example, if a standard lens is used), such distortion correction may be omitted.

[0028] The image acquisition unit 11 can correct the ground image so that its orientation matches that of the map image, based on the orientation of the aircraft 10 when the visual sensor 108 captured the ground image. The map images stored in the map database 12 represent the map along a specific direction. Typically, the map image represents the map with north at the top. On the other hand, the direction corresponding to the top of the ground image depends on the orientation of the aircraft 10 at the time of capture (particularly the yaw angle). Therefore, the image acquisition unit 11 corrects the orientation of the ground image to match that of the map image.

[0029] In one example, the image acquisition unit 11 uses the attitude angles (roll angle, pitch angle, and yaw angle) acquired from the IMU 109 to calculate the orientation of the aircraft 10 when it takes a ground image. The image acquisition unit 11 applies the yaw angle to a two-dimensional rotation matrix with the center point of the ground image as the rotation center, and rotates the ground image so that the upward direction of the ground image corresponds to a specific direction (for example, north). The image acquisition unit 11 may correct the tilt of the ground image caused by the tilt of the aircraft 10 based on at least one of the roll angle and pitch angle.

[0030] If a stabilizer such as a gimbal is attached to the visual sensor 108 and a ground image is obtained in accordance with the orientation of the map image, then correction of the orientation of the ground image may be omitted.

[0031] Figure 4 shows an example of a process for correcting a ground image. In this example, the visual sensor 108 is a fisheye camera, and the image acquisition unit 11 acquires a ground image 201 from the fisheye camera. The image acquisition unit 11 performs distortion correction on the ground image 201 to generate an intermediate corrected image 202 that is similar to an image taken with a standard lens. The image acquisition unit 11 performs orientation correction on the intermediate corrected image 202 to finally generate a corrected ground image 203. In this example, the image acquisition unit 11 rotates the intermediate corrected image 202 (ground image 201), which has north pointing to the right, 90 degrees counterclockwise so that north is at the top, similar to a map image. As a result, a corrected ground image 203 is obtained. In the example in Figure 4, this ground image 203 is used in subsequent processing.

[0032] Returning to Figure 3, in step S14, the map acquisition unit 13 acquires a target map image from the map database 12 based on the past position of the aircraft 10.

[0033] In one example, the map acquisition unit 13 acquires the last position obtained during the control of the aircraft 10 as its past position. If the aircraft 10 has just taken off, the map acquisition unit 13 may acquire the takeoff position indicated by the flight route stored in the auxiliary storage device 103 as its past position. While the aircraft 10 is in flight, the map acquisition unit 13 may acquire the last position of the aircraft 10 estimated by the estimation system as its past position. The map acquisition unit 13 may acquire the past position using a technology other than the estimation system. For example, the map acquisition unit 13 may acquire the last position of the aircraft 10 obtained by the GNSS device 107 as its past position. Alternatively, the map acquisition unit 13 may acquire the position of the aircraft 10 estimated by dead reckoning navigation using the flight route, flight time, speed, control variables, etc., as its past position.

[0034] The map acquisition unit 13 determines a geographical range to be used to estimate the position of the aircraft 10 based on its past position. For example, the map acquisition unit 13 may determine a predetermined range centered on the past position as its geographical range. Alternatively, the map acquisition unit 13 may determine the geographical range based on the past position and the speed and direction of travel of the aircraft 10. The map acquisition unit 13 acquires a partial image from the map images in the map database 12 that corresponds to the determined geographical range as the target map image. For example, the map acquisition unit 13 generates the target map image by stitching together multiple map tiles that correspond to the determined geographical range. As described above, the target map image includes the three-dimensional position of each location within the determined geographical range.

[0035] In step S15, the altitude calculation unit 14 calculates the altitude of the aircraft 10 above ground level. In one example, the altitude calculation unit 14 calculates the altitude above ground level as the difference between the altitude of the aircraft 10 above sea level (the altitude of the aircraft 10 relative to sea level) obtained by the altimeter 110 and the elevation of the reference position indicated by the target map image. In the following example, the altitude calculation unit 14 will be described on the premise that it uses a point located at the center of the target map image as the reference position.

[0036] In step S16, the altitude conversion unit 15 converts the altitude above ground of the aircraft 10 to a virtual focal length based on the ground altitude and the GSD of the target map image.

[0037] Figure 5 is a diagram illustrating the process of converting altitude above ground to virtual focal length. Figure 5 shows the geometric relationship of the image in the physical world (real space) 301 and the corresponding geometric relationship of the image in the image world (pixel space) 302. Figure 5 also shows a reference position 311 on the ground located directly below the visual sensor 108 and an arbitrary point 312 located a distance D (m) from the reference position 311. The distance between the visual sensor 108 and the reference position 311 in the physical world 301 is considered to be the altitude above ground H (m) of the flying object 10. This physical world 301 is then replaced by the image world 302. In the image world 302, the distance between the virtual visual sensor 208 and the reference position 311 is the virtual focal length f outin pixels, and the pixel distance from the reference position 311 to the point 312 is R (in pixels).

[0038] As shown in FIG. 5, the virtual triangle in the physical world 301 and the virtual triangle in the image world 302 are in a similar relationship. From this similarity relationship, D / H=R / f out holds. In order for R (in pixels) to correspond to D (in meters), let g be the GSD of the target map image map expressed in units of m / pixel, then the relationship D=R·g map needs to hold. From these relational expressions, R·g map / H=R / f out is obtained, and the following expression (1) is derived from this expression. f out =H / g map …(1)

[0039] The altitude conversion unit 15 applies the calculated ground altitude H to expression (1) to calculate the virtual focal length f out . For example, when the ground altitude H is 100 meters and the GSD of the target map image is 0.5 m / pixel, the virtual focal length f out is 100 / 0.5=200 pixels.

[0040] Returning to FIG. 3, in step S17, the normalization unit 16 normalizes the ground image to a reference scale based on the virtual focal length, and generates a normalized image. As described above, the ground image to be normalized is the ground image corrected by the image acquisition unit 11. Step S17 will be described in detail with reference to FIG. 6. FIG. 6 is a flowchart showing an example of processing for normalizing a ground image.

[0041] In step S171, the incident angle calculation unit 161 calculates the incident angle for each of a plurality of pixels constituting the ground image, based on the virtual focal length f out .

[0042] Figure 7 is a diagram illustrating the process of calculating the angle of incidence. Figure 7 shows the center point 321 of the ground image 320, the target pixel 322 which is the pixel for which the angle of incidence θ is to be calculated, and the reference point 323 which indicates the position of the virtual vision sensor 208. The reference point 323 corresponds to the position of the vision sensor 108. The coordinates of the center point 321 are (c x ,c y Let's represent this as (u,v) and the coordinates of the target pixel 322 be (u,v). The reference point 323 is located above the center point 321 at a virtual focal length f out Assume they are located at a distance of . The angle of incidence θ is given by the reference point 323 as the vertex, the target pixel 322, the reference point 323, and the center point (c x ,c y The angle between the optical axis of the virtual visual sensor 208 along the vertical direction and the light ray corresponding to the target pixel can also be said to be the angle between the optical axis of the virtual visual sensor 208 along the vertical direction and the light ray corresponding to the target pixel. The pixel distance R, which is the distance from the center point 321 to the target pixel 322, is given by x = uc x and y=vc y Let R = √(x 2 +y 2 It is expressed by ). The unit of the pixel distance R is pixels. Based on these assumptions, the incident angle θ is obtained by the following equation (2). θ = arctan(R / f) out ) …(2)

[0043] The incident angle calculation unit 161 calculates the incident angle θ for each of the multiple pixels (target pixels) that make up the ground image using equation (2).

[0044] Returning to Figure 6, in step S172, the pixel normalization unit 162 normalizes the pixel distance R to a reference scale based on the incident angle θ for each of the multiple pixels that make up the ground image.

[0045] Figure 8 is a diagram illustrating the process of normalizing the pixel distance. Similar to Figure 7, Figure 8 also shows the center point 321, the target pixel 322, and the reference point 323. In Figure 8, the target pixel 322 is the pixel whose pixel distance is to be normalized. The pixel normalization unit 162 uses a pre-set mapping function f based on the projection model of the visual sensor 108 (for example, a projection model specific to the lens of the visual sensor 108) to normalize the pixel distance (the distance from the center point 321 to the target pixel 322). The mapping function f is a function for calculating the normalized pixel distance based on the angle of incidence. The projection model is a mathematical model that defines the rules by which the visual sensor 108 (the lens of the visual sensor 108) projects light rays from three-dimensional space onto a two-dimensional image plane. If the angle of incidence is represented as θ and the normalized pixel distance is represented as r', the pixel normalization unit 162 calculates the pixel distance r' using the following equation (3). As a result, the pixel distance R is normalized to the pixel distance r'. The mapping function f may be expressed by a mathematical formula or represented by a lookup table. r'=f(θ) …(3)

[0046] The pixel normalization unit 162 applies the incident angle θ to the mapping function f for each of the multiple pixels that make up the ground image, normalizing the pixel distance R to a reference scale. As a result, a normalized pixel distance r' is obtained for each individual pixel.

[0047] Returning to Figure 6, in step S173, the image normalization unit 163 normalizes the original position of each of the multiple pixels constituting the ground image to a reference scale based on the normalized pixel distance r', thereby setting the normalized position of the pixel.

[0048] Figure 9 is a diagram illustrating the process of normalizing the position of a pixel. Similar to Figure 7, Figure 9 shows the center point 321 and the target pixel 322 of the ground image 320. In Figure 9, the target pixel 322 is the pixel whose position is to be normalized. The azimuth angle φ indicating the position of the target pixel 322 as seen from the center point 321 is given by x = ucx and y=vc y This is obtained by φ = arctan2(y,x). The image normalization unit 163 calculates the coordinates (u',v'), which are the normalized position of the target pixel 322, using the following equations (4) and (5). Through this normalization, the ground image 320 is transformed into the ground image 320'. This ground image 320' corresponds to the normalized image described later. u'=c x +r'·cosφ …(4) v'=c y +r'·sinφ …(5)

[0049] Equations (4) and (5) show the process of normalizing the original position (u,v) of the target pixel 322 to a reference scale based on the normalized pixel distance r' of the target pixel 322. The image normalization unit 163 sets the normalized position (u',v') of the target pixel 322 through this process.

[0050] The image normalization unit 163 sets the normalized position of each of the multiple pixels (target pixels) that make up the ground image based on equations (4) and (5).

[0051] Returning to Figure 6, in step S174, the image normalization unit 163 generates a coordinate transformation table that shows the correspondence between the original position and the normalized position for each of the multiple pixels that make up the ground image. The coordinate transformation table is a data structure for transforming the coordinates (position) of each pixel in the ground image to the normalized position (coordinates). The coordinate transformation table shows the combination of the original position (u,v) and the normalized position (u',v') for all pixels in the ground image.

[0052] In step S175, the image normalization unit 163 converts the ground image to a reference scale based on the coordinate transformation table to generate a normalized image. Based on the coordinate transformation table, the image normalization unit 163 places each of the multiple pixels constituting the ground image at a normalized position, normalizing the ground image to the reference scale. The image normalization unit 163 obtains the normalized ground image as the normalized image. If gaps occur between the placed pixels, the image normalization unit 163 performs interpolation processing such as bilinear interpolation or bicubic interpolation to fill the gaps with additional pixels.

[0053] In this way, the image normalization unit 163 normalizes the ground image to a reference scale based on the normalized pixel distance of each of the multiple pixels.

[0054] Returning to Figure 3, in step S18, the position estimation unit 17 estimates the position of the aircraft 10 based on the normalized image and the target map image. For example, the position estimation unit 17 estimates the position of the aircraft 10 by performing matching, which involves comparing the two images.

[0055] The position estimation unit 17 may perform feature point matching to identify the correspondence between one or more feature points in the normalized image and one or more feature points in the target map image. In both the normalized image and the target map image, feature points indicate features of terrain or man-made objects. For example, individual feature points indicate various features such as contours, colors, and positional relationships of rivers, coastlines, roads, buildings, etc.

[0056] The position estimation unit 17 may perform feature point matching using a pre-trained model that receives two images and performs predetermined inference. The pre-trained model may be implemented by a graph neural network (GNN), which is a deep learning model designed to process data with a graph structure. For example, the pre-trained model may be implemented by an Attention GNN with an attention mechanism incorporated. The pre-trained model is generated in advance by machine learning using training data and implemented in the aircraft 10.

[0057] The trained model may be generated to perform feature point matching, which identifies the correspondence between one or more feature points in the normalized image and one or more feature points in the target map image. In this case, the position estimation unit 17 inputs both the normalized image and the target map image into the trained model and obtains the result of feature point matching between these two images. The position estimation unit 17 projects the normalized image onto the target map image so that one or more feature points in the normalized image match one or more feature points in the target map image. The position estimation unit 17 estimates the position of the center of the projected normalized image as the position of the flying object 10.

[0058] The trained model may be generated to perform the process of extracting one or more feature points from an image. In this case, the position estimation unit 17 inputs the normalized image into the trained model and extracts one or more feature points from the normalized image. The position estimation unit 17 also inputs the target map image into the trained model and extracts one or more feature points from the target map image. The position estimation unit 17 performs feature point matching between one or more feature points from the normalized image and one or more feature points from the target map image. In this example, the position estimation unit 17 performs feature point matching using a method different from that of the trained model (e.g., nearest neighbor method). The position estimation unit 17 projects the normalized image onto the target map image so that one or more feature points from the normalized image match one or more feature points from the target map image. The position estimation unit 17 estimates the position of the center of the projected normalized image as the position of the flying object 10.

[0059] The trained model may be generated to perform a process of identifying the position corresponding to the center of the normalized image from the target map image. In this case, the position estimation unit 17 inputs both the normalized image and the target map image into the trained model and identifies the position corresponding to the center of the normalized image from the target map image. The position estimation unit 17 estimates the position output from the trained model as the position of the aircraft 10.

[0060] The position estimation unit 17 may perform pattern matching between the normalized image and the target map image without using a trained model, and estimate the position of the aircraft 10 based on the results of the pattern matching.

[0061] Figure 10 is a diagram illustrating the normalization of the ground image and the estimation of the aircraft's position, and corresponds to steps S17 and S18 described above. Figure 10 shows the corrected ground image 203 in the same way as in Figure 4. In this example, the normalization unit 16 normalizes the corrected ground image 203 to generate a normalized image 210. The position estimation unit 17 performs matching between the normalized image 210 and the target map image 220 to estimate the position of the aircraft 10.

[0062] Returning to Figure 3, in step S19, the flight control unit 18 controls the aircraft 10 based on the estimated position. The flight control unit 18 continues to fly the aircraft 10 toward its destination based on the estimated position and flight route, while communicating with the ground station as needed.

[0063] As shown in step S20, flight control using the estimation system is performed until the aircraft 10 lands.

[0064] If the aircraft 10 is in flight (NO in step S20), the process returns to step S12. In the iterative process, the image acquisition unit 11 acquires a new ground image and corrects this ground image (steps S12, S13). The map acquisition unit 13 acquires a new target map image based on past positions (for example, the position estimated in step S18 above) (step S14). The altitude calculation unit 14 calculates the current ground altitude of the aircraft 10 (new ground altitude) (step S15), and the altitude conversion unit 15 converts this ground altitude into a virtual focal length (step S16). The normalization unit 16 normalizes the new ground image to a reference scale based on the virtual focal length and generates a new normalized image (step S17). The position estimation unit 17 estimates the position of the aircraft 10 based on the new normalized image and the new target map image (step S18), and the flight control unit 18 further controls the aircraft 10 based on the estimated position (step S19).

[0065] In response to landing (YES in step S20), the aircraft 10 terminates the processing flow S1.

[0066] [Differentiation] The technology relating to this disclosure has been described in detail above based on various examples. However, this disclosure is not limited to the examples given above. The technology relating to this disclosure can be modified in various ways without departing from its essence.

[0067] (Another system configuration) Figure 11 shows another example of the application of the estimation system. In this example, at least part of the estimation system is applied to a ground server 30 connected to the aircraft 20 via a communication network.

[0068] The aircraft 20 includes an image acquisition unit 21, an altitude acquisition unit 22, a transmission unit 23, and a flight control unit 24 as functional components.

[0069] The image acquisition unit 21 is a functional module similar to the image acquisition unit 11 described above. The altitude acquisition unit 22 is a functional module that acquires the altitude of the aircraft 20 above sea level from an altimeter. The transmission unit 23 is a functional module that transmits flight data obtained from various sensors of the aircraft 20 to the server 30. For example, the flight data includes ground images and altitude above sea level. The flight control unit 24 is a functional module that controls the flight of the aircraft 20 based on the estimated position received from the server 30. The flight control unit 24 corresponds to the flight control unit 18 described above. The hardware components of the aircraft 20 are the same as those of the aircraft 10.

[0070] The server 30 comprises a receiving unit 31, an altitude conversion unit 32, a map database 33, a map acquisition unit 34, a normalization unit 35, and a position estimation unit 36 ​​as functional components. The normalization unit 35 includes an incident angle calculation unit 351, a pixel normalization unit 352, and an image normalization unit 353.

[0071] The receiving unit 31 is a functional module that receives flight data from the aircraft 20. The altitude conversion unit 32 is a functional module that calculates the altitude of the aircraft 20 above ground based on the altitude of the aircraft 20 above sea level and the target map image, and converts that altitude to a virtual focal length. The altitude conversion unit 32 corresponds to the combination of the altitude calculation unit 14 and the altitude conversion unit 15 described above. The map database 33, map acquisition unit 34, normalization unit 35, incident angle calculation unit 351, pixel normalization unit 352, and image normalization unit 353 are functional modules similar to the map database 12, map acquisition unit 13, normalization unit 16, incident angle calculation unit 161, pixel normalization unit 162, and image normalization unit 163 described above. The position estimation unit 36 ​​is a functional module that estimates the position of the aircraft 20 based on the normalized image and the target map image, and transmits the estimated position to the aircraft 20. The position estimation unit 36 ​​corresponds to the position estimation unit 17 described above.

[0072] Figure 12 shows an example of the hardware configuration of server 30. For example, server 30 comprises a processor 381, main memory 382, ​​auxiliary storage 383, communication control device 384, input device 385, and output device 386. The processor 381 is an arithmetic unit that executes the operating system and application programs, and is, for example, a CPU or GPU. The main memory 382 consists of, for example, ROM and RAM. The auxiliary storage 383 consists of, for example, a hard disk or flash memory, and generally stores more data than the main memory 382. The communication control device 384 consists of, for example, a network card or wireless communication module. The input device 385 consists of, for example, a keyboard, mouse, touch panel, etc. The output device 386 consists of, for example, a monitor (touch panel in one example) and a speaker.

[0073] Each functional module of the server 30 is implemented by an estimated program 390 pre-stored in the auxiliary storage device 383. Each functional module is implemented by loading the estimated program 390 into the processor 381 or main memory 382 and having the processor 381 execute the estimated program 390. The processor 381 operates the communication control device 384, input device 385, or output device 386 according to the estimated program 390 to read and write data to the main memory 382 or auxiliary storage device 383.

[0074] The estimation program 390 may be provided on a non-temporary recording medium such as a CD-ROM, DVD-ROM, or semiconductor memory. Alternatively, the estimation program 390 may be provided via a communication network as a data signal superimposed on a carrier wave.

[0075] Server 30 may consist of a single server 30 or multiple servers 30. When multiple servers 30 are used, these servers 30 are connected via a communication network such as the Internet or an intranet to logically construct a single server 30.

[0076] Processing flow S1 can also be realized through the cooperation of the aircraft 20 and the server 30. Processing flow S1 in this modified example will be described with reference again to Figures 3 and 6.

[0077] In step S11, the flight control unit 24 of the aircraft 20 starts flight control.

[0078] In step S12, the image acquisition unit 21 of the aircraft 20 acquires a ground image.

[0079] In step S13, the image acquisition unit 21 corrects the ground image. The transmission unit 23 transmits flight data, including the corrected ground image, to the server 30. The server 30 receives the flight data using the receiving unit 31.

[0080] In step S14, the map acquisition unit 34 of the server 30 acquires a target map image from the map database 33 based on the past position of the aircraft 20.

[0081] In step S15, the altitude conversion unit 32 of the server 30 calculates the altitude of the aircraft 20 above ground level. The altitude acquisition unit 22 of the aircraft 20 acquires the altitude above sea level, and the transmission unit 23 transmits flight data including that altitude above sea level to the server 30. The receiving unit 31 of the server 30 receives the flight data. The altitude conversion unit 32 calculates the difference between that altitude above sea level and the elevation of the reference position indicated by the target map image as the altitude above ground level.

[0082] In step S16, the altitude conversion unit 32 of the server 30 converts its altitude relative to the ground into a virtual focal length.

[0083] In step S17, the normalization unit 35 of the server 30 normalizes the ground image to a reference scale based on its virtual focal length and generates a normalized image. The normalization unit 35 (incident angle calculation unit 351, pixel normalization unit 352, and image normalization unit 353) performs a series of processes shown in Figure 6.

[0084] In step S18, the position estimation unit 36 ​​of the server 30 estimates the position of the aircraft 20 based on the normalized image and the target map image. The position estimation unit 36 ​​transmits the estimated position to the aircraft 20. The flight control unit 24 of the aircraft 20 receives the estimated position.

[0085] In step S19, the flight control unit 24 of the aircraft 20 controls the aircraft 20 based on its estimated position.

[0086] As shown in step S20, flight control using the estimation system is performed until the aircraft 20 lands.

[0087] (Other variations) In the example above, the image normalization units 163 and 353 generate a coordinate transformation table and then normalize the ground image, but the image normalization unit may normalize the ground image without using a coordinate transformation table.

[0088] In the example above, the aircraft 10 or the server 30 calculates the altitude above ground. If the aircraft has a function to directly measure the altitude above ground, the process of calculating the altitude above ground may be omitted.

[0089] In the above example, the map acquisition units 13 and 34 acquire the target map image from the map databases 12 and 33 based on the past position of the aircraft 10. As a variation of this, the map acquisition unit may acquire the entire map image stored in the map database as the target map image. Alternatively, the map acquisition unit may acquire a target map image from the map database that shows the entire flight route of the aircraft.

[0090] In the above example, the estimation system is used for automatic or autonomous flight, but the estimation system may be used for other purposes. For example, the estimation system may display the estimated position of the aircraft on a display device to present the estimated position to users such as pilots, drone operators, or air traffic controllers. Users can then control or monitor the aircraft by referring to the estimate.

[0091] The processing steps for a method executed by at least one processor are not limited to the examples above. For example, some of the steps described above may be omitted, or each step may be performed in a different order. Also, any two or more of the steps described above may be combined, or some of the steps may be modified or deleted. Alternatively, other steps may be performed in addition to each of the steps described above.

[0092] In comparing the relative magnitudes of two numerical values ​​in this disclosure, either of the two criteria, "greater than or equal to" and "greater than," may be used, or either of the two criteria, "less than or equal to" and "less than," may be used.

[0093] In this disclosure, the expression "at least one processor executes a first process, a second process, ... and the nth process," or a corresponding expression, refers to a concept that includes cases where the entity executing the n processes from the first process to the nth process, i.e., the processor, changes along the way. In other words, this expression refers to a concept that includes both cases where all n processes are executed by the same processor and cases where the processor changes at an arbitrary rate for the n processes.

[0094] [Note] As can be seen from the various examples above, this disclosure includes the following aspects: (Note 1) Equipped with at least one processor, The at least one processor, The aircraft acquires ground images taken from above using its onboard visual sensors. A target map image, including the 3D position of each location within the geographical area used to estimate the position of the aforementioned flying object, is obtained from a map database. Based on the altitude of the aircraft above ground and the ground sample distance of the target map image, the altitude above ground is converted into a virtual focal length expressed in pixels. Based on the virtual focal length, the ground image is normalized to a reference scale which is the scale of the target map image to generate a normalized image. The position of the flying object is estimated based on the normalized image and the target map image. Estimation system. (Note 2) The at least one processor, For each of the multiple pixels that make up the aforementioned ground image, Based on the pixel distance from the center point of the ground image to the pixel and the virtual focal length, the angle of incidence formed by the pixel, the reference point corresponding to the position of the visual sensor, and the center point of the ground image is calculated, where the angle of incidence is the angle with the reference point as its vertex. The incident angle is applied to a pre-set mapping function based on the projection model of the visual sensor to normalize the pixel distance to the reference scale. The ground image is normalized to the reference scale based on the normalized pixel distance of each of the plurality of pixels. The estimation system described in Appendix 1. (Note 3) The at least one processor, For each of the plurality of pixels, the original position of the pixel is normalized to the reference scale based on the normalized pixel distance of the pixel, and the normalized position of the pixel is set. The ground image is normalized to the reference scale by placing each of the plurality of pixels at the normalized position. The estimation system described in Appendix 2. (Note 4) The at least one processor, A coordinate transformation table is generated for each of the aforementioned multiple pixels, showing the correspondence between the original position and the normalized position. Based on the coordinate transformation table, each of the plurality of pixels is placed in the normalized position. The estimation system described in Appendix 3. (Note 5) The at least one processor, Based on the orientation of the aircraft when the visual sensor captured the ground image, the ground image is corrected so that the orientation of the ground image is the same as the orientation of the target map image. The corrected ground image is normalized to the reference scale. The estimation system described in any one of the appendices 1 to 4. (Note 6) The aforementioned target map image includes the elevation of each location within the aforementioned geographical area, The at least one processor calculates the difference between the altitude of the aircraft above sea level and the altitude of the reference location indicated by the target map image as the altitude above ground. The estimation system described in any one of the appendices 1 to 5. (Note 7) The aforementioned map database stores map images covering a wider area than the aforementioned geographical area. The at least one processor, The geographical range is determined based on the past position of the aforementioned aircraft, A portion of the aforementioned map image corresponding to the geographical area is acquired as the target map image. The estimation system described in any one of the appendices 1 to 6. (Note 8) An estimation method performed by an estimation system comprising at least one processor, The steps include acquiring ground images taken from above using a visual sensor mounted on the aircraft, The steps include obtaining a target map image from a map database that includes the three-dimensional position of each location within a geographical area used to estimate the position of the aforementioned flying object, The steps include converting the altitude of the aircraft above ground and the ground sample distance of the target map image into a virtual focal length expressed in pixels, The steps include: normalizing the ground image to a reference scale which is the scale of the target map image based on the virtual focal length, thereby generating a normalized image; A step of estimating the position of the flying object based on the normalized image and the target map image, An estimation method that includes [this]. (Note 9) The steps include acquiring ground images taken from above using a visual sensor mounted on the aircraft, The steps include obtaining a target map image from a map database that includes the three-dimensional position of each location within a geographical area used to estimate the position of the aforementioned flying object, The steps include converting the altitude of the aircraft above ground and the ground sample distance of the target map image into a virtual focal length expressed in pixels, The steps include: normalizing the ground image to a reference scale which is the scale of the target map image based on the virtual focal length, thereby generating a normalized image; A step of estimating the position of the flying object based on the normalized image and the target map image, An estimation program that causes a computer to execute a command.

[0095] According to appendices 1, 8, and 9, the altitude of the aircraft that captured the ground image is converted into a virtual focal length expressed in pixels, based on the ground sample distance of the target map image. This conversion makes it possible to process the ground image, target map image, and altitude using pixels as a common indicator, thus enabling accurate normalization of the ground image to the scale of the target map image. By using the normalized image and target map image with matching scales, the position of the aircraft can be accurately estimated.

[0096] According to Appendix 2, for each of the multiple pixels that make up the ground image, the pixel distance is normalized to a reference scale based on the angle of incidence calculated based on the virtual focal length and the mapping function corresponding to the visual sensor. Then, the ground image is normalized to a reference scale based on the normalized individual pixel distances. By separating the calculation of the angle of light rays based on a virtual perspective projection from the use of a mapping function that reflects the projection model of the visual sensor, the ground image can be normalized while adapting to various types of visual sensors.

[0097] According to Appendix 3, the ground image is normalized to a reference scale by placing each pixel in a normalized position based on the normalized pixel distance. By performing normalization based on the reference scale for each pixel, a normalized image can be accurately generated.

[0098] According to Appendix 4, a coordinate transformation table is generated for each pixel showing the correspondence between its original position and its normalized position, and the position of each pixel is normalized based on this coordinate transformation table. Since the normalization of each pixel is managed comprehensively by the coordinate transformation table, the normalization of ground images can be performed efficiently.

[0099] According to Appendix 5, the orientation of the ground image is aligned with the orientation of the target map image, and then the ground image is normalized to a reference scale. This configuration allows for easy position estimation based on the normalized image and the target map image, without being affected by the attitude of the aircraft when the ground image was taken.

[0100] According to Appendix 6, the altitude of the aircraft above ground is calculated based on the aircraft's altitude above sea level and the elevation shown in the target map image. This configuration allows for accurate acquisition of the altitude above ground without incurring special costs for the aircraft's altimeter itself (for example, without introducing an altimeter capable of measuring altitude above ground).

[0101] According to Appendix 7, partial images corresponding to the aircraft's past positions are acquired as the target map image. By using such a limited target map image, the processing load on the processor when estimating the aircraft's position can be reduced. [Explanation of Symbols]

[0102] 10...Flight unit, 11...Image acquisition unit, 12...Map database, 13...Map acquisition unit, 14...Altitude calculation unit, 15...Altitude conversion unit, 16...Normalization unit, 17...Position estimation unit, 18...Flight control unit, 20...Flight unit, 21...Image acquisition unit, 22...Altitude acquisition unit, 23...Transmission unit, 24...Flight control unit, 30...Server, 31...Receiver unit, 32...Altitude conversion unit, 33...Map database, 34...Map acquisition unit, 35...Normalization unit, 36...Position estimation unit, 120...Estimation program, 161...Incident angle calculation unit, 162...Pixel normalization unit, 163...Image normalization unit, 201...Ground image, 202...Intermediate correction image, 203...Corrected ground image, 210...Normalized image, 220...Target map image, 351...Incident angle calculation unit, 352...Pixel normalization unit, 353...Image normalization unit, 390...Estimation program.

Claims

1. Equipped with at least one processor, The at least one processor, The aircraft acquires ground images taken from above using its onboard visual sensors. A target map image, including the three-dimensional position of each location within the geographical area used to estimate the position of the aforementioned flying object, is obtained from a map database. Based on the altitude of the aircraft above ground and the ground sample distance of the target map image, the altitude above ground is converted into a virtual focal length expressed in pixels. For each of the multiple pixels that make up the aforementioned ground image, Based on the pixel distance from the center point of the ground image to the pixel and the virtual focal length, the angle of incidence formed by the pixel, the reference point corresponding to the position of the visual sensor, and the center point of the ground image is calculated, where the angle of incidence is the angle with the reference point as its vertex. The incident angle is applied to a mapping function predetermined based on the projection model of the visual sensor to normalize the pixel distance to a reference scale which is the scale of the target map image. Based on the normalized pixel distance of each of the plurality of pixels, the ground image is normalized to the reference scale to generate a normalized image. The position of the flying object is estimated based on the normalized image and the target map image. Estimation system.

2. The at least one processor, For each of the plurality of pixels, the original position of the pixel is normalized to the reference scale based on the normalized pixel distance of the pixel, and the normalized position of the pixel is set. The ground image is normalized to the reference scale by placing each of the plurality of pixels at the normalized position. The estimation system according to claim 1.

3. The at least one processor, A coordinate transformation table is generated for each of the aforementioned multiple pixels, showing the correspondence between the original position and the normalized position. Based on the coordinate transformation table, each of the plurality of pixels is placed in the normalized position. The estimation system according to claim 2.

4. The at least one processor, Based on the orientation of the aircraft when the visual sensor captured the ground image, the ground image is corrected so that the orientation of the ground image is the same as the orientation of the target map image. The corrected ground image is normalized to the reference scale. The estimation system according to any one of claims 1 to 3.

5. The aforementioned target map image includes the elevation of each location within the aforementioned geographical area, The at least one processor calculates the difference between the altitude of the aircraft above sea level and the altitude of the reference location indicated by the target map image as the altitude above ground. The estimation system according to any one of claims 1 to 3.

6. The aforementioned map database stores map images covering a wider area than the aforementioned geographical area. The at least one processor, The geographical range is determined based on the past position of the aforementioned aircraft, A portion of the aforementioned map image corresponding to the geographical area is acquired as the target map image. The estimation system according to any one of claims 1 to 3.

7. An estimation method performed by an estimation system comprising at least one processor, The steps include acquiring ground images taken from above using a visual sensor mounted on the aircraft, The steps include obtaining a target map image from a map database that includes the three-dimensional position of each location within a geographical area used to estimate the position of the aforementioned flying object, The steps include converting the altitude of the aircraft above ground and the ground sample distance of the target map image into a virtual focal length expressed in pixels, For each of the multiple pixels that make up the aforementioned ground image, Based on the pixel distance from the center point of the ground image to the pixel and the virtual focal length, the angle of incidence formed by the pixel, the reference point corresponding to the position of the visual sensor, and the center point of the ground image is calculated, where the angle of incidence is the angle with the reference point as its vertex. The steps include: applying the incident angle to a pre-set mapping function based on the projection model of the visual sensor to normalize the pixel distance to a reference scale which is the scale of the target map image; The steps include: normalizing the ground image to the reference scale based on the normalized pixel distance of each of the plurality of pixels to generate a normalized image; A step of estimating the position of the flying object based on the normalized image and the target map image, An estimation method that includes [this].

8. The steps include acquiring ground images taken from above using a visual sensor mounted on the aircraft, The steps include obtaining a target map image from a map database that includes the three-dimensional position of each location within a geographical area used to estimate the position of the aforementioned flying object, The steps include converting the altitude of the aircraft above ground and the ground sample distance of the target map image into a virtual focal length expressed in pixels, For each of the multiple pixels that make up the aforementioned ground image, Based on the pixel distance from the center point of the ground image to the pixel and the virtual focal length, the angle of incidence formed by the pixel, the reference point corresponding to the position of the visual sensor, and the center point of the ground image is calculated, where the angle of incidence is the angle with the reference point as its vertex. The steps include: applying the incident angle to a pre-set mapping function based on the projection model of the visual sensor to normalize the pixel distance to a reference scale which is the scale of the target map image; The steps include: normalizing the ground image to the reference scale based on the normalized pixel distance of each of the plurality of pixels to generate a normalized image; A step of estimating the position of the flying object based on the normalized image and the target map image, An estimation program that causes a computer to execute a command.

Images