Estimation system, estimation method, and estimation program

Abstract

Determining the location of an aircraft without using a Global Navigation Satellite System (GNSS). The estimation system includes at least one processor that acquires target ground images obtained from the air by a visual sensor mounted on the air vehicle, acquires one or more target ground maps from a storage device, each map including location information of each location within a predetermined geographical range, and estimates the position of the air vehicle based on the target ground images and the one or more target ground maps.

Description

[Technical Field]

[0001] One aspect of the present disclosure relates to an estimation system, an estimation method, and an estimation program. [Background technology]

[0002] There are known technologies for identifying the geographical position of an aircraft. For example, Patent Document 1 describes an unmanned aircraft equipped with a Global Positioning System (GPS) device as a location information acquisition unit. GPS is a type of Global Navigation Satellite System (GNSS). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-194954 Summary of the Invention [Problem to be solved by the invention]

[0004] There is a need for a mechanism to determine the position of an aircraft without using the Global Navigation Satellite System (GNSS). [Means for solving the problem]

[0005] An estimation system according to one aspect of the present disclosure includes at least one processor that acquires a target ground image obtained from the air by a visual sensor mounted on an air vehicle, acquires one or more target ground maps from a storage device, each map including location information of each location within a predetermined geographical area, and estimates a position of the air vehicle based on the target ground image and the one or more target ground maps.

[0006] In this aspect, the position of the aircraft is estimated using a target ground image acquired from the sky by the aircraft's visual sensor and a target ground map prepared in advance, thereby enabling the position of the aircraft to be identified without using a global navigation satellite system (GNSS). [Effects of the Invention]

[0007] According to one aspect of the present disclosure, the position of an air vehicle can be determined without using a global navigation satellite system (GNSS). [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 10 is a diagram illustrating an example of application of the estimation system. [Figure 2] FIG. 2 is a diagram illustrating an example of the hardware configuration of an aircraft. [Figure 3] 10 is a flowchart illustrating an example of processing executed by an air vehicle. [Figure 4] FIG. 10 is a diagram illustrating an example of estimating the position of an aircraft by feature point matching. [Figure 5] FIG. 10 illustrates another example of application of the estimation system. [Figure 6] FIG. 2 illustrates an example of a hardware configuration of a server. DETAILED DESCRIPTION OF THE INVENTION

[0009] Various examples of the present disclosure will be described in detail below with reference to the accompanying drawings. In the description of the drawings, the same or equivalent elements are designated by the same reference numerals, and redundant description will be omitted.

[0010] [System Overview] The estimation system according to the present disclosure is a computer system that estimates the position of an airborne vehicle using a method different from the Global Navigation Satellite System (GNSS). The position of an airborne vehicle refers to the geographical location of the airborne vehicle. The estimation system estimates the position of the airborne vehicle based on ground images acquired from the sky by a visual sensor mounted on the airborne vehicle and a pre-prepared ground map. An airborne vehicle refers to an artificial object that can move in the air. The airborne vehicle may be a manned aircraft such as an airplane or helicopter, or an unmanned aircraft such as a drone.

[0011] In one example, the estimation system is used as an alternative when the position of an air vehicle cannot be obtained using GNSS. For example, the estimation system estimates the position of an air vehicle when GNSS is unavailable due to various factors, such as GNSS device failure, reduced GNSS reliability, the effects of external disturbances, attacks such as GNSS jamming and GNSS spoofing, and interference or tampering by malicious third parties. Reduced GNSS reliability can occur when a GNSS receiver receives signals from fewer satellites than usual. The estimation system estimates the position of the air vehicle based on ground images and ground maps, allowing the air vehicle to be located without using GNSS.

[0012] [System Configuration] FIG. 1 is a diagram illustrating an example of an application of the estimation system. In this example, the estimation system is applied to an air vehicle 10 that flies automatically or autonomously. The air vehicle 10 can also be considered an example of applying the estimation system to an edge device. The air vehicle 10 includes functional components such as a positioning unit 11, a flight control unit 12, a monitoring unit 13, an image acquisition unit 14, a map database 15, and a position estimation unit 16. The positioning unit 11 is a functional module that determines the position of the air vehicle 10 using GNSS. The flight control unit 12 is a functional module that controls flight based on the position of the air vehicle 10. The monitoring unit 13 is a functional module that monitors the operation of the GNSS. The image acquisition unit 14 is a functional module that acquires ground images (also referred to as "target ground images" in this disclosure) using a visual sensor, which are used to estimate the position of the air vehicle 10. The map database 15 is a functional module that stores one or more ground maps used to estimate the position of the air vehicle 10. The position estimation unit 16 is a functional module that estimates the position of the air vehicle 10 based on the target ground images and the target ground map. The position estimation unit 16 uses a trained model 20 for its estimation process.

[0013] In the present disclosure, the ground map stored in the map database 15 is also referred to as a "target ground map." Typically, the map database 15 stores multiple target ground maps. The target ground map is image data showing a predetermined geographical range. The target ground map includes location information for each of multiple locations located within that geographical range. For example, a geographical location is associated with each pixel of the target ground map as location information, and the location information for each location is embedded in the target ground map. The geographical location is expressed, for example, by latitude and longitude.

[0014] FIG. 2 is a diagram illustrating an example of the hardware configuration of the aircraft 10. For example, the aircraft 10 includes a processor 101, a main memory 102, an auxiliary memory 103, a communication control device 104, an input device 105, an output device 106, a GNSS device 107, a visual sensor 108, an inertial measurement unit (IMU) 109, and an altimeter 110. The processor 101 is a computing device that executes an operating system and application programs, such as a CPU or GPU. The main memory 102 is composed of, for example, ROM and RAM. The auxiliary memory 103 is composed of, for example, a hard disk or flash memory, and generally stores larger amounts of data than the main memory 102. The communication control device 104 is composed of, for example, a wireless communication module. The input device 105 is composed of, for example, a touch panel or various switches. The output device 106 is composed of, for example, a light, a speaker, or a touch panel. The GNSS device 107 is a device that receives signals from multiple satellites to determine the position of the aircraft 10. The visual sensor 108 may be, for example, a monocular camera, an infrared (IR) camera, a stereo camera, an event camera, a scanning light detection and ranging (LiDAR) sensor, or a synthetic aperture radar (SAR) sensor. Correspondingly, the ground image (target ground image) acquired by the visual sensor 108 may be, for example, a color image, a grayscale image, an infrared image, a depth image, point cloud data, or a SAR image. In one example, the visual sensor 108 is mounted on the air vehicle 10 so as to face vertically downward. The IMU 109 is a device that detects three-dimensional inertial motion (translational motion and rotational motion) and is configured using an acceleration sensor and a gyro sensor. The altimeter 110 is a device that measures the altitude of the current location and may be, for example, a barometric altimeter or a radio altimeter.

[0015] Each functional module of the aircraft 10 is realized by an estimation program 120 pre-stored in the auxiliary storage device 103. Each functional module is realized by loading the estimation program 120 into the processor 101 or the main storage device 102 and having the processor 101 execute the estimation program 120. In accordance with the estimation program 120, the processor 101 operates the communication control device 104, the input device 105, the output device 106, the GNSS device 107, the visual sensor 108, the IMU 109, and the altimeter 110, and reads and writes data from and to the main storage device 102 or the auxiliary storage device 103.

[0016] The estimation program 120 may be provided by being recorded on a non-transitory recording medium such as a CD-ROM, a DVD-ROM, a semiconductor memory, etc. Alternatively, the estimation program 120 may be provided via a communication network as a data signal superimposed on a carrier wave.

[0017] [Pre-trained model] The trained model 20 is a computational model that receives ground images and ground maps and performs predetermined inference. The trained model 20 may be realized by a graph neural network (GNN), which is a deep learning model designed to process data with a graph structure. For example, the trained model 20 is realized by an attention GNN that incorporates an attention mechanism. The trained model 20 is generated in advance by machine learning using training data and is implemented in the aircraft 10.

[0018] In one example, the trained model 20 performs feature point matching, as a predetermined inference, to identify correspondences between one or more feature points in a ground image and one or more feature points in a ground map. In both the ground image and the ground map, feature points indicate features of terrain or man-made objects. For example, individual feature points indicate various characteristics, such as the contours, colors, and relative positions of rivers, coastlines, roads, buildings, etc. In this example, the trained model 20 accepts both the ground image and the ground map, performs feature point matching based on the image data, and outputs the execution result. If a specific area shown in the ground image is included in the ground map, the execution result indicates that the feature points in the ground image correspond to the feature points in the ground map. If a specific area shown in the ground image is not included in the ground map, the execution result indicates that the feature points in the ground image do not correspond to the feature points in the ground map. In this example, each data record of the training data includes a ground image, a ground map, and a ground truth for feature point matching. The trained model 20 is generated by supervised learning using the training data. The trained model 20 is robust to changes in lighting conditions, weather, or seasons due to pre-training, and exhibits generalization even in untrained regions or environments. In addition, the trained model 20 can perform stable feature extraction even when the image is degraded or contains occlusions.

[0019] In another example, the trained model 20 executes a process of extracting one or more feature points from an image as a predetermined inference. In this example, the trained model 20 accepts a ground image, extracts one or more feature points from the ground image, and outputs the extracted feature points. The trained model 20 also accepts a ground map, extracts one or more feature points from the ground map, and outputs the extracted feature points. In this way, the trained model 20 extracts one or more feature points from each of the ground image and the ground map. In this example, each data record of the training data includes a ground image or a ground map and a correct answer for the feature points. The trained model 20 is generated by supervised learning using the training data.

[0020] In yet another example, the trained model 20 executes a process of identifying a location corresponding to a ground image from a ground map as a predetermined inference. In this example, the trained model 20 receives both a ground image and a ground map, and identifies a location corresponding to the ground image from the ground map based on the image data. For example, the trained model 20 identifies a location corresponding to the center of the ground image from the ground map. The trained model 20 outputs the identified location. If the geographical range shown in the ground image differs from the geographical range represented on the ground map, the trained model 20 outputs an execution result indicating that the location corresponding to the ground image could not be identified. In this example, each data record of the training data includes a ground image, a ground map, and a correct answer for the location corresponding to the ground image. The trained model 20 is generated by supervised learning using the training data.

[0021] [System Operation] Referring to Figure 3, a process executed by the aircraft 10 will be described as an example of an estimation method according to the present disclosure. Figure 3 is a flowchart showing an example of this process as a process flow S1. From the perspective of the trained model 20, the process flow S1 corresponds to the inference phase (operation phase) of machine learning.

[0022] In step S11, the positioning unit 11 and flight control unit 12 cooperate to start flight control using GNSS. For example, the flight control unit 12 causes the aircraft 10 to take off to head toward a destination along a flight route that follows a predetermined number of waypoints (WP). In response to the takeoff, the positioning unit 11 starts identifying the position using GNSS. In one example, the flight control unit 12 references a flight route pre-stored in the auxiliary storage device 103 and flies the aircraft 10 based on the identified position and the flight route. The flight control unit 12 controls the flight while communicating with a ground station such as a control room as necessary.

[0023] In step S12, the monitoring unit 13 determines whether the GNSS device 107 is operating normally. For example, the monitoring unit 13 obtains information about the GNSS device 107 by checking the state of the GNSS device 107 or obtaining an event signal output from the GNSS device 107. Then, the monitoring unit 13 determines whether the GNSS device 107 is functioning normally based on the information.

[0024] If the GNSS is operating normally (YES in step S12), the process proceeds to step S13. In step S13, the positioning unit 11 and flight control unit 12 cooperate to control the aircraft 10 using the GNSS. The positioning unit 11 identifies its position using the GNSS, and the flight control unit 12 continues to fly the aircraft 10 toward its destination based on the identified position and a predetermined flight route, while communicating with a ground station as necessary. After step S13, the process proceeds to step S18.

[0025] If an abnormality has occurred in the GNSS (NO in step S12), the process proceeds to step S14. Then, from step S14 to step S17, the estimation system performs position estimation. As described above, GNSS abnormalities can occur due to various factors, such as a failure of the GNSS device 107, a decrease in the reliability of the GNSS, the influence of external disturbances, attacks such as GNSS jamming and GNSS spoofing, and interference or tampering by a malicious third party. In such cases, the determination of the position of the aircraft 10 is switched from the GNSS to the estimation system.

[0026] In step S14, the image acquisition unit 14 acquires a target ground image. The image acquisition unit 14 controls the visual sensor 108 to capture an image of the ground from above. The image acquisition unit 14 acquires the image obtained by the visual sensor 108 as the target ground image.

[0027] In step S15, the position estimation unit 16 acquires one or more target ground maps used to estimate the position of the aircraft 10 from the map database 15. The position estimation unit 16 may acquire the position of the aircraft 10 last determined by GNSS as a reference position and select one or more target ground maps from multiple target ground maps in the map database 15 based on this reference position. For example, the position estimation unit 16 may select one target ground map including the reference position, or may select one target ground map including the reference position and one or more target ground maps adjacent to this target ground map. The position estimation unit 16 may select one or more target ground maps corresponding to the flight route. For example, the position estimation unit 16 may select one or more target ground maps estimated to correspond to the current position of the aircraft 10 based on dead reckoning navigation using the flight route, flight time, speed, control amount, etc. Alternatively, the position estimation unit 16 may select all target ground maps in the map database 15. In either case, the position estimation unit 16 can acquire one or more target ground maps through such selection processing.

[0028] In step S16, the position estimation unit 16 estimates the position of the aircraft 10 based on the target ground image and one or more target ground maps. For each of the one or more target ground maps, the position estimation unit 16 inputs the target ground image and the target ground map into the trained model 20 and performs a predetermined inference. The trained model 20 outputs the inference result for each target ground map. The position estimation unit 16 estimates the position of the aircraft 10 based on the predetermined inference result for each target ground map. As described above, there can be various methods of inference using the trained model 20, and correspondingly, various examples of the processing of the position estimation unit 16 can be given.

[0029] In one example, the trained model 20 performs feature point matching to identify correspondence between one or more feature points of the ground image and one or more feature points of the ground map. In this case, for each of one or more target ground maps, the position estimation unit 16 inputs both the target ground image and the target ground map into the trained model 20 and obtains a result of feature point matching between the target ground image and the target ground map. The position estimation unit 16 identifies a geographical range corresponding to the target ground image from one or more target ground maps based on the result of feature point matching for each of the one or more target ground maps. Then, the position estimation unit 16 estimates the position of the air vehicle 10 based on the identified geographical range. In one example, the position estimation unit 16 selects one target ground map whose feature points match with the target ground image, or selects one target ground map whose feature points match most closely with the target ground image. The position estimation unit 16 performs projective transformation on the target ground image so that one or more feature points of the target ground image match one or more feature points of the target ground map, and projects the transformed target ground image onto the selected target ground map. Through this process, the position estimation unit 16 identifies a geographical range corresponding to the target ground image. The position estimation unit 16 estimates the position of the center of gravity of that geographical range, i.e., the position corresponding to the center of gravity of the projectively transformed target ground image, as the position of the aircraft 10. The position of the center of gravity corresponds to the center of the target ground image.

[0030] FIG. 4 is a diagram illustrating an example of estimating the position of an aircraft 10 through feature point matching using a trained model 20. In this example, it is assumed that the map database 15 stores multiple target ground maps 200, and the image acquisition unit 14 acquires a target ground image 210. Under this assumption, it is assumed that the position estimation unit 16 performs processing on one target ground map 201 corresponding to the area covered by the target ground image 210. The position estimation unit 16 inputs the target ground image 210 and the target ground map 201 to the trained model 20, and acquires a feature point matching result 290 between multiple feature points 220 of the target ground image 210 and multiple feature points 230 of the target ground map 201. When the position estimation unit 16 selects the target ground map 201, the position estimation unit 16 performs projective transformation on the target ground image 210 so that the multiple feature points 220 of the target ground image 210 match the multiple feature points 230 of the target ground map 201, and projects the transformed target ground image 210′ ​​onto the target ground map 201. The target ground image 210' corresponds to the geographical range corresponding to the target ground image 210. The position estimation unit 16 estimates the position 240 of the center of gravity of the target ground image 210' as the position of the flying object 10.

[0031] In another example, the trained model 20 executes a process of extracting one or more feature points from an image. In this case, the position estimation unit 16 inputs a target ground image into the trained model 20 and extracts one or more feature points from the target ground image. Furthermore, for each of one or more target ground maps, the position estimation unit 16 inputs the target ground map into the trained model 20 and extracts one or more feature points from the target ground map. Then, for each of the one or more target ground maps, the position estimation unit 16 performs feature point matching between one or more feature points of the target ground image and one or more feature points of the target ground map. In this example, the position estimation unit 16 performs feature point matching using a method (e.g., nearest neighbor method) different from that of the trained model 20. Based on the result of feature point matching for each of the one or more target ground maps, the position estimation unit 16 identifies a geographical range corresponding to the target ground image from the one or more target ground maps. Then, the position estimation unit 16 estimates the position of the aircraft 10 based on the identified geographical range. In one example, the position estimation unit 16 selects one target ground map whose feature points match with the target ground image, or selects one target ground map whose feature points match most closely with the target ground image. The position estimation unit 16 performs projective transformation on the target ground image so that one or more feature points of the target ground image match one or more feature points of the target ground map, and projects the transformed target ground image onto the selected target ground map. Through this process, the position estimation unit 16 identifies a geographical range corresponding to the target ground image. The position estimation unit 16 estimates the position of the center of gravity of that geographical range as the position of the aircraft 10.

[0032] Referring again to FIG. 4 , an example of performing feature point matching using a method different from that of the trained model 20 will be described. The position estimation unit 16 inputs a target ground image 210 into the trained model 20 and extracts multiple feature points 220. The position estimation unit 16 also inputs a target ground map 201 into the trained model 20 and extracts multiple feature points 230. The position estimation unit 16 performs matching between the multiple feature points 220 and the multiple feature points 230 using a method different from that of the trained model 20, and obtains an execution result 290. If the position estimation unit 16 selects the target ground map 201, the position estimation unit 16 performs projective transformation on the target ground image 210 so that the multiple feature points 220 of the target ground image 210 match the multiple feature points 230 of the target ground map 201, and projects the transformed target ground image 210′ ​​onto the target ground map 201. The position estimation unit 16 estimates the position 240 of the center of gravity of the target ground image 210′ ​​as the position of the aircraft 10.

[0033] In yet another example, the trained model 20 executes a process of identifying a position corresponding to a ground image from a ground map. In this case, for each of one or more target ground maps, the position estimation unit 16 inputs both the target ground image and the target ground map into the trained model 20 and identifies a position corresponding to the target ground image from the target ground map. The position estimation unit 16 then estimates the position of the aircraft 10 based on the results of identifying the position for each of the one or more target ground maps. Typically, when the trained model 20 processes one target ground map together with the target ground image, it outputs a position corresponding to the target ground image, and when the trained model 20 processes another target ground map together with the target ground image, it outputs an execution result indicating that it was unable to identify a position corresponding to the target ground image. The position estimation unit 16 estimates the position output from the trained model 20 as the position of the aircraft 10. In the example of Figure 4, the position estimation unit 16 inputs both the target ground image 210 and the target ground map 201 into the trained model 20, the trained model 20 directly infers the position 240 from the input data, and the position estimation unit 16 finally estimates the position 240 as the position of the aircraft 10.

[0034] As shown in steps S14 to S16, in response to determining that the GNSS is not operating normally, the image acquisition unit 14 acquires a target ground image, and the position estimation unit 16 acquires one or more target ground maps and estimates the position of the aircraft 10.

[0035] In step S17, the flight control unit 12 controls the aircraft 10 based on the estimated position. The flight control unit 12 continues to fly the aircraft 10 toward the destination based on the estimated position and flight route, while communicating with a ground station as necessary. After step S17, the process proceeds to step S18.

[0036] As shown in step S18, flight control using the GNSS or estimation system is performed until the aircraft 10 lands. If the aircraft 10 is in flight (NO in step S18), the process returns to step S12. In the repeated step S12, the monitoring unit 13 again determines the operation of the GNSS.

[0037] If the GNSS is operating normally or has started to operate normally again (YES in step S12), then in step S13, the positioning unit 11 and flight control unit 12 cooperate to perform flight control using the GNSS.

[0038] If an abnormality has occurred in the GNSS (NO in step S12), in step S14, the image acquisition unit 14 acquires a new target ground image (step S14). In step S15, the position estimation unit 16 newly acquires one or more target ground maps. In the repeated step S15, the position estimation unit 16 may acquire the position of the aircraft 10 last determined by the GNSS as the reference position, or may acquire the position of the aircraft 10 last estimated by the estimation system as the reference position. In either case, the position estimation unit 16 can select one or more target ground maps from the map database 15 based on the reference position. As described above, the position estimation unit 16 may select one or more target ground maps corresponding to the flight route, or may select all target ground maps in the map database 15. In step S16, the position estimation unit 16 re-estimates the position of the aircraft 10 based on the new target ground image and the one or more newly acquired target ground maps. In step S17, the flight control unit 12 controls the aircraft 10 based on the newly estimated position.

[0039] In response to landing (YES in step S18), the aircraft 10 ends the processing flow S1.

[0040] [Variations] The technology according to the present disclosure has been described in detail above based on various examples. However, the present disclosure is not limited to the above examples. The technology according to the present disclosure can be modified in various ways without departing from the spirit of the present disclosure.

[0041] (different system configuration) 5 is a diagram showing another example of application of the estimation system, in which at least a part of the estimation system is applied to a ground server 40 connected to an air vehicle 30 via a communication network.

[0042] The flying object 30 has, as its functional components, a positioning unit 31, a flight control unit 32, a monitoring unit 33, and an image transmission unit 34. The positioning unit 31, the flight control unit 32, and the monitoring unit 33 are similar to the positioning unit 11, the flight control unit 12, and the monitoring unit 13 described above, respectively. The image transmission unit 34 is a functional module that transmits target ground images captured by a visual sensor to a server 40. The hardware components of the flying object 30 are similar to those of the flying object 10 described above.

[0043] The server 40 has, as functional components, an image receiving unit 41, a map database 42, and a position estimation unit 43. The image receiving unit 41 is a functional module that receives target ground images from the aircraft 30. The map database 42 is similar to the map database 15 described above. The position estimation unit 43 is a functional module that estimates the position of the aircraft 30 based on the target ground images and the target ground map, and transmits the estimated position to the aircraft 30. The position estimation unit 43 uses a trained model 50 for its estimation process. The trained model 50 is similar to the trained model 20 described above.

[0044] FIG. 6 is a diagram showing an example of the hardware configuration of the server 40. For example, the server 40 includes a processor 401, a main memory device 402, an auxiliary memory device 403, a communication control device 404, an input device 405, and an output device 406. The processor 401 is a computing device that executes an operating system and application programs, and is, for example, a CPU or GPU. The main memory device 402 is composed of, for example, a ROM and RAM. The auxiliary memory device 403 is composed of, for example, a hard disk or flash memory, and generally stores larger amounts of data than the main memory device 402. The communication control device 404 is composed of, for example, a network card or a wireless communication module. The input device 405 is composed of, for example, a keyboard, a mouse, a touch panel, etc. The output device 406 is composed of, for example, a monitor (a touch panel in one example) and a speaker.

[0045] Each functional module of the server 40 is realized by an estimation program 410 stored in advance in the auxiliary storage device 403. Each functional module is realized by having the processor 401 or the main storage device 402 read the estimation program 410 and have the processor 401 execute the estimation program 410. The processor 401 operates the communication control device 404, the input device 405, or the output device 406 in accordance with the estimation program 410, and reads and writes data from and to the main storage device 402 or the auxiliary storage device 403.

[0046] The estimation program 410 may be provided by being recorded on a non-transitory recording medium such as a CD-ROM, a DVD-ROM, a semiconductor memory, etc. Alternatively, the estimation program 410 may be provided via a communication network as a data signal superimposed on a carrier wave.

[0047] The server 40 may be configured as one server 40 or as multiple servers 40. When multiple servers 40 are used, these servers 40 are connected via a communication network such as the Internet or an intranet, thereby logically constructing one server 40.

[0048] The process flow S1 can also be realized by cooperation between the flying object 30 and the server 40. Referring again to Figure 3, the process flow S1 in this modified example will be described.

[0049] In step S11, the positioning unit 31 and flight control unit 32 of the flying object 30 cooperate to start flight control using GNSS.

[0050] In step S12, the monitoring unit 33 of the flying object 30 determines whether the GNSS is operating normally.

[0051] If the GNSS is operating normally (YES in step S12), the process proceeds to step S13. In step S13, the positioning unit 31 and the flight control unit 32 cooperate to control the flying object 30 using the GNSS. After step S13, the process proceeds to step S18.

[0052] If an abnormality has occurred in the GNSS (NO in step S12), the process proceeds to step S14. Then, from step S14 to step S17, the estimation system performs position estimation.

[0053] In step S14, the image transmission unit 34 of the flying object 30 controls the visual sensor 108 to capture an image of the ground from the sky. The image transmission unit 34 transmits the image obtained from the visual sensor 108 to the server 40. In the server 40, the image reception unit 41 receives the image as a target ground image. This reception process is an example of a process for acquiring a target ground image.

[0054] In step S15, the position estimation unit 43 of the server 40 acquires one or more target ground maps used to estimate the position of the flying object 30 from the map database 42. Similar to the position estimation unit 16, the position estimation unit 43 may acquire one or more target ground maps by various methods. The position estimation unit 43 may receive information necessary to select one or more target ground maps from the flying object 30. For example, the position estimation unit 43 may receive the position of the flying object 30 last determined by GNSS as the reference position, or may receive the flight route, or may receive the flight route and flight time.

[0055] In step S16, the position estimation unit 43 estimates the position of the aircraft 30 based on the target ground image and one or more target ground maps. For each of the one or more target ground maps, the position estimation unit 43 inputs the target ground image and the target ground map into the trained model 50 and performs a predetermined inference. The trained model 50 outputs the inference result for each target ground map. The position estimation unit 43 estimates the position of the aircraft 30 based on the predetermined inference result for each target ground map. As with the trained model 20, the inference method used by the trained model 50 can vary, and correspondingly, the processing procedure of the position estimation unit 43 can also vary, as with the position estimation unit 16. The position estimation unit 43 transmits the estimated position to the aircraft 30.

[0056] As shown in steps S14 to S16, in response to determining that the GNSS is not operating normally, the image receiving unit 41 acquires a target ground image, and the position estimation unit 43 acquires one or more target ground maps and estimates the position of the aircraft 30.

[0057] In step S17, the flight control unit 32 receives the estimated position transmitted from the server 40, and controls the flying object 30 based on this estimated position. After step S17, the process proceeds to step S18.

[0058] As shown in step S18, flight control using the GNSS or estimation system is performed until the flying object 30 lands.

[0059] (Other variations) In the above example, the position estimation unit estimates the position of the aircraft using a trained model, but the position estimation unit may estimate the position of the aircraft without using a trained model. For example, the position estimation unit may perform pattern matching between the target ground image and each of one or more target ground maps, and estimate the position of the aircraft based on the results of the pattern matching.

[0060] Although the estimation system is used as an alternative to GNSS in the above example, the estimation system may be used independently of GNSS. For example, the aircraft may fly based on a position estimated by the estimation system without using GNSS. Alternatively, the estimation system may be used to calculate the reliability of GNSS. In this example, the aircraft or the server includes a determination unit that compares the aircraft's position determined by GNSS with the aircraft's position estimated by the estimation system to determine the reliability of GNSS. For example, the determination unit may calculate an index indicating the reliability of GNSS using a predetermined formula based on the difference between the two positions. Alternatively, the determination unit may determine that the reliability of GNSS is high if the difference is less than a predetermined threshold, and that the reliability of GNSS is low if the difference is equal to or greater than the threshold.

[0061] Although the above example illustrates the use of the estimation system for automatic or autonomous flight, the estimation system may also be used for other purposes. For example, the estimation system may display the estimated position of the air vehicle on a display device to present the estimated position to a user, such as a pilot, drone operator, or air traffic controller. The user can refer to the estimate to navigate or monitor the air vehicle.

[0062] The processing steps of the method executed by at least one processor are not limited to the above examples. For example, some of the above steps may be omitted, or the steps may be executed in a different order. Furthermore, any two or more of the above steps may be combined, or some of the steps may be modified or deleted. Alternatively, other steps may be executed in addition to the above steps.

[0063] In the present disclosure, when comparing the magnitude of two numerical values, either of the two criteria "greater than or equal to" and "greater than" may be used, or either of the two criteria "less than or equal to" and "less than" may be used.

[0064] In this disclosure, the expression "at least one processor executes a first process, executes a second process, ... executes an nth process" or a corresponding expression indicates a concept including a case where the entity executing the n processes from the first process to the nth process, i.e., the processor, changes midway through. In other words, this expression indicates a concept including both a case where all n processes are executed by the same processor and a case where the processor changes among the n processes according to an arbitrary policy.

[0065] [Note] As can be seen from the various examples above, the present disclosure includes the following aspects. (Appendix 1) at least one processor; the at least one processor: Acquires target ground images from the sky using a visual sensor mounted on the aircraft, retrieving from a storage device one or more target ground maps including location information for each location within a predetermined geographic area; estimating a position of the air vehicle based on the target ground image and the one or more target ground maps; Estimation system. (Appendix 2) The at least one processor determines, as an estimate of the position of the air vehicle, For each of the one or more target ground maps, input the target ground image and the target ground map into a trained model that receives the ground image and the ground map and performs a predetermined inference, and perform the predetermined inference; estimating a position of the air vehicle based on results of the predetermined inference for each of the one or more target ground maps; 10. The estimation system of claim 1. (Appendix 3) the trained model performs, as the predetermined inference, feature point matching to identify correspondences between one or more feature points of the ground image and one or more feature points of the ground map; the at least one processor: For each of the one or more target ground maps, input the target ground image and the target ground map into the trained model to obtain a result of feature point matching between the target ground image and the target ground map; Identifying a geographical area corresponding to the target ground image from the one or more target ground maps based on the result of the feature point matching for each of the one or more target ground maps; estimating a position of the air vehicle based on the identified geographic range; 1. The estimation system according to claim 2. (Appendix 4) The trained model extracts one or more feature points from each of the ground image and the ground map as the predetermined inference; the at least one processor: Inputting the target ground image into the trained model to extract one or more feature points of the target ground image; For each of the one or more target ground maps, input the target ground map into the trained model to extract one or more feature points of the target ground map; For each of the one or more target ground maps, perform feature point matching between the one or more feature points of the target ground image and the one or more feature points of the target ground map; Identifying a geographical area corresponding to the target ground image from the one or more target ground maps based on the result of the feature point matching for each of the one or more target ground maps; estimating a position of the air vehicle based on the identified geographic range; 1. The estimation system according to claim 2. (Appendix 5) The trained model identifies, as the predetermined inference, a position corresponding to the ground image from the ground map; the at least one processor: For each of the one or more target ground maps, input the target ground image and the target ground map into the trained model, and identify the location corresponding to the target ground image from the target ground map; estimating a position of the air vehicle based on the results of identifying the position with respect to each of the one or more target ground maps; 1. The estimation system according to claim 2. (Appendix 6) the at least one processor: determining whether the global positioning satellite system of the aircraft is operating normally; In response to determining that the global positioning satellite system is not operating normally, acquiring the target ground image, acquiring the one or more target ground maps, and estimating the position of the air vehicle. 6. The estimation system according to any one of appendices 1 to 5. (Appendix 7) the storage device stores a plurality of the target ground maps; the at least one processor selecting the one or more target ground maps from the plurality of target ground maps based on a last determined position of the air vehicle by the global positioning satellite system. 10. The estimation system according to claim 6. (Appendix 8) 1. An estimation method performed by an estimation system comprising at least one processor, comprising: acquiring a target ground image from the sky using a visual sensor mounted on the aircraft; retrieving from a storage device one or more target ground maps including location information for each location within a predetermined geographic range; estimating a position of the air vehicle based on the target ground image and the one or more target ground maps; Estimation methods including: (Appendix 9) acquiring a target ground image from the sky using a visual sensor mounted on the aircraft; retrieving from a storage device one or more target ground maps including location information for each location within a predetermined geographic range; estimating a position of the air vehicle based on the target ground image and the one or more target ground maps; An estimation program that causes a computer to execute the above.

[0066] According to Supplements 1, 8, and 9, the position of an aircraft is estimated using a target ground image obtained from the sky by the aircraft's visual sensor and a target ground map prepared in advance. Therefore, the position of the aircraft can be determined without using a global navigation satellite system (GNSS). By using the target ground image and target ground map, the position can be determined more easily than by a method using GNSS. In addition, because visual sensors are cheaper than GNSS devices, a mechanism for estimating the position of an aircraft can be realized at lower cost.

[0067] According to Supplementary Note 2, a predetermined inference is performed using a trained model, and the position of the aircraft is estimated based on the results of the inference. By using the trained model, even if there is a difference between the time when the target ground map was created and the time when the target ground image was acquired, changes on the ground that occur over time can be absorbed and inference can be performed robustly and accurately. Therefore, the position of the aircraft can be estimated with high accuracy.

[0068] According to Supplementary Note 3, feature point matching between the target ground image and the target ground map is performed using a trained model. By using the trained model, even if there is a difference between the time when the target ground map was created and the time when the target ground image was acquired, changes on the ground that occur over time can be absorbed and feature point matching can be performed robustly and accurately. Therefore, the position of the aircraft can be estimated with high accuracy.

[0069] According to Supplementary Note 4, feature points are extracted from both the target ground image and the target ground map using the trained model. Using the trained model enables feature points to be extracted with high accuracy, which also improves the accuracy of subsequent processing, such as feature point matching and aircraft position estimation.

[0070] According to Supplementary Note 5, the position of the aircraft is estimated directly from the target ground image and target ground map using a trained model. By using the trained model, the position of the aircraft can be estimated robustly and accurately by absorbing changes on the ground that occur over time, such as when the target ground map was created and when the target ground image was acquired.

[0071] According to Supplementary Note 6, when the Global Navigation Satellite System (GNSS) is not operating normally, the position of the aircraft is estimated using ground images acquired from the sky by the aircraft's visual sensor and a pre-prepared ground map. This mechanism makes it possible to estimate the position of the aircraft and continue flying safely even when the aircraft falls into a situation where it cannot use GNSS.

[0072] According to Supplementary Note 7, one or more target ground maps are selected based on the last position of the aircraft determined by the Global Navigation Satellite System (GNSS), thereby reducing the number of maps to be compared with the target ground image, thereby reducing the processing load on the processor and enabling the position of the aircraft to be estimated quickly. [Explanation of symbols]

[0073] 10...aircraft, 11...positioning unit, 12...flight control unit, 13...monitoring unit, 14...image acquisition unit, 15...map database, 16...position estimation unit, 20...trained model, 30...aircraft, 31...positioning unit, 32...flight control unit, 33...monitoring unit, 34...image transmission unit, 40...server, 41...image receiving unit, 42...map database, 43...position estimation unit, 50...trained model, 107...GNSS device, 108...visual sensor, 120...estimation program, 200, 201...target ground map, 210...target ground image, 220, 230...feature points, 240...position of aircraft, 410...estimation program.

Claims

1. at least one processor; the at least one processor: Determine whether the aircraft's global positioning satellite system is operating normally; in response to determining that the global positioning satellite system is not operating normally, Acquires target ground images from the sky using a visual sensor mounted on the aircraft, acquiring the last determined position of the aircraft by the global positioning satellite system as a reference position; selecting one or more target ground maps from a storage device that stores a plurality of target ground maps including location information of each location within a predetermined geographical range based on the reference position; estimating a position of the air vehicle based on the target ground image and the one or more target ground maps; Estimation system.

2. The at least one processor determines, as an estimate of the position of the air vehicle, For each of the one or more target ground maps, input the target ground image and the target ground map into a trained model that receives the ground image and the ground map and performs a predetermined inference, and perform the predetermined inference; estimating a position of the air vehicle based on results of the predetermined inference for each of the one or more target ground maps; The estimation system of claim 1 .

3. the trained model performs, as the predetermined inference, feature point matching to identify correspondences between one or more feature points of the ground image and one or more feature points of the ground map; the at least one processor: For each of the one or more target ground maps, input the target ground image and the target ground map into the trained model to obtain a result of feature point matching between the target ground image and the target ground map; Identifying a geographical area corresponding to the target ground image from the one or more target ground maps based on the result of the feature point matching for each of the one or more target ground maps; estimating a position of the air vehicle based on the identified geographic range; The estimation system according to claim 2 .

4. The trained model extracts one or more feature points from each of the ground image and the ground map as the predetermined inference; the at least one processor: Inputting the target ground image into the trained model to extract one or more feature points of the target ground image; For each of the one or more target ground maps, input the target ground map into the trained model to extract one or more feature points of the target ground map; For each of the one or more target ground maps, performing feature point matching between the one or more feature points of the target ground image and the one or more feature points of the target ground map; Identifying a geographical area corresponding to the target ground image from the one or more target ground maps based on the result of the feature point matching for each of the one or more target ground maps; estimating a position of the air vehicle based on the identified geographic range; The estimation system according to claim 2 .

5. The trained model identifies, as the predetermined inference, a position corresponding to the ground image from the ground map; the at least one processor: For each of the one or more target ground maps, input the target ground image and the target ground map into the trained model, and identify the location corresponding to the target ground image from the target ground map; estimating a position of the air vehicle based on the results of the location determination for each of the one or more target ground maps; The estimation system according to claim 2 .

6. 1. An estimation method performed by an estimation system comprising at least one processor, comprising: determining whether the global positioning satellite system of the aircraft is operating normally; in response to determining that the global positioning satellite system is not operating normally, Acquires target ground images from the sky using a visual sensor mounted on the aircraft, acquiring the last determined position of the aircraft by the global positioning satellite system as a reference position; selecting one or more target ground maps from a storage device that stores a plurality of target ground maps including location information of each location within a predetermined geographical range based on the reference position; estimating a position of the air vehicle based on the target ground image and the one or more target ground maps; Steps and Estimation methods including:

7. A step of determining whether the global positioning satellite system of the aircraft is operating normally; in response to determining that the global positioning satellite system is not operating normally, Acquires target ground images from the sky using a visual sensor mounted on the aircraft, acquiring the last determined position of the aircraft by the global positioning satellite system as a reference position; selecting one or more target ground maps from a storage device that stores a plurality of target ground maps including location information of each location within a predetermined geographical range based on the reference position; estimating a position of the air vehicle based on the target ground image and the one or more target ground maps; Steps and An estimation program that causes a computer to execute the above.

Images