Synchronization of aircraft sensor systems

Abstract

To provide a method, apparatus, and system for adjusting operation of a number of aircraft with higher accuracy.SOLUTION: The invention provides a method, apparatus, system, and computer program product for operating an aerial imaging system 200. A first altitude 226 of a first aircraft 204 is determined, by a computer system 210, using first images 212 of a key point 220 generated by the first aircraft and stereo depth triangulation 224. The first altitude 226 is compared by the computer system with a second altitude 230 of a second aircraft 206 determined by the second aircraft 206, to form a comparison 232. An offset 238 between the first altitude and the second altitude is determined using the comparison 232. The first altitude or the second altitude is adjusted based on the offset 238. Multiple images of one region are obtained from the first aircraft at the first altitude and from the second aircraft at the second altitude.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates generally to aircraft, and more particularly to methods, apparatus, and systems for coordinating the operation of some aircraft with increased precision. [Background technology]

[0002] Photogrammetry is the use of photographs to make measurements. In surveying and mapping, photogrammetry uses images to measure distances between objects. For example, photogrammetry can be used to plot contour lines on a topographical map. Photogrammetry can also be used to create point cloud models of a three-dimensional model of an environment. For example, a point cloud model can be generated from photographs of a field of crops, a park, a city block, a stadium, a terrain, or any other object of interest.

[0003] An aircraft equipped with a camera system can fly over an area, such as a park, a crop field, a city, or other area. The aircraft generates images of the area where overlap exists. Features in the overlap area are used to accurately triangulate and identify key points in the images as part of the process of generating a model.

[0004] This process of using photogrammetry to take measurements of an area is time-consuming. The process involves downloading imagery from an aircraft and processing the data to generate a point cloud of the area. This type of area mapping can take more time than desired.

[0005] It would therefore be desirable to have a method and apparatus that takes into account at least some of the above-mentioned problems, as well as other possible problems, such as overcoming the technical challenges associated with generating data for surveying and mapping an area. Summary of the Invention

[0006] One embodiment of the present disclosure provides an aerial imaging system including a computer system and a controller within the computer system. The controller is configured to determine a first altitude of a first aircraft using a first image of keypoints generated by the first aircraft and stereo depth triangulation. The controller is configured to compare the first altitude to a second altitude determined for the second aircraft using at least one of stereo depth triangulation using a second image of keypoints generated by the second aircraft and measurements made by a sensor system on the second aircraft to generate a comparison. The controller is configured to use the comparison to determine an offset between the first altitude determined for the first aircraft and the second altitude measured by the second aircraft.

[0007] Another embodiment of the present disclosure provides an aerial imaging system including a first camera system, an altitude sensor system, and an aircraft controller. The first camera system is mounted on a first aircraft and generates first images during flight of the first aircraft. The altitude sensor system is mounted on the first aircraft and detects a first altitude of the first aircraft. The controller is configured to determine a first distance from the first aircraft to the key point using first images of the key point generated by the first camera system associated with the first aircraft during flight of the first aircraft and stereo depth triangulation. The controller is configured to receive, from the second aircraft, a second distance to the key point from the second aircraft and a second altitude measured for the second aircraft. The controller is configured to compare the first distance and first altitude from the first aircraft to the key point and the second distance and second altitude from the second aircraft to generate a comparison. The controller is configured to use the comparison to determine an offset between the first altitude of the first aircraft and the second altitude of the second aircraft, the offset being used to adjust the altitude of the first aircraft.

[0008] Yet another embodiment of the present disclosure provides a method for operating an airborne imaging system. A first altitude of a first aircraft is determined by a computer system using stereo depth triangulation and a first image of key points generated by the first aircraft during the first aircraft's flight. The first altitude is compared by the computer system to a second altitude of a second aircraft determined by a second aircraft to create a comparison. An offset between the first altitude and the second altitude is determined by the computer system using the comparison. At least one of the first altitude or the second altitude is adjusted by the computer system based on the offset. The computer system obtains multiple images of an area from the first aircraft at the first altitude and from the second aircraft at the second altitude.

[0009] Yet another embodiment of the present disclosure provides a computer program product for operating an airborne imaging system. The computer program product includes a computer-readable storage medium having first, second, and third program codes stored thereon. The program codes are executable by a computer system and cause the computer system to determine a first distance from the first aircraft to the keypoint using a first image of the keypoint generated by a first aircraft during the first aircraft's flight and stereo depth triangulation. The second program code is executable by the computer system to cause the computer system to compare the first distance from the first aircraft to the keypoint and a first altitude of the first aircraft measured by the first aircraft with a second distance from the second aircraft to the keypoint and a second altitude of the second aircraft measured by the second aircraft, and to generate a comparison. The third program code is executable by the computer system to cause the computer system to determine an offset between the first altitude of the first aircraft and the second altitude measured by the second aircraft using the comparison and to adjust the altitude of the first aircraft using the offset.

[0010] These features and functions can be achieved independently in various embodiments of the present disclosure or can be combined in yet other embodiments, further details of which can be seen with reference to the following description and drawings.

[0011] The novel features believed characteristic of the illustrative embodiments are set forth in the appended claims. However, the illustrative embodiments, together with preferred modes of use, further objects and features thereof, will best be understood by reference to the following detailed description of illustrative embodiments of the present disclosure taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a pictorial illustration of image generation by an aerial imaging system in accordance with an illustrative embodiment; [Figure 2] 1 is an example of an imaging environment, in accordance with an illustrative embodiment. [Figure 3] 1 is an illustration of an example block diagram of an aircraft in accordance with an illustrative embodiment; [Figure 4] 1 is an illustration of a block diagram of an aircraft system in accordance with an illustrative embodiment; [Figure 5] 1 is an example of image-based triangulation in accordance with an illustrative embodiment; [Figure 6] 1 is an illustration of a flow diagram of a process for operating an aerial imaging system in accordance with an illustrative embodiment. [Figure 7] 10 is an example of a flow diagram of a process for performing an action using an offset in accordance with an illustrative embodiment; [Figure 8] 10 is an illustration of a flow diagram of a process for determining distance to keypoints in accordance with an illustrative embodiment; [Figure 9] 1 is an illustration of a flow diagram of a process performed by an aircraft to generate information used to operate an airborne imaging system in accordance with an illustrative embodiment; [Figure 10]1 is an illustration of a flow diagram of a process for operating an aerial imaging system in accordance with an illustrative embodiment. [Figure 11] 1 is another example of a flow diagram of a process for operating an airborne imaging system in accordance with an illustrative embodiment. [Figure 12] 1 is an example block diagram of a data processing system in accordance with an illustrative embodiment; DETAILED DESCRIPTION OF THE INVENTION

[0013] The exemplary embodiments recognize and take into account one or more different considerations. For example, the exemplary embodiments recognize and take into account that flight time can be reduced by flying the aircraft at a higher altitude so that the camera system generates images of a larger area within the region being mapped. However, the exemplary embodiments recognize and take into account that while the time required to generate data for the region is reduced, the resolution of the data is also reduced.

[0014] The exemplary embodiments recognize and take into account that it may take many hours for a single aircraft to cover multiple areas within a region being mapped. Post-processing of these images is performed when the images are downloaded after the flight. As a result, there may be several datasets from an aircraft making multiple flights to generate multiple images of a region. The exemplary embodiments also recognize and take into account that it may take several hours to process the images in a dataset from a single flight. Furthermore, the exemplary embodiments recognize and take into account that once a dataset is processed, it is stitched together and reprocessed. Thus, the exemplary embodiments recognize and take into account that it may take several days to process and refine a dataset to generate an acceptable point cloud for the region.

[0015] The exemplary embodiments recognize and take into account that another solution may include using several aircraft equipped with multiple camera systems to fly over an area and generate multiple images of the area. The exemplary embodiments recognize and take into account that using several aircraft may reduce flight time. However, the exemplary embodiments recognize and take into account that with multiple aircraft generating images of an area, there may be errors that increase the difficulty in generating an accurate point cloud from the images.

[0016] The exemplary embodiments recognize and take into account that sensor errors can reduce the accuracy of the point cloud and increase the time required to create a point cloud from images. For example, the exemplary embodiments recognize and take into account that several aircraft may fly at different altitudes from one another. As a result, stitching images generated from several aircraft may be more difficult to form accurately and may take longer than desired.

[0017] The exemplary embodiments recognize and take into account that differences in altitude can be taken into account. The exemplary embodiments recognize and take into account that when multiple images of an area generated by an aircraft are generated at different altitudes, knowing the altitude of the aircraft when generating the images is important for accurately stitching the images together to form a point cloud. The exemplary embodiments recognize and take into account that current sensors for determining the altitude of aircraft, such as UAS, are less accurate than desired for processing images from unmanned aerial systems to generate a point cloud.

[0018] For example, the exemplary embodiments recognize and take into account that barometric altitude sensors may not be as accurate as desired. For example, variations in temperature, air pressure, or both may occur across different regions of a region. As a result, different UASs applying the region's patterns may obtain different altitude readings, even if the UASs are flying at the same altitude. The exemplary embodiments recognize and take into account that this issue is compounded when aircraft fly at different altitudes.

[0019] As another example, the exemplary embodiment recognizes and takes into account that global positioning system (GPS) units in aircraft may also be less accurate than desired. As a UAS flies over different areas of a region to generate images, signal strength may vary due to cloud cover and other environmental factors, affecting the accuracy of the altitude determined by the global positioning system unit or the UAS.

[0020] Thus, the illustrative embodiments provide a method, apparatus, and system for reducing errors in determining altitude between several aircraft. In these instances, a camera system can be used to synchronize several aircraft. For example, the camera system can be used to generate images of key features as several aircraft fly over them and generate images that include them.

[0021] In one example, a first distance from a first aircraft to a keypoint is determined using a first image of the keypoint generated by the first aircraft during the first aircraft's flight and a first altitude of the first aircraft measured by the first aircraft. To create a comparison, the first distance from the first aircraft to the keypoint and the first altitude of the first aircraft relative to the keypoint are compared with a second distance from a second aircraft to the keypoint and a second altitude of the second aircraft. Using this comparison, an offset between the first altitude of the first aircraft and the second altitude measured by the second aircraft is determined. This offset is used to adjust the altitude of the first aircraft.

[0022] In this example, the offset is used in several different ways. For example, the offset can be used to adjust the altitude of one of the two aircraft. In another example, the offset can be used to adjust the altitude of the metadata of images taken by at least one of the first aircraft or the second aircraft. These and other actions can be performed using the offset. Furthermore, the offset can be determined many times depending on the particular mission.

[0023] 1 , a pictorial representation of image generation by an aerial imaging system is shown in accordance with an illustrative embodiment. In this example, aerial imaging system includes unmanned aerial vehicle 100 and unmanned aerial vehicle 102.

[0024] As shown, unmanned aerial vehicle 100 and unmanned aerial vehicle 102 are part of aerial imaging system 103 and may be operated to generate images of object of interest 104. In this example, object of interest 104 is an area or region of land. In another example, object of interest 104 may be a building, a dam, or other suitable object of interest.

[0025] In this example, unmanned aerial vehicle 100 travels along flight path 106, and unmanned aerial vehicle 102 travels along flight path 108 to generate images of target object 104 or at least a portion of target object 104. As shown, flight path 106 results in unmanned aerial vehicle 100 flying over a portion of target object 104. Flight path 108 results in unmanned aerial vehicle 102 flying over a different portion of target object 104.

[0026] As shown, unmanned aerial vehicles 100 and 102 generate images at each of the waypoints, such as waypoint 112 and waypoint 114 along flight path 106 and flight path 108 .

[0027] As shown, the image generated by unmanned aerial vehicle 100 at waypoint 112 and the image generated by unmanned aerial vehicle 102 at waypoint 114 overlap to include keypoint 122. In this example, keypoint 122 is a keypoint that can be used to account for the difference in altitude between unmanned aerial vehicle 100 and unmanned aerial vehicle 102.

[0028] For example, it may be desirable for unmanned aerial vehicles 100 and 102 to generate images at waypoint 110 at the same altitude as they fly over object of interest 104. However, the altitudes measured by unmanned aerial vehicles 100 and 102 may have errors, such that even though they are commanded to fly at the same altitude, they may actually be flying at different altitudes. Such errors may be due to inaccuracies in the sensors used to measure altitude, such as calibration errors. Such errors may introduce inaccuracies into the datasets used to generate a point cloud of an object or area of ​​land. If the images in the datasets are at the incorrect altitude, combining the datasets to form the point cloud will result in an inaccurate point cloud.

[0029] In this example, a different altitude may be specified to account for the difference in actual altitude. For example, this difference, also referred to as an offset, may be used to adjust the altitude of unmanned aerial vehicle 100 so that unmanned aerial vehicle 102 fly at the same altitude. In another example, this adjustment may be made by both unmanned aerial vehicle 100 and unmanned aerial vehicle 102 based on an offset of the difference in altitude between unmanned aerial vehicle 100 and unmanned aerial vehicle 102.

[0030] Thus, the offset can be used to calibrate the altitude sensor of at least one of unmanned aerial vehicles 100 and 102. In this illustrated example, the altitude measured by one or both of the unmanned aerial vehicles may be erroneous, causing the two vehicles to fly at different altitudes when the same altitude is desired. Thus, images generated by the unmanned aerial vehicles can be used to determine their actual altitudes.

[0031] For example, unmanned aerial vehicle 100 may generate two or more images of keypoint 122. Unmanned aerial vehicle 100 may identify the presence of keypoint 122 in images generated at waypoint 112 and waypoint 114. In this example, a first image may be generated at waypoint 112 and a second image may be generated at waypoint 114. Unmanned aerial vehicle 100 may use these images to determine a first distance from waypoint 112 to keypoint 122.

[0032] In this example, the keypoint 122 may be an object or an object feature. The keypoint 122 may be any object or object feature having a size that can be used as a reference point. In one example, the keypoint 122 may be an object or feature represented by a pixel or by the center of a group of pixels that capture the same keypoint. In this example, the first distance may be determined using stereo depth triangulation or some other suitable technique.

[0033] Additionally, unmanned aerial vehicle 100 may measure a first altitude for unmanned aerial vehicle 100 using a barometric altitude sensor associated with unmanned aerial vehicle 100. This measurement may be made at waypoint 112.

[0034] In this example, the unmanned aerial vehicle 102 also identifies a keypoint 122 in images generated by the unmanned aerial vehicle 102 at the waypoint 114. The unmanned aerial vehicle 102 determines a second distance from the waypoint 114 to the keypoint 122 using the images generated by the unmanned aerial vehicle 102 and stereo depth triangulation. In this example, the unmanned aerial vehicle 102 may measure a second altitude of the unmanned aerial vehicle 102 at the waypoint 114. The second altitude may be measured using a barometric altitude sensor associated with the unmanned aerial vehicle 102.

[0035] As shown, unmanned aerial vehicle 102 transmits the second distance to unmanned aerial vehicle 100. Additionally, unmanned aerial vehicle 102 transmits to unmanned aerial vehicle 100 a second altitude of unmanned aerial vehicle 102 measured by unmanned aerial vehicle 102.

[0036] Unmanned aerial vehicle 100 can then determine the offset using the distance from each of the unmanned aerial vehicles to key point 122 and the altitude measured by each of the unmanned aerial vehicles. In another example, the altitude can be calculated using the imagery to determine the offset between the altitudes measured using barometric altitude sensors in unmanned aerial vehicles 100 and 102.

[0037] The offset may then be used by at least one of unmanned aerial vehicles 100 or 102 to make an adjustment to the altitude at which one or both of the unmanned aerial vehicles are flying. This adjustment may be made to the altitude recorded in the metadata generated for the image or may be used to calibrate an altitude sensor.

[0038] As used herein, the phrase "at least one of" when used in conjunction with a list of items means that different combinations of one or more of the listed items may be used and that only one of each listed item is required. In other words, "at least one of" means that any combination of items and any number of items from the list may be used, but not all of the listed items are required. An item may be a specific object, thing, or category.

[0039] For example, without limitation, "at least one of item A, item B, and item C" may include item A, item A and item B, or item B. This example may also include item A, item B, and item C, or item B and item C. Of course, any combination of these items may also be present. In some instances, "at least one of" may be, for example, without limitation, two of item A; one of item B; ten of item C; four of item B and seven of item C; or other suitable combinations.

[0040] The illustration of this example in FIG. 1 is not intended to limit the manner in which other examples may be implemented. In another example, flight path 106 and flight path 108 can generate images including keypoint 122 at the same waypoint. In addition to or instead of keypoint 122, other keypoints can be used to determine the offset. The offset can be determined multiple times during a mission in which unmanned aerial vehicles 100 and 102 generate images of object of interest 104. For example, flight path 106 and flight path 108 can be selected such that images generated by the unmanned aerial vehicles include other keypoints located on object of interest 104. In another example, the unmanned aerial vehicle can fly outside of object of interest 104 to generate images including keypoints for use in determining the offset. As another example, processing of images in determining the offset can be performed at a location other than one or both of the unmanned aerial vehicles. For example, this processing can be performed using a remote data processing system, a cloud service, or some other component separate from the unmanned aerial vehicles.

[0041] 2, an illustration of an imaging environment is shown in accordance with an illustrative embodiment. In this example, aerial imaging system 103 is an example of an embodiment of aerial imaging system 200 in imaging environment 202.

[0042] As shown, aerial imaging system 200 includes a first aircraft 204 and may also include a second aircraft 206. First aircraft 204 and second aircraft 206 are selected from at least one of an airplane, an unmanned aerial system, an unmanned aerial vehicle, a drone, a rotorcraft, a spacecraft, or any other suitable type of aircraft. As shown, first aircraft 204 and second aircraft 206 may be the same type of aircraft or different types of aircraft.

[0043] As shown, controller 208 within computer system 210 is configured to control the operation of at least one of first aircraft 204 or second aircraft 206 .

[0044] The controller 208 may be implemented in software, hardware, firmware, or a combination thereof. If software is used, the operations performed by the controller 208 may be implemented in program code configured to execute on hardware, such as a processor unit. If firmware is used, the operations performed by the controller 208 may be implemented in program code and data, stored in persistent memory, and executed on a processor unit. If hardware is used, the hardware may include circuitry that operates to perform the operations in the controller 208.

[0045] In these instances, the hardware may take the form of at least one of a circuit system, an integrated circuit, an application-specific integrated circuit (ASIC), a programmable logic device, or any other suitable type of hardware configured to perform certain operations. When a programmable logic device is used, the device may be configured to perform certain operations. The device may be later reconfigured or may be permanently configured to perform certain operations. Programmable logic devices include, for example, programmable logic arrays, programmable array logic, field programmable logic arrays, field programmable gate arrays, and other suitable hardware devices. Additionally, these processes may be implemented in organic components integrated with inorganic components, and components may be composed entirely of non-human organic components. For example, these processes may be implemented as circuits in organic semiconductors.

[0046] Computer system 210 is a physical hardware system that includes one or more data processing systems. When multiple data processing systems are present within computer system 210, the data processing systems communicate with each other using a communication medium. This communication medium may be a network. The data processing systems may be selected from at least one of a computer, a server computer, a tablet computer, or any other suitable data processing system.

[0047] As shown, first aircraft 204 generates first image 212 and second aircraft 206 generates second image 214. First image 212 may be generated by first camera system 216 aboard first aircraft 204. Second image 214 may be generated by second camera system 218 aboard second aircraft 206. Each of these camera systems is a physical hardware system and may include software. These camera systems may include one or more cameras or sensors capable of generating images. These cameras may be selected from at least one of a visible light camera, an infrared camera, an ultraviolet sensor, a visible light sensor, a near-infrared sensor, a short-wave infrared sensor, a mid-wave infrared sensor, a long-wave infrared (LWIR) sensor, a bolometer, an electro-optical camera, an infrared sensor, a thermal sensor, or any other suitable type of sensor.

[0048] Controller 208 may identify a key point 220 in a first image 212 generated by a first camera system 216 onboard first aircraft 204. Key point 220 may be one of an area of ​​land, a mountain, a tower, a building, a lake, a dam, a marker, a pattern, a very high frequency (VHF) omnidirectional range (VOR) antenna, or any other suitable object.

[0049] Identifying the keypoints 220 can be done in a number of different ways. For example, the controller 208 can use an artificial intelligence system 244 or at least one of a variety of image processing techniques to identify the keypoints 220 in the first image 212 and the second image 214.

[0050] The artificial intelligence system 244 is a system that performs intelligent behavior and may be based on the functions of the human brain. The artificial intelligence system 244 may include at least one of an artificial neural network, a cognitive system, a Bayesian network, fuzzy logic, an expert system, a natural language system, or any other suitable system. Machine learning is used to train the artificial intelligence system 244. Machine learning involves inputting data into a process that allows the process to adjust and improve the function of the artificial intelligence system 244.

[0051] A machine learning model in the artificial intelligence system 244 is a type of artificial intelligence model that can learn without being explicitly programmed. A machine learning model can learn based on training data input into the machine learning model. A machine learning model can learn using various types of machine learning algorithms. The machine learning algorithms include at least one of supervised learning, unsupervised learning, feature learning, sparse dictionary learning, anomaly detection, association rules, or other types of learning algorithms. Examples of machine learning models include artificial neural networks, decision trees, support vector machines, Bayesian networks, genetic algorithms, and other types of models. These machine learning models can be trained using data and processed with additional data to obtain a desired output.

[0052] As shown, controller 208 can determine a first distance 222 from first aircraft 204 to keypoint 220 using first image 212 and stereo depth triangulation 224 of keypoint 220 generated by first aircraft 204 during the flight of first aircraft 204. In this example, stereo depth triangulation 224 performed using the image of keypoint 220 is also referred to as image triangulation 225.

[0053] In this example, stereo depth triangulation 224 is a triangulation process that uses stereo analysis to determine the distance to an object in an image based on several images of the object, such as keypoint 220, taken from different views or positions.

[0054] In this example, first distance 222 is the distance from first aircraft 204 to keypoint 220. More specifically, first distance 222 is from an imaging device of first aircraft 204 to keypoint 220. For example, first distance 222 may be determined using stereo depth triangulation 224 performed using a first pixel 223 of keypoint 220 in first image 212. Second distance 228 may be determined using stereo depth triangulation 224 performed using a second pixel 229 of keypoint 220 in second image 214. This determination may be made using controller 208. In determining first distance 222, this distance may be determined with respect to first aircraft 204 captured at a position between a first one of multiple first images 212 and a second one of multiple first images 212. First distance 222 may be determined using the first image and the second image.

[0055] In this example, the first pixel 223 and the second pixel 229 may be for one or more dimensions or characteristics of the keypoint 220. For example, if the keypoint 220 is a mountain, the pixels identified in the image may be for at least one of a pixel at the base of the mountain, a pixel to the height of the mountain, or another attribute of the mountain. As another example, if the keypoint 220 is a crossroads, the pixels may be for the width of the crossroads. If the keypoint 220 is a satellite dish, the pixels may be a line of pixels across the diameter of the satellite dish.

[0056] As shown, the controller 208 may compare a first distance 222 from the first aircraft 204 to the key point 220 and a first altitude 226 of the first aircraft 204 as measured by the first aircraft 204 to a second distance 228 from the second aircraft 206 to the key point 220 and a second altitude 230 of the second aircraft 206 as measured by the second aircraft 206 to make a comparison 232. A ratio of these values ​​may be used to determine an offset 238 between the first altitude 226 and the second altitude 230. The first distance 222 and the second distance 228 are considered correct.

[0057] In this example, first altitude 226 is measured by first aircraft 204 using first altitude sensor system 234 in first aircraft 204. In this example, second altitude 230 may be measured by second altitude sensor system 236 in second aircraft 206. The altitude sensor system may include at least one of a barometric pressure sensor, a Global Positioning System (GPS) receiver, or some other suitable type of sensor.

[0058] In this example, controller 208 may use comparison 232 to determine an offset 238 between first altitude 226 of first aircraft 204 measured by first aircraft 204 and second altitude 230 measured by second aircraft 206. In this example, offset 238 is used to adjust the altitude of at least one of first aircraft 204 or second aircraft 206.

[0059] As shown, the controller 208 can use the offset 238 to perform a set of actions 242. The set of actions 242 can be selected from at least one of: adjusting altitude information of points in point clouds obtained from images generated of an environment by the first aircraft 204 and the second aircraft 206; executing a mission for the first aircraft 204 and the second aircraft 206; controlling the first aircraft 204 and the second aircraft 206; synchronizing sensor systems measuring altitude within the first aircraft 204 and the second aircraft 206; adjusting routes for the first aircraft 204 and the second aircraft 206; synchronizing altitudes flown by the first aircraft 204 and the second aircraft 206; and adjusting formation flying of the first aircraft 204 and the second aircraft 206. In adjusting the altitude information of the points, altitudes recorded for the points can be adjusted so that coordinates of the points in point clouds generated from different data sets generated by different aircraft are properly aligned.

[0060] As used herein, a "set of" when used with respect to an item means one or more items. For example, a "set of actions 242" is one or more actions 242.

[0061] In certain instances, one or more technical solutions exist that overcome technical challenges associated with generating data for surveying and mapping an area. As a result, the one or more technical solutions may provide the technical effect of enabling corresponding altitudes between two or more aircraft generating data, such as imagery. In this manner, if the altitudes between two or more aircraft generating data sets can be adjusted to be the same, processing of these data sets can occur more quickly. An offset can be used to adjust the altitude at which the aircraft actually fly. Additionally, this instance may also be used to adjust the altitude stored in the metadata of images generated by the aircraft.

[0062] The examples can be used to determine corrections for altitude measurements between two or more aircraft, thereby enabling other types of missions to be performed more efficiently and accurately. For example, in precision flights, such as using drones for air shows, the examples can be used to determine offsets between measurements made between two or more drones, allowing drones flying in formation or performing flight maneuvers to operate with tighter tolerances and distances relative to each other. The examples can be used to perform these and other missions more accurately.

[0063] In one instance, the identification of offset 238 may be used in a process in which multiple aircraft 207 perform a mission in which a data set in the form of images is generated from some of the multiple aircraft 207. For example, first aircraft 204 and second aircraft 206 may generate such images. In addition to these two aircraft, one or more aircraft of multiple aircraft 207 may also operate to generate the images.

[0064] A data set including images generated by the multiple aircraft 207 may then be processed. Additionally, the altitude for at least one of the multiple aircraft 207 may be corrected to increase accuracy when processing the images. For example, the images may be processed to generate a point cloud. With the corrected altitude for one or more of the multiple aircraft 207, the accuracy in the point cloud of the object may be improved compared to current technology. Each set of images from each of the multiple aircraft 207 may be processed to generate multiple point clouds of the object. The object may be, for example, a field, an area of ​​land, a tree, a building, or other suitable object. The point clouds may be combined to form a single point cloud for the object. By correcting the point clouds for altitude, the single point cloud generated for the object may have a desired accuracy.

[0065] Computer system 210 is configured to perform at least one of the steps, operations, or actions described in the various examples using software, hardware, firmware, or a combination thereof. As a result, computer system 210 operates as a special-purpose computer system that enables controller 208 within computer system 210 to operate the airborne imaging system with a desired performance level. In particular, controller 208 transforms computer system 210 into a special-purpose computer system compared to currently available general-purpose computer systems that do not have controller 208.

[0066] The illustration of the aerial imaging environment in FIG. 2 is not meant to imply physical or architectural limitations to the manner in which an exemplary embodiment may be implemented. Other components in addition to or in place of those shown may be used. Some components may be unnecessary. Also, the blocks are presented to illustrate some functional components. When implemented in an exemplary embodiment, one or more of these blocks may be combined, divided into different blocks, or combined and divided.

[0067] For example, controller 208 and computer system 210 are shown as being external to first aircraft 204 and second aircraft 206. In one example, computer system 210 and controller 208 may be located within first aircraft 204. In another example, computer system 210 and controller 208 may be distributed between first aircraft 204 and second aircraft 206. In other words, controller 208 and computer system 210 in different data processing systems may be distributed within aircraft in airborne imaging system 200.

[0068] In yet another example, controller 208 and computer system 210 may be remotely located relative to first aircraft 204 and second aircraft 206, or may be distributed at a location selected from at least one of the remote location, first aircraft 204, or second aircraft 206. The remote location may be, for example, a ground station or another aircraft.

[0069] Furthermore, one or more aircraft may be part of the airborne imaging system 200 in addition to or instead of the first aircraft 204 and the second aircraft 206. These aircraft may be of the same type or different types in different examples.

[0070] 3 is an example block diagram of an aircraft depicted in accordance with an illustrative embodiment, in which aircraft 300 illustrates example components that may be used to implement first aircraft 204 and second aircraft 206.

[0071] In this example, aircraft 300 includes airframe 301, electronics 302, propulsion system 304, and steering system 306. Airframe 301 is the mechanical structure of aircraft 300. Airframe 301 may include, for example, a fuselage, airframe supports, wings, landing gear, and other physical structures. In this example, airframe 301 carries electronics 302, propulsion system 304, and steering system 306.

[0072] The electronics 302 has several different components. As shown, the electronics 302 includes a sensor system 308, a communication system 310, and a flight control system 312.

[0073] In this example, sensor system 308 includes several different components. As shown, sensor system 308 includes an advanced sensor system 314, a camera system 318, and other suitable sensors.

[0074] Altitude sensor system 314 is a physical sensor system and may include software. Altitude sensor system 314 is an example of an altitude sensor system that may be used to implement first altitude sensor system 234 and second altitude sensor system 236 in FIG. 2. Altitude sensor system 314 operates to measure the altitude of aircraft 300. For example, altitude sensor system 314 may include a barometric altimeter, a Global Positioning System (GPS) receiver, or any other suitable type of sensor capable of measuring the altitude of aircraft 300.

[0075] As shown, camera system 318 is a physical system that may include software. Camera system 318 is an example of a camera system that may be used to implement first camera system 216 and second camera system 218 in Figure 2. Camera system 318 may include at least one of a visible light camera, an infrared camera, or some other suitable type of camera.

[0076] Examples of other types of sensors that may be included in sensor system 308 are at least one of a gyroscope, an accelerometer, an angle of attack (AoA) vane, or other suitable types of sensors. These and other sensors in sensor system 308 communicate with flight control system 312. Information generated by sensor system 308 may be used by flight control system 312 to control the operation of aircraft 300 and navigate aircraft 300 along a flight path.

[0077] As shown, propulsion system 304 is a physical system that generates force to move aircraft 300. In this example, the force takes the form of thrust. Propulsion system 304 may include at least one of a jet engine, a turbofan, a turboprop, a ramjet, a rotor system, an electric propulsion system, a hybrid propulsion system, or other suitable engine or propulsion device capable of generating thrust for aircraft 300.

[0078] In this example, steering system 306 may be configured to steer aircraft 300 along a flight path to reach a target point. Steering system 306 may operate autonomously or under manned control. In this example, steering system 306 may respond to signals from flight controls 324 in flight control system 312 and may use feedback or other control systems to guide aircraft 300 along the flight path.

[0079] As shown, communication system 310 is a physical device, such as a radio transceiver and antenna. Communication system 310 can exchange information with a remote computer system. Communication system 310 can operate to send and receive information to a remote location, such as another aircraft or a ground station.

[0080] As shown, flight control system 312 may determine one or more flight paths for aircraft 300 to reach a desired location based on signals received from navigation system components. Flight control system 312 may calculate, generate, and send navigation commands, such as data signals, to steering system 306 to guide aircraft 300 along the flight paths.

[0081] In this example, flight control system 312 includes several components. As shown, flight control system 312 includes a map system 322, a flight controller 324, a processor unit 316, and a memory 326.

[0082] As shown, mapping system 322 may be part of a map-based flight control system that provides location information for natural and man-made features within an area. Map system 322 may communicate with other components within flight control system 312 to support navigation of aircraft 300. This functionality may include providing map information for route calculations, but may also include independent navigation capabilities.

[0083] For example, map system 322 may provide a map-based navigation system that stores a map of the driving environment including one or more objects. The map-based navigation system may be coupled to a camera and configured to determine the vehicle's location by comparing the stored objects with the visible surroundings, and may provide location data in the absence of Global Positioning System (GPS) data or other location information.

[0084] In this example, processor unit 316 is a physical device that may communicate with at least one of map system 322, flight control system 312, sensor system 308, propulsion system 304, or steering system 306, as well as various other components, systems, and subsystems that may be present in aircraft 300. Processor unit 316 may be an internal processor within a computer system that supports various functions, such as navigation functions or image processing functions, such as computer system 210 in FIG. 2. Processor unit 316 may be configured to control the operation of at least one of map system 322, flight control system 312, sensor system 308, propulsion system 304, or steering system 306 of aircraft 300.

[0085] As shown, processor unit 316 may perform processing and computational functions in support of at least one of navigating, identifying key points, determining distances to key points, determining offsets relative to altitudes measured by altitude sensor system 314, or other suitable functions. Processor unit 316 may include several different processors that cooperate to perform the operations of controller 208 of FIG. 2 described herein. For example, an internal processor within aircraft 300 controls the operation of aircraft 300, while another processor assigned to controller 208 controls identifying key points in images, determining distances to key points using the images, comparing a first distance and a first altitude of aircraft 300 to a second distance and a second altitude of a second aircraft to create a comparison, determining an offset between the first altitude and the second altitude, and performing one or more actions using the offset.

[0086] Flight controls 324 may operate to control components within aircraft 300, such as flight control system 312, sensor system 308, propulsion system 304, or steering system 306. Flight controls 324 communicates with processor unit 316, aircraft 300, flight control system 312, sensor system 308, steering system 306, and various other components of the devices and systems described herein.

[0087] As shown, flight controls 324 may include any hardware, software, or any combination thereof for controlling the various components within aircraft 300 and flight control system 312 described herein, including, but not limited to, microprocessors, microcontrollers, application specific integrated circuits, programmable gate arrays, and any other digital and / or analog components, and combinations thereof, along with inputs and outputs for communicating control signals, drive signals, power signals, sensor signals, and other suitable types of signals.

[0088] In this example, memory 326 is a physical device and may include a local memory or a remote storage device that stores a log of data for flight control system 312, including, but not limited to, images, determined distances, identification of key points, locations of key points, altitude, and aircraft status information generated by sensor system 308. Other information that may be stored in memory 326 includes at least one of direction, speed, flight path, steering specifications, global positioning system coordinates, sensor readings, and other suitable information. Information stored in memory 326 may be accessed by at least one of processor unit 316 or flight control device 324, in this example.

[0089] 3, this component placement may vary in other examples. For example, sensors of sensor system 308 may be located in or on aircraft 300. Further, at least one of sensor system 308, flight control system 312, or communication system 310 may share components, such as memory, sensors, processors, or controllers. Additionally, one or more sensors of sensor system 308 may be removably coupled to aircraft 300, or components within this module may be integrated into airframe 301 for aircraft 300 in any desired manner.

[0090] Flight control system 312 may also include the components described above as being part of electronics 302, as well as other sensors in sensor system 308. As another example, the other sensors may also include other flight instrumentation, processing circuitry, communications circuitry, optical systems including cameras and other sensors necessary or useful in the operation of an unmanned aerial system or other autonomous or manually piloted aircraft. Thus, the placement of the various components may be configured as desired by a designer or operator and, therefore, is not limited to the specific examples described or shown herein.

[0091] 4 is an example block diagram of an aircraft system 400 in accordance with an illustrative embodiment. In this diagram, aircraft system 400 illustrates example components that may be implemented in at least one of first aircraft 204 or second aircraft 206 in FIG. 2 .

[0092] As shown, aircraft system 400 includes an aircraft processor 424 communicatively coupled to at least one of a memory device 428, a flight control unit 426, a wireless transceiver 432, and a navigation system 442. Aircraft processor 424 may be configured to perform one or more operations based, at least in part, on instructions (e.g., software) and one or more databases stored in memory device 428 (e.g., a hard drive, flash memory, etc.). Aircraft system 400 may further include other desired components, such as a wireless transceiver 432 coupled to an antenna 434, to communicate data between aircraft system 400 and a remote device 438. In this example, remote device 438 may be a human-machine interface or another portable electronic device, such as a smartphone, tablet, or laptop computer, or a controller at a location such as a base station. Aircraft system 400 may also communicate with another aircraft via wireless transceiver 432, for example, to facilitate cooperative operations.

[0093] In some aspects, aircraft system 400 can exchange data (e.g., processed data, raw data, etc.) with remote device 438 and / or other equipment on the aircraft over network 436. In some aspects, wireless transceiver 432 can be configured to communicate using one or more wireless standards, such as Bluetooth (e.g., short-wavelength ultra-high frequency (UHF) radio waves in the 2.4-2.485 GHz Industrial, Scientific, and Medical (ISM) band), near field communication (NFC), Wi-Fi (e.g., the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standard), etc. Remote device 438 can facilitate monitoring and / or control of aircraft system 400 and its payload, including ISR payload 440.

[0094] Aircraft processor 424 is operatively coupled to flight controls 426 and may control the operation of various of actuators 430 (e.g., those controlling the movement and locking of any flight surface, such as at least one of the movable flight control surfaces, electric motor 404 via electronic speed controller (ESC) 415, electric motor 406 via electronic speed controller (ESC) 416, electric motor 408 via electronic speed controller (ESC) 417, or engine 418 via engine controller 422) in response to commands from an operator, an autopilot, a navigation system 442, or other higher-level systems via radio transceiver 432. In some aspects, aircraft processor 424 and flight controls 426 may be integrated into a single component or circuit. During operation, flight controls 426 may dynamically (i.e., in real time or near real time) and independently adjust thrust during various phases of flight via ESCs 416 or engine controls 422 (as the case may be) to control the roll, pitch, or yaw of first air vehicle 204 or second air vehicle 206 in Figure 2. If rotors (e.g., propellers) having rotor blades are used, flight controls 426 may vary the revolutions per minute (RPM) of the rotors and / or, if desired, the pitch of the rotor blades. For example, electric motors 408 may be controlled by adjusting the power provided to each electric motor from a power source (e.g., a battery pack or battery bank) via ESCs 416.

[0095] The aircraft processor 424 may be operatively coupled to a navigation system 442, which may include a GPS 442a communicatively coupled to an INS 442b and / or an IMU 442c, which may include one or more gyros and accelerometers. The GPS 442a provides absolute drift-free position values ​​that can be used to reset the INS solution or blended with the INS solution by using a mathematical algorithm such as a Kalman filter. The navigation system 442 may communicate inertial stabilization data to the aircraft processor 424, among other things.

[0096] To gather data and / or monitor the area, flight control system 312 may be equipped with additional ones of sensors 410 (e.g., ISR payload 440) including, for example, one or more cameras 440a (e.g., optical instruments for recording or capturing images and / or video, including light detection and ranging (LiDAR) devices), audio devices 440b (e.g., microphones, echolocation sensors, etc.), and other sensors 440c for facilitating ISR functions and providing ISR data (e.g., photographs, video, audio, sensor readings, etc.). ISR payload 440 is operably coupled to aircraft processor 424 to facilitate communication of ISR data (sensor data) between ISR payload 440 and aircraft processor 424. The ISR data may be used to navigate and / or otherwise control aircraft system 400. In certain aspects, ISR payload 440 may be rotatably and pivotally coupled to, for example, the underside of the airframe (or another structural component, such as a rotor boom or wing) via a gimbal system so that ISR payload 440 can more easily point downward to monitor objects below and / or on the ground. Data may be dynamically or periodically communicated over network 436 from aircraft system 400 via wireless transceiver 432 to remote device 438, or may be stored in memory device 428 for later access or processing.

[0097] 5, an example of image-based triangulation is shown, according to an exemplary embodiment. In this example, the triangulation is similar to stereo triangulation, in which images are taken at different positions by an imaging system. In this example, a stereo camera system is not required.

[0098] As shown, a first image is taken by the aircraft at a first position 500 and a second image is taken at a second position 502. In this illustrated example, these images are similar to the right and left images taken by a stereo imaging system.

[0099] As shown, the two images are taken by an aircraft on flight plane 504, which is the plane of the aircraft traveling on flight plane 504 in the direction of line 505. The images include key points 506 on the ground plane 508.

[0100] These images can be used to perform triangulation to determine distance Z 510 from a point on the ground 512 to the key point 506. Distance Z 510 is the distance from a point to a line. This distance is the shortest distance from a given point to any point on the line. As a result, the measured distance is perpendicular to the line.

[0101] In this example, image plane 514 is a plane passing through the focal lengths for images taken at first location 500 and second location 502. In this illustrated example, focal length f 516 is the focal length of the imaging system in the aircraft at point 512. Image plane 514 is separated from flight plane 504 by imaging system focal length f 516. In this example, point XL 522 and point XR 524 are points on image plane 514. Distance Di 515 is the distance between point XL 522 and point XR 524 on image plane 514.

[0102] As shown, distance h 530 is the altitude, in this example, the distance from the flight plane 504 to the ground. As shown, d 532 is the distance from key point 506 to distance h 530. As shown in Figure 5, this distance is the shortest distance from a given point, key point 506, to any point on the line, distance h 530. In this example, d 532 can be determined from extrapolating the distance between pixels based on the resolution of the camera system.

[0103] In this example, the distance Z 510 from point 512 to keypoint 506 can be referred to as the depth. Distance Z 510 can be calculated as follows: Z=hf / Di where Z is the distance Z 510, h is the distance h 530, f is the focal length f 516, and Di is the distance Di 515. Using Z, the value of the distance h 530 can be determined as follows: Z2=d2h2

[0104] As a result, the measured altitude can be compared to the calculated altitude (h 530). This comparison can be used to determine an offset. This offset can then be used to calibrate the sensors measuring altitude within the aircraft. This type of ranging and calibration can be performed by each aircraft in the fleet.

[0105] In another example, the measured altitude from a first aircraft may match the calculated altitude, while the measured altitude from a second aircraft may not match the calculated altitude. In this case, the measured altitude of the second aircraft can be corrected so that the altitude at which the operation is performed is the expected altitude for both aircraft. In another example, both measured altitudes may be inaccurate compared to the calculated altitude. In this case, the offset for both aircraft can be identified and used to calibrate or correct the altitudes of the aircraft.

[0106] In another example, a first aircraft may calculate a first distance Z1 to a keypoint and measure a first altitude ma1. A second aircraft may calculate a second distance Z2 to the keypoint and measure a second altitude ma2. One of the measured altitudes, e.g., ma1, may be assumed to be correct for calibration purposes. The ratio of the measured altitude to the distance to the keypoint may be used to determine the offset. For example, ma2 = Z2ma1 / z1, where ma2 is the offset.

[0107] Referring now to FIG. 6, an example flow diagram of a process for operating an airborne imaging system is shown in accordance with an illustrative embodiment. The process of FIG. 6 may be implemented in hardware, software, or both. If implemented in software, the process may take the form of program code executed by one or more processor units located in one or more hardware devices in one or more computer systems. For example, the process may be implemented in controller 208 in computer system 210 of FIG. 2. In this example, controller 208 in computer system 210 may be located in at least one of the first aircraft, the second aircraft, a remote location, or some other aircraft or vehicle.

[0108] The process begins by determining a first distance from the first aircraft to the keypoint using a first image of the keypoint generated by the first aircraft during the first aircraft's flight and stereo depth triangulation (operation 600). Operation 600 may determine the keypoint using an object recognition process that may be implemented by at least one of an artificial intelligence system, a machine learning model, a computer vision system, a genetic algorithm, an interpretation tree, or other suitable technique or process.

[0109] The process compares a first distance from a first aircraft to the key point and a first altitude of the first aircraft measured by the first aircraft with a second distance from a second aircraft to the key point and a second altitude of the second aircraft measured by the second aircraft to generate a comparison (operation 602).

[0110] The process uses the comparison to determine an offset between the first altitude of the first aircraft and the second altitude measured by the second aircraft (operation 604). The process then terminates. This offset may be used to adjust the altitude of the first aircraft. The adjustment may be made by adjusting the altitude at which the first aircraft flies. In another example, the altitude may be adjusted in metadata associated with the image generated by the first aircraft.

[0111] 7, a flow diagram of a process for performing an action using an offset is shown in accordance with an example embodiment. This flow diagram illustrates additional operations that may be performed in the process of the flow diagram shown in FIG.

[0112] The process begins by performing a set of actions using the offset (operation 700), which is performed after the offset is identified in operation 604 of Figure 6. The process then ends.

[0113] In this exemplary embodiment, the set of operations performed in operation 700 is selected from at least one of: adjusting altitude information of points in point clouds obtained from images of the surrounding environment generated by the first aircraft and the second aircraft; controlling the missions of the first aircraft and the second aircraft; adjusting the routes of the first aircraft and the second aircraft; synchronizing the altitudes at which the first aircraft and the second aircraft fly; coordinating formation flying of the first aircraft and the second aircraft; or other suitable action.

[0114] 8, a flow diagram of a process for determining distance to keypoints is shown in accordance with an illustrative embodiment. The process shown in this figure is an example of one way in which operation 600 of FIG. 6 may be performed.

[0115] The process begins by identifying keypoints in an image (operation 800). In this example, at least two images are used for this determination, each of which is generated at a different location.

[0116] The process identifies pixels of keypoints in these images (operation 802). These pixels may be for one or more features of the keypoint. For example, the pixels may be a line of pixels along the bottom or side of the keypoint. In another example, the pixels may be the circumference of a region within the keypoint.

[0117] The process performs stereo depth triangulation using the identified pixels for the keypoints in the images (operation 804), after which the process ends. The result of operation 804 is the distance from the aircraft to the keypoints.

[0118] Referring now to Figure 9, an example flow diagram of a process performed by an aircraft to generate information used to operate an airborne imaging system is shown in accordance with an illustrative embodiment. The process of Figure 9 may be implemented in hardware, software, or both. If implemented in software, the process may take the form of program code executed by one or more processor units located in one or more hardware devices in one or more computer systems. For example, the process may be implemented in second aircraft 206 of Figure 2. For example, controller 208 and computer system 210 may be distributed such that some operations are performed in second aircraft 206.

[0119] The process of Figure 9 can be performed using determined or known parameters for both aircraft. These parameters include focal length, field of view, image sensor dimensions, number of pixels, latitude, longitude, speed, altitude, vehicle orientation, and camera orientation. These parameters can be used to perform triangulation, as described above with respect to Figure 5. For example, focal length and stereo triangulation variables can be used to calculate distances from a pair of images to points such as landmarks.

[0120] The field of view is used to determine whether a given point, such as a landmark, is visible in several images. If the field of view is known, it is possible to predict the area on the ground that will be captured. This parameter can be used to set up routes for two aircraft in which overlapping images are captured.

[0121] The image sensor dimensions and pixel count are values ​​that can be used to calculate points XL 522 and XR 524 in Figure 5. The image sensor dimensions are used to determine the angular distance of the key points from the centerline.

[0122] The process begins by generating an image (operation 900). The process identifies dominant features on the ground in the image (operation 902). In operation 902, dominant features are found in two or more images. Dominant features can take several different forms. For example, dominant features could be road signs, objects, or patterns made by people on the ground.

[0123] The process involves measuring the pixel length of a key feature in an image, act 904. The pixel length of a key feature may be constant or may vary from image to image.

[0124] The process involves performing stereo depth triangulation using two consecutive images of the keypoint to determine the distance from the aircraft to the keypoint (operation 906). In operation 906, the stereo depth triangulation may also use the altitude measured by the aircraft.

[0125] The process transmits the location of the key features, the identification of the key features, the distance of the key features, and the measured altitude for the second aircraft to the first aircraft, which also uses these locations to generate an image (operation 908), with the process then terminating.

[0126] Referring now to Figure 10, an example flow diagram of a process for operating an airborne imaging system is shown in accordance with an illustrative embodiment. The process of Figure 10 may be implemented in hardware, software, or both. If implemented in software, the process may take the form of program code executed by one or more processor units located in one or more hardware devices in one or more computer systems. For example, the process may be implemented in first aircraft 204 of Figure 2. For example, controller 208 in computer system 210 may be distributed such that some operations are performed in first aircraft 204.

[0127] The process begins by receiving key feature locations, key feature identification, key feature distance, and a measured altitude for the second aircraft (operation 1000). The information received from the second aircraft is used in a process to determine an offset in the altitude measured by the first aircraft. In other words, even if the measurements show both aircraft at the same altitude, the actual altitudes may be different. This information is used to determine an offset between the actual altitude of the aircraft based on the measured altitude of that aircraft. In one example, the offset assumes that the measurement of the first aircraft is the correct altitude and that the second aircraft has an error that must be adjusted.

[0128] The process then generates images of the identified key features at the locations received from the second aircraft (operation 1002). In operation 1002, these images may be generated when the first aircraft flies close enough to the locations of the keypoints to generate images of the keypoints. In one instance, the flight to this location may be part of a predetermined flight path. In another instance, the first aircraft may change its flight path to generate images of the keypoints.

[0129] The process includes performing stereo depth triangulation on the keypoints using images generated by the first aerial vehicle (operation 1004). In operation 1004, a first distance to the keypoints of the first aerial vehicle is determined from performing the stereo depth triangulation.

[0130] The process includes determining a first ratio of a first distance to a key point for a first aircraft to an altitude measured by the first aircraft (operation 1006). The process includes determining a second ratio of a second distance to a key point for a second aircraft to an altitude measured for the second aircraft (operation 1008). The process includes comparing the first ratio to the second ratio to determine an offset between a first altitude of the first aircraft and a second altitude of the second aircraft (operation 1010). For example, in operation 1010, the ratio may be: h_b / h_a=i_a / i_b

[0131] Where ha = pixels of the keypoint feature covered by drone A's camera sensor; hb = number of pixels of the keypoint feature covered by drone B's camera sensor; ia = calculated altitude of drone A; ib = measured altitude of drone B. In this way, the offset of ib = measured altitude of drone B from ib = calculated altitude of drone A can be determined.

[0132] The process adjusts the altitude of the first aircraft so that their altitude ratio matches that of the second aircraft (operation 1012). The process then ends. This process results in the measured altitudes of both aircraft being the same. This process does not mean that these altitudes are necessarily practical, and both have the same error or inaccuracy.

[0133] 11 , an example flow diagram of a process for operating an aerial imaging system is shown in accordance with an illustrative embodiment. The process of FIG. 11 may be implemented in hardware, software, or both. If implemented in software, the process may take the form of program code executed by one or more processor units located in one or more hardware devices in one or more computer systems. For example, the process may be implemented in controller 208 in computer system 210 of aerial imaging system 200 of FIG. 2. In this example, controller 208 in computer system 210 may be located in at least one of the first aircraft, the second aircraft, a remote location, or some other aircraft or vehicle.

[0134] The process begins by determining an altitude of the first aircraft to the keypoints using stereo depth triangulation with a first image of the keypoints generated by the first aircraft (operation 1100). The process compares a first altitude determined for the first aircraft using stereo depth triangulation with the first image to a second altitude determined for the second aircraft using either stereo depth triangulation with a second image of the keypoints generated by the second aircraft or measurements made by a sensor system on board the second aircraft, generating a comparison (operation 1102).

[0135] The process uses the comparison to determine an offset between the first altitude determined for the first aircraft and the second altitude measured by the second aircraft (operation 1104), with the process then terminating.

[0136] The offset determined in operation 1104 may be used to perform a set of actions, which may be selected from at least one of: adjusting altitude information of points in point clouds obtained from images of the surrounding environment generated by the first and second aircraft; controlling the missions of the first and second aircraft; adjusting the routes of the first and second aircraft; synchronizing the altitudes at which the first and second aircraft fly; synchronizing sensor systems that measure the altitudes of the first and second aircraft; adjusting the formation flying of the first and second aircraft; or other suitable actions.

[0137] The flow diagrams and block diagrams of different depicted embodiments illustrate the architecture, functionality, and operation of some possible implementations of apparatus and methods in example embodiments. In this regard, each block in a flow diagram or block diagram may represent at least one of a module, a segment, a function, or a portion of an operation or step. For example, one or more blocks may be implemented as program code, hardware, or a combination of program code and hardware. If implemented in hardware, the hardware may take the form of, for example, an integrated circuit that is manufactured or configured to perform one or more processes in the flow diagrams or block diagrams. If implemented as a combination of program code and hardware, the implementation may take the form of firmware. Each block in a flow diagram or block diagram may be implemented using a dedicated hardware system performing various processes or various combinations of dedicated hardware and program code executed by the dedicated hardware.

[0138] In some alternative implementations of the exemplary embodiments, one or more functions noted in the blocks may occur out of the order noted in the figures. For example, in some cases, two blocks shown in succession may be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. Also, other blocks may be added in addition to the blocks shown in a flow diagram or block diagram.

[0139] Referring now to Figure 12, a block diagram of a data processing system is depicted in accordance with an illustrative embodiment. Data processing system 1200 may be used to implement computer system 210 in Figure 2. In this example, data processing system 1200 includes a communications framework 1202 that provides communications between a processor unit 1204, a memory 1206, a persistent storage device 1208, a communications unit 1210, input / output (I / O) devices 1212, and a display 1214. In this example, communications framework 1202 takes the form of a bus system.

[0140] Processor unit 1204 is responsible for executing instructions for software that may be loaded into memory 1206. Processor unit 1204 includes one or more processors. For example, processor unit 1204 may be selected from at least one of a central processing unit (CPU), a graphics processing unit (GPU), a physics processing unit (PPU), a digital signal processor (DSP), a network processor, or other suitable types of processor. Furthermore, processor unit 1204 may be implemented using one or more heterogeneous processor systems in which a main processor resides on a single chip with secondary processors. As another example, processor unit 1204 may be a symmetric multiprocessor system in which several processors of the same type are included on a single chip.

[0141] Memory 1206 and persistent storage 1208 are examples of storage device(s) 1216. A storage device is any hardware that can store information (e.g., but not limited to, data, program code in functional form, or other suitable information) on a temporary and / or persistent basis. Storage device 1216, in these examples, may also be referred to as a computer-readable storage device. In these examples, memory 1206 may be, for example, a random access memory or any other suitable volatile or non-volatile storage device. Persistent storage 1208 may take various forms, depending on the particular implementation.

[0142] For example, persistent storage 1208 may comprise one or more components or devices. For example, persistent storage 1208 may be a hard drive, a solid-state drive (SSD), a flash memory, a rewritable optical disk, a rewritable magnetic tape, or some combination of the above. The medium used by persistent storage 1208 may also be removable. For example, a removable hard drive may be used for persistent storage 1208.

[0143] In these examples, communications unit 1210 provides for communication with other data processing systems or devices. In these examples, communications unit 1210 is a network interface card.

[0144] Input / output devices 1212 allow for the input and output of data with other devices that may be connected to data processing system 1200. For example, input / output devices 1212 may provide a connection for user input through at least one of a keyboard, a mouse, or some other suitable input device. Additionally, input / output devices 1212 may send output to a printer. Display 1214 provides a mechanism for displaying information to a user.

[0145] Instructions for at least one of the operating system, applications, or programs may be located in storage device(s) 1216, which are in communication with processor unit 1204 through communications framework 1202. The processes of the different embodiments may be executed by processor unit 1204 using computer-implemented instructions that may be located in a memory, such as memory 1206.

[0146] These instructions are referred to as program code, computer usable program code, or computer readable program code, which may be read and executed by a processor in processor unit 1204. In different embodiments, the program code may be embodied in different physical or computer readable storage media, such as memory 1206 or persistent storage 1208.

[0147] Program code 1218 may be loaded onto or transferred to data processing system 120 for execution by processor unit 1204 and may be located in a functional form on a removable computer readable medium 1220. In these examples, program code 1218 and computer readable medium 1220 form computer program product 1222. In the example, computer readable medium 1220 is computer readable storage medium 1224.

[0148] In these examples, computer readable storage medium 1224 is not a medium that propagates or transmits program code 1218, but rather a physical or tangible storage device used to store program code 1218. Computer readable storage medium 1220, as used herein, is not itself to be interpreted as a transitory signal such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave or other transmission medium propagating through a waveguide or other transmission medium (e.g., an optical pulse passing through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0149] Alternatively, program code 1218 may be transferred to data processing system 1200 using a computer readable signal medium. The computer readable signal medium may be, for example, a propagated data signal containing program code 1218. For example, the computer readable signal medium may be at least one of an electromagnetic signal, an optical signal, or any other suitable type of signal. These signals may be transmitted over a connection, such as a wireless connection, an optical fiber cable, a coaxial cable, a wire, or any other suitable type of connection.

[0150] Additionally, as used herein, “computer-readable medium 1220” may refer to either the singular or the plural. For example, program code 1218 may be located in computer-readable medium 1220 in the form of a single storage device or storage system. In another example, program code 1218 may be located in computer-readable medium 1220 distributed across several data processing systems. In other words, some instructions in program code 1218 may be located in one data processing system, while other instructions in program code 1218 may be located within a single data processing system. For example, a portion of program code 1218 may be located in computer-readable medium 1220 in a server computer, while another portion of program code 1218 may be located in computer-readable medium 1220 located on a set of client computers.

[0151] The different components illustrated for data processing system 1200 are not meant to provide architectural limitations to the manner in which different embodiments may be implemented. In some instances, one or more of the components may be incorporated into or otherwise form a part of another component. For example, in some instances, memory 1206, or portions thereof, may be integrated within processor unit 1204. The different illustrative embodiments may be implemented in a data processing system including components in addition to or instead of those illustrated for data processing system 1200. Other components illustrated in FIG. 12 may differ from the illustrated example. The different embodiments may be implemented using any hardware device or system capable of running program code 1218.

[0152] Thus, examples provide a method, apparatus, system, and computer program product for operating an airborne imaging system. Using a first image of a keypoint generated by a first aircraft during a flight of the first aircraft and stereo depth triangulation, a first distance from the first aircraft to the keypoint is determined by a computer system. To create a comparison, the computer system compares the first distance from the first aircraft to the keypoint and a first altitude of the first aircraft measured by the first aircraft with a second distance from a second aircraft to the keypoint and a second altitude of the second aircraft measured by the second aircraft. Using this comparison, the computer system determines an offset between the first altitude of the first aircraft and the second altitude measured by the second aircraft, where the offset is used to adjust the altitude of the first aircraft.

[0153] By determining the offset, altitude adjustments can be made to at least one of the first aircraft or the second aircraft to perform various missions more accurately or with less processing resource usage. For example, if the altitudes are more closely correlated with each other, images and data sets generated between different aircraft can be processed more quickly to generate a point cloud of an object of interest. In other instances, altitudes can be adjusted so that precision flights or other missions can be performed more accurately and safely.

[0154] The description of different exemplary embodiments is presented for purposes of illustration and description and is not intended to be exhaustive or limited to the disclosed forms of embodiments. Various examples describe components that perform actions or operations. In some examples, the components may be configured to perform the described actions or operations. For example, a component may have a configuration or design of structure that provides the component with the ability to perform the actions or operations described as being performed by the component in the examples. Furthermore, to the extent that the terms "including," "has," "contains," and variations thereof are used herein, such terms are intended to be inclusive in the same manner as the open transitional phrase "comprises," without excluding additional or other elements.

[0155] Additionally, the present disclosure includes embodiments according to the following clauses:

[0156] Clause 1. An airborne imaging system (200), comprising: a computer system (210); a controller (208) within a computer system (210), said controller (208) comprising: determining a first altitude (226) of a first aircraft (204) using a first image (212) and stereo depth triangulation (224) of keypoints (220) generated by the first aircraft (204); comparing the first altitude (226) with a second altitude (230) determined for the second aircraft (206) by either the stereo depth triangulation (224) using a second image (214) of the keypoints (220) generated by a second aircraft (206) or measurements made by a sensor system (308) of the second aircraft (206) to generate a comparison (232); and an airborne imaging system (200) configured to use the comparison (232) to determine an offset (238) between the first altitude (226) determined for the first aircraft (204) and the second altitude (230) measured by the second aircraft (206).

[0157] Clause 2. The aerial imaging system (200) of clause 1, wherein the controller (208) is configured to perform a set of actions (242) using the offset (238).

[0158] Clause 3. The airborne imaging system (200) described in Clause 2, wherein the set of actions (242) is selected from at least one of: adjusting altitude information of points in a point cloud obtained from images of the surrounding environment generated by the first aircraft (204) and the second aircraft (206); controlling the missions of the first aircraft (204) and the second aircraft (206); adjusting the routes of the first aircraft (204) and the second aircraft (206); synchronizing the altitudes at which the first aircraft (204) and the second aircraft (206) fly; synchronizing sensor systems that measure the altitudes of the first aircraft (204) and the second aircraft (206); or adjusting the formation flying of the first aircraft (204) and the second aircraft (206).

[0159] Clause 4. An aerial imaging system (200) described in any one of the preceding clauses, wherein the controller (208) is configured to identify the key points (220) in the first image (212) generated by a first camera system (216) associated with the first aircraft (204).

[0160] Clause 5. An aerial imaging system (200) described in any one of the preceding clauses, wherein the stereo depth triangulation (224) is performed using pixels of the key point (220) in the first image (212) or the second image (214).

[0161] Clause 6. The aerial imaging system (200) of any one of the preceding clauses, wherein the first aircraft (204) and the second aircraft (206) are selected from at least one of an airplane, an unmanned aerial system, an unmanned aerial vehicle, a drone, a rotorcraft, or a spacecraft.

[0162] Clause 7. The aerial imaging system (200) of any one of the preceding clauses, wherein the key point (220) is one of an area of ​​land, a mountain, a tower, a building, a lake, a dam, a marker, a pattern, and a VOR antenna.

[0163] Clause 8. In the airborne imaging system (200), a first camera system (216) mounted on the first aircraft (204), the first camera system (216) generating first images (212) during flight of the first aircraft (204); an altitude sensor system (234, 314) onboard the first aircraft (204), the altitude sensor system (234, 314) measuring a first altitude (226) of the first aircraft (204); a controller (208) for the first aircraft (204), determining, during flight of the first aircraft (204), a first distance (222) from the first aircraft (204) to the keypoint (220) using the first image (212) of the keypoint (220) generated by the first camera system (216) associated with the first aircraft (204) and stereo depth triangulation (224); receiving from a second aircraft (206) a second distance (228) from said second aircraft (206) to said key point (220) and a second altitude (230) measured for said second aircraft (206); comparing the first distance (222) and the first altitude (226) from the first aircraft (204) to the key point (220) with the second distance (228) and the second altitude (230) from the second aircraft (206) to the key point (220) to generate a comparison (232); and a controller (208) configured to use the comparison (232) to determine an offset (238) between the first altitude (226) of the first aircraft (204) and the second altitude (230) of the second aircraft (206), the offset (238) being used to adjust the altitude of the first aircraft (204); and An airborne imaging system (200) comprising:

[0164] Clause 9. The aerial imaging system (200) of clause 8, further comprising performing a set of actions (242) using the offset (238).

[0165] Clause 10. The aerial imaging system (200) described in Clause 9, wherein the set of actions (242) is selected from at least one of: adjusting altitude information of points in a point cloud obtained from images of the surrounding environment generated by the first aircraft (204) and the second aircraft (206); controlling the missions of the first aircraft (204) and the second aircraft (206); adjusting the routes of the first aircraft (204) and the second aircraft (206); synchronizing the altitudes at which the first aircraft (204) and the second aircraft (206) fly; synchronizing sensor systems that measure the altitudes of the first aircraft (204) and the second aircraft (206); or adjusting the formation flying of the first aircraft (204) and the second aircraft (206).

[0166] Clause 11. An aerial imaging system (200) described in any one of clauses 8 to 10, further comprising identifying the key points (220) within the first image (212) generated by the first camera system (216).

[0167] Clause 12. An aerial imaging system (200) described in any one of clauses 8 to 11, wherein the first distance (222) is determined using stereo depth triangulation (224) performed using a first pixel (223) of the keypoint (220) in the first image (212), and the second distance (228) is determined using stereo depth triangulation (224) performed using a second pixel (229) of the keypoint (220) in the second image (214).

[0168] Clause 13. An aerial imaging system (200) as described in any one of clauses 8 to 12, wherein the altitude sensor system (234, 314) includes at least one of a barometric pressure sensor or a global positioning system receiver.

[0169] Clause 14. A method of operating an airborne imaging system (200), comprising: determining (600) a first altitude (226) of the first aircraft (204) using a first image (212) and stereo depth triangulation (224) of keypoints (220) generated by the first aircraft (204) during flight of the first aircraft (204); comparing (602) by the computer system (210) the first altitude (226) with a second altitude (230) of the second aircraft (206) determined by the second aircraft (206) to generate a comparison (232); determining (604) an offset (238) between the first altitude (226) and the second altitude (230) by the computer system (210) using the comparison (232); adjusting, by the computer system (210), at least one of the first altitude (226) or the second altitude (230) based on the offset (238); and The computer system (210) obtains multiple images of an area from the first aircraft (204) at the first altitude (226) and from the second aircraft (206) at the second altitude (230). A method comprising:

[0170] Clause 15. The method of clause 14, further comprising performing (700), by the computer system (210), a set of actions (242) using the offset (238).

[0171] Clause 16. The method of clause 15, wherein the set of actions (242) is selected from at least one of: adjusting altitude information of points in a point cloud obtained from images of the surrounding environment generated by the first aircraft (204) and the second aircraft (206); controlling the missions of the first aircraft (204) and the second aircraft (206); adjusting the routes of the first aircraft (204) and the second aircraft (206); synchronizing the altitudes at which the first aircraft (204) and the second aircraft (206) fly; synchronizing sensor systems that measure the altitudes of the first aircraft (204) and the second aircraft (206); or adjusting the formation flying of the first aircraft (204) and the second aircraft (206).

[0172] Clause 17. The method of any one of clauses 14 to 16, further comprising identifying, by the computer system (210), the key points (220) in the first image (212) generated by a first camera system (216) associated with the first aircraft (204).

[0173] Clause 18. The method of any one of clauses 14 to 17, wherein the second altitude (230) is determined using stereo depth triangulation (224) performed using a second image (214) of the key point (220) generated by the second aircraft (206).

[0174] Clause 19. The method of any one of clauses 14 to 18, wherein the second altitude (230) is determined using an altitude sensor system (236, 314) associated with the second aircraft (206).

[0175] Clause 20. The method of any one of clauses 14 to 19, wherein the first aircraft (204) and the second aircraft (206) are selected from at least one of an airplane, an unmanned aerial system, an unmanned aerial vehicle, a drone, a rotorcraft, or a spacecraft.

[0176] Clause 21. A computer program product (1222) for operating an airborne imaging system (200), comprising: a computer-readable storage medium (1224); first program code stored on the computer-readable storage medium and executable by a computer system (210), the first program code causing the computer system (210) to determine a first distance (222) from the first aircraft (204) to the keypoint (220) using a first image (212) of the keypoint (220) generated by the first aircraft (204) during flight of the first aircraft (204) and stereo depth triangulation (224); second program code stored on the computer-readable storage medium and executable by the computer system (210), the second program code causing the computer system (210) to compare the first distance (222) from the first aircraft (204) to the key point (220) and a first altitude (226) of the first aircraft (204) measured by the first aircraft (204) with a second distance (228) from a second aircraft (206) to the key point (220) and a second altitude (230) of the second aircraft (206) measured by the second aircraft (206) to generate a comparison (232); and and third program code stored on the computer-readable storage medium and executable by the computer system, the third program code causing the computer system to use the comparison to determine an offset between the first altitude of the first aircraft and the second altitude measured by the second aircraft, the offset being used to adjust the altitude of the first aircraft. A computer program product (1222) comprising:

[0177] Numerous modifications and variations will be apparent to those skilled in the art. Furthermore, different exemplary embodiments may offer different features as compared to other preferred embodiments. The selected embodiment or embodiments have been chosen and described in order to best explain the principles, practical applications of the embodiments, and to enable others skilled in the art to appreciate the disclosure of the various embodiments, including various modifications suited to the particular use contemplated.

Claims

1. In an airborne imaging system (200), a first camera system (216) mounted on the first aircraft (204), the first camera system (216) generating a first image (212) during flight of the first aircraft (204); an altitude sensor system (234, 314) onboard the first aircraft (204), the altitude sensor system (234, 314) measuring a first altitude (226) of the first aircraft (204); a controller (208) for the first aircraft (204), determining a first distance (222) from the first aircraft (204) to the keypoint (220) using the first image (212) of the keypoint (220) generated by the first camera system (216) associated with the first aircraft (204) during flight of the first aircraft (204) and stereo depth triangulation (224); receiving, from a second aircraft (206), a second distance (228) from the second aircraft (206) to the key point (220) and a second altitude (230) measured for the second aircraft (206); comparing the first distance (222) and the first altitude (226) from the first aircraft (204) to the key point (220) with the second distance (228) and the second altitude (230) from the second aircraft (206) to the key point (220) to generate a comparison (232); and a controller configured to use the comparison to determine an offset between the first altitude of the first aircraft and the second altitude of the second aircraft, the offset being used to adjust the altitude of the first aircraft; and An airborne imaging system (200) comprising:

2. The aerial imaging system (200) of claim 1, further comprising performing a set of actions (242) using the offset (238).

3. 3. The airborne imaging system of claim 2, wherein the set of actions is selected from at least one of: adjusting altitude information of points in point clouds obtained from images of the surrounding environment generated by the first aircraft and the second aircraft; controlling missions of the first aircraft and the second aircraft; adjusting routes of the first aircraft and the second aircraft; synchronizing altitudes at which the first aircraft and the second aircraft fly; synchronizing sensor systems that measure altitudes of the first aircraft and the second aircraft; or adjusting formation flying of the first aircraft and the second aircraft.

4. The aerial imaging system (200) of any one of claims 1 to 3, further comprising identifying the key points (220) in the first image (212) produced by the first camera system (216).

5. 5. The aerial imaging system (200) of claim 1, wherein the first distance (222) is determined using stereo depth triangulation (224) performed using a first pixel (223) of the keypoint (220) in the first image (212), and the second distance (228) is determined using stereo depth triangulation (224) performed using a second pixel (229) of the keypoint (220) in the second image (214).

6. The airborne imaging system (200) of any one of claims 1 to 5, wherein the altitude sensor system (234, 314) includes at least one of a barometric pressure sensor or a global positioning system receiver.

7. A method of operating an airborne imaging system (200), comprising: determining (600) a first altitude (226) of the first aircraft (204) using a first image (212) and stereo depth triangulation (224) of key points (220) generated by the first aircraft (204) during flight of the first aircraft (204); comparing (602) by the computer system (210) the first altitude (226) with a second altitude (230) of the second aircraft (206) determined by the second aircraft (206) to generate a comparison (232); determining (604) an offset (238) between the first altitude (226) and the second altitude (230) using the comparison (232); adjusting, by the computer system (210), at least one of the first altitude (226) or the second altitude (230) based on the offset (238); and The computer system (210) obtains multiple images of an area from the first aircraft (204) at the first altitude (226) and from the second aircraft (206) at the second altitude (230). A method comprising:

8. The method of claim 7, further comprising performing, by the computer system, a set of actions using the offset.

9. 9. The method of claim 8, wherein the set of actions (242) is selected from at least one of: adjusting altitude information of points in point clouds obtained from images of the surrounding environment generated by the first aircraft (204) and the second aircraft (206); controlling missions of the first aircraft (204) and the second aircraft (206); adjusting routes of the first aircraft (204) and the second aircraft (206); synchronizing altitudes at which the first aircraft (204) and the second aircraft (206) fly; synchronizing sensor systems that measure altitudes of the first aircraft (204) and the second aircraft (206); or adjusting formation flying of the first aircraft (204) and the second aircraft (206).

10. 10. The method of claim 7, further comprising identifying, by the computer system, the key points in the first image produced by a first camera system associated with the first aircraft.

11. 11. The method of claim 7, wherein the second altitude is determined using stereo depth triangulation performed using a second image of the keypoint generated by the second aircraft.

12. The method of any one of claims 7 to 11, wherein the second altitude (230) is determined using an altitude sensor system (236, 314) associated with the second aircraft (206).

13. 13. The method of any one of claims 7 to 12, wherein the first aerial vehicle (204) and the second aerial vehicle (206) are selected from at least one of an airplane, an unmanned aerial system, an unmanned aerial vehicle, a drone, a rotorcraft, or a spacecraft.

14. A computer program product (1222) for operating an airborne imaging system (200), comprising: a computer-readable storage medium (1224); first program code stored on the computer-readable storage medium and executable by a computer system (210), the first program code causing the computer system (210) to determine a first distance (222) from the first aircraft (204) to the keypoint (220) using first images (212) of the keypoint (220) generated by the first aircraft (204) during flight of the first aircraft (204) and stereo depth triangulation (224); second program code stored on the computer-readable storage medium and executable by the computer system (210), the second program code causing the computer system (210) to compare the first distance (222) from the first aircraft (204) to the key point (220) and a first altitude (226) of the first aircraft (204) measured by the first aircraft (204) with a second distance (228) from a second aircraft (206) to the key point (220) and a second altitude (230) of the second aircraft (206) measured by the second aircraft (206) to generate a comparison (232); and and third program code stored on the computer-readable storage medium and executable by the computer system, the third program code causing the computer system to determine, using the comparison, an offset between the first altitude of the first aircraft and the second altitude measured by the second aircraft, the offset being used to adjust the altitude of the first aircraft. A computer program product (1222) comprising:

Images