System and method for low cost above-ground altitude and topographic data generation

Abstract

Systems and methods for low cost HAGL measurements and topographic database updates are provided. In one example, a system includes an image capture device configured to capture a first image and a second image and a navigation system configured to generate pose data. The system also includes a processor communicatively coupled to the image capture device and the navigation system, and a non-transitory computer readable medium communicatively coupled to the processor, the non-transitory computer readable medium storing instructions, the instructions, when executed by the processor, cause the processor to: estimate an optical flow between the first image and the second image; determining a per-image rigid flow for each depth interval in a set of depth intervals based on the pose data, wherein each depth interval corresponds to a particular depth range; and generating a dense depth map by determining, for each pixel, a depth interval that minimizes a difference between the estimated optical flow and the determined rigid flow for the respective pixel.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Indian Provisional Patent Application No. 202311075998 filed on November 7, 2023 and entitled “SYSTEMS AND METHOD FOR LOST-COST HEIGHT ABOVE GROUND LEVELAND TERRAIN DATA GENERATION”, the contents of which are incorporated herein by reference in their entirety. Background Art

[0003] Manned and unmanned aircraft require height above ground level (HAGL) information for safe operations during take-off, descent and landing phases, as well as during the low-altitude cruise phase. For autonomous systems, the availability of HAGL with integrity is crucial for safe operation. Current solutions use different sensors (e.g., radar, lidar, and laser-based rangefinders) to obtain HAGL, but these systems typically provide acceptable performance only over an altitude of a few hundred meters, and the cost of sensors with increased range is very high. In addition, current radar altimeters or laser-based rangefinders include active sensors that emit signals in the radio frequency band. These emitted signals may cause interference to other systems in the area (e.g., telecommunication systems such as 5G base stations) and may be detected by other systems.

[0004] For the reasons stated above, as well as other reasons provided below, there is a need for improved techniques for obtaining HAGL measurements and terrain data for aircraft. Summary of the invention

[0005] In one aspect, a system includes an image capture device configured to capture a first image and a second image. The system also includes a navigation system configured to generate pose data. The system also includes one or more processors communicatively coupled to the image capture device and the navigation system. The system also includes a non-transitory computer-readable medium communicatively coupled to the one or more processors, wherein the non-transitory computer-readable medium stores one or more instructions that, when executed by the one or more processors, cause the one or more processors to: estimate an optical flow between a first image and a second image; determine a per-image rigid flow for each depth interval in a set of depth intervals based on the pose data, wherein each depth interval corresponds to a specific depth range; and generate a dense depth map by determining a depth interval for each pixel that minimizes the difference between the estimated optical flow and the determined rigid flow for the corresponding pixel.

[0006] In another aspect, a system includes one or more inputs communicatively coupled to an image capture device and a navigation system. The system also includes one or more processors. The system also includes a non-transitory computer-readable medium communicatively coupled to the one or more processors, wherein the non-transitory computer-readable medium stores one or more instructions that, when executed by the one or more processors, cause the one or more processors to: estimate an optical flow between a first image and a second image received from the image capture device; determine a per-image rigid flow for each depth interval in a set of depth intervals based on pose data received from the navigation system, wherein each depth interval corresponds to a specific depth range; determine a depth interval for each pixel that minimizes a difference between the estimated optical flow and the determined rigid flow for the corresponding pixel; and generate a dense depth map based on the depth interval determined for each pixel.

[0007] In another aspect, a method includes capturing a first image and a second image. The method also includes generating first pose data corresponding to the first image and second pose data corresponding to the second image. The method also includes estimating the optical flow between the first image and the second image. The method also includes determining a per-image rigid flow for each depth interval in a plurality of depth intervals. The method also includes determining a depth interval in a plurality of depth intervals for each corresponding pixel in a plurality of pixels, the depth interval minimizing the difference between the estimated optical flow and the determined rigid flow for the corresponding pixel. The method also includes generating a dense depth map based on the depth intervals determined for the plurality of pixels. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Embodiments of the present invention may be more readily understood, and further advantages and uses thereof more apparent, when consideration is given to the specification and the following drawings, in which:

[0009] Figure 1 is a block diagram of an example system;

[0010] Figure 2 is an illustration of an example dense depth map;

[0011] Figure 3 is a flowchart of an example method for generating a dense depth map;

[0012] Figure 4 is a diagram of an example method of determining rigid flow; and

[0013] Figure 5 is a flow chart of an example method for generating HAGL measurements from a dense depth map.

[0014] According to common practice, the various features described are not necessarily drawn to scale, but rather are used to emphasize features relevant to the invention. Reference characters denote similar elements throughout the figures and text. DETAILED DESCRIPTION

[0015] In the following detailed description, reference is made to the accompanying drawings which form a part thereof and which illustrate by way of specific illustrative embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention, and it is understood that other embodiments may be utilized and logical, mechanical and electrical changes may be made without departing from the scope of the present invention. Therefore, the following detailed description should not be considered in a limiting sense.

[0016] Many types of aircraft (manned and unmanned) are equipped with autonomous subsystems and include downward-facing cameras as part of the platform. The systems and methods described herein can take advantage of these already available features of aircraft to provide a low-cost passive system that can be used for dense depth map generation, HAGL estimation, and terrain database updates. These features can be used for many applications, such as, for example, real-time HAGL sensors, terrain collision avoidance systems, real-time connected aircraft database updates, real-time analytics (e.g., vegetation analysis, disaster management, etc.), and the like.

[0017] Figure 1 is a block diagram of an example system 100 in which the techniques described herein may be implemented. Figure 1 In the illustrated example, system 100 includes various components including an image capture device 102, a navigation system 104, and a computing device 106 including one or more processors 108 and at least one memory 110. Figure 1 A particular number of components are shown in FIG. 1 , but it should be understood that this is merely an example and a different number of image capture devices 102 , navigation systems 104 , and computing devices 106 , or components thereof, may also be included.

[0018] In some examples, the system 100 is included in or on a vehicle. In such examples, the vehicle may include an aircraft, and some of the principles described throughout this disclosure are explained with reference to an aircraft. However, the term vehicle is intended to include all such vehicles that fall within the ordinary meaning of the term as understood by one of ordinary skill in the art, including, but not limited to, air-traveling vehicles (e.g., commercial, non-commercial, or recreational aircraft), unmanned vehicles (e.g., drones, urban air mobility vehicles), and space-traveling vehicles.

[0019] exist Figure 1 In the example shown, image capture device 102 is communicatively coupled to computing device 106. In some examples, image capture device 102 is a camera. In some such examples, image capture device 102 is a monocular camera or a multispectral camera. In other examples, image capture device 102 is an infrared (IR) sensor. It should be understood that other types of cameras or sensors may also be used.

[0020] The image capture device 102 is configured to capture images of the environment in the field of view frame of the image capture device 102. The image capture device 102 is configured to capture images at a sampling rate while the image capture device 102 (and the vehicle including the image capture device 102) is moving. In some examples, the sampling rate is adjustable and can be adjusted based on multiple factors, including but not limited to the speed, position, obstacles, driving stage, etc. of the vehicle including the image capture device 102. In some examples, the sampling rate is adjusted so that the consecutively captured images overlap by at least fifty percent. In some examples, the sampling rate can be adjusted by one or more processors 108. In other examples, a different processor or controller is used to adjust the sampling rate of the image capture device 102.

[0021] For ease of description, image capture device 102 will be discussed in the context of a first image and a second image captured by image capture device 102. It should be understood that the terms first image and second image may apply to any two images that are captured successively by image capture device 102, or have sufficient correspondence (e.g., overlap by approximately fifty percent) such that they may be meaningfully processed using the techniques described herein. For example, a first image may be processed together with a second image, and separately from a third image that also has sufficient correspondence with the first image.

[0022] Image capture device 102 is configured to capture a first image at a first time and a first pose and a second image at a second time different from the first time and a second pose different from the first pose. Image capture device 102 is configured to provide the first image and the second image to computing device 106 for further processing as described herein.

[0023] exist Figure 1 In the example shown, the navigation system 104 is also communicatively coupled to the computing device 106. In some examples, the navigation system 104 is configured to determine the position and rotation values ​​of the system 100 or a vehicle including the system 100 according to time. In some examples, the navigation system 104 is configured to receive global navigation satellite system (GNSS) and inertial measurement unit (IMU) inputs and output the pose data of the system 100 or a vehicle including the system 100 according to time. In some such examples, the pose data includes a position estimate and a rotation (or orientation) determined using a Kalman filter or other navigation filter. In the context of the system 100, the navigation system 104 is configured to output the first pose data of the system 100, which corresponds to the first time when the first image is captured by the image capture device 102. Similarly, the navigation system 104 is configured to output the second pose data of the system 100, which corresponds to the second time when the second image is captured by the image capture device 102. The first pose data and the second pose data may be in a frame including a body of a vehicle of the system 100 .

[0024] exist Figure 1 In the example shown, the computing device 106 includes one or more processors 108 and at least one memory 110. The one or more processors 108 are communicatively coupled to the image capture device 102, the navigation system 104, and the at least one memory 110. The one or more processors 108 are configured to execute various instructions stored on the at least one memory 110, which cause the one or more processors 108 to implement the techniques and methods described herein.

[0025] exist Figure 1 In the example shown, at least one memory 110 includes optical flow instructions 112. One or more processors 108 are configured to execute the optical flow instructions 112 to estimate optical flow between a first image and a second image captured by the image capture device 102. In some examples, the estimated optical flow indicates an apparent motion of features in the second image relative to the first image, the apparent motion caused by relative motion of the system 100 relative to the environment captured in the first image and the second image. The optical flow instructions 112 utilized by the system 100 may include any suitable technique known in the art for estimating optical flow between two images.

[0026] The accuracy of the optical flow estimated by the one or more processors 108 executing the optical flow instructions 112 is determined based on the sampling rate of the image capture device 102 and the accuracy of the specific optical flow instructions 112 utilized. If a particular system 100 requires higher accuracy, the sampling rate of the image capture device 102 can be increased so that the first image has a greater percentage of overlap with the second image, and the specific optical flow instructions 112 selected can be selected to have increased accuracy. It should be understood that the sampling rate of the image capture device 102 and the optical flow instructions 112 will also be selected based on the available resources (computation, memory, etc.) available to the system 100.

[0027] exist Figure 1 In the example shown, at least one memory 110 includes rigid flow instructions 114. The one or more processors 108 are configured to execute the rigid flow instructions 114 to determine a per-image rigid flow for each of a plurality of depth intervals, as will be discussed in greater detail below.

[0028] In some examples, when executing rigid flow instructions 114, one or more processors 108 are configured to determine a difference between the first pose and the second pose using the first pose data and the second pose data from the navigation system 104. The determination provides an indication of an amount of motion that has occurred between the first pose of the image capture device 102 and the second pose of the image capture device 102. In some examples, the difference between the first pose and the second pose is referred to as a delta pose.

[0029] In some examples, when executing rigid flow instructions 114, one or more processors 108 are configured to transform pose data output by navigation system 104 from a frame of the body of the vehicle including image capture device 102 to a frame of image capture device 102. In some examples, the transformation is based on a known fixed relationship between image capture device 102 and the body of the vehicle. In some examples, the known fixed relationship between image capture device 102 and the body of the vehicle is determined during calibration of system 100.

[0030] For a given pixel in the second image, it is unclear what depth should be associated with the pixel of the dense depth map 120. When executing the rigid flow instructions 114, the one or more processors 108 are configured to propose a depth hypothesis for each pixel based on each of the plurality of depth intervals. For each different depth hypothesis, a different rigid flow is calculated using the pose information and the depth hypothesis for the given pose. The depth value for the rigid flow determination is provided by the depth hypothesis proposal (depth interval), and the rotation / position transformation is provided from the navigation system 104.

[0031] In some examples, each depth interval in the depth intervals used for the hypothesis defines a range of depth values ​​within the depth range. The depth range is the complete range of potential values ​​for the depth to be evaluated. The depth range is defined by a minimum depth value and a maximum depth value. The depth range is divided into different depth intervals for evaluation. In some examples, the total depth range is divided into depth intervals of equal size. In other examples, the total depth range is divided into depth intervals of different sizes. In any case, the depth interval is defined by a minimum depth interval value and a maximum depth interval value.

[0032] In some examples, the total depth range, the size of each depth interval, the number of depth intervals, and / or the depth values ​​covered by each depth interval are selected based on one or more previous HAGL measurements. In some such examples, the previous HAGL measurements are determined by computing device 106 using the techniques discussed herein. In some examples, the previous HAGL measurements are provided by an altimeter 124 and / or a HAGL sensor 125 communicatively coupled to computing device 106. In some examples, the most recent previous HAGL measurement may be used as the center of the depth range. In other examples, a trend or average of multiple previous HAGL measurements may be used to define the center of the depth range.

[0033] An example scenario is where the previous HAGL measurement is a height above ground of 1000 m, the depth range is 200 m, and the number of depth intervals is 20. If the depth range is equally divided into 20 depth intervals, each depth interval has a range of 20 m, with the lowest range depth interval ranging from 800 m to 820 m, and the highest range depth interval ranging from 1180 m to 1200 m. The depth range and / or number of depth intervals may also be selected using, for example, other factors such as desired / required accuracy, the sampling rate of the image capture device 102, the speed of the vehicle comprising the system 100, available resources, etc.

[0034] In other examples, one or more processors 108 do not utilize previously known information when executing rigid stream instructions 114. Such examples may take longer to complete due to the need to evaluate a greater range of depths and a greater number of depth intervals. By selecting the depth range and / or the number of depth intervals, the starting point for evaluation is more targeted and the resolution may be improved compared to examples that do not utilize previously known information.

[0035] exist Figure 1In the example shown, at least one memory 110 includes dense depth map instructions 116. When executing the dense depth map instructions 116, the one or more processors 108 are configured to determine a specific depth interval for each pixel of the second image that minimizes the difference between the estimated optical flow and the determined rigid flow for the pixel. In some examples, each depth interval is associated with a different depth value that is unique to the specific depth interval. When generating the dense depth map 120, the depth value associated with the specific depth interval determined to minimize the difference between the estimated optical flow and the determined rigid flow for the corresponding pixel is then used for the corresponding pixel. In general, the dense depth map 120 captures a per-pixel depth that corresponds to a vertical distance from a point on the ground to an image plane of the image capture device 102.

[0036] In some examples, one or more processors 108 are configured to determine in parallel for at least two pixels the depth interval that minimizes the difference between the estimated optical flow and the determined rigid flow, which can improve computational efficiency. In some examples, one or more processors 108 are configured to sequentially determine for at least two pixels the depth interval that minimizes the difference between the estimated optical flow and the determined rigid flow. It should be understood that one or more processors 108 can be configured to determine the depth interval based on available resources using a combination of parallel or sequential processing.

[0037] In some examples, a dense depth map 120 generated by one or more processors 108 when executing dense depth map instructions 116 is in a frame of the image capture device 102. The dense depth map 120 includes all pixels of the second image that are evaluated, and the characteristics of each of the pixels correspond to characteristics of a specific depth interval. In some examples, the dense depth map 120 is stored in at least one memory 110 for future use or future reference. In some examples, the computing device 106 is configured to display the dense depth map 120 on a display 126 that is communicatively coupled to the computing device 106. Figure 2 An example dense depth map 120 displayed on a display 126 is shown in FIG.

[0038] In general, the greater the number of depth intervals that need to be evaluated for each pixel, the longer the evaluation will take. For example, if available resources are scarce or accuracy and precision are critical (e.g., when the system 100 is close to the ground), the depth range and / or the number of depth intervals may need to be adjusted. In some examples, if the previous HAGL measurement is below a first threshold (e.g., associated with an increased risk of an obstacle or hitting the ground), the depth range is reduced and / or the number of depth intervals is increased, which will improve the accuracy and precision of the evaluation, but will increase the computational load due to the more detailed evaluation. Similarly, if the previous HAGL measurement is above a second threshold (e.g., associated with a reduced risk of an obstacle or hitting the ground), the depth range is increased and / or the number of depth intervals is reduced, which will reduce the accuracy and precision of the evaluation and reduce the computational load. It should be understood that similar changes to the depth range and / or the number of depth intervals may also be made using thresholds for other factors (such as the desired / required accuracy, the sampling rate of the image capture device 102, the speed of the vehicle including the system 100), for example.

[0039] In some examples, at least one memory 110 optically includes HAGL instructions 118. In such examples, one or more processors 108 are configured to execute the HAGL instructions 118 to determine a HAGL of a vehicle including the system 100. When executing the HAGL instructions 118, the one or more processors 108 are configured to generate a transformed dense depth map by transforming a dense depth map 120 from a frame of the image capture device 102 to a frame of a body of the vehicle including the image capture device 102. In some examples, the transformation of the dense depth map 120 to the frame of the body is based on a known fixed relationship between the image capture device 102 and the body of the vehicle. In some examples, the known fixed relationship between the image capture device 102 and the body of the vehicle is determined during calibration of the system 100.

[0040] When executing HAGL instructions 118, one or more processors 108 are also configured to determine a depth (or depth interval) for a center pixel of the transformed dense depth map. In some examples, determining the depth for a center pixel of the transformed dense depth map includes determining a depth value associated with a depth interval for the center pixel.

[0041] When executing the HAGL instructions 118, the one or more processors 108 are also configured to convert the depth for the center pixel of the transformed dense depth map into a HAGL measurement value. As discussed above, a depth interval is associated with a specific depth range defined by a lowest depth interval value and a highest depth interval value. In some examples, the HAGL measurement value generated by the one or more processors 108 when executing the HAGL instructions 118 corresponds to the value of the center of the depth interval. In some such examples, the error of the HAGL measurement value corresponds to the absolute value of the difference between the center value and the lowest depth interval value or the highest depth interval value of the specific depth interval.

[0042] In some examples, at least one memory 110 optionally includes terrain instructions 119. In such examples, one or more processors 108 are configured to execute terrain instructions 119 in order to generate terrain database 122 or to modify terrain database 122.

[0043] In some examples, when executing terrain instructions 119, one or more processors 108 are configured to perform processes that facilitate real-time terrain database updates, which are suitable for real-time situational awareness, path planning, obstacle avoidance, and / or emergency landing. In such examples, one or more processors 108 are configured to send the dense depth map 120 along with the associated pose data to a terrestrial receiver via a wireless communication link (e.g., a satellite communication link). The dense depth map 120 and the pose data can be pushed or otherwise sent to connected aircraft in the area, and if there is a difference between the terrain data stored in the terrain database and the dense depth map 120, the terrain database can be modified based on the dense depth map 120 and the pose data.

[0044] In some examples, when executing terrain instructions 119, one or more processors 108 are configured to perform a process for facilitating offline terrain database updates, which are applicable to subsequent missions. In such examples, one or more processors 108 are configured to send the dense depth map 120 along with the associated pose data to a terrestrial receiver via a wired communication link (e.g., an Ethernet cable) or a wireless communication link (e.g., a WiFi or cellular communication link). The dense depth map 120 and the pose data can then be loaded into a terrain database (e.g., terrain database 122) on the onboard device before the subsequent mission.

[0045] In some examples, the system 100 also includes an altimeter 124 and / or a HAGL sensor 125. In such examples, the altimeter 124 may include a radar altimeter or another alternative type of altimeter, and the HAGL sensor 125 may include a laser rangefinder or another alternative type of sensor. The HAGL measurements generated by the one or more processors 108 when executing the HAGL instructions 118 may be used to evaluate the HAGL measurements generated using the altimeter 124 and / or the HAGL sensor 125. For example, the accuracy of the HAGL measurements generated using the altimeter 124 and / or the HAGL sensor 125 may be compared to the HAGL measurements generated by the one or more processors 108. In some examples, the HAGL measurements generated by the one or more processors 108 when executing the HAGL instructions 118 may be used in conjunction with the HAGL measurements generated using the altimeter 124 and / or the HAGL sensor 125. For example, HAGL measurements generated using the altimeter 124 and / or HAGL sensor 125 may be combined with HAGL measurements generated by one or more processors 108 in a Kalman filter or other navigation filter to improve the accuracy and reliability of the HAGL measurements.

[0046] In some examples, the image capture device 102 is configured to capture a third image at a third time different from the first time and the second time and at a third pose different from the first pose and the second pose. In some such examples, processing similar to the processing described herein may be performed on the first image and the third image. In some examples, post-processing is performed, and the depth of a particular point may be compared based on the depth estimate determined using the first image and the second image and the depth estimate determined using the first image and the third image. If there is a match between the depth estimates of the particular point, the depth estimate may be considered to have been verified because the same depth estimate was produced using different image combinations. However, if there is a mismatch between the depth estimates of the particular point, the depth estimate may be deleted because there is an inconsistency in the depth estimate. In some examples, the amount of mismatch is compared to a threshold to determine whether the amount of mismatch is sufficient to indicate that the depth estimate should be deleted. The post-processing described above is used to form a more robust solution by retaining one of the depth estimates or by a judicious combination of depth estimates.

[0047] Figure 3 A flowchart of an example method 300 for generating a dense depth map is shown. Figure 1 Common features discussed with respect to the example systems in may include similar characteristics to features discussed with respect to method 300, and vice versa. In some examples, blocks of method 300 are performed by system 100 described above.

[0048] For ease of explanation, Figure 3The blocks of the flowchart in FIG. 300 have been arranged in a generally sequential manner; however, it should be understood that such an arrangement is merely exemplary, and it should be recognized that the method 300 (and Figure 3 The processing associated with the blocks (shown in FIG. 1 ) may be performed in a different order (eg, where at least some of the processing associated with the blocks is performed in parallel in an event-driven manner).

[0049] Method 300 includes capturing a first image and a second image (block 302). The first image and the second image are captured at different times using the same image capture device. For example, the first image may be captured using the image capture device at a first time, and the second image may be captured using the image capture device at a second time. In some examples, the image capture device is a monocular camera or a multispectral camera. In other examples, the image capture device is an infrared (IR) sensor or other type of sensor.

[0050] The method 300 also includes estimating an optical flow between the first image and the second image (block 304). In some examples, estimating the optical flow between the first image and the second image includes indicating an apparent motion of features in the second image relative to the first image, the apparent motion caused by relative motion of the system relative to an environment captured in the first image and the second image. In some examples, the estimate of the optical flow is determined using any suitable technique known in the art.

[0051] The method 300 also includes generating pose data corresponding to the first image and the second image (box 306). In some examples, the pose data is generated by a navigation system or a navigation filter. The pose data corresponding to the first image can be the pose of the vehicle including the image capture device when the first image was captured. Similarly, the pose data corresponding to the second image can be the pose of the vehicle including the image capture device when the second image was captured.

[0052] Method 300 also includes determining a per-image rigid flow for each depth bin (block 308). In some examples, determining the rigid flow includes proposing a depth hypothesis for each pixel in the second image, where each depth hypothesis is represented by a depth bin. Further details about this step are provided below. Figure 4 Available in.

[0053] The method 300 also includes determining, for each respective pixel, a depth interval that minimizes a difference between the estimated optical flow and the determined rigid flow for the respective pixel (block 310). In some examples, determining the depth interval for each respective pixel includes determining the depth interval for at least two of the pixels in parallel. In some examples, determining the depth interval for each respective pixel includes determining the depth interval for at least two of the pixels sequentially.

[0054] The method 300 also includes generating a dense depth map based on the depth interval determined for each of the pixels (block 312). In some examples, the dense depth map includes each corresponding pixel of the second image, the corresponding pixel having a depth value associated with the depth interval determined for the corresponding pixel. For example, the depth value can be a center value of a range of values ​​represented by the depth interval.

[0055] Figure 4 A flow chart of an example method 400 for determining rigid flow is shown. Figure 1 Common features discussed with respect to the example systems in may include similar characteristics to features discussed with respect to method 400, and vice versa. In some examples, blocks of method 400 are performed by system 100 described above.

[0056] For ease of explanation, Figure 4 The blocks of the flowchart in FIG. 4 have been arranged in a generally sequential manner; however, it should be understood that such an arrangement is merely exemplary, and it should be recognized that the method 400 (and Figure 4 The processing associated with the blocks (shown in FIG. 1 ) may be performed in a different order (eg, where at least some of the processing associated with the blocks is performed in parallel in an event-driven manner).

[0057] Method 400 includes determining a difference between a pose estimate for a first image and a pose estimate for a second image (block 402). In some examples, determining the difference between the pose estimates includes an indication of an amount of motion that occurred between a first pose of an image capture device when the first image was taken and a second pose of the image capture device when the second image was taken. In some examples, pose data provided by a navigation system or a navigation filter is converted from frames of a body of a vehicle including a camera to camera frames in order to make such a determination.

[0058] Method 400 also includes dividing the depth range into depth intervals (box 404). In some examples, the depth range defines the entire range of depths being evaluated. The depth interval includes a smaller range of depth values ​​to be evaluated within the depth range. In some examples, the depth range is equally divided into N depth intervals. In other examples, the depth range is unequally divided into N depth intervals based on the desired accuracy or precision, previous above-ground height measurements, or similar factors that may affect the desired resolution of the dense depth map.

[0059] The method 400 also includes determining a per-image rigid flow for each depth bin based on a difference between the pose estimate of the first image and the pose estimate of the second image (block 406). In some examples, determining the per-image rigid flow includes determining a movement of the rigid object from the first image to the second image. By using the difference between the pose estimate of the first image and the pose estimate of the second image, the movement in the pixel coordinate system can be associated with, for example, a movement in a GPS coordinate system.

[0060] Figure 5 A flow chart of an example method 500 for generating above-ground height measurements based on a dense depth map is shown. Figure 1 Common features discussed with respect to the example systems in may include similar characteristics to features discussed with respect to method 500, and vice versa. In some examples, blocks of method 500 are performed by system 100 described above.

[0061] For ease of explanation, Figure 5 The blocks of the flowchart in FIG. 5 have been arranged in a generally sequential manner; however, it should be understood that such an arrangement is merely exemplary, and it should be recognized that the method 500 (and Figure 5 The processing associated with the blocks (shown in FIG. 1 ) may be performed in a different order (eg, where at least some of the processing associated with the blocks is performed in parallel in an event-driven manner).

[0062] The method 500 also includes converting a dense depth map from a camera frame to a body frame (block 502). In some examples, the conversion of the dense depth map to a body frame is based on a known fixed relationship between the image capture device and the body of the vehicle. In some examples, the known fixed relationship between the image capture device and the body of the vehicle is determined during calibration of the system.

[0063] Method 500 also includes determining a depth for a center pixel of the dense depth map in the fuselage frame (block 504). In some examples, determining the depth for the center pixel of the transformed dense depth map includes determining a depth value associated with a depth interval for the center pixel.

[0064] By using the techniques described herein, the systems and methods provide a low-cost solution for dense depth map generation, which can be applied to above ground height estimation, terrain database updates, landing zone detection, obstacle avoidance, etc. The systems and methods are passive and can therefore be used in environments where active systems and methods are not ideal because they can be detected (e.g., in combat zones) or cause interference (e.g., interfering with cellular systems). The systems and methods described herein also provide consistent performance at higher altitudes (e.g., up to 3000m or more) compared to traditional sensor-based techniques.

[0065] In various aspects, the system elements or examples described throughout this disclosure (such as systems, computing devices, or components thereof) may be implemented on one or more computer systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), or similar devices including hardware that executes code to implement those elements, processes, or examples, the code being stored on a non-transitory data storage device. These devices include or work with software programs, firmware, or other computer-readable instructions for implementing various methods, processing tasks, calculations, and control functions used in distributed antenna systems.

[0066] These instructions are usually stored on any suitable computer storage medium for storing computer-readable instructions or data structures. Computer-readable media can be implemented as any available medium that can be accessed by a general or special-purpose computer or processor or any programmable logic device. Suitable processor-readable media may include storage devices or memory media, such as magnetic media or optical media. For example, storage or memory media may include conventional hard disks, CD-ROMs, volatile or non-volatile media such as random access memories (RAM) (including but not limited to synchronous dynamic random access memories (SDRAM), double data rate (DDR) RAM, RAMBUS dynamic RAM (RDRAM), static RAM (SRAM) etc.), read-only memories (ROM), electrically erasable programmable ROM (EEPROM) and flash memory etc. Suitable processor-readable media may also include transmission media (such as electrical signals, electromagnetic signals or digital signals) transmitted via communication media (such as networks and / or wireless links).

[0067] The techniques described herein may be implemented in digital electronic circuit systems, or using programmable processors (e.g., special purpose processors or general purpose processors, such as computers), firmware, software, or a combination thereof. Apparatus embodying these techniques may include appropriate input and output devices, a programmable processor, and a storage medium tangibly embodying program instructions for execution by the programmable processor. Processes embodying these techniques may be performed by executing a program of instructions by a programmable processor to perform a desired function by operating on input data and producing appropriate outputs. These techniques may be advantageously implemented in one or more programs executable on a programmable system, the programmable system including at least one programmable processor coupled to receive data and instructions from a data storage system, at least one input device, and at least one output device, and to transmit data and instructions to a data storage system, at least one input device, and at least one output device. Generally speaking, the processor will receive instructions and data from a read-only memory and / or a random access memory. Storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including (by way of example) semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and DVD disks. Any of the foregoing may be supplemented by, or incorporated in, specially designed application specific integrated circuits (ASICs).

[0068] Example Implementation

[0069] Embodiment 1 includes a system comprising: an image capture device configured to capture a first image and a second image; a navigation system configured to generate pose data; one or more processors communicatively coupled to the image capture device and the navigation system; and a non-transitory computer-readable medium communicatively coupled to the one or more processors, wherein the non-transitory computer-readable medium stores one or more instructions that, when executed by the one or more processors, cause the one or more processors to: estimate an optical flow between the first image and the second image; determine a per-image rigid flow for each depth interval in a set of depth intervals based on the pose data, wherein each depth interval corresponds to a specific depth range; and generate a dense depth map by determining a depth interval for each pixel that minimizes a difference between the estimated optical flow and the determined rigid flow for the corresponding pixel.

[0070] Embodiment 2 includes a system according to embodiment 1, wherein the one or more instructions, when executed by one or more processors, further cause the one or more processors to generate an image-based above-ground height measurement based on the dense depth map and pose data by transforming the dense depth map from a camera frame to a body frame coordinate system.

[0071] Embodiment 3 includes a system according to any one of embodiments 1 to 2, wherein the one or more instructions, when executed by one or more processors, further cause the one or more processors to generate terrain data based on the dense depth map and the pose data.

[0072] Embodiment 4 includes a system according to embodiment 3, wherein the one or more instructions, when executed by one or more processors, also cause the one or more processors to update terrain data in a terrain database in real time for situational awareness, path planning, obstacle avoidance and / or emergency landing.

[0073] Embodiment 5 includes a system according to any one of embodiments 1 to 4, wherein the one or more instructions, when executed by one or more processors, further cause the one or more processors to select the set of depth intervals based on previous above ground height measurements.

[0074] Embodiment 6 includes a system according to any one of Embodiments 2 to 5, wherein the system also includes an altimeter or other above ground height sensor configured to determine a second above ground height measurement; wherein the one or more instructions, when executed by one or more processors, further cause the one or more processors to: combine the image-based above ground height measurement with the second above ground height measurement; and / or compare the image-based above ground height measurement with the second above ground height measurement.

[0075] Embodiment 7 includes the system of any one of Embodiments 1-6, wherein the image capture device, the navigation system, the one or more processors, and the non-transitory computer readable medium are located on a vehicle.

[0076] Embodiment 8 includes the system of embodiment 7, wherein the sampling rate of the image capture device is adjusted based on the speed of the vehicle, the position of the vehicle, and / or the driving stage.

[0077] Embodiment 9 includes a system according to any one of embodiments 1 to 8, wherein the image capture device is also configured to capture a third image; wherein the one or more instructions, when executed by one or more processors, also cause the one or more processors to: generate a second dense depth map based on a second optical flow and a second rigid flow determined using the first image, the third image, and pose data corresponding to the first image and the third image; compare the dense depth map with depth estimates of specific points in the second dense depth map; and delete or verify or judiciously combine the depth estimates based on the comparison.

[0078] Embodiment 10 includes the system of any one of Embodiments 1 to 9, wherein the one or more processors are configured to determine the respective depth intervals for at least two pixels in parallel.

[0079] Embodiment 11 includes a system comprising: one or more input terminals, which are communicatively coupled to an image capture device and a navigation system; one or more processors; and a non-transitory computer-readable medium, which is communicatively coupled to the one or more processors, wherein the non-transitory computer-readable medium stores one or more instructions, which when executed by the one or more processors cause the one or more processors to: estimate an optical flow between a first image and a second image received from the image capture device; determine a per-image rigid flow for each depth interval in a set of depth intervals based on pose data received from the navigation system, wherein each depth interval corresponds to a specific depth range; determine a depth interval for each pixel, which minimizes the difference between the estimated optical flow and the determined rigid flow for the corresponding pixel; and generate a dense depth map based on the depth interval determined for each pixel.

[0080] Embodiment 12 includes a system according to embodiment 11, wherein the one or more instructions, when executed by one or more processors, further cause the one or more processors to generate an image-based above-ground height measurement based on the dense depth map and pose data by transforming the dense depth map from a camera frame to a body frame coordinate system.

[0081] Embodiment 13 includes a system according to any one of embodiments 11 to 12, wherein the one or more instructions, when executed by one or more processors, further cause the one or more processors to generate terrain data based on the dense depth map and the pose data.

[0082] Embodiment 14 includes a system according to any one of Embodiments 11 to 13, wherein the one or more instructions, when executed by one or more processors, further cause the one or more processors to select the set of depth intervals based on previous above ground height measurements.

[0083] Embodiment 15 includes the system of any one of Embodiments 11 to 14, wherein the one or more processors are configured to determine the respective depth intervals for at least two pixels in parallel.

[0084] Embodiment 16 includes a method comprising: capturing a first image and a second image; generating first pose data corresponding to the first image and second pose data corresponding to the second image; estimating an optical flow between the first image and the second image; determining a per-image rigid flow for each depth interval in a plurality of depth intervals; determining a depth interval in a plurality of depth intervals for each corresponding pixel in a plurality of pixels, the depth interval minimizing a difference between the estimated optical flow and the determined rigid flow for the corresponding pixel; and generating a dense depth map based on the depth intervals determined for the plurality of pixels.

[0085] Embodiment 17 includes the method according to embodiment 16, which further includes: determining the difference between the first pose data corresponding to the first image and the second pose data corresponding to the second image; dividing the depth range into multiple depth intervals; and determining a per-image rigid flow for each depth interval in the multiple depth intervals based on the determined difference between the first pose data corresponding to the first image and the second pose data corresponding to the second image.

[0086] Embodiment 18 includes a method according to any one of Embodiments 16 to 17, wherein a dense depth map is generated in a frame of an image capture device, the method further comprising: transforming the dense depth map from the frame of the image capture device into a frame of a body of a vehicle to generate a transformed dense depth map; and determining a depth interval for a center pixel of the transformed dense depth map.

[0087] Embodiment 19 includes the method of any one of embodiments 16 to 18, further comprising adjusting a sampling rate for capturing the first image and the second image based on a speed of the vehicle, a position of the vehicle, and / or a driving phase.

[0088] Embodiment 20 includes the method according to any one of Embodiments 16 to 19, further comprising updating terrain data in a terrain database based on the dense depth map.

[0089] Although specific embodiments have been illustrated and described herein, one of ordinary skill in the art will recognize that any arrangement calculated to achieve the same purpose may be substituted for the specific embodiments shown. This application is intended to cover any modifications or variations of the present invention. Therefore, it is apparent that the present invention is limited only by the claims and their equivalents.

Claims

1. A system (100), comprising: an image capture device (102) configured to capture a first image and a second image; A navigation system (104) configured to generate pose data; one or more processors (108) communicatively coupled to the image capture device (102) and the navigation system (104); and A non-transitory computer-readable medium (110) communicatively coupled to the one or more processors (108), wherein the non-transitory computer-readable medium (110) stores one or more instructions (112, 114, 116) that, when executed by the one or more processors (108), cause the one or more processors (108) to: estimating an optical flow between the first image and the second image; determining a per-image rigid flow for each depth bin in a set of depth bins based on the pose data, wherein each depth bin corresponds to a particular depth range; as well as A dense depth map (120) is generated by determining, for each pixel, a depth interval that minimizes a difference between the estimated optical flow and the determined rigid flow for the corresponding pixel.

2. The system (100) of claim 1, wherein the one or more instructions (118), when executed by the one or more processors (108), further cause the one or more processors (108) to generate an image-based above-ground height measurement based on the dense depth map (120) and the pose data by transforming the dense depth map (120) from a camera frame to a body frame coordinate system.

3. The system (100) of claim 1, wherein the one or more instructions (119), when executed by the one or more processors (108), further cause the one or more processors (108) to generate terrain data based on the dense depth map (120) and the pose data.

Images