Landing area assessment
By capturing and analyzing landing area images in real time through a multi-sensor system, obstacles can be identified and avoided, solving the problem of insufficient obstacle detection in the aircraft landing area and achieving a safe landing.
Patent Information
- Application Number
- CN202111214970.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-10-19
- Filing Date
- 2021-10-19
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2041-10-19
AI Technical Summary
In the existing technology, there may be undetected obstacles in the landing area of the aircraft, which may lead to damage and passenger injury.
Employing a multi-sensor system, including imaging and navigation sensors, it captures real-time images of the landing area and identifies and locates obstacles using position data, generating navigation commands to avoid obstacles.
Effectively identify and avoid obstacles in the landing area to ensure the safe landing of the aircraft and reduce the risk of damage and injury.
Smart Images

Figure CN114387825B_ABST
Abstract
Description
Technical Field
[0001] This disclosure generally relates to aircraft, and more specifically to assessing the landing area of an aircraft. Background Technology
[0002] The assumption for landing zones of aircraft such as airplanes and helicopters is that the landing zone is flat and generally free of obstacles. However, in practice, there may be obstacles or objects in the landing zone that could impede or prevent the aircraft from landing. In the worst case, undetectable obstacles in the landing zone could damage the aircraft and / or cause bodily injury to the passengers on board. Summary of the Invention
[0003] The subject matter of this application was developed in response to the current state of the art, particularly in response to the problems and shortcomings associated with conventional methods used for landing area assessment. Therefore, the subject matter of this application has been developed for assessing landing areas, overcoming at least some of the aforementioned shortcomings of the prior art.
[0004] This document discloses a system. The system includes one or more imaging sensors coupled to an aircraft. The system also includes one or more navigation sensors coupled to the aircraft in proximity to the one or more imaging sensors. The system further includes an imaging module configured to capture images of the aircraft's landing area while the aircraft is in flight. The captured images are captured using the one or more imaging sensors. The system further includes a position determination module configured to associate a corresponding location data from a plurality of location data with each of the captured images. The plurality of location data is determined using the one or more navigation sensors coupled to the aircraft. The system also includes an object recognition module configured to identify and locate one or more objects relative to the aircraft within the landing area using the captured images and the location data associated with the captured images.
[0005] This document further discloses a method. The method includes capturing images of the aircraft's landing area while the aircraft is in flight. The captured images are acquired using one or more imaging sensors coupled to the aircraft. The method also includes associating a corresponding location data from a plurality of location data with each of the captured images. The plurality of location data is determined using one or more navigation sensors coupled to the aircraft. The method further includes using the captured images and the location data associated with the captured images to identify and locate one or more objects within the landing area relative to the aircraft.
[0006] Furthermore, this document discloses a computer program product comprising a computer-readable storage medium containing program code. The program code is executable by a processor to receive images of the landing area of the aircraft from one or more imaging sensors coupled to the aircraft. These images are captured using one or more imaging sensors while the aircraft is in flight. The program code is also executable by a processor to associate a corresponding location data among a plurality of location data with each of the captured images. The plurality of location data is determined using one or more navigation sensors coupled to the aircraft. The program code is further executable by a processor to identify and locate one or more objects relative to the aircraft within the landing area using the captured images and the location data associated with the captured images.
[0007] The features, structures, advantages, and / or characteristics of the subject matter of this disclosure described herein can be combined in any suitable manner in one or more examples and / or embodiments. Numerous specific details are provided in the following description to provide a thorough understanding of the examples of the subject matter of this disclosure. Those skilled in the art will recognize that the subject matter of this disclosure can be practiced without one or more of the specific features, details, components, materials, and / or methods of a particular example or embodiment. In other instances, additional features and advantages that are not present in all examples or embodiments may be recognized in certain examples and / or embodiments. Furthermore, in some instances, well-known structures, materials, or operations have not been shown or described in detail to avoid obscuring aspects of the subject matter of this disclosure. The features and advantages of the subject matter of this disclosure will become more fully apparent from the following description and the appended claims, or may be learned by practice of the subject matter described below. Attached Figure Description
[0008] To facilitate a clearer understanding of the advantages of the subject matter, a more specific description of the subject matter briefly described above will be provided with reference to specific embodiments shown in the accompanying drawings. It should be understood that these drawings depict only typical embodiments of the subject matter and should not be considered as limiting its scope. The subject matter will be described and illustrated with additional specificity and detail using the drawings, wherein:
[0009] Figure 1 This is a schematic block diagram of a system for landing area assessment according to one or more examples of this disclosure;
[0010] Figure 2 This is a schematic block diagram of an apparatus for landing area assessment according to one or more examples of this disclosure;
[0011] Figure 3 Example embodiments of a system for landing area assessment according to one or more examples of this disclosure are depicted;
[0012] Figure 4A Example embodiments of landing zone assessment outputs according to one or more examples of this disclosure are depicted;
[0013] Figure 4B Another example embodiment of the output of landing area assessment according to one or more examples of this disclosure is depicted;
[0014] Figure 5 This is a schematic flowchart of a method for landing area assessment according to one or more examples of this disclosure;
[0015] Figure 6A This is the first part of an illustrative flowchart of a method for landing area assessment according to one or more examples of this disclosure; and
[0016] Figure 6B This refers to a method for landing area assessment based on one or more examples of this disclosure. Figure 6A Continuing with the schematic flowchart. Detailed Implementation
[0017] Throughout this specification, references to "an embodiment," "embodiment," or similar language mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment of this disclosure. Throughout this specification, the phrases "in one embodiment," "in an embodiment," and similar language may, but do not necessarily all refer to the same embodiment. Similarly, the use of the term "implementation" means an implementation having a particular feature, structure, or characteristic described in connection with one or more embodiments of this disclosure; however, unless explicitly associated otherwise, an implementation may be associated with one or more embodiments.
[0018] Figure 1 This is a schematic block diagram of a system 100 for landing zone assessment according to one or more examples of this disclosure. As described herein, system 100 is a multi-sensor-based landing zone assessment system designed to assist aircraft landing (manual or autonomous). A high-resolution digital surface model of the landing zone is generated via sensor fusion based on data from various sensors (e.g., imaging and / or navigation) placed on the aircraft. Based on the generated digital surface model of the landing zone, a series of semantic representations are generated to assess the terrain conditions for landing (e.g., obstacles above the ground, surface roughness, surface normals, etc.). Navigation instructions and / or suggestions can be generated and provided to autonomous aircraft 110, 112 (e.g., unmanned aerial vehicles) and / or pilots (onboard or remotely) to avoid any objects, hazards, or other structures within the landing zone, while enabling aircraft 110, 112 to land without prior knowledge of the landing zone area.
[0019] System 100 includes one or more information processing devices 102, one or more messaging devices 104, one or more data networks 106, one or more servers 108, and one or more aircraft 110, 112. Even if a specific number of information processing devices 102, messaging devices 104, data networks 106, one or more servers 108, and aircraft 110, 1112 are depicted in... Figure 1 However, those skilled in the art will recognize that, according to this disclosure, system 100 may include any number of information processing devices 102, messaging devices 104, data networks 106, servers 108, and aircraft 110, 112.
[0020] The information processing device 102 of system 100 may include one or more of the following: desktop computer, laptop computer, tablet computer, smartphone, smart speaker (e.g., Amazon). Google Apple Security systems, set-top boxes, game consoles, smart TVs, smartwatches, fitness trackers or other wearable activity tracking devices, optical head-mounted displays (e.g., virtual reality headsets, smart glasses, etc.), high-definition multimedia interfaces (“HDMI”) or other electronic display dongles, personal digital assistants, digital cameras, camcorders, or other computing devices including processors (e.g., central processing units (“CPU”), processor cores, field-programmable gate arrays (“FPGA”) or other programmable logic, application-specific integrated circuits (“ASIC”), controllers, microcontrollers and / or other semiconductor integrated circuit devices), volatile memory and / or non-volatile storage media.
[0021] In some embodiments, the information processing device 102 may be located on aircraft 110, 112 and communicatively coupled to one or more other information processing devices 102, one or more servers 108, and / or one or more aircraft 110 via data network 106, as described below. The information processing device 102 may include a processor, processor core, etc., configured to execute various programs, program code, systems, applications, instructions, functions, etc., for sending and receiving messages, comparing information, storing and retrieving data in a database, etc. In some embodiments, the program code may execute on a graphics processing unit (“GPU”).
[0022] In one embodiment, landing device 104 is configured to capture images of the landing area of aircraft 110, 112 in real time while aircraft 110, 112 are in flight. One or more imaging sensors coupled to aircraft 110, 112 can be used to capture the images. In a further embodiment, landing device 104 is configured to associate a corresponding location data from a plurality of location data with each of the captured images. One or more navigation sensors coupled to aircraft 110, 112 can be used to determine the plurality of location data. In some embodiments, landing device 104 is configured to use the captured images and the location data associated with the captured images to identify and locate one or more objects within the landing area relative to aircraft 110, 112. Reference is made below. Figure 2 The landing device 104 is described in more detail, including its various modules.
[0023] In various embodiments, the landing device 104 is located on the aircraft 110, 112 as part of a flight information system or other computing system associated with the aircraft 110, 112; located on or communicatively coupled to a ground-based computing system; and / or some combination thereof.
[0024] In some embodiments, the landing device 104 is implemented as a hardware appliance that may be mounted or deployed on the information processing device 102, the server 108, the aircraft 110, 112, and / or other locations on the data network 106. In some examples, the landing device 104 includes hardware devices such as security hardware dongles or other hardware appliances (e.g., set-top boxes, network devices, etc.) that are connected via a wired connection (e.g., a Universal Serial Bus (“USB”) connection) or a wireless connection (e.g., [missing information]). Wi-Fi, near field communication (“NFC”, etc.) are attached to devices, laptops, servers 108, tablets, smartphones, security systems, etc.; attached to electronic display devices (e.g., televisions or monitors using HDMI ports, DisplayPort ports, Mini DisplayPort ports, VGA ports, DVI ports, etc.); and so on. The hardware of the landing device 104 includes a power interface, a wired and / or wireless network interface, a graphics interface attached to a display, and / or a semiconductor integrated circuit device as described below, configured to perform the functions described herein regarding the landing device 104.
[0025] In some embodiments, landing device 104 includes semiconductor integrated circuit devices (e.g., one or more chips, dies, or other discrete logic hardware), such as field-programmable gate arrays (“FPGAs”), or in some examples, other programmable logic, firmware for the FPGA or other programmable logic, and microcode for execution on a microcontroller, application-specific integrated circuit (“ASIC”), processor, processor core, etc. In one embodiment, landing device 104 is mounted on a printed circuit board having one or more wires or connections (e.g., connections to volatile memory, non-volatile storage media, network interfaces, peripherals, graphics / display interfaces, etc.). Hardware devices include one or more pins, pads, or other electrical connections configured to send and receive data (e.g., communicating with one or more wires on the printed circuit board, etc.), and one or more hardware circuits and / or, in some examples, other circuits configured to perform various functions of landing device 104.
[0026] In some embodiments, the semiconductor integrated circuit devices or other hardware devices of the landing device 104 include one or more volatile storage media and / or are communicatively coupled thereto. These volatile storage media may include, but are not limited to, random access memory (“RAM”), dynamic RAM (“DRAM”), cache, etc. In one embodiment, the semiconductor integrated circuit devices or other hardware devices of the landing device 104 include one or more non-volatile storage media and / or are communicatively coupled thereto. These non-volatile storage media may include, but are not limited to: NAND flash memory, NOR flash memory, nanometer random access memory (nano RAM or NRAM), nanowire-based memory, silicon oxide-based sub-10 nanometer process memory, graphene memory, silicon-oxide-nitride-oxide-silicon (“SONOS”), resistive RAM (“RRAM”), programmable metallization cell (“PMC”), conductive bridge RAM (“CBRAM”), magnetoresistive RAM (“MRAM”), dynamic RAM (“DRAM”), phase-changing RAM (“PRAM” or “PCM”), magnetic storage media (e.g., hard disk, magnetic tape), optical storage media, etc.
[0027] In another embodiment, the landing device 104 comprises software services and software applications. The landing device, system, modules, etc., may be executed in the memory of another device and communicate with the illustrated computing system, flight information system, aircraft 110, 112, or other information processing device 102 via computer-to-computer communication hosted for processing images, position data, etc., to assess the landing area.
[0028] In one embodiment, data network 106 includes a digital communication network for transmitting digital communications. In some examples, data network 106 includes a wireless network such as a wireless cellular network, or a local wireless network such as a Wi-Fi network. Networks, near field communication (“NFC”) networks, ad hoc networks, etc. In some examples, data network 106 includes a wide area network (“WAN”), a storage area network (“SAN”), a local area network (LAN), a fiber optic network, the Internet, or other digital communication networks. In some examples, data network 106 includes two or more networks. In one example, data network 106 includes one or more servers, routers, switches, and / or other network devices. In some examples, data network 106 includes one or more computer-readable storage media, such as hard disk drives, optical drives, non-volatile memory, RAM, etc.
[0029] In one example, the wireless connection is a mobile phone network. In another example, the wireless connection uses a Wi-Fi network based on any of the Institute of Electrical and Electronics Engineers (“IEEE”) 802.11 standards. Alternatively, in some examples, the wireless connection is... Connection. Furthermore, in various examples, the wireless connection employs radio frequency identification (“RFID”) communication, which includes protocols developed by the International Organization for Standardization (“ISO”), the International Electrotechnical Commission (“IEC”), and the United States Testing and Research Institute (USTRICO). DASH7 TM Alliance and EPCGlobal TM RFID
[0030] Alternatively, wireless connectivity can be based on the IEEE 802 standard. Connection. In one embodiment, the wireless connection employs a technology developed by Sigma. Design Connection. Alternatively, wireless connectivity can be achieved using a technology from Cochrane, Canada. Innovations defines and / or connect.
[0031] The wireless connection can be an infrared connection, which includes at least the following: The connection conforms to the defined Infrared Physical Layer Specification (“IrPHY”). Alternatively, the wireless connection may be cellular telephone network communication. All standards and / or connection types include the latest versions and revisions of the standards and / or connection types as of the date of this application.
[0032] In one embodiment, one or more servers 108 are implemented as blade servers, mainframe servers, tower servers, rack servers, etc. In some examples, one or more servers 108 are configured as mail servers, network servers, application servers, FTP servers, media servers, data servers, web servers, file servers, virtual servers, etc. In some examples, one or more servers 108 are communicatively coupled (e.g., networked) to one or more information processing devices 102 via a data network 106. In some examples, one or more servers 108 store data in data structures such as databases, data repositories, etc.
[0033] In some embodiments, one or more servers 108 may be located on aircraft 110, 112, on the ground as part of a ground-based computing system, etc. One or more servers 108 may include storage for storing images, video, and / or location data and / or for processing video / image data to identify / locate objects within the landing zone of aircraft 110, 112 (e.g., programs, applications, algorithms, libraries, etc.).
[0034] In one embodiment, aircraft 110 may include a fixed-wing aircraft, such as an airplane, glider, or drone. Another type of aircraft 112 may include a rotorcraft, such as a helicopter, gyroplane, gyroscope, or rotary kite. In either case, aircraft 110 or 112 may be operated by a pilot on board or remotely or autonomously controlled using an onboard flight control system.
[0035] In one embodiment, such as Figure 3 As described in more detail herein, aircraft 110 and 112 include one or more imaging sensors coupled to aircraft 110 and 112. Imaging sensors may include electro-optic cameras (e.g., stereo EO cameras), light detection and ranging (“LIDAR”) sensors, etc. For simplicity, the subject matter disclosed herein refers to imaging sensors as LIDAR sensors / cameras, but those skilled in the art will recognize that various imaging sensors / cameras can be used alone or in combination with LIDAR cameras. Imaging sensors may be placed on aircraft 110 and 112 at locations that minimize errors, interference, distortion, etc., caused by the movement of aircraft 110 and 112.
[0036] In a further embodiment, aircraft 110, 112 include navigation sensors, position sensors, positioning sensors, etc., coupled to aircraft 110, 112 at a location near the imaging sensor. As used herein, a navigation sensor can refer to a navigation device that uses a computer, motion sensors (e.g., accelerometers), and rotation sensors (e.g., gyroscopes) to continuously calculate the position, orientation, and velocity (direction and speed of motion) of a moving object via dead reckoning. Navigation sensors may include inertial navigation systems (“INS”), global positioning systems (“GPS”) sensors, altimeters, gyroscopes, barometers, speedometers, and / or other position or positioning sensors that, when aircraft 110, 112 are in flight, are able to determine, estimate, calculate, etc., the position, location, motion, orientation, etc. of aircraft 110, 112 relative to the ground (e.g., landing zone), relative to objects on the ground, and relative to other aircraft 110, 112, etc.
[0037] In some embodiments, each imaging sensor in the imaging sensor array is coupled adjacently to the corresponding navigation sensor to the aircraft 110, 112. For example, the imaging sensor and the corresponding navigation sensor may be located together within the same housing coupled to / coupled to the aircraft 110, 112 (e.g., coupled to the underside of the aircraft 110, 112). In such an embodiment, the imaging sensor and the corresponding navigation sensor are positioned such that they are no more than 0.5 meters apart from each other.
[0038] In some embodiments, the imaging sensor and the navigation sensor are communicatively or operatively coupled such that when an image is captured by the imaging sensor, position or location data is typically captured simultaneously, allowing the captured image to be correlated with the captured position data. In some embodiments, aircraft 110, 112 include at least two imaging sensors and at least two corresponding navigation sensors. In one embodiment, at least one imaging sensor is located below aircraft 110, 112 and oriented along a horizontal plane, while another at least one imaging sensor is located below aircraft 110, 112 and oriented along a vertical plane, providing a view of the landing area when aircraft 110, 112 approach and are above the landing area.
[0039] In some embodiments, the number and orientation of one or more imaging sensors are determined based on one or more specifications of the aircraft 110, 112. In some embodiments, one or more specifications of the aircraft 110, 112 include the type of aircraft 110, 112, the size of the aircraft 110, 112, the shape of the aircraft 110, 112, and the configuration of the landing gear of the aircraft 110, 112. For example, a larger aircraft 110, 112 may allow for more imaging and navigation sensors than a smaller aircraft 110, 112. In another example, the imaging and navigation sensors may be positioned on the aircraft 110, 112 based on its shape and / or fuselage configuration. In this way, various aircraft manufacturers and models can be supported, and the imaging sensor settings (e.g., resolution, frame rate, etc.) can be adjusted based on aircraft specifications.
[0040] Figure 2 This is a schematic block diagram of an apparatus 200 for landing area assessment according to one or more examples of this disclosure. In one embodiment, apparatus 200 includes an instance of landing apparatus 104. In some embodiments, landing apparatus 104 includes an imaging module 202, a location determination module 204, an object recognition module 206, and an output module 208, which will be described in more detail below.
[0041] In one embodiment, imaging module 202 is configured to capture images of the landing area of aircraft 110, 112 in real time using one or more imaging sensors coupled to aircraft 110, 112 while in flight. As used herein, landing area may refer to an area on the ground where aircraft 110, 112 is configured to land, such as a runway, emergency landing site, and / or helicopter landing pad.
[0042] When aircraft 110 and 112 are in flight, imaging module 202 can capture images continuously, periodically, at predetermined intervals, etc. For example, imaging module 202 can use imaging sensors to capture images in real time (e.g., at 30 frames per second, 60 frames per second, etc.), per millisecond, per microsecond, per second, every 5 seconds, every 10 seconds, every 30 seconds, every minute, etc.
[0043] In some embodiments, the rate at which the imaging module 202 captures images is determined based on the position of the aircraft 110, 112 and the specifications of the imaging sensor (e.g., the rate at which the imaging sensor can capture images). (e.g., the imaging module 202 may increase the rate at which it captures images as the aircraft 110, 112 gets closer to the landing area.)
[0044] In some embodiments, the imaging module 202 does not use the imaging sensor to capture images until the aircraft 110, 112 is in a specific mode (e.g., landing mode or landing gear is detected to be deployed), within a predetermined proximity to the landing zone (e.g., within 5 miles), detects that the aircraft 110, 112 has begun to land (e.g., detects that the aircraft's altitude or elevation is decreasing or detects that the speed of the aircraft 110, 112 is decreasing), or detects that the aircraft has reached its destination (e.g., based on the flight path and position of the aircraft 110, 112), etc.
[0045] In some embodiments, the position determination module 204 is configured to associate a corresponding position data from a plurality of position data with each of the captured images, the plurality of position data being determined using one or more navigation sensors coupled to the aircraft 110, 112. For example, the position determination module 204 may read, access, or otherwise determine position data from an inertial navigation system (INS) sensor associated with the imaging sensor (e.g., within the same housing) at approximately the same time that the imaging module 202 captures an image using an imaging sensor.
[0046] For example, the location determination module 204 can align the LIDAR image with the INS location data in time and space by attaching, assigning, correlating, etc., the most recent INS signal to the LIDAR image.
[0047] In one embodiment, object recognition module 206 is configured to use captured images and location data associated with the captured images to identify and locate one or more objects relative to the aircraft within the landing area. For example, object recognition module 206 may use various image processing algorithms, processes, methods, techniques, programs, etc., to process a series of LIDAR images to identify objects that may exist within the landing area.
[0048] In one embodiment, object recognition module 206 applies a point cloud transformation to a captured image, such as a LiDAR image, to generate a point cloud. As used herein, a point cloud can refer to a set of data points in space. In such an embodiment, object recognition module 206 transforms image data captured by imaging module 202 using imaging sensors from the coordinate system of one or more imaging sensors to the coordinate system of aircraft 110, 112.
[0049] In a further embodiment, the object recognition module 206 filters the generated point cloud by identifying and removing statistical outliers from the point cloud to generate a cleaned point cloud. For example, the object recognition module 206 may apply a statistical outlier removal algorithm to the generated point cloud to remove noise points from the generated point cloud to create a cleaned point cloud.
[0050] In various embodiments, the object recognition module 206 downsamples the cleaned point cloud. For example, the object recognition module 206 uses cube pixels to downsample the cleaned point cloud to improve the efficiency of further processing of the cleaned point cloud.
[0051] In a further embodiment, the object recognition module 206 uses a mapping of multiple location data to the time-aligned captured image to localize (e.g., geo-register) the cleaned point cloud to world coordinates. For example, the object recognition module 206 can use INS location data that is temporally aligned with the cleaned point cloud data to geo-register the cleaned point cloud to working coordinates. In such an embodiment, each point in the cleaned point cloud is associated with latitude, longitude, and altitude values derived from a corresponding location data from multiple location data sets, and each point will have its latitude, longitude, altitude / height, etc., data registered to real-world coordinates.
[0052] In a further embodiment, the object recognition module 206 generates a surface model of the landing area based on a localized / geo-registered point cloud. As used herein, the surface model may refer to the surface topography of the landing area. In such an embodiment, the object recognition module 206 dynamically performs surface topography analysis on the surface model to further process the surface topography of the landing area to identify objects within the landing area.
[0053] For example, in one embodiment, the object recognition module 206 may apply a plane fitting algorithm to determine ground objects within the landing area by fitting one or more planes to a localized point cloud using Random Sample Consensus (“RANSAC”) (assuming the landing area is a flat surface).
[0054] In another example embodiment, the object recognition module 206 applies a surface roughness algorithm to calculate the surface roughness index (“SRI”) of the landing area to highlight corners and edges within the landing area based on a localized point cloud. In such an example, the object recognition module 206 uses a 2.5D digital surface model to calculate the SRI to highlight corners / edges on the ground.
[0055] In a further example embodiment, the object recognition module 206 applies a surface normal algorithm to compute the surface normal of each point in the localized point cloud of the landing area using singular value decomposition (“SVD”). In such an example, the object recognition module 206 computes the surface normal of each point in the localized point cloud, which generates a surface normal representation within the landing area.
[0056] Based on this disclosure, those skilled in the art will recognize other surface analysis algorithms that can be used to analyze the surface structure of landing areas, such as landmark localization, 3D intelligence, surveillance and reconnaissance (“ISR”), runway measurement, land surveying, 3D airport mapping, etc. Furthermore, the generated surface models can be used for other applications, such as disaster relief, weapon effectiveness analysis, etc.
[0057] In a further embodiment, the object recognition module 206 may use various artificial intelligence / machine learning algorithms, such as supervised and unsupervised machine learning algorithms, to process the captured image data, generate point clouds, process and analyze the point clouds, and identify objects in the landing area based on previously generated historical or training data and the point clouds generated by machine learning.
[0058] In some embodiments, output module 208 is configured to generate and provide one or more navigation instructions to avoid one or more objects within the landing zone upon landing of aircraft 110, 112. In one embodiment, output module 208 may derive navigation instructions from a surface model. In some embodiments, the navigation instructions include instructions for maneuvering aircraft 110, 112 to avoid objects in the landing zone, such as left, right, forward, backward, up, down, slower, faster, higher, lower, etc.
[0059] In some embodiments, aircraft 110 and 112 are autonomous, and output module 208 transmits, sends, or otherwise provides navigation commands to the flight control system of autonomous aircraft 110 and 112, enabling autonomous aircraft 110 and 112 to manipulate themselves to avoid objects in the landing area according to the navigation commands.
[0060] In some embodiments, output module 208 provides, for example, a visual graphic of the surface model of the landing area on a display within aircraft 110, 112. In some embodiments, the visual graphic highlights one or more identified objects within the landing area, making the one or more identified objects distinguishable from areas of the landing area without objects. For example, output module 208 may mark the following areas of the landing area on a graphical representation of the landing area presented on the display: the area where objects are located, the area where aircraft 110, 112 has safely landed, etc.
[0061] Figure 3 Example embodiments of a system 300 for landing area assessment according to one or more examples of this disclosure are depicted. In one embodiment, system 300 includes aircraft 110, 112, such as helicopter 301. Helicopter 301 includes two imaging sensors 302 coupled to the underside of helicopter 301. The imaging sensors may include a LiDAR camera.
[0062] In one embodiment, one imaging sensor 302 is oriented along a generally horizontal axis A, while another imaging sensor 302 is oriented along a generally vertical axis B. The imaging sensors 302 may be associated with a navigation sensor 304, such as an INS sensor (e.g., within the same housing), which captures position data of the helicopter 301 at substantially the same time as capturing images using a LiDAR camera.
[0063] In some embodiments, system 300 includes a landing device 104 as described above, configured to capture and process LiDAR images to identify one or more objects 308 located within the field of view 310 of the LiDAR camera in landing area 306. Identified objects may include vehicles, people, structures, materials, pallets, crates, boxes, bags, containers, cones, building materials, debris, etc. Landing device 104 may be located on helicopter 301 (e.g., as part of a flight control system), on the ground (as part of ground computing system 312, which is communicatively coupled to helicopter 301 via data network 106), and / or some combination of both.
[0064] Landing device 104 uses LiDAR imagery and INS navigation / position data to identify object 308 in landing area 306 and provides output to assist in avoiding object 308 during helicopter 301 landing. For example, the output may include navigation commands / instructions / directions (e.g., provided to the pilot on board or away from the helicopter, or provided to the autonomous flight control system), a graphical or visual display of the landing area with highlighted object 308, etc. Figures 4A-4B As shown.
[0065] Figure 4A Example embodiment 400 depicts the output of a landing zone-based assessment according to one or more examples of this disclosure. Figure 4A The diagram depicts a point cloud 402 generated, processed, cleaned, and analyzed by the landing device 104 based on images captured using an imaging sensor. In the example output, the landing device 104 has identified at least one object 404 in the landing zone that aircraft 110 and 112 should avoid during landing. Therefore, the landing device 104 can generate and provide navigation instructions for aircraft 110 and 112 to avoid object 404 during their landing.
[0066] Figure 4B Another example embodiment 450 depicts the output of a landing zone-based assessment according to one or more examples of this disclosure. Figure 4BAnother point cloud 402 is depicted, highlighting different areas of the landing zone based on the safety criteria of aircraft 110, 112 landing in these areas without impacting any objects within the landing zone. For example, unsafe areas of landing zone 452, such as areas where objects are located within the landing zone, may be colored, or otherwise highlighted differently from safe areas of landing zone 454 (e.g., areas where no objects are located within the landing zone), so that the pilots or other controllers of aircraft 110, 112 can easily see whether there are objects in the landing zone and areas of the landing zone that should be avoided during landing.
[0067] Figure 5 This is a schematic flowchart of a method 500 for landing area assessment according to one or more examples of this disclosure. Method 500 begins and captures images of the landing areas of aircraft 110 and 112 in real time while they are in flight. The captured images are obtained using one or more imaging sensors coupled to aircraft 110 and 112. Method 500 associates a corresponding position data from a plurality of position data with each of the captured images. The plurality of position data is determined using one or more navigation sensors coupled to aircraft 110 and 112.
[0068] Method 500 uses captured images and location data associated with the captured images to identify and locate one or more objects within the landing area relative to aircraft 110, 112, and method 500 terminates. In some embodiments, imaging module 202, location determination module 204, and object recognition module 206 perform the individual steps of method 500.
[0069] Figure 6A This is the first part of a schematic flowchart of a method 600 for landing area assessment according to one or more examples of this disclosure. Method 600 begins and captures images of the landing areas of aircraft 110, 112 in real time while the aircraft 110, 112 is in flight. The captured images are obtained using one or more imaging sensors coupled to aircraft 110, 112. Method 600 associates a corresponding position data from a plurality of position data with each of the captured images. The plurality of position data is determined using one or more navigation sensors coupled to aircraft 110, 112.
[0070] Method 600 applies a point cloud transformation (606) to the captured image to generate a point cloud. The image data is transformed from the coordinate system of one or more imaging sensors to the coordinate system of the aircraft 110, 112. Method 600 filters the generated point cloud (608) to generate a cleaned point cloud by identifying and removing statistically abnormal data from the point cloud.
[0071] Method 600 uses a mapping of location data to time-aligned values for each captured image to downsample the cleaned point cloud 610 and localizes the cleaned point cloud 612 to world coordinates. Each point in the cleaned point cloud is associated with latitude, longitude, and altitude derived from the location data. Method 600 generates 614 a surface model of the landing area based on the localized point cloud. The surface model includes the surface topography of the landing area. One or more navigation commands are derived from the surface model.
[0072] Figure 6B This is a method 600 for landing area assessment based on one or more examples of this disclosure. Figure 6A Continuing with the schematic flowchart, method 600 continues and dynamically performs 616 surface topography analysis to further process the surface topography of the landing area. Surface topography analysis may include plane fitting 616A for determining ground objects within the landing area by fitting one or more planes to a localized point cloud using Random Sample Consensus (“RANSAC”).
[0073] Surface topography analysis may include a surface roughness index (“SRI”) for calculating the surface roughness 616B of corners and edges within the landing area based on a localized point cloud. Surface topography analysis may include surface normals 616C for calculating the surface normal of each point in the localized point cloud of the landing area using singular value decomposition (“SVD”).
[0074] Method 600 identifies and locates 618 one or more objects within the landing zone relative to aircraft 110, 112 based on surface terrain analysis. The method provides 620 one or more navigation instructions to avoid one or more objects within the landing zone during landing of aircraft 110, 112, and method 600 terminates. In some embodiments, imaging module 202, position determination module 204, object recognition module 206, and output module 208 perform the steps of method 600.
[0075] Furthermore, this disclosure includes embodiments pursuant to the following provisions:
[0076] Clause 1. A system comprising:
[0077] One or more imaging sensors coupled to the aircraft;
[0078] One or more navigation sensors of the aircraft are coupled in the vicinity of the one or more imaging sensors;
[0079] An imaging module configured to capture images of the landing area of the aircraft while the aircraft is in flight, wherein the captured images are captured using one or more imaging sensors;
[0080] A position determination module configured to associate a corresponding location data point from a plurality of location data points with each of the captured images, wherein the plurality of location data points are determined using the one or more navigation sensors coupled to the aircraft; and
[0081] An object recognition module is configured to use the captured image and the location data associated with the captured image to identify and locate one or more objects relative to the aircraft within the landing area.
[0082] Clause 2. The system according to Clause 1, wherein each of the one or more imaging sensors is coupled adjacently to the aircraft to a corresponding navigation sensor of the one or more navigation sensors.
[0083] Clause 3. The system according to any one of Clauses 1 or 2, wherein one of the one or more imaging sensors and its corresponding navigation sensor are positioned adjacent to each other within a housing, the housing being coupled / coupled to the underside of the aircraft.
[0084] Clause 4. The system according to any one of Clauses 1-3, wherein the imaging sensor and its corresponding navigation sensor are positioned such that they are no more than 0.5 meters apart.
[0085] Clause 5. The system according to any one of Clauses 1-4, wherein the aircraft is an autonomous aircraft that processes one or more navigation commands to avoid the one or more objects within the landing area without user interaction.
[0086] Clause 6. The system according to any one of Clauses 1-5, wherein the aircraft includes at least a first imaging sensor oriented along a horizontal plane and a second imaging sensor oriented along a vertical plane.
[0087] Clause 7. The system according to any one of Clauses 1-6, wherein the number and orientation of the one or more imaging sensors are determined based on one or more specifications of the aircraft, the one or more specifications including at least one of the following: the type of the aircraft, the size of the aircraft, the shape of the aircraft, or the construction of the aircraft's landing gear.
[0088] Clause 8. A method comprising:
[0089] The image of the landing area of the aircraft is captured while the aircraft is in flight, wherein the captured image is captured using one or more imaging sensors coupled to the aircraft;
[0090] Associate one location data point from a plurality of location data points with each of the captured images, wherein the plurality of location data points are determined using one or more navigation sensors coupled to the aircraft; and
[0091] The captured images and the location data associated with the captured images are used to identify and locate one or more objects relative to the aircraft within the landing area.
[0092] Clause 9. The method described in accordance with Clause 8 further includes:
[0093] A point cloud is generated by applying a point cloud transformation to the captured image data from the first coordinate system of the one or more imaging sensors to the second coordinate system of the aircraft.
[0094] The generated point cloud is filtered by identifying and removing statistically abnormal data from the point cloud to generate a cleaned point cloud; and
[0095] The cleaned point cloud is downsampled.
[0096] Clause 10. The method according to any one of Clauses 8 or 9 further comprises localizing the cleaned point cloud to world coordinates using the location data and a time-aligned mapping of each of the captured images, each point in the cleaned point cloud being associated with latitude, longitude, and altitude values derived from the location data, wherein the localization generates a localized point cloud.
[0097] Clause 11. The method according to any one of Clauses 8-10 further comprises generating a surface model of the landing area based on the localized point cloud, the surface model including the surface terrain of the landing area, wherein one or more navigation instructions are derived from the surface model.
[0098] Clause 12. The method described under any one of Clauses 8-11 further includes providing one or more navigation instructions for avoiding the one or more objects within the landing area when the aircraft lands.
[0099] Clause 13. A computer program product comprising a computer-readable storage medium containing program code executable by a processor for:
[0100] Images of the landing area of the aircraft are received from one or more imaging sensors coupled to the aircraft, wherein the images are captured using the one or more imaging sensors while the aircraft is in flight;
[0101] Associate one location data point from a plurality of location data points with each of the captured images, wherein the plurality of location data points are determined using one or more navigation sensors coupled to the aircraft; and
[0102] The captured images and the location data associated with the captured images are used to identify and locate one or more objects relative to the aircraft within the landing area.
[0103] Clause 14. The computer program product according to Clause 13, wherein the program code is further executable by the processor to:
[0104] A point cloud is generated by applying a point cloud transformation to the captured image data from the first coordinate system of the one or more imaging sensors to the second coordinate system of the aircraft.
[0105] The generated point cloud is filtered by identifying and removing statistically abnormal data from the point cloud to generate a cleaned point cloud; and
[0106] The cleaned point cloud is downsampled.
[0107] Clause 15. A computer program product according to any one of Clauses 13 or 14, wherein the program code is further executable by the processor to localize the cleaned point cloud to world coordinates using a temporal alignment mapping of the plurality of location data with the captured image, wherein the localization generates a localized point cloud; and
[0108] Each point in the cleaned point cloud is associated with a latitude, longitude, and altitude value derived from a corresponding location data from the plurality of location data.
[0109] Clause 16. The computer program product according to any one of Clauses 13-15, wherein the program code is further executable by the processor for generating a surface model of the landing area based on the localized point cloud, wherein the surface model includes a surface terrain model of the landing area, and one or more navigation instructions are derived from the surface model.
[0110] Clause 17. A computer program product according to any one of Clauses 13-16, wherein the program code is further executable by the processor to provide a visual graphic of the surface model of the landing area, the visual graphic highlighting one or more identified objects within the landing area such that the one or more identified objects are distinguishable from areas of the landing area without objects.
[0111] Clause 18. The computer program product according to any one of Clauses 13-17, wherein the program code is further executable by the processor to perform surface topography analysis for further surface topography processing of the landing area.
[0112] Clause 19. The computer program product according to any one of Clauses 13-18, wherein the surface topography analysis includes one or more of the following:
[0113] Plane fitting is used to determine the ground objects within the landing area by fitting one or more planes to the localized point cloud using Random Sample Consensus (“RANSAC”).
[0114] Used to calculate the Surface Roughness Index (“SRI”) of the landing area to highlight the surface roughness of corners and edges within the landing area based on the localized point cloud; and
[0115] Surface normals used to compute the surface normals of each point in the localized point cloud of the landing area using singular value decomposition (“SVD”).
[0116] Clause 20. A computer program product according to any one of Clauses 13-19, wherein the program code is further executable by the processor to determine and provide one or more navigation instructions for avoiding the one or more objects in the landing zone during the landing of the aircraft.
[0117] In the above description, certain terms such as “upper,” “lower,” “upper part,” “lower part,” “horizontal,” “vertical,” “left,” “right,” “above,” “below,” etc., may be used. These terms are used, where applicable, to provide a clear description when dealing with relative relationships. However, these terms are not intended to imply absolute relationships, positioning, and / or orientation. For example, relative to an object, the “upper” surface can simply become the “lower” surface by flipping the object over. Nevertheless, it is still the same object. Furthermore, unless otherwise explicitly stated, the terms “including,” “contains,” “has,” “comprises,” “consisting,” and their variations mean “including but not limited to.” Unless otherwise explicitly stated, the list of listed items does not imply that any or all items are mutually exclusive and / or mutually inclusive. Unless otherwise explicitly stated, the terms “a,” “an,” and “the” also mean “one or more.” Furthermore, the term “multiple” can be defined as “at least two.”
[0118] Additionally, instances of an element being "coupled" to another element in this specification can include direct and indirect coupling. Direct coupling can be defined as one element coupling to another element and forming some kind of contact with it. Indirect coupling can be defined as coupling between two elements that are not in direct contact with each other but have one or more other elements between the coupled elements. Furthermore, as used herein, fixing one element to another element can include direct fixing and indirect fixing. Also, as used herein, "adjacent" does not necessarily mean contact. For example, an element may be adjacent to another element without contacting that element.
[0119] As used in this article, the phrase "at least one" when used with a list of items means that different combinations of one or more of the listed items may be used, and that only one of the items in the list may be required. The item can be a specific object, thing, or category. In other words, "at least one" means that any combination of items or multiple items can be used from the list, but not all items in the list may be required. For example, "at least one of items A, B, and C" could mean item A; item A and item B; item B; item A, item B, and item C; or item B and item C. In some cases, "at least one of items A, B, and C" could mean, for example, but not limited to, two items from item A, one item from item B, and ten items from item C; four items from item B and seven items from item C; or some other suitable combination.
[0120] Unless otherwise stated, the terms “first,” “second,” etc., are used merely as identifiers in this document and are not intended to impose any order, position, or hierarchy requirements on the items referred to by these terms. Furthermore, references to items such as “second” do not require or exclude the existence of items such as “first” or lower numbered items and / or items such as “third” or higher numbered items.
[0121] As used herein, a system, apparatus, structure, article, element, component, or hardware “configured to” perform a specified function is indeed capable of performing the specified function without any changes, rather than merely having the potential to perform the specified function after further modification. In other words, a system, apparatus, structure, article, element, component, or hardware “configured to” perform the specified function is specifically selected, created, implemented, utilized, programmed, and / or designed for the specified purpose. As used herein, “configured to” indicates existing characteristics of the system, apparatus, structure, article, element, component, or hardware that enable the system, apparatus, structure, article, element, component, or hardware to perform the specified function without further modification. For the purposes of this disclosure, a system, apparatus, structure, article, element, component, or hardware described as “configured to” perform a particular function may additionally or alternatively be described as “suitable” and / or “operable to” perform that function.
[0122] The illustrative flowcharts included herein are generally presented as logical flowcharts. Therefore, the depicted sequence and labeled steps indicate one embodiment of the proposed method. Other steps and methods that are functionally, logically, or effectively equivalent to one or more steps or portions thereof of the illustrated method are conceivable. Furthermore, the format and symbols used are provided to interpret the logical steps of the method and are understood not to limit the scope of the method. Although various arrow types and line styles may be used in the flowcharts, they should be understood not to limit the scope of the corresponding methods. In practice, some arrows or other connectors may be used only to indicate the logical flow of the method. For example, an arrow may indicate a wait or monitoring period of unspecified duration between enumeration steps of the depicted method. Additionally, the order in which a particular method occurs may or may not strictly follow the order of the corresponding steps shown.
[0123] Implementations of various modules may take the form of entirely hardware embodiments, entirely software embodiments (including firmware, resident software, microcode, etc.), or embodiments combining software and hardware aspects, all of which are generally referred to herein as “circuit,” “module,” or “system.” Furthermore, embodiments may take the form of a program product contained in one or more computer-readable storage devices that store machine-readable code, computer-readable code, and / or program code (hereinafter referred to as “code”). The storage device may be tangible, non-transitory, and / or non-transferable. The storage device may not contain signals. In one embodiment, the storage device uses only signals for accessing the code.
[0124] Modules can be implemented as hardware circuits that include custom-designed very large-scale integrated (VLSI) circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. Modules can also be implemented in programmable hardware devices such as field-programmable gate arrays, programmable array logic, programmable logic devices, etc.
[0125] Modules can also be implemented in code and / or software for execution by various types of processors. An identified code module may, for example, comprise one or more physical or logical blocks of executable code, which may be organized, for example, as objects, procedures, or functions. However, the executable files of the identified modules do not need to be physically located together, but may include different instructions stored in different locations that, when logically connected, constitute the module and achieve the module's stated purpose.
[0126] In practice, a code module can be a single instruction or many instructions, and can even be distributed across several different code segments, different programs, and several memory devices. Similarly, operational data can be identified and illustrated within this module, and can be implemented in any suitable form and organized within any suitable type of data structure. Operational data can be collected as a single dataset or can be distributed across different locations, including across different computer-readable storage devices. Where the module or part of a module is implemented in software, the software portion is stored on one or more computer-readable storage devices.
[0127] Modules may utilize any combination of one or more computer-readable media. A computer-readable medium may be a computer-readable storage medium. A computer-readable storage medium may be a storage device for storing code. A storage device may be, for example, but not limited to, electronic, magnetic, optical, electromagnetic, infrared, holographic, micromechanical, or semiconductor systems, apparatuses, or devices, or any suitable combination thereof.
[0128] More specific examples of storage devices (a non-exhaustive list) will include the following devices: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable optical disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium can be any tangible medium that can contain or store programs for use by or in connection with an instruction execution system, apparatus, or device.
[0129] The code used to implement the operations of the embodiments can be written in any combination of one or more programming languages, including object-oriented programming languages such as Python, Ruby, Java, Smalltalk, C++, etc., as well as traditional procedural programming languages such as the "C" programming language, and / or machine languages such as assembly language. The code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0130] The subject matter of this disclosure may be implemented in other specific forms without departing from the spirit or essential characteristics of this disclosure. The described embodiments should be considered illustrative rather than restrictive in all respects. All modifications falling within the meaning and scope of the equivalents of the claims should be included within their scope.
Claims
1. A system (100 / 300) comprising: One or more imaging sensors (302) coupled to the aircraft (110); One or more navigation sensors (304) are coupled to the aircraft (110) in the vicinity of the one or more imaging sensors (302). An imaging module (202) is configured to capture images of the landing area (306) of the aircraft (110) while the aircraft (110) is in flight, wherein the captured images are captured using the one or more imaging sensors (302); A position determination module (204) configured to associate a corresponding position data from a plurality of position data with each of the captured images, wherein the plurality of position data is determined using one or more navigation sensors (304) coupled to the aircraft (110); and Object recognition module (206), the object recognition module is configured to: A point cloud is generated by applying a point cloud transformation to the captured image data, which is then transformed from the first coordinate system of the one or more imaging sensors to the second coordinate system of the aircraft. The captured images and the location data associated with the captured images are used to identify and locate one or more objects within the landing area (306) relative to the aircraft (110); The generated point cloud is filtered (608) by identifying and removing statistically abnormal data from the point cloud to generate a cleaned point cloud; The cleaned point cloud is downsampled (610); The cleaned point cloud is localized to world coordinates using the location data and a time-aligned mapping of each of the captured images, wherein each point in the cleaned point cloud is associated with a latitude, longitude, and altitude value derived from the location data, and the localization generates a localized point cloud; and A surface model of the landing area is generated based on the localized point cloud. The surface model includes the surface terrain of the landing area, wherein one or more navigation commands are derived from the surface model.
2. The system according to claim 1, wherein, Each of the one or more imaging sensors (302) is coupled adjacently to the aircraft (110) to a corresponding navigation sensor in the one or more navigation sensors (304).
3. The system according to claim 2, wherein, One of the one or more imaging sensors (302) and its corresponding navigation sensor (304) are positioned adjacent to each other within a housing coupled to the underside of the aircraft (110).
4. The system according to claim 3, wherein, The imaging sensor (302) and its corresponding navigation sensor (304) are positioned such that they are no more than 0.5 meters apart.
5. The system according to any one of claims 1 to 4, wherein, The aircraft (110) is an autonomous aircraft that processes one or more navigation commands to avoid one or more objects within the landing area (306) without user interaction.
6. The system according to any one of claims 1 to 4, wherein, The aircraft (110) includes at least a first imaging sensor (302) oriented along a horizontal plane (A) and a second imaging sensor oriented along a vertical plane (B).
7. The system according to any one of claims 1 to 4, wherein, The number and orientation of the one or more imaging sensors (302) are determined based on one or more specifications of the aircraft (110), including the type of the aircraft, the size of the aircraft, the shape of the aircraft, and the construction of the aircraft's landing gear.
8. A method (500 / 600) comprising: While the aircraft (110) is in flight, an image of the landing area (306) of the aircraft (110) is captured (502), wherein the captured image is captured using one or more imaging sensors (302) coupled to the aircraft (110); Associate one of the multiple location data with each of the captured images (504), wherein the multiple location data are determined using one or more navigation sensors (304) coupled to the aircraft (110); Point cloud is generated by applying a point cloud transformation to the captured images, transforming the image data from a first coordinate system of the one or more imaging sensors to a second coordinate system of the aircraft; and The captured images and the location data associated with the captured images are used to identify (506) and locate one or more objects within the landing area (306) relative to the aircraft (110). The generated point cloud is filtered (608) by identifying and removing statistically abnormal data from the point cloud to generate a cleaned point cloud; The cleaned point cloud is downsampled (610); The cleaned point cloud is localized to world coordinates using the location data and a time-aligned mapping of each of the captured images, wherein each point in the cleaned point cloud is associated with a latitude, longitude, and altitude value derived from the location data, and the localization generates a localized point cloud; and A surface model of the landing area is generated based on a localized point cloud. The surface model includes the surface terrain of the landing area, wherein one or more navigation commands are derived from the surface model.
9. The method of claim 8, further comprising providing one or more navigation instructions for avoiding the one or more objects within the landing area during the landing of the aircraft.
10. A computer program product comprising a computer-readable storage medium containing program code, the program code being executable by a processor to: Images of the landing area (306) of the aircraft (110) are received from one or more imaging sensors (302) coupled to the aircraft (110), wherein, The images were captured using the one or more imaging sensors (302) while the aircraft (110) was in flight; Associate one of the multiple location data with each of the captured images (204), wherein the multiple location data are determined using one or more navigation sensors (304) coupled to the aircraft (110); Point cloud is generated by applying a point cloud transformation to the captured images, transforming the image data from a first coordinate system of the one or more imaging sensors to a second coordinate system of the aircraft; and Using the captured images and the location data associated with the captured images to identify and locate (206) one or more objects within the landing area (306) relative to the aircraft (110); and The program code can be further executed by the processor for: The generated point cloud is filtered by identifying and removing statistically abnormal data from the point cloud to generate a cleaned point cloud; and The cleaned point cloud is downsampled; The program code is further executable by the processor to localize the cleaned point cloud to world coordinates using a temporal alignment mapping of the plurality of location data and the captured image, wherein the localization generates a localized point cloud; and Each point in the cleaned point cloud is associated with a latitude, longitude, and altitude value derived from a corresponding location data point among the plurality of location data; and The program code can be further executed by the processor to generate a surface model of the landing area based on the localized point cloud, wherein the surface model includes a surface terrain model of the landing area, and one or more navigation instructions are derived from the surface model.
11. The computer program product according to claim 10, wherein, The program code can further be executed by the processor to provide a visual graph of the surface model of the landing area, the visual graph highlighting one or more identified objects within the landing area such that the one or more identified objects are distinguishable from areas of the landing area without objects.
12. The computer program product according to claim 10, wherein, The program code can further be executed by the processor to perform surface topography analysis for further surface topography processing of the landing area.
13. The computer program product according to claim 12, wherein, The surface topography analysis includes one or more of the following: Used to determine the plane fitting of ground objects within the landing area by fitting one or more planes to the localized point cloud using random sample consistency; Used to calculate the surface roughness index of the landing area to highlight the surface roughness of corners and edges within the landing area based on the localized point cloud; as well as Surface normals used to compute the surface normals of each point in the localized point cloud of the landing area using singular value decomposition.
14. The computer program product according to claim 10, wherein, The program code may further be executed by the processor to determine and provide one or more navigation instructions for avoiding one or more objects in the landing zone when the aircraft lands.
Citation Information
Patent Citations
Building group orientated unmanned aerial vehicle autonomous navigation method
CN109163718A
Landing hazard avoidance display
US20160137309A1
Backup Navigation System for Unmanned Aerial Vehicles
US20190080142A1
