Automatic landing surface terrain assessment and related systems and methods

Abstract

Automatic landing surface terrain assessment and related systems and methods are disclosed herein. A representative method includes receiving a request to land a movable object, and in response to the request, identifying a target landing area on the landing surface based on at least one image of the landing surface obtained by the movable object. The method may also include guiding the movable object to land at the target landing area.

Description

Technical Field

[0001] The present technology generally relates to devices for automatically measuring or otherwise assessing the topography of a landing surface or other target surface, and related systems and methods. Background Art

[0002] An unmanned aerial vehicle (UAV) is an aerial vehicle that does not have a human pilot on board. UAVs can be operated remotely by a human controller outside the aircraft, or autonomously by an onboard computer. The use of UAVs is rapidly expanding in military, commercial, scientific, recreational, and other applications. Summary of the invention

[0003] The following summary is provided for the convenience of the reader and identifies several representative embodiments of the disclosed technology.

[0004] A computer-implemented method for automatically identifying landing surface features according to a representative embodiment includes: receiving a request to land a movable object; in response to the request, identifying a target landing area on the landing surface based on at least one image of the landing surface obtained by the movable object; and guiding the movable object to land in the target landing area. In a specific embodiment, the method may include: obtaining depth information corresponding to each pixel of the at least one image of the landing surface, wherein the each pixel corresponds to each portion of the landing surface. In any of the foregoing embodiments, the method may include: generating a depth map including depth information for each pixel, the depth map may represent the height of the corresponding portion of the landing surface. In any of the foregoing embodiments, the at least one image may include at least two images, and the method may include performing stereo matching on the at least two images of the landing surface to generate depth information for matching pixels of the at least two images.

[0005] In any of the foregoing embodiments, the method may include identifying a plurality of candidate regions within a pixel frame of the at least one image of the landing surface. The method may also include, for a single candidate region, identifying a best fit plane for pixels within the candidate region, and determining a cost value associated with the best fit plane based on a cost function. The method may also include selecting at least one of the candidate regions as corresponding to the target landing region based on the cost value. In any of the foregoing embodiments, the cost value associated with the best fit plane may represent a degree of deviation of the pixel from the best fit plane. In any of the foregoing embodiments, the cost function may be based on a sum of squares of deviations of the pixel from the best fit plane. In any of the foregoing embodiments, the method may include determining whether the best fit plane is tilted by comparing a normal vector of the best fit plane with a gravity vector, and discarding candidate regions where the best fit plane is tilted. In any of the foregoing embodiments, the method may include discarding candidate regions where the cost function is greater than a threshold. In any of the foregoing embodiments, the method may include (if all candidate regions are discarded because the cost function is greater than a threshold) directing the movable object to another location and capturing an image of another landing surface at the other location.

[0006] In any of the foregoing embodiments, the method may include adjusting the cost function to include cost function information of nearby candidate areas. In any of the foregoing embodiments, the process of adjusting the cost function may include applying a smoothing function to adjust the cost function of each candidate area. In any of the foregoing embodiments, the smoothing function may be a Gaussian blur function. In any of the foregoing embodiments, the selection process may include selecting a candidate area with the lowest cost function among the candidate areas. In any of the foregoing embodiments, the process of determining multiple candidate areas may include: scanning a pixel frame with a sliding window to generate multiple candidate areas, wherein the size of the sliding window is greater than or equal to the size of the movable object. In any of the foregoing embodiments, the shape of a single candidate area may conform to the shape of the movable object.

[0007] In other embodiments, a representative device includes at least one motor and a controller programmed with instructions. When the instructions are executed: receive a request to land the device, and in response to the request, identify a target landing area on the landing surface based on at least one image of the landing surface obtained by the device. The instructions may also use the at least one motor to guide the device to land in the target landing area. In a specific embodiment, the device may be an unmanned aerial vehicle (UAV), a manned aircraft, an autonomous vehicle, a self-balancing vehicle, or a robot. In any of the foregoing embodiments, when the instructions are executed, a new image of the landing surface may be captured as the device descends toward the target landing area, and a depth map may be updated to include depth information for each pixel of the new image of the landing surface, wherein each pixel corresponds to each portion of the landing surface. The instructions may determine based on the depth map whether the target landing area continues to meet the landing conditions as the device descends.

[0008] In other embodiments, an unmanned aerial vehicle (UAV) device includes a plurality of motors, a plurality of propellers (wherein each propeller is carried by a corresponding respective motor), at least one camera, and a controller. The controller may be programmed with instructions that, when executed, identify a target landing area on a landing surface based on at least one image of the landing surface captured by at least one camera, and use the motor to guide the UAV device to land at the target landing area. In a specific embodiment, the at least one camera may include two bottom-view cameras, and the instructions, when executed, may use the two bottom-view cameras to simultaneously capture two images of the landing surface. In any of the foregoing embodiments, the instructions, when executed, may use at least one camera to capture two images of the landing surface at two different points in time, wherein the two images of the landing surface are captured from different perspectives. In any of the foregoing embodiments, the instructions, when executed, may guide the UAV device to a predetermined height suitable for automatic terrain measurement. In any of the foregoing embodiments, the instructions, when executed, may determine whether at least one image of the landing surface was taken when the drone device was directly above the center of the landing surface, and if the at least one image of the landing surface was not taken when the drone device was directly above the center of the landing surface, apply perspective projection to the at least one image of the landing surface. In any of the foregoing embodiments, the instructions, when executed, may direct the movable object to move to a position directly above the target landing area.

[0009] In other embodiments, a non-transitory machine-readable medium storing instructions, the instructions, when executed, receive a request to land a movable object, and in response to the request, identify a target landing area on a landing surface based on at least one image of the landing surface obtained by the movable object. The instructions may guide the movable object to land at the target landing area. In a specific embodiment, the instructions, when executed, may: monitor the altitude of the movable object as the movable object descends toward the target landing area; and if the altitude of the movable object undergoes a disruptive change, stabilize the flight attitude of the movable object before continuing to descend. In any of the foregoing embodiments, the instructions, when executed, may: detect a disruptive change in the state of an inertial navigation system or a forward-looking camera of the movable object; and pause the descent of the movable object until the altitude of the movable object no longer undergoes a disruptive change. In any of the foregoing embodiments, the instructions, when executed, may: monitor the altitude of the movable object as the movable object descends toward the target landing area; and if the altitude of the movable object undergoes a disruptive change, determine that the target landing area is currently unsuitable for landing. Then, the movable object may be guided to hover at the current height. In any of the aforementioned embodiments, when the instruction is executed, if the altitude of the movable object continues to undergo disruptive changes over a period of time, the movable object may be guided to rise to a higher altitude. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a partially schematic isometric illustration of an unmanned aerial vehicle having a control system configured to automatically assess a landing surface in accordance with a representative embodiment of the present technology.

[0011] Figure 2 is a partial schematic diagram of a portion of the bottom of a drone configured in accordance with an embodiment of the present technology.

[0012] Figure 3 is a partial schematic diagram of a control system according to a representative embodiment of the present technology, the control system including a first controller onboard a drone and configured to control the drone.

[0013] Figure 4A is a flow chart illustrating the planning phase of a process for landing a movable object in accordance with a representative embodiment of the present technology.

[0014] Figure 4B is a flow chart illustrating the descent phase of a process for landing a movable object in accordance with a representative embodiment of the present technology.

[0015] Figure 5 is a partial schematic diagram of a sliding window and corresponding candidate landing points according to a representative embodiment of the present technology.

[0016] Figure 6 is a flow chart illustrating a process for identifying flat areas in a pixel frame corresponding to a target landing point in accordance with a representative embodiment of the present technology.

[0017] Figure 7 is a partial schematic diagram of a sample Gaussian blur filter for a candidate region according to a representative embodiment of the present technology.

[0018] Figure 8 is a partial schematic diagram of a sample perspective projection according to a representative embodiment of the present technology.

[0019] Fig. 9 is a partial schematic diagram of the hardware architecture of a computing device that performs the above-described processes according to a representative embodiment of the present technology. DETAILED DESCRIPTION

[0020] 1. Overview

[0021] Landing a drone is challenging. For example, an operator can remotely control the landing process of a drone. The operator controls the drone to maneuver to a target location, such as a location near the operator. The drone captures a live view video stream of the ground and sends the live view video stream in real time to a device viewed by the operator. Based on the live view of the ground, the operator identifies a suitable landing point (also called a target landing area) and manually controls the drone to land at the landing point.

[0022] However, this landing process requires the operator to be very skilled in controlling the drone. It is challenging to control the drone landing process based on the live video feed from the drone, especially when the operator cannot directly visually perceive the flight trajectory and attitude of the drone. Therefore, the drone may be damaged during clumsy landings caused by human error during the landing process.

[0023] The present technology generally relates to devices for automatically measuring or otherwise evaluating the topography of a landing surface or other target surface, and related systems and methods. Such devices may include, for example, drones, manned aircraft, self-driving cars, self-balancing vehicles, robots, etc. In a specific embodiment, the drone can automatically evaluate the landing surface based on computer visualization technology. During the landing process, the drone captures visual signals (e.g., images or videos) of the landing surface. Based on the visual signals, the drone detects the topographic features of the landing surface and identifies places or areas with flat or smooth surfaces suitable for landing. The drone then lands on the landing point to improve the landing process.

[0024] Implementations of the technology may include an onboard visual sensor (such as a camera) to assess the landing surface, for example based on stereo matching techniques. In a specific embodiment, depth information is extracted from the image and used to reconstruct the 3D terrain and select a target landing point. The advantage of this automatic landing surface measurement technology over the traditional drone landing process is that the drone (or other type of movable object) can automatically and efficiently identify the target landing point without operator intervention. Accordingly, this can enable the drone to land at the landing point without damaging itself due to an uneven or tilted landing surface.

[0025] For the sake of clarity, the following description does not set forth details of structures or processes that are well known and commonly associated with drones (or other types of movable objects) and corresponding systems and subsystems, but that may unnecessarily obscure some important aspects of the technology of the present disclosure. In addition, although the following disclosure sets forth several embodiments of different aspects of the technology, several other embodiments may have configurations or components that are different from those described in this section. Accordingly, the technology may have other embodiments that have additional elements and / or do not have the following references. Figure 1-9 Several elements are described.

[0026] Provided Figures 1 to 9 The accompanying drawings are not intended to limit the scope of the claims in this application unless otherwise specified.

[0027] Multiple embodiments of the technology described below can take the form of computer executable instructions or controller executable instructions, including routines executed by programmable computers or controllers. Those skilled in the relevant art will understand that the technology can be implemented on systems different from those computer or controller systems shown and described below. The technology can be embodied in a special-purpose computer or data processor that is specially programmed, configured or constructed to execute one or more computer executable instructions described below. Therefore, the terms "computer" and "controller" commonly used herein refer to any data processor, and may include Internet devices and handheld devices (including palmtop computers, wearable computers, cellular or mobile phones, multi-processor systems, processor-based or programmable consumer electronics, network computers, mini computers, etc.). The information processed by these computers and controllers can be presented on any suitable display medium (including cathode ray tube (CRT) displays or liquid crystal displays (LCD)). Instructions for executing computer or controller executable tasks can be stored in or on any suitable computer-readable medium, including hardware, firmware, or a combination of hardware and firmware. Instructions can be contained in any suitable storage device, including, for example, flash drives, universal serial bus (USB) devices, and / or other suitable media.

[0028] 2. Representative Examples

[0029] Figure 1 is a partially schematic isometric illustration of a representative drone 100 configured in accordance with embodiments of the present technology. Drone 100 may include a fuselage 110, which may in turn include a central portion 111 and one or more outer portions 112. Figure 1 In the representative embodiment shown, the fuselage 110 includes four outer portions 112 (e.g., arms 113) that are spaced apart from each other as they extend away from the central portion 111. In other embodiments, the fuselage 110 can include other numbers of outer portions 112. In any of these embodiments, each outer portion 112 can support components of a propulsion system 120 that drives the drone 100. For example, each arm 113 can support a respective motor 121 that drives a respective propeller 122. The drone can control each group of motors 121 and respective propellers 122 independently of the remaining motors 121 and propellers 122.

[0030] The fuselage 110 can carry a payload 130, such as an imaging device 131. In specific embodiments, the imaging device 131 can include a camera, such as a camera configured to capture video data, still data, or both. The camera can be sensitive to wavelengths in any of a variety of suitable bands, including visible bands, ultraviolet bands, infrared bands, or combinations thereof. In other embodiments, the payload 130 can include other types of sensors, other types of cargo (such as packages or other deliverables), or both. In many of these embodiments, the payload 130 is supported relative to the fuselage 110 using a gimbal 150, which enables the payload 130 to be independently positioned relative to the fuselage 110. Thus, for example, when the payload 130 includes the imaging device 131, the imaging device 131 can move relative to the fuselage 110 to track a target. As shown in FIG. Figure 1 As shown, when the drone 100 is not flying, the landing gear 114 can support the drone 100 in a position to protect the payload 130 .

[0031] In a representative embodiment, the drone 100 includes a control system 140 having some components onboard the drone 100 and some components not onboard the drone 100. For example, the control system 140 may include a first controller 141 onboard the drone 100, and a second controller 142 (e.g., a human-operated ground-based controller) located remotely from the drone 100 and connected to the first controller 141 via a communication link (e.g., a wireless link). The first controller 141 may include an onboard computer-readable medium 143a that executes instructions directing the actions of the drone 100 (including, but not limited to, the operation of the propulsion system 120 and the imaging device 131). The second controller 142 may include an offboard computer-readable medium 143b and one or more input / output devices 148, such as a display 144 and a control device 145. The operator manipulates the control device 145 to remotely control the drone 100 and receives feedback from the drone 100 via the display 144 and / or other devices. In other representative embodiments, the drone 100 may operate autonomously, in which case the second controller 142 may be excluded, or may be used only for operator override functions. The onboard computer readable medium 143a may be removable from the drone 100. The offboard computer readable medium 143b may be removable from the second controller 142, for example, detachable from one or more input / output devices 148.

[0032] exist Figure 1 In the representative embodiment shown, drone 100 includes two forward-looking cameras 171 and 172. Cameras 171 and 172 capture images or videos in the human visible spectrum range, infrared range, ultraviolet range, or a combination thereof. Drone 100 may include more sensors (e.g., on the bottom). Figure 2 1 is a partial schematic diagram of a portion of the bottom of a representative drone 100 configured in accordance with an embodiment of the present technology. At the bottom of the fuselage 110, the drone may also include two bottom view cameras 173 and 174. In addition, the drone may include two ultrasonic sensors 177 and 178. The ultrasonic sensors 177 and 178 may detect objects or landing surfaces below the drone 100 and measure the distance to the objects or the ground by sending and receiving ultrasonic waves. Although Figure 1 and Figure 2 The drone shown in FIG. 1 includes four cameras 171-174 and two ultrasonic sensors 177-178, but drones in other embodiments may include any suitable number of cameras or ultrasonic sensors. In still other embodiments, drone 100 may include other types of sensors.

[0033] Figure 31 is a schematic diagram of a first controller 141, which may include a processor 146, a memory 147, and an input / output device 148. A control unit 149 (also referred to as a controller) directs the operation of various components on the drone 100, including the propulsion system 120 described above. A computer readable medium 143 (which may be housed in or include elements of any of the above-described components) contains instructions that, when executed, direct the behavior of the propulsion system 120 and other drone systems. The first communication device 161a is configured to provide wireless communication with a corresponding second communication device 161b carried by the second controller 142 via a communication link 160.

[0034] Using bottom-view cameras 173-174 or ultrasonic sensors 177-178, drone 100 is able to determine its flying altitude relative to ground level (e.g., local circular level) and evaluate the topography of one or more candidate landing surfaces. Based on the evaluation, the drone identifies a landing point within the candidate landing surface that is suitable for landing (e.g., substantially flat).

[0035] Figure 4A and 4B is a flow chart illustrating a process 400 for landing a movable object in accordance with a representative embodiment of the present technology. Figure 4A The planning phase of process 400 is shown, while Figure 4B Shown is the descent phase of process 400. Process 400 may be performed by various types of movable objects, such as a drone, a manned aircraft, an autonomous vehicle / self-balancing vehicle, or a robot.

[0036] Go to Figure 4A At block 405 of process 400, the movable object (e.g., a drone) receives an instruction to land at a target location. At block 410, the movable object navigates closer to the target location. Receipt of the instruction may be triggered by various mechanisms. In some embodiments, for example, an operator presses a "return home" button of the second controller 142 or a "return home" button on the display 144 of the second controller 142 (e.g., Figure 3 ). The second controller 142 generates and sends instructions to the flying drone. The transmitted instructions may include the coordinates of the target location. For example, the second controller 142 may include the position coordinates of the second controller 142 itself as the target location. Alternatively, the second controller 142 may include in the instruction a predetermined return location recorded in the memory of the second controller 142.

[0037] In some embodiments, the instruction does not necessarily include the coordinates of the target location. For example, when the drone begins operation and takes off, the drone can detect and record the coordinates of the take-off location. Once the drone receives the instruction, the drone can use the coordinates of the take-off location as the recorded target location for navigation.

[0038] In some embodiments, when the battery level of the drone falls below a threshold level, the drone itself can generate a command to land so that the drone can return to the landing point before the battery is exhausted to avoid losing power and falling.

[0039] At block 415, the movable object descends to a predetermined height suitable for automated terrain assessment or measurement. The height (also referred to as elevation) relative to the ground beneath the object (e.g., a building, ship, or platform) or drone is calculated. In some embodiments, the bottom view cameras 173 and 174 (e.g., Figure 2 The predetermined height is determined based on the pixel resolution of the image (shown in FIG. 1 ). A three-dimensional (3D) reconstruction of the terrain may be based on images captured by bottom-view cameras 173 and 174. Each pixel in the image represents a landing area. The higher the drone hovers, the larger the landing area represented by each pixel, and thus the lower the resolution of the 3D terrain reconstruction. The predetermined height is typically small enough so that the resolution of the 3D terrain reconstruction is high enough to identify an appropriate landing point for the drone.

[0040] On the other hand, the predetermined height is large enough so that the field of view of the bottom view cameras 173 and 174 covers a suitably large landing area. If the predetermined height is too small, the cameras 173-174 can only cover a smaller landing area and may not include any suitable landing point. In some embodiments, the predetermined height is about 3 meters.

[0041] The drone may use various types of depth sensors to determine the current flight altitude. For example, the depth sensor used to determine the flight altitude may include a single camera, a pair of cameras, an ultrasonic sensor, an IMU, or a laser rangefinder. In some embodiments, the drone may use bottom view cameras 173 and 174 (e.g., Figure 2 As shown in Figure 2, the flight altitude is determined. Figure 2 As shown, bottom view cameras 173 and 174 are disposed at the bottom of drone 100 and are separated by a known distance. In other words, bottom view cameras 173 and 174 capture images or videos of an object or the ground from different perspectives. Drone 100 can use stereo matching techniques (e.g., semi-global block matching) to find corresponding pixels in the image or video frames captured by cameras 172 and 174, and perform a 3D reconstruction process by triangulation. In other embodiments, cameras 172 and 174 can be replaced by one or more external cameras (e.g., Figure 1 The imaging device 131 shown in FIG. 1 is replaced.

[0042] In yet another embodiment, a drone can use a single bottom-view camera to capture images or videos of an object or the ground from different perspectives. For example, the drone's camera captures a first image of the ground from a first perspective. The drone then moves to a different location, and the camera captures a second image of the ground from a second perspective. The drone can then use stereo matching techniques to find corresponding pixels in the two images and perform a three-dimensional reconstruction through triangulation.

[0043] In yet another embodiment, the drone may use one or more ultrasonic sensors (e.g., Figure 2 The ultrasonic sensors 177 and 178 shown in the figure determine the flight altitude. The ultrasonic sensor of the drone transmits ultrasonic waves to the objects or the ground below. The transmitted ultrasonic waves are reflected by the objects or the ground. The ultrasonic sensor receives the reflected ultrasonic waves and determines the altitude based on the travel time of the ultrasonic waves.

[0044] In addition, the drone 100 can use an onboard inertial measurement unit (IMU) to determine the flight altitude. The IMU may include, for example, one or more accelerometers, gyroscopes, magnetometers, or a combination thereof. The IMU provides acceleration information that can be used to calculate velocity and position information through integration operations.

[0045] In some embodiments, the drone 100 may use a combination of multiple components (such as ultrasonic sensors 177 and 178, bottom view cameras 173 and 174, and IMU) to determine the flight altitude. In some other embodiments, the drone 100 dynamically switches between components such as ultrasonic sensors 177 and 178, bottom view cameras 173 and 174, and IMU to determine the flight altitude.

[0046] At block 416, the movable object uses one or more cameras (e.g., Figure 1 The image is captured by the camera 172 and 174 of the drone 100 shown. At decision block 417, the movable object determines whether the image was captured when the drone was in a horizontal state. In other words, the movable object determines whether the image was captured with respect to the landing surface directly below the drone. If not, then at block 418, the movable object applies perspective projection, which will be discussed later. Figure 8 If yes, the process proceeds directly to step 420.

[0047] At block 420 , the movable object determines the size of the sliding window in the pixel frames of cameras 172 and 174 that corresponds to the size of the target landing point in the ground frame. Figure 55 is a partial schematic diagram of sliding windows and corresponding suitable target landing points according to a representative embodiment of the present technology. Target landing point 515 is located in ground frame 510. Target landing point 515 has a radius R. Representative circular sliding window 525 is located in pixel frame 520. Sliding window 525 has a radius r. Although Figure 5 The target landing point 515 and the sliding window 525 are shown as having circular shapes, but the target landing point 515 and the sliding window 525 may have other suitable shapes, such as square, rectangular, etc.

[0048] Pixel frame 520 represents the pixel area in bottom view cameras 173 and 174 that capture the image or video of ground frame 510 on the ground. In other words, each pixel of pixel frame 520 contains visual information of the corresponding area within ground frame 510 on the ground. The target landing point 515 in ground frame 510 is large enough for the drone to land. Therefore, the size of the target landing point 515 can depend on the size of the drone (e.g., "footprint" or floor plan size). The floor plan is the outline of the drone (or other type of movable object) or the whole from top to bottom. In some embodiments, the size of the target landing point 515 is equal to the size of the drone. Similarly, the shape of the target landing point 515 can also be related to the shape of the drone. In some embodiments, for example, the shape of the target landing point 515 can generally conform to the shape of the drone (e.g., floor plan shape).

[0049] Sliding window 525 represents the area of ​​pixel frame 520 that captures the image of target landing point 515. Figure 5 As shown, the radius r of the sliding window 525 depends on the size of the drone, the current altitude, and the focal length of cameras 172 and 174: r = f*R / H, where R is the radius of the target landing point, f is the focal length, and H is the focal length plus the current flight altitude measured between the camera and the landing surface.

[0050] Back to Figure 4A At block 425, the movable object scans the pixel frame using a sliding window to identify a flat area corresponding to the target landing point. Figure 6 Details of the process performed in block 425 are described.

[0051] Then at decision block 430, the movable object determines whether a flat area is identified. If at decision block 430 the drone identifies no flat area within the pixel frame, then block 485 includes navigating around the target location. Process 400 further returns to block 416 to capture images of another area to identify a possible landing site.

[0052] If the drone identifies a flat area corresponding to the target landing point, process 400 proceeds to Figure 4BThe descent stage is shown. Go to Figure 4B , at box 435, the movable object navigates and descends to a flat area identified as a target landing point. In addition, at box 440, the movable object monitors the altitude in real time during the descent process. At decision box 445, the movable object determines whether there is a disruptive change in the flight altitude. Disruptive changes in the flight altitude can be detected by bottom-view cameras 173 and 174, IMU, or a combination thereof. Since the drone is gradually descending, such disruptive changes may indicate changes in the environment, particularly changes in the landing surface. For example, an obstacle may have moved onto the landing surface.

[0053] In some embodiments, if the change in flight altitude is greater than the predicted change in flight altitude based on the current descent speed of the movable object, the movable object determines that the change is disruptive (e.g., indicating the presence of a downdraft). In some other embodiments, the movable object determines that the change is disruptive if the change in flight altitude is faster than the maximum descent speed at which the movable object is designed to operate.

[0054] If there is no disruptive change in flight altitude, then at block 450, the movable object continues to descend until the drone lands. If there is a disruptive change in flight altitude, then at block 455, the movable object checks another sensor of the drone (e.g., an inertial navigation system or a forward-looking camera such as a Figure 1 171 and 172)). Decision block 460 includes determining whether there are further disruptive changes in the state of the inertial navigation system or the forward-looking camera. If there are further disruptive changes in the state of the inertial navigation system or the forward-looking camera, this generally indicates that the flight altitude of the drone is unstable. Therefore, at block 465, the movable object stabilizes the flight altitude of the drone before continuing to descend.

[0055] If there are no disruptive changes in the state of the inertial navigation system or the forward-looking camera, this generally indicates that the drone itself is descending smoothly. In contrast, disruptive changes in flight altitude are caused by environmental changes (e.g., the presence of a person or object at the landing site). At box 470, the movable object pauses the descent process and hovers. At decision box 475, the machine determines whether the flight altitude reading returns to a normal level within a specific (e.g., pre-selected) time period. If so, then at box 480, the movable object continues to descend until the drone lands. Otherwise, process 400 returns to box 416 ( Figure 4A The process of identifying another target landing point is repeated.

[0056] In some embodiments, a controller (e.g., Figure 1 and 3The first controller 141 shown in FIG. 1 may be programmed with instructions that, when executed, perform the following operations: Figure 4A and 4B The apparatus may include one or more steps shown in process 400. For example, the apparatus may include a structure and one or more motors carried by the structure. The structure may include, for example, at least a portion of a drone, a manned aircraft, an autonomous vehicle, a self-balancing vehicle, or a robot. The apparatus may further include a controller programmed to perform one or more steps in process 400.

[0057] Figure 6 6 is a flow chart illustrating a process 600 for identifying a flat area in a pixel frame corresponding to a target landing point according to a representative embodiment of the present technology. At block 605, a movable object (e.g., a drone) identifies a candidate area within a pixel frame of an image by moving a sliding window to a position within the pixel frame. For example, based on the above reference Figure 5 The described techniques determine the size of the sliding window. Each candidate region occupies a unique portion of the pixel frame. The candidate regions may, but do not necessarily, overlap each other. In some embodiments, the techniques may determine a step size (e.g., measured in pixels) for moving the sliding window. For example, after a candidate region is identified, the next candidate region may be determined by moving the sliding window from the current candidate region by the determined step size.

[0058] At block 610, the movable object obtains position coordinates and depth coordinates of a single pixel within the candidate region. For example, a single pixel may have three coordinates. The x and y coordinates represent the position of the pixel within the pixel frame. The z coordinate represents the depth information of a portion of the landing surface represented by the pixel. Thus, the x, y, and z coordinates of the pixel reproduce the topography of the landing area represented by the candidate region. In some embodiments, the image may be captured based on images acquired by two cameras (e.g., Figure 1 In one embodiment, the depth information is obtained by using stereo matching techniques (e.g., semi-global block matching) to obtain depth information from images captured simultaneously by cameras 172 and 174 (e.g., cameras 172 and 174 shown in FIG. 1 ) or two sequential images captured by a single camera. In other embodiments, another distance sensor (e.g., an ultrasonic sensor) can be used to obtain depth information.

[0059] At block 615, the movable object identifies a "best fit" plane that fits the coordinates of the pixels of the candidate region, and a cost function associated with the best fit plane, the cost function representing the degree of deviation of the pixels from the plane. The best fit plane is the plane with the minimum cost value among the possible fit planes. In other words, the degree of deviation indicates whether the pixels of the candidate region can be considered a flat plane. If the degree of deviation is small, the actual landing surface represented by the candidate region is considered to be flat and suitable for proper landing.

[0060] In some embodiments, the movable object may use an algorithm (e.g., the Levenberg-Marquardt algorithm) to find the best-fit plane and its associated cost value.

[0061] This algorithm minimizes the sum of the squares of the deviation values of pixel Y i from the fitting plane function f.

[0062]

[0063] The fitting plane function can be expressed as:

[0064]

[0065] The residual vector r is:

[0066]

[0067] The cost function for calculating the cost value is:

[0068]

[0069] The residuals can also be further attenuated according to, for example, the following algorithm:

[0070] (J T J + λI)δ = J T r,

[0071] where J is the Jacobian matrix of the derivative of the residuals with respect to each parameter, λ is the attenuation parameter (the adaptive balance between two steps), and r is the residual vector.

[0072] The attenuation process first initializes the value of the attenuation parameter and calculates the residual r and the Jacobian matrix J at the initial parameter guess. Then, the attenuation process calculates the metric g = J T J + λI, and the cost gradient

[0073] Then, the attenuation process calculates the new residual r at the point given by and calculates the cost C new at the new point new = 1 / 2r new 2 . If C new < C, the process accepts the step size x = x new , and sets r = r new and λ = λ / λ down . Otherwise, the process rejects the step size and keeps the old parameter guess x and the old residual r, and adjusts λ = λ / λ upThe decay procedure checks for convergence. If convergence exists, the procedure returns x as the best fit parameters. If convergence does not exist but the step size is accepted, the procedure finds the Jacobian matrix J at the new parameter values ​​and returns the computed metric g and the cost gradient.

[0074] At decision block 620, the movable object determines whether all candidate regions within the pixel frame have been processed. If not, process 600 returns to block 605 to identify another candidate region within the pixel frame. If all candidate regions within the pixel frame have been processed, then at block 625, the movable object discards candidate regions with cost values ​​greater than a threshold. The discarded regions are considered to be uneven because the associated cost values ​​are large, meaning that their pixels are not close to the best fit plane. In other words, the corresponding landing surface portion is uneven or not flat enough.

[0075] At block 630, the movable object further discards tilted candidate regions. The tilted candidate regions represent tilted landing surfaces that are not suitable for proper landing. In some embodiments, for each candidate region, the movable object may compare the direction of the gravity vector (which may be determined by, for example, an IMU) with the direction of the normal vector of the best fit plane of the candidate region. If the angle between the directions of the gravity vector and the normal vector exceeds a threshold, the corresponding landing surface is too tilted to be used for landing.

[0076] At block 640, the movable object may apply a smoothing function to adjust the cost functions of each of the remaining candidate regions so that the adjusted cost functions include information about the cost functions of the nearby candidate regions. The reason for applying the smoothing function is that, in at least some embodiments, the goal is not just to select the candidate region with the lowest cost function (which means a flat surface), but to select a candidate region whose nearby areas are also relatively flat. This is beneficial because a drone or other device may deviate from the planned path during landing and may land in a nearby area instead of the selected candidate region. Therefore, selecting a candidate region whose nearby areas are also relatively flat reduces the probability that the drone will be damaged during landing.

[0077] In some embodiments, the smoothing function may be, for example, a Gaussian blur function. Figure 7 is a partial schematic diagram of a high-speed blur filter according to a representative embodiment of the present technology. Figure 7 The 3×3 Gaussian blur filter shown in FIG. 6 can be applied, for example, to block 640 of process 600. Figure 7 As shown, the Gaussian blur filter takes 1 / 4 of the original cost function of the candidate area, 1 / 8 of the cost function of the upper, lower, left, and right adjacent areas, and 1 / 16 of the cost function of the four corner areas.

[0078] return Figure 6, at block 650, the movable object selects the candidate region with the smallest cost value calculated according to the adjusted cost function as the flat region corresponding to the target landing point. The cost value of the adjusted cost function adjusted by a smoothing function (e.g., a Gaussian blur filter) indicates not only whether the candidate region is flat, but also the flatness of the nearby region. Then, according to Figure 4A and 4B The process shown, the UAV can descend to the identified target landing point.

[0079] like Figure 4A and Figure 4B As shown in process 400, if the image is not captured while the drone is level, the movable object may apply a perspective projection to the image. Figure 8 FIG. 1 is a partial schematic diagram of a perspective projection according to a representative embodiment of the present technology. Figure 8 As shown, the pixel frame 820 of the captured image does not correspond to the landing surface directly below the drone. The area 825 within the pixel frame 820 has a corresponding perspective projection area 815 on the ground frame 810. In various embodiments, the area 825 can have different shapes, such as a circle, an ellipse, a square, a matrix, etc. Therefore, the projection area 815 can have a shape such as a circle, an ellipse, a square, a rectangle, etc. For example, the area 825 can be circular and the projection area 815 can be elliptical.

[0080] Assuming that [u, v] is the coordinate of a pixel in the region 825 in the pixel frame 820, the coordinate of the corresponding point on the perspective projection region 815 is expressed as [x w ,y w , z w ]. The perspective projection operation is:

[0081]

[0082] K is the intrinsic matrix, which includes parameters related to the camera, such as focal length, image sensor format, principal point of the camera lens, and lens distortion. R and T include extrinsic parameters, which represent the transformation from world coordinates [x w ,y w , z w ] to the camera coordinates [u, v]. R is the rotation matrix. T represents the position of the origin of the world coordinate system expressed in terms of the coordinates of the camera center coordinate system. If the coordinate system is chosen so that the origin is at the center of the projection area, then z w =0,x w 2 +y w 2 =1.

[0083] in other words, therefore,

[0084]

[0085] Fig. 9 900 is a partial schematic diagram of the hardware architecture of a computing device that can perform the above process according to a representative embodiment of the present technology. The computing device 900 can perform some or all of the processor-executable processing steps described in detail below. In various embodiments, the computing device 900 includes a processor subsystem including one or more processors 902. The processor 902 can be or can include one or more programmable general-purpose or special-purpose microprocessors, digital signal processors (DSPs), programmable controllers, application-specific integrated circuits (ASICs), programmable logic devices (PLDs), etc., or a combination of these hardware-based devices.

[0086] The computing device 900 may also include memory 904, a network adapter 910, a cluster access adapter 912, and a storage device adapter 914, all of which are interconnected through an interconnect device 908. The interconnect device 908 may include, for example, a system bus, a peripheral component interconnect (PCI) bus, a HyperTransport or Industry Standard Architecture (ISA) bus, a Small Computer System Interface (SCSI) bus, a Universal Serial Bus (USB), the Institute of Electrical and Electronics Engineers (I9E) standard 1394 bus (sometimes referred to as "FireWire"), or any other data communication system.

[0087] Cluster access adapter 912 includes one or more ports suitable for coupling computing device 900 to other devices. In the illustrated embodiment, Ethernet may be used as the cluster protocol and interconnect medium, but other types of protocols and interconnects may be utilized within the cluster architecture described herein.

[0088] The computing device 900 may be implemented as a single processor or multi-processor system executing an operating system 906, wherein the operating system 906 may implement a high-level module (e.g., a hypervisor) to logically organize information on a storage device into a hierarchy of named directories, files, and a special type of file called a virtual disk. The computing device 900 may also include a graphics processing unit for graphics processing tasks or for parallel processing of non-graphics tasks.

[0089] The memory 904 may include storage locations addressable by the processor 902 and adapters 910, 912, and 914 to store processor executable code and data structures. The processor 902 and adapters 910, 912, and 914 may in turn include processing elements and / or logic circuits configured to execute software code and manipulate data structures. An operating system 906 (part of which typically resides in the memory and is executed by the processor 902) functionally organizes the computing device 900 by, in particular, configuring the processor 902 to make calls. It will be apparent to those skilled in the art that other processing and memory implementations including various computer-readable storage media may be used to store and execute program instructions related to the present technology.

[0090] The network adapter 910 may include multiple ports to couple the computing device 900 to one or more clients via a point-to-point link, a wide area network, a virtual private network implemented on a public network (such as the Internet), or a shared local area network. Therefore, the network adapter 910 may include mechanical, electronic, and signaling circuits required to connect the computing device 900 to the network. Illustratively, the network may be embodied as an Ethernet network or a Fiber Channel (FC) network. The client may communicate with the computing device on the network by exchanging discrete frames or data packets according to a predefined protocol (e.g., TCP / IP).

[0091] The storage device adapter 914 can cooperate with the operating system 906 to access the information requested by the client. The information can be stored in an attached array of any type of writable storage media, such as a disk or tape, an optical disk (e.g., CD-ROM or DVD), flash memory, a solid state disk (SSD), an electronic random access memory (RAM), micro-electromechanical and / or any other similar medium suitable for storing information (including data and parity information). The storage device adapter 914 may include multiple ports with input / output (I / O) interface circuits that are coupled to the disks through I / O interconnects (e.g., conventional high-performance fiber channel (FC) link topologies). In various embodiments, the cluster adapter 912 and the storage device adapter 914 may be implemented as one adapter that is configured to connect to a switching structure (e.g., a storage network switch) to communicate with other devices and mass storage devices.

[0092] A feature of several of the above embodiments is that the drone can automatically assess or measure the landing surface using computer visualization techniques. The drone can use its visual sensors (such as cameras) to measure the landing surface, for example using stereo matching techniques. Depth information can be extracted from the image and used to reconstruct the three-dimensional terrain and select the target landing point. The advantage of this arrangement is that the drone can automatically and efficiently identify the target landing point without the intervention of the operator. Therefore, the drone can descend correctly to the landing point without damaging itself due to uneven or tilted landing surfaces.

[0093] It can be understood from the above that specific embodiments of the present technology have been described herein for illustrative purposes, but various modifications may be made without departing from the present technology. For example, terrain measurement technology can be applied to other devices besides drones, such as manned aircraft, self-driving cars, self-balancing vehicles, robots, wearable devices (such as smart glasses, virtual reality (VR) or augmented reality (AR) head-mounted displays). For example, a self-driving car or robot may include one or more cameras for capturing images of the ground surface in front of the self-driving car or robot. The self-driving car or robot measures terrain conditions by extracting depth information from the image. Based on the terrain conditions, the self-driving car or robot automatically selects a route with a smooth driving surface.

[0094] In the above, the landing site is described as comprising the ground in the content of several embodiments. In other embodiments, the landing site may comprise an object (eg, a ship, a vehicle or a platform), water or other suitable sites.

[0095] The technology may also be generally directed to wearable devices such as smart glasses, virtual reality (VR) or augmented reality (AR) head-mounted displays, etc. The technology may automatically measure or otherwise assess the topography of a surface in front of such a wearable device or a user wearing the wearable device, and identify a target surface suitable for the user to travel to. The technology may also indicate a target surface to a user using a wearable device or other device capable of outputting target surface information to the user.

[0096] Certain aspects of the techniques described in the context of specific embodiments may be combined or excluded in other embodiments. For example, in some embodiments, if the drone is able to land on a tilted surface, the drone does not need to discard tilted candidate regions. In some embodiments, if the drone is equipped with a mechanism that automatically orients the camera to capture images downward regardless of the drone's flight attitude, the drone does not need to apply a perspective projection to the captured images. In other embodiments, for example, if the drone is able to descend to the landing point in an accurate manner, the drone does not apply a smoothing function to the cost function of the candidate region.

[0097] In addition, although advantages associated with certain embodiments of the present technology have been described in the context of these embodiments, other embodiments may also exhibit such advantages, and not all embodiments need to exhibit such advantages to fall within the scope of the present technology. Therefore, the present disclosure and associated technology may include other embodiments not explicitly shown or described herein.

[0098] To the extent any material incorporated herein conflicts with the present disclosure, the present disclosure controls.

[0099] At least a portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.

Claims

1. A method for controlling a movable object to automatically land, characterized in that: include: receiving a request to land the movable object at a target location; guiding the movable object to approach the target location; Based on at least one image of the landing surface obtained by the movable object, identifying a flat area on the landing surface as a target landing area, wherein the method comprises: generating a depth map of depth information of each pixel in the at least one image of the landing surface to represent the height of the corresponding part of the landing surface, wherein the each pixel corresponds to each part of the landing surface; and Guiding the movable object to land in the target landing area, wherein, when the movable object lands toward the target landing area, capturing a new image of the landing surface; updating the depth map to include depth information for each pixel of the new image of the landing surface; and based on the depth map, determining whether the target landing area continues to meet landing conditions when the movable object lands.

2. The method according to claim 1, characterized in that: Also includes: After guiding the movable object to approach the target location, the movable object is further guided to descend to a predetermined height suitable for automatic terrain assessment or measurement.

3. The method according to claim 1, characterized in that: Also includes: When the movable object fails to identify a target landing area on the landing surface, the movable object navigates around the target location to identify other possible landing points.

4. The method according to claim 3, characterized in that: After the step of navigating the movable object around the target location, the method further includes: directing the movable object to another location; An image of another landing surface at the another location is captured to identify a possible landing point.

5. The method according to claim 1, characterized in that When the at least one image of the landing surface is not taken when the movable object is located directly above the center of the landing surface, perspective projection is applied to the at least one image of the landing surface.

6. The method according to claim 1, characterized in that The at least one image comprises at least two images, and the method further comprises: Stereo matching is performed on the at least two images of the landing surface to generate depth information of matching pixels of the at least two images.

7. The method according to claim 1, characterized in that Also includes: A best-fit plane is identified based on the position coordinates and depth coordinates of the pixels in the depth map, a cost value associated with the best-fit plane is determined based on a cost function, and the target landing area is determined based on the cost value.

8. The method according to claim 7, characterized in that The cost value associated with the best fit plane represents a degree of deviation of a pixel from the best fit plane.

9. The method according to claim 7, characterized in that: When all pixels are discarded because the cost function is greater than the threshold, then: directing the movable object to another location; and An image of another landing surface at the another location is captured.

10. The method according to claim 7, characterized in that Also includes: Whether the best fit plane is tilted is determined by comparing the normal vector of the best fit plane to the gravity vector.

11. The method according to claim 1, characterized in that At least one image of the landing surface is acquired by a bottom-view camera of the movable object.

12. A movable object comprising: Multiple motors; a plurality of propellers, wherein each propeller is carried by a corresponding respective motor; At least one camera; as well as A controller programmed with instructions that, when executed: receiving a request to land the movable object at a target location; guiding the movable object to approach the target location; a flat area of ​​the landing surface identified by the movable object as a target landing area; guiding the movable object to land in the target landing area; wherein, when the movable object descends toward the target landing area, the altitude of the movable object is monitored; When the altitude of the movable object undergoes disruptive changes, determining that the target landing area is currently unsuitable for landing; and guiding the movable object to hover at the current altitude, or when the altitude of the movable object continues to undergo disruptive changes over a period of time, guiding the movable object to rise to a higher altitude; The disruptive changes include situations where a person or an object appears on the target landing area.

Images