Air vehicle control to avoid collisions with objects on ground in event of potential failure

Abstract

The invention relates to air vehicle control to avoid collisions with objects on the ground in the event of a potential fault. Techniques for vehicle control are disclosed. These techniques include collision trajectory prediction of an air vehicle and identification of objects beneath the UAV using one or more sensors. The techniques also include determining a risk that the air vehicle will collide with the object in the event that the vehicle is later losing propulsive force, and preventing the air vehicle from colliding with the object based on the determined risk.

Description

Technical Field

[0001] Aspects of the present disclosure relate to vehicle control, and more particularly, aspects of the invention relate to air vehicle (eg, unmanned aerial vehicle (UAV)) control to avoid collision with objects on the ground in the event of an air vehicle malfunction. Background Art

[0002] Safety and risk assessment of UAVs and other aerial vehicles is critical. To reduce this risk, government regulations in various jurisdictions require UAV operators to conduct risk assessments (e.g., specific operational risk assessments (SORA)) on the operation of the UAV. This can include a ground risk assessment that determines the risk of the UAV's operation to objects protruding from the ground (e.g., any animate or inanimate object, including people, animals, obstacles, structures, vehicles, and any other suitable objects). For example, a UAV losing propulsion may cause the aerial vehicle to lose control and fall to the ground, or otherwise impact (e.g., collide) with an object on the ground. In one aspect, loss of power, loss of engine, mid-air collision, artillery impact, or any other causal occurrence may result in loss of propulsion. One or more techniques described herein may be used to reduce the impact of potential UAV failures (e.g., loss of propulsion) on objects on the ground. Summary of the invention

[0003] An embodiment includes a method. The method includes receiving a flight status of an aerial vehicle. The method also includes identifying an object below the aerial vehicle using one or more sensors. The method also includes determining a risk that the aerial vehicle will collide with the object if the aerial vehicle later loses propulsion. The method also includes preventing the aerial vehicle from colliding with the object based on the determined risk.

[0004] Embodiments also include a non-transitory computer readable medium containing computer program code that, when executed by operation of one or more computer processors, performs operations. The operations include receiving a flight status of an air vehicle. The operations also include identifying an object below the air vehicle using one or more sensors. The operations also include determining a risk that the air vehicle will collide with the object if the air vehicle later loses propulsion. The operations also include preventing the air vehicle from colliding with the object based on the determined risk.

[0005] Embodiments also include an unmanned aerial vehicle (UAV) comprising a computer processor and a memory having instructions stored thereon, which when executed on the computer processor, perform operations. The operations include identifying a flight state of the UAV. The operations also include receiving a corrected flight state of the UAV, wherein the corrected flight state is calculated based on: identifying an object below the UAV using one or more sensors and determining a risk that the UAV will collide with the object if the UAV later loses propulsion. The operations also include automatically controlling the UAV to operate using the corrected flight state. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] In order that the manner in which the above-described features may be understood in detail, a more particular description, briefly summarized above, may be had by reference to example aspects, some of which are illustrated in the accompanying drawings.

[0007] Figure 1 An unmanned aerial vehicle system for avoiding collision of an aerial vehicle with an object on the ground is shown according to one aspect.

[0008] Figure 2 A block diagram of a controller for controlling an unmanned aerial vehicle to avoid collision with objects on the ground is shown according to one aspect.

[0009] Figure 3 is a flow chart illustrating controlling an unmanned aerial vehicle to avoid collision with objects on the ground according to one aspect.

[0010] Figure 4 An example of controlling an unmanned aerial vehicle to avoid collision with objects on the ground is shown according to one aspect.

[0011] Figure 5 is a flow chart illustrating preventing an unmanned aerial vehicle from colliding with an object on the ground according to one aspect. DETAILED DESCRIPTION

[0012] The present disclosure describes techniques for preventing an aerial vehicle (e.g., a UAV) from impacting (e.g., colliding) with an object below the aerial vehicle (e.g., on the ground) in the event of a loss of propulsion. In embodiments, the object may include any animate or inanimate object, including a person, an animal, an obstacle, a structure, a vehicle, and any other suitable object. As described above, reducing the ground risk of a UAV is extremely important. Existing solutions typically attempt to reduce the ground risk by reducing the negative effects of a UAV colliding with an object on the ground. For example, a flight termination system (FTS) (e.g., a bulky parachute or airbag) may be included in a UAV and used to slow the UAV's descent rate or reduce the force of a collision with an object on the ground. But these FTS add significant weight and drag to the UAV, greatly reducing its range (as well as other disadvantages).

[0013] As another example, a ground risk safety buffer zone may be used to limit the UAV's operating area to effectively avoid critical ground objects.

[0014] A ground risk safety buffer can be defined as an area around the UAV's operating area that operates as the last area to accommodate UAVs that have exited the operating area and the emergency area. The UAV is expected to land within the boundaries of the ground risk safety buffer area (e.g., using FTS controlled landing). The size of this buffer depends on a combination of hardware, software, and procedural measures employed by the UAV. In general, the higher the robustness, the smaller the buffer required, and the buffer is often calculated manually (e.g., before the UAV operates). But this significantly limits the UAV's operating area, as population density and other factors affect the required ground risk safety buffer. In addition, this can also increase fuel and operating costs by changing the UAV's flight path. For example, avoiding a large safety buffer may require a modified and inefficient flight path, which increases fuel and operating costs.

[0015] Other existing solutions to reduce the impact of failures of UAVs focus on improving the reliability of the UAV to avoid loss of propulsion or other failures. For example, existing solutions can add redundancy to navigation or propulsion to avoid the UAV losing propulsion. But if a loss of propulsion does occur, this does not eliminate the risk of objects on the ground.

[0016] One or more technologies described herein detect exposed objects below an aerial vehicle. For example, an aerial vehicle, a ground station, or any other suitable control system can use artificial intelligence (AI) machine learning (ML) technology to detect exposed objects and their trajectories on the ground in real time during the operation of the aerial vehicle. The control system can also calculate the potential landing trajectory of the aerial vehicle in real time. Therefore, when the aerial vehicle has the risk of colliding with the exposed objects on the ground, the control system can take measures to prevent collision. For example, the control system can change the aerial vehicle trajectory when the aerial vehicle is in operation to reduce any risk of collision in the case of assuming a sudden loss of propulsion later. Alternatively, or in addition, the control system can warn the pilot or other operator of the risk so that (one or more) evasive actions can be taken. In this way, collisions on the ground are restrained to avoid exposed objects without the need to equip the aerial vehicle with bulky parachutes or airbags, and at the same time reduce the originally necessary safety buffer.

[0017] One or more of these technologies may provide significant technical advantages. For example, these technologies significantly reduce the risk of an aerial vehicle crashing into an object in the event of a loss of propulsion or other failure, while still providing flexibility to the operator by reducing the size or necessary safety buffer. In addition, one or more of these technologies may significantly reduce pilot workload by making corrections autonomously, and may eliminate the burden of installing an FTS (e.g., a bulky parachute or airbag), which in turn reduces the weight and fuel required for the aerial vehicle.

[0018] In the current disclosure, reference is made to various aspects. However, it should be understood that the present disclosure is not limited to the specifically described aspects. On the contrary, any combination of the following features and elements (whether or not related to different aspects) is contemplated as implementing and practicing the teachings provided herein. Additionally, when describing the elements of aspects in the form of "at least one of A and B", it will be understood that only element A, only element B, and aspects including elements A and B can all be expected. In addition, although some aspects can achieve advantages that are superior to other possible solutions and / or superior to the prior art, whether a specific advantage is achieved by a given aspect does not limit the present disclosure. Therefore, the aspects, features, aspects, advantages, and advantages disclosed herein are merely illustrative and are not considered to be elements or limitations of the appended claims, unless explicitly stated in one or more claims. Similarly, reference to "the present invention" should not be interpreted as a summary of any inventive subject matter disclosed herein, and should not be considered to be elements or limitations of the appended claims, unless explicitly stated in the claims.

[0019] As will be appreciated by those skilled in the art, the aspects described herein may be embodied as a system, method, or computer program product. Thus, aspects may take the form of entirely hardware aspects, entirely software aspects (including firmware, resident software, microcode, etc.), or aspects combining software and hardware aspects, all of which may generally be referred to herein as "circuits," "modules," "services," or "systems." Furthermore, aspects described herein may take the form of a computer program product embodied in one or more computer-readable storage media having computer-readable program code embodied thereon.

[0020] Program code embodied on a computer readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0021] The computer program code for performing the operations of aspects of the present disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​(e.g., Java, Smalltalk, C++, etc.), and traditional procedural programming languages ​​(e.g., "C" programming language or similar programming languages). The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0022] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products according to aspects of the present disclosure. It should be understood that each frame of the flowchart illustration and / or block diagram and the combination of frames in the flowchart illustration and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine, so that instructions executed by a processor of a computer or other programmable data processing device create a device for implementing the function / action specified in (one or more) frames of the flowchart illustration and / or block diagram.

[0023] These computer program instructions may also be stored in a computer-readable medium, which can direct a computer, other programmable data processing apparatus, or other device to work in a specific manner, so that the instructions stored in the computer-readable medium produce a manufactured product including instructions for implementing the functions / actions specified in (one or more) boxes of the flowchart illustration and / or block diagram.

[0024] The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operating steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, such that the instructions executed on the computer, other programmable data processing apparatus, or other device provide a process for implementing the functions / actions specified in (one or more) boxes of the flowchart illustration and / or block diagram.

[0025] The flowchart illustrations and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various aspects of the present disclosure. In this regard, each box in the flowchart illustration or block diagram can represent a module, section or part of the code, which includes one or more executable instructions for implementing the specified (one or more) logical functions. It should also be noted that in some alternative embodiments, the functions shown in the box may not occur in the order shown in the accompanying drawings. For example, in fact, the two boxes shown in succession can be executed substantially simultaneously, or these boxes can sometimes be executed in reverse order or out of order, depending on the functions involved. It will also be noted that each box of the block diagram and / or flowchart illustration, and the combination of boxes in the block diagram and / or flowchart illustration can be implemented by a system based on special-purpose hardware or a combination of special-purpose hardware and computer instructions that performs a specified function or action.

[0026] While the foregoing is directed to aspects of the present disclosure, other and further aspects of the disclosure may be devised without departing from the basic scope thereof, and the scope of which is to be determined by the claims that follow.

[0027] Figure 1 An unmanned aerial vehicle system 100 is shown for avoiding collisions of an aerial vehicle with objects on the ground according to one aspect. Before starting a flight, a pilot 102 (e.g., a human pilot) may initiate flight of the aerial vehicle. For example, the pilot 102 may do so by providing commands 104 to the unmanned aerial vehicle system to initiate the flight and control the flight during operation. The commands 104 may include joystick commands, a predefined route for the aerial vehicle (e.g., within a specific airspace), or any other suitable command or commands.

[0028] The air vehicle (e.g., UAV) may then take off and begin operations. Once the UAV is in the air, the command and control unit 140 may calculate a change in the current flight state (FS) of the air vehicle (e.g., altitude, speed, heading) to cause the UAV to follow a desired route, whether this is a predefined route or in response to an ongoing command (e.g., a joystick command) from the pilot 102. The command and control unit 140 provides a desired flight state (ΔFS) 142 to the dynamic protection module 130. In an embodiment, the ΔFS includes both a change in flight state and no change in flight state. For example, the command and control unit 140 may repeatedly (e.g., continuously, at defined intervals, or using any other suitable technique) identify the ΔFS (e.g., with respect to a continuation of a current flight plan or a modified flight plan).

[0029] In parallel, the exposed object detector 114 identifies potentially exposed objects (e.g., on the ground or under the UAV). For example, the exposed object detector 114 may receive data from one or more sensors 112. The sensor 112 may include an infrared sensor, a radar sensor, a LIDAR sensor, a visible spectrum sensor (e.g., a visible spectrum camera), or any other suitable sensor. In one aspect, as further discussed below, the exposed object detector 114 uses a suitable ML technique to identify exposed objects. For example, the exposed object detector 114 may use a deep learning neural network (DNN) or another suitable ML technique (e.g., a supervised ML model) to use data from the sensor 112 to identify the object. This may include image object detection (e.g., from one or more visible spectrum cameras) or any other suitable ML technique. The use of ML is merely an example, and the exposed object detector 114 may use any suitable technique or combination of techniques known to a person of ordinary skill in the art without excessive experimentation to identify exposed objects.

[0030] In one aspect, the exposed object detector 114 provides object type and position data 116 to the trajectory estimator. The trajectory estimator 120 estimates a four-dimensional (e.g., in three dimensions over time) trajectory of the exposed object detected at block 114. For example, the trajectory estimator 120 may identify a movement pattern of the object. In one aspect, the exposed object detector 114 estimates the object trajectory using a suitable ML technique. This is merely an example, and any suitable technique or combination of techniques known to one of ordinary skill in the art without undue experimentation may be used.

[0031] Alternatively or in addition, the trajectory estimator 120 (or any other suitable module) may calculate a dynamically sized safety buffer around the exposed object identified at block 114. For example, the trajectory estimator 120 may calculate the real-time safety buffer size requirement based on characteristics of the object (e.g., potential danger to the object, speed of movement of the object, likelihood of movement of the object, and any other suitable characteristics) and characteristics of the aerial vehicle (e.g., UAV). For example, the trajectory estimator 120 will calculate a different dynamic safety buffer for a standing person than for a bicycle rider or a fast-moving car, and the buffer will depend on the speed, size, mass, and other suitable characteristics of the aerial vehicle.

[0032] In one aspect, the flight control module 122 may operate actuators 124 in the airborne vehicle to control the current FS, and the navigation module 126 may provide the current FS information to the dynamic protection module 130. The dynamic protection module 130 uses any or all of the current FS data, the desired ΔFS 142, and the estimated object trajectory 128 to calculate whether the desired ΔFS 142 may result in a risky situation (e.g., a situation where a collision between the airborne vehicle and the object may occur) under a hypothetical loss of propulsion.

[0033] If such a collision risk is detected, dynamic protection module 130 may take a number of actions to prevent an unwanted collision between the air vehicle and the object. For example, dynamic protection module 130 may provide warning 134 to command and control module 140, which in turn may provide a warning to pilot 102 (e.g., a stick vibration, a reaction force, or any other suitable warning known to one of ordinary skill in the art without undue experimentation).

[0034] Alternatively, or in addition, the dynamic protection module 130 may automatically calculate a corrected ΔFS' 132 to reduce the risk of collision. The dynamic protection module 130 may provide the corrected ΔFS' 132 (e.g., along with any warnings 134) to the command and control module 140. This may allow the pilot 102 to control the air vehicle to follow the suggested change in flight conditions indicated by the corrected ΔFS' 132.

[0035] In addition, the dynamic protection module 130 can provide the corrected ΔFS' 132 to the flight control 122. The flight control 122 can control the actuator 124 (e.g., change the altitude, speed, heading, or any other aspect) to modify the flight state. In one aspect, the action taken by the dynamic protection module 130 when a collision risk is detected (e.g., whether to provide any or all of a warning, a recommended flight state change, or an automatic flight state change) can be configured. For example, if a possible collision risk is detected, the pilot 102 or any other suitable human operator can configure the unmanned aerial vehicle system 100 to provide the desired behavior. In addition, the dynamic protection module 130 can be configured to provide different responses (e.g., more or less aggressive flight state changes, favoring changes in one or more specific aspects of the flight state (e.g., altitude, speed, heading) over other aspects, or any other suitable response known to a person of ordinary skill in the art without undue experimentation).

[0036] Figure 2 A block diagram of a controller 200 for controlling an unmanned aerial vehicle to avoid collision with an object on the ground is shown according to one aspect. In one aspect, the controller 200 includes a processor 202, a memory 210, and a network component 220. The memory 210 can take the form of any non-transitory computer-readable medium known to one of ordinary skill in the art without undue experimentation. The processor 202 typically retrieves and executes programming instructions stored in the memory 210. The processor 202 represents a single central processing unit (CPU), multiple CPUs, a single CPU with multiple processing cores, a graphics processing unit (GPU) with multiple execution paths, etc.

[0037] The network component 220 includes components necessary for the controller 200 to interface with a suitable communication network. For example, the network component 220 may include wired, WiFi, or cellular network interface components and associated software. Although the memory 210 is shown as a single entity, those of ordinary skill in the art will understand without undue experimentation that the memory 210 may include one or more memory devices having memory frames associated with physical addresses, such as random access memory (RAM), read-only memory (ROM), flash memory, or other types of volatile and / or non-volatile memory.

[0038] The memory 210 generally includes program code for performing various functions associated with the use of the controller 200. The program code is generally described as various functional "applications" or "modules" within the memory 210, but, as one of ordinary skill in the art will appreciate without undue experimentation, alternative embodiments may have different functions and / or combinations of functions. Within the memory 210, the object detector service 212 facilitates exposed object detection (e.g., as described above with respect to Figure 1 114 shown in FIG. 1 and described below with respect to Figure 3 2. The trajectory estimator service 214 facilitates estimating the exposed object trajectory (e.g., as described above with respect to Figure 1 120 and the following description of Figure 3 Dynamic protection service 216 facilitates providing dynamic protection (e.g., as described above with respect to an object below the aerial vehicle (e.g., an object on the ground below the UAV) in the event of a hypothetical loss of propulsion. Figure 1 130 as shown in FIG. 1 and as described below with respect to FIG. Figure 3 further described).

[0039] although Figure 2 The object detector service 212, trajectory estimator service 214, and dynamic protection service 216 are described as being located in the memory 210, but for the sake of clarity, this representation is provided only as an illustration. More generally, the controller 200 may include one or more computing platforms, such as, for example, computer servers, which may be located in the same location, or may form an interactively linked but distributed system, such as a cloud-based system (e.g., a public cloud, a private cloud, a hybrid cloud, or any other suitable cloud-based system). For example, the functionality of the object detector service 212, the trajectory estimator service 214, and the dynamic protection service 216 may be spread across an air vehicle, one or more ground control systems, and any other suitable computing system. Therefore, the processor 202 and the memory 210 may correspond to distributed processors and memory resources within one or more computing environments. Therefore, it should be understood by those of ordinary skill in the art without undue experimentation that any or all of the object detector service 212, the trajectory estimator service 214, and the dynamic protection service 216 may be stored away from the controller 200 in the distributed memory resources of one or more computing environments.

[0040] Figure 3 FIG. 3 is a flowchart 300 illustrating controlling an unmanned aerial vehicle to avoid collision with an object on the ground according to one aspect. At block 302, an object detector service (e.g., Figure 2 The object detector service 212 shown in FIG. 1 identifies one or more exposed objects. For example, as described above with respect to Figure 1As described in block 114 shown in FIG. 1 , the object detector service may use suitable ML techniques to identify objects beneath the aerial vehicle.

[0041] In one aspect, the object detector service may use one or more onboard sensors (including, but not limited to, one or more cameras) to detect static and moving objects. The object detector service may use one or more cameras over the visible spectrum to detect and classify exposed objects based on purely visible data. Alternatively, or in addition, the object detector service may use additional information, such as LIDAR point clouds, radar, data, or infrared imaging sensors. These are merely examples. In addition, the object detection service may use ML for object detection and classification (e.g., a DNN or other suitable neural network or another trained ML model). This is also merely an example, and the object detection service may use any suitable technique known to a person of ordinary skill in the art without undue experimentation (e.g., algorithmic techniques that do not use ML).

[0042] At block 304, a trajectory estimator service (e.g., Figure 2 The trajectory estimator service 214 shown in ) predicts the trajectory of one or more exposed objects. For example, as described above with respect to Figure 1 As described in block 120 shown in FIG. 3 , the trajectory estimator service may use suitable ML techniques to identify the trajectory of the moving object identified in block 302 .

[0043] In one aspect, the trajectory estimator service computes 4-dimensional trajectories of these objects to predict their future positions (e.g., using one or more suitable ML models). In addition, the trajectory estimator service can use different techniques (e.g., different trained ML models) for different classes of objects. For example, different techniques can be used for walking pedestrians, bicyclists, motorcyclists, cars, boats, or any other suitable moving objects.

[0044] In one aspect, the trajectory estimator service uses a temporal data sequence (e.g., a video or LIDAR frame sequence) to track exposed objects, identify the state of the exposed objects (e.g., static or moving), and predict the trajectory of any moving exposed objects. Any suitable ML technique known to a person of ordinary skill in the art without excessive experimentation can be used. For example, an ML model (e.g., a DNN or other trained ML model) can be trained to obtain a sequence of data frames for each of the detected objects and infer a prediction of the future trajectory of the object. In one aspect, multiple data sources (e.g., data from multiple sensors or multiple types of sensor data) can be used to enhance the accuracy of the prediction. However, this is merely an example, and the trajectory estimator service can use any suitable technique known to a person of ordinary skill in the art without excessive experimentation (e.g., any suitable ML technique or non-ML technique).

[0045] At block 306, a dynamic protection service (e.g., Figure 2 The dynamic protection service 216 shown in FIG. 2 receives the desired flight status. For example, as described above with respect to Figure 1 As described in block 142 shown in , the pilot may provide ongoing commands (e.g., joystick commands), predefined routes, or both to control an air vehicle (e.g., a UAV). The dynamic protection service may receive a desired flight state reflecting these commands (e.g., a desired change to the current flight state or a desire to maintain flight conditions).

[0046] At block 308, the dynamic protection service identifies the flight state. Figure 1 As described in block 126 shown in FIG. 1 , the dynamic protection service may receive the current flight state (e.g., attitude, altitude, speed, wind speed, latitude, longitude, orientation, etc.). In an embodiment, the current flight state reflects any changes in the flight state resulting from the correction. For example, the current flight state may be reflected in the flight state provided by the dynamic protection module 130 to the flight control unit 122. Figure 1 The change after the correction ΔFS′ 132 is shown in FIG.

[0047] At block 310, the dynamic protection service calculates a predicted collision point with any object identified at block 302, as well as a predicted uncertainty level in the event of a hypothetical loss of propulsion. For example, the dynamic protection service may use the identified objects and trajectories (e.g., as determined at blocks 302 and 304 above), the current flight state (e.g., as received at block 308 above), and the desired flight state (e.g., as received at block 306 above) to estimate when or whether the air vehicle is likely to collide with an exposed object if the air vehicle were to lose propulsion.

[0048] In one aspect, the dynamic protection service can use any or all of a plurality of different models to predict the collision point, including an uncontrolled gliding model of a fixed-wing air vehicle, a parabolic free fall model, an uncontrolled rotation model, or any other suitable model known to those of ordinary skill in the art without excessive experimentation. In one aspect, the dynamic protection service uses all of these models to predict the collision. Alternatively, the dynamic protection service selects one or more preferred models (e.g., based on user commands, air vehicle characteristics, environmental characteristics, or any other suitable factors). In addition, the dynamic protection service can estimate the uncertainty of the prediction based on the input values.

[0049] Additionally, in an embodiment, the dynamic protection service may use the predicted trajectories for the identified objects but not for all objects (e.g., using predicted trajectories for some or no objects) to calculate the predicted collision point. However, this may reduce the accuracy of the predicted collision point, requiring a larger safety buffer for corrected flight conditions (e.g., when preventing a collision at block 318, described below). That is, the use of predicted trajectories may not be necessary or appropriate in all situations, but where available, it may improve the accuracy of the predicted collision point and reduce the necessary safety buffer to avoid a collision between an air vehicle and an object.

[0050] At block 312 , the dynamic protection service checks for potential collisions with exposed objects. At block 314 , if the dynamic protection service predicts a collision, flow moves to block 318 .

[0051] At block 318, the dynamic protection service prevents collisions. Figure 5 This is discussed further. For example, the dynamic protection service may provide a warning or other alert indicating a risk of collision to a pilot or other human operator. Alternatively, or in addition, the dynamic protection service may automatically control the air vehicle to avoid a collision or reduce the likelihood of a collision occurring.

[0052] Returning to block 314, if the dynamic protection service predicts no collision with any exposed objects, flow proceeds to block 316. At block 316, the air vehicle continues the desired flight (e.g., based on the desired flight state received at block 306). For example, the air vehicle may move as directed by the pilot (e.g., based on joystick control or using a predefined route).

[0053] In an embodiment, Figure 3 The illustrated techniques are performed continuously in real time during flight. One or more controllers (e.g., Figure 2The controller 200 shown in FIG. 2 continuously monitors and estimates trajectories for exposed objects. Then, in the event of a hypothetical loss of propulsion, the dynamic protection service identifies the risk of a potential collision with the exposed object and prevents the collision.

[0054] Figure 4 An example of controlling an unmanned aerial vehicle to avoid collision with objects on the ground is shown. Figure 4 An example urban environment 400 is shown in which an aerial vehicle (e.g., a UAV) 410 is operating. If no action is taken, the aerial vehicle 410 is traveling in a flight state 412. An object detector service (e.g., Figure 2 424, and another car 426. For example, the object detector service 212 shown in FIG. 4 identifies three potentially exposed objects: a car 422, a person 424, and another car 426. For example, the object detector service can use a suitable ML model (e.g., a DNN) to detect these objects from one or more sensors (e.g., a camera, an infrared sensor, a LIDAR, a radar, or any other suitable sensor) associated with the air vehicle 410. The object detector service can further use the ML model to classify the objects (e.g., classify the car 422, the person 424, and the other car 426).

[0055] In one aspect, a trajectory estimator service (e.g., Figure 2 4 ) estimates the trajectory of each of objects 422, 424, and 426. For example, the trajectory estimator service estimates the trajectory of car 422 as vector 432A leading to position 432B. The trajectory estimator service estimates the trajectory of person 424 as vector 434A leading to position 434B. The trajectory estimator service estimates the trajectory of car other 426 as vector 436A leading to position 436B.

[0056] In one aspect, a dynamic protection service (e.g., Figure 2 442) determines whether the air vehicle 410 is at risk of colliding with any of the objects 422, 424, or 426 and takes steps to avoid the collision. For example, the dynamic protection service calculates the estimated falling trajectory of the air vehicle 410 under the assumed loss of propulsion as the curved dashed line 442.

[0057] The dynamic protection service predicts that if the air vehicle 410 suddenly loses propulsion, the air vehicle 410 is at risk of colliding with the car 422, and takes measures to avoid the collision. For example, the dynamic protection service can send a command to the air vehicle 410 (e.g., to an actuator associated with the air vehicle 410) to change the flight state to the new flight state 414. The sudden loss of propulsion along the new flight state 414 will result in a different descent trajectory 444, in which the air vehicle 410 will not be at risk of colliding with the car 422 (or any other exposed object). Alternatively, or in addition, the dynamic protection service can provide a warning to the pilot to encourage the pilot to change the flight state of the air vehicle 410. Figure 4 The illustration in is merely an example, and, as one of ordinary skill in the art would appreciate without undue experimentation, one or more of the techniques described herein may be applied to a variety of aerial vehicles (e.g., a variety of manned or unmanned flying vehicles) in a variety of environments.

[0058] Figure 5 is a flow chart according to one aspect, the flow chart illustrating preventing an unmanned aerial vehicle from colliding with an object on the ground. In one aspect, Figure 5 Corresponding to Figure 3 Block 318 is shown. At block 502, a dynamic protection service (e.g., Figure 2 The dynamic protection service 216 shown in ) determines whether to perform automatic correction.

[0059] As described above, in one aspect, if a hypothetical loss of propulsion occurs, the dynamic protection service can automatically correct the flight state of the aerial vehicle (e.g., UAV) to avoid the risk of collision with the exposed object. For example, the dynamic protection service can determine a modified flight state of the aerial vehicle that reduces the risk of collision, and can control the aerial vehicle (e.g., using appropriate actuators) to travel on the modified flight state. Alternatively, or in addition, the dynamic protection service can provide a warning to a human operator (e.g., a pilot) to control the aerial vehicle to reduce the risk of collision.

[0060] In one aspect, the automatic correction is configurable (e.g., by a human administrator). Alternatively, or in addition, the automatic correction depends on the characteristics of the air vehicle, the flight environment, or any other suitable factor known to one of ordinary skill in the art without undue experimentation. For example, automatic correction may be enabled for specific air vehicles, various types of air vehicles, or flight environments, and disabled for other scenarios. If automatic correction is available, the process proceeds to block 504. If automatic correction is not available, the process proceeds to block 506, discussed further below.

[0061] At block 504, the dynamic protection service modifies the command to avoid the collision. In one aspect, the dynamic protection service generates a corrected flight state 508. For example, as described above in Figure 4 As shown in FIG. 5 , corrected flight state 508 (e.g., Figure 4 The corrected flight state 414 shown in FIG. 4 may reduce the risk of collision with one or more exposed objects.

[0062] In one aspect, the dynamic protection service can calculate an appropriate safety buffer zone to reduce the risk of a collision. The safety buffer zone can be based on characteristics of the air vehicle (e.g., speed, size, descent velocity), exposed objects, environment (e.g., wind speed), and any other suitable characteristics known to one of ordinary skill in the art without undue experimentation. In addition, the safety buffer zone can be based on the certainty of the dynamic protection service's collision prediction. For example, as described above with respect to Figure 3 As discussed in block 310 shown in , the dynamic protection service can calculate a certainty level of its prediction of the collision point with the exposed object. This certainty level can be used to generate a safety buffer for the corrected flight state 508. A higher certainty level may require a smaller safety buffer, while a lower certainty level may require a larger safety buffer. In addition, as described above, using object trajectories can reduce the required safety buffer.

[0063] In addition, the dynamic protection service can use any one or all of a plurality of different fall models to predict the fall trajectory of the aerial vehicle and generate a corrected flight state. The dynamic protection service can use one or more of an uncontrolled gliding model, a parabolic free fall model, an uncontrolled rotation model, or any other suitable model known to a person of ordinary skill in the art without excessive experiments for a fixed-wing aerial vehicle. In an embodiment, the dynamic protection service selects one or more preferred models (e.g., based on user commands, aerial vehicle characteristics, environmental characteristics, or any other suitable factors known to a person of ordinary skill in the art without excessive experiments). In addition, the dynamic protection service can calculate the collision point on the ground when applying different potential modifications to the corresponding flight parameters (e.g., speed, altitude, heading, etc.) so as to find a preferred non-collision trajectory based on (customer) predefined preferences (e.g., using an iterative process).

[0064] At block 510, the dynamic protection service modifies the flight state of the air vehicle. In one aspect, the dynamic protection service provides one or more flight control modules (e.g., Figure 1 The flight control module 122 shown provides corrected flight conditions 508. This controls appropriate actuators (e.g., Figure 1 The actuator 124 shown in FIG. 1 is used to change the flight state of the aerial vehicle.

[0065] Returning to block 504, in one aspect, the dynamic protection service proceeds to block 506 and sends one or more warnings to the human operator regarding a potential collision with the exposed object. The dynamic protection service may send the warning(s) in addition to or in lieu of automatically modifying the flight state of the air vehicle (e.g., if automatic correction is not available).

[0066] At block 506, the dynamic protection service may provide the pilot with a warning or a stick reaction force (e.g., Figure 1 144). The pilot can then control the air vehicle to reduce the risk of collision. In addition, in one aspect, the dynamic protection service can provide the pilot with a corrected flight state (e.g., in a human-readable format) to help the pilot control the air vehicle. The pilot can follow the corrected flight state (e.g., altitude, speed, direction) to reduce the risk of collision.

Claims

1. A method comprising: Receiving the flight status of the air vehicle (306); identifying an object beneath the aerial vehicle using one or more sensors (302); determining a risk that the aerial vehicle will collide with the object in the event that the aerial vehicle later loses propulsion (310); and The aerial vehicle is prevented from colliding with the object based on the determined risk (318).

2. The method of claim 1, wherein the aerial vehicle comprises an unmanned aerial vehicle (UAV) (100, 410), the method further comprising: A predicted trajectory of the object is determined (304, 432A, 434A, 436A), wherein determining the risk that the UAV will collide with the object is also based at least in part on the predicted trajectory (310).

3. The method of claim 2, wherein identifying the object uses a first one or more trained machine learning (ML) models (302) and determining the predicted trajectory uses a second one or more trained ML models (304).

4. The method of claim 3, wherein the first and second one or more trained ML models each comprise a corresponding deep learning neural network (DNN) (302, 304).

5. The method of claim 1, wherein the one or more sensors (112) include at least one of: (i) a visible spectrum camera, (ii) an infrared sensor, (iii) a LIDAR, or (iv) a radar.

6. The method of claim 1, wherein determining the risk that the aerial vehicle will collide with the object in the event that the aerial vehicle later loses propulsion comprises using a fall model (310) of the aerial vehicle.

7. The method of claim 6, wherein the fall model (310) comprises at least one of: (i) an uncontrolled gliding model of a fixed-wing air vehicle, (ii) a parabolic free fall model, or (iii) an uncontrolled rotation model.

8. The method of claim 1 , wherein preventing the aerial vehicle from colliding with the object comprises: generating a corrected flight state for the air vehicle (508); and The airborne vehicle is automatically controlled to operate using the corrected flight conditions (510).

9. The method of claim 8, wherein generating the corrected flight state of the air vehicle further comprises: A safety buffer zone is generated for the air vehicle and the object, wherein the corrected flight state is based on the safety buffer zone (508).

10. The method of claim 1 , wherein preventing the aerial vehicle from colliding with the object comprises: A warning is transmitted to a pilot of the air vehicle, the warning reflecting the risk that the air vehicle will collide with the object if the air vehicle later loses propulsion (506).

11. The method of claim 10, wherein preventing the aerial vehicle from colliding with the object further comprises: generating a corrected flight state for the air vehicle (508); and At least a portion of the corrected flight conditions is provided to the pilot (506, 508).

12. A non-transitory computer readable medium containing computer program code, which, when executed by operation of one or more computer processors (210), performs operations comprising: Receiving the flight status of the air vehicle (306); identifying an object beneath the aerial vehicle using one or more sensors (302); determining a risk that the aerial vehicle will collide with the object in the event that the aerial vehicle later loses propulsion (310); and The aerial vehicle is prevented from colliding with the object based on the determined risk (318).

13. The non-transitory computer-readable medium of claim 12, wherein the aerial vehicle comprises an unmanned aerial vehicle (UAV) (100 / 410), the operations further comprising: and determining a predicted trajectory of the object (304, 432A, 434A, 436A), wherein determining the risk that the UAV will collide with the object is also based on the predicted trajectory (310), wherein identifying the object uses a first one or more trained machine learning (ML) models (302) and determining the predicted trajectory uses a second one or more trained ML models (304).

14. The non-transitory computer-readable medium of claim 12, wherein determining the risk that the aerial vehicle will collide with the object in the event that the UAV later loses propulsion comprises using a fall model (310) of the aerial vehicle.

15. The non-transitory computer readable medium of claim 12, wherein preventing the aerial vehicle from colliding with the object comprises: generating a corrected flight state for the air vehicle (508); and The airborne vehicle is automatically controlled to operate using the corrected flight conditions (510).

16. An unmanned aerial vehicle (UAV), comprising: Computer processor (202); and A memory (210) having stored thereon instructions that, when executed on the computer processor, perform operations including: Identifying a flight state of the UAV (306); receiving a corrected flight state for the UAV (508), wherein the corrected flight state is calculated based on: identifying an object (110) below the UAV using one or more sensors and determining a risk (310) that the UAV will collide with the object if the UAV later loses propulsion; and The UAV is automatically controlled to operate using the corrected flight conditions (510).

17. The UAV of claim 16, wherein the corrected flight state (508) is based at least in part on determining a predicted trajectory (304, 432A, 434A, 436A) of the object and wherein identifying the object uses a first one or more trained machine learning (ML) models (302) and determining the predicted trajectory uses a second one or more trained ML models (304).

18. A UAV according to claim 16, wherein determining the risk that the UAV will collide with the object in the event that the UAV later loses propulsion includes using a fall model (310) of the UAV.

19. The UAV of claim 16, wherein the UAV comprises the one or more sensors, the operations further comprising: Sensor data regarding the object is captured using the one or more sensors (110).

20. The UAV of claim 19, wherein the one or more sensors (110) include at least one of: (i) a visible spectrum camera, (ii) an infrared sensor, (iii) a LIDAR, or (iv) a radar.

Images