Generate flight plans for semi-autonomous drones

The system for dynamically grouping semi-autonomous drones addresses the challenges of unreliability and complex coordination in traditional UAVs by enabling autonomous task planning and adaptive asset management, enhancing efficiency and resilience in drone operations.

JP2024544459AActive Publication Date: 2024-12-03ANDURIL IND INC
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
JP2024516740
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-09-17
Filing Date
2022-08-31
Publication Date
2024-12-03
Estimated Expiration
2042-08-31

AI Technical Summary

Technical Problem

Traditional unmanned aerial vehicles (UAVs) or drones face issues such as unreliability, slower speed, limited range, and complex coordination of multiple assets, requiring human intervention for task management.

Method used

A system for dynamically grouping semi-autonomous drones that includes a communications interface and processors to receive tasks, determine asset capabilities, and autonomously plan and execute operations, allowing for efficient coordination and adaptation in response to failures or changes.

Benefits of technology

Enhances the efficiency and resilience of drone operations by enabling autonomous task planning and dynamic asset reconfiguration, improving resource utilization and reducing reliance on human control.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024544459000001_ABST
    Figure 2024544459000001_ABST
Patent Text Reader

Abstract

A system for generating an environment for operations with assets includes one or more processors configured to obtain data associated with one or more tasks performed by the assets, where 1) the assets include semi-autonomous drones, and 2) the data associated with the tasks includes flight plans of other drones, determine discrete representations of geographic locations, the discrete representations including discrete elements each corresponding to a volume associated with the geographic location, annotate the discrete representations with the flight plans of the other drones to create an annotated representation, determine a first flight plan for one drone, the first flight plan determined based on the annotated representation, and communicate information regarding the first flight plan to the at least one other asset.
Need to check novelty before this filing date? Find Prior Art

Description

[Background technology]

[0001] Conventional unmanned aerial vehicles (UAVs) or drones are useful for performing many tasks. Such drones may perform surveillance, delivery of commercial packages or weapons, mapping remote or inaccessible areas, and / or other missions. Although useful, such drones suffer from many drawbacks. For example, drones are typically remotely piloted, may be unreliable, may be slower than desired, may have limited range, and / or may have other issues that adversely affect performance. Furthermore, coordination of multiple assets, including one or more drones, is complex and generally requires a user at a control center to control and coordinate the assets in connection with the performance of a task. [Brief description of the drawings]

[0002] Various embodiments of the invention are disclosed in the following detailed description and the accompanying drawings.

[0003] [Figure 1] FIG. 1 illustrates a system for performing operations in accordance with various embodiments of the present application.

[0004] [Diagram 2] 1 is a block diagram illustrating a device for configuring or controlling operations in accordance with various embodiments of the present application.

[0005] [Diagram 3] FIG. 1 is a block diagram illustrating a device for performing at least a portion of the operations in accordance with various embodiments of the present application.

[0006] [Figure 4A] FIG. 1 illustrates a system for performing at least a portion of the operations in accordance with various embodiments of the present application.

[0007] [Figure 4B]FIG. 1 illustrates a system for performing at least a portion of the operations in accordance with various embodiments of the present application.

[0008] [Figure 4C] FIG. 1 illustrates a system for performing at least a portion of the operations in accordance with various embodiments of the present application.

[0009] [Figure 5A] FIG. 1 illustrates a system for performing at least a portion of the operations in accordance with various embodiments of the present application.

[0010] [Figure 5B] FIG. 1 illustrates a system for performing at least a portion of the operations in accordance with various embodiments of the present application.

[0011] [Figure 5C] FIG. 1 illustrates a system for performing at least a portion of the operations in accordance with various embodiments of the present application.

[0012] [Figure 6] 1 illustrates a user interface for configuring, monitoring, and / or controlling operations in accordance with various embodiments of the present application.

[0013] [Figure 7A] 1 illustrates a method for structuring work in accordance with various embodiments of the present application.

[0014] [Figure 7B] 1 illustrates a method for structuring work in accordance with various embodiments of the present application.

[0015] [Figure 8A] 1 illustrates a method for performing at least one task of a work in accordance with various embodiments herein.

[0016] [Figure 8B]1 illustrates a method for performing at least one task of a work in accordance with various embodiments herein.

[0017] [Figure 8C] 1 illustrates a method for performing at least one task of a work in accordance with various embodiments herein.

[0018] [Figure 9A] 1 illustrates a method for implementing a plan associated with an operation in accordance with various embodiments of the present application.

[0019] [Figure 9B] 1 illustrates a method for implementing a plan associated with an operation in accordance with various embodiments of the present application.

[0020] [Figure 9C] 1 illustrates a method for implementing a plan associated with an operation in accordance with various embodiments of the present application.

[0021] [Figure 9D] 1 illustrates a method for implementing a plan associated with an operation in accordance with various embodiments of the present application.

[0022] [Figure 9E] 1 illustrates a method for implementing a plan associated with an operation in accordance with various embodiments of the present application.

[0023] [Figure 10A] 1 illustrates a method for implementing a plan associated with an operation in accordance with various embodiments of the present application.

[0024] [Figure 10B] 1 illustrates a method for implementing a plan associated with an operation in accordance with various embodiments of the present application.

[0025] [Figure 11]1 illustrates a method for structuring work in accordance with various embodiments of the present application.

[0026] [Figure 12A] 1 illustrates a method for implementing a plan associated with an operation in accordance with various embodiments of the present application.

[0027] [Figure 12B] 1 illustrates a method for implementing a plan associated with an operation in accordance with various embodiments of the present application.

[0028] [Figure 12C] 1 illustrates a method for implementing a plan associated with an operation in accordance with various embodiments of the present application.

[0029] [Figure 13] 1 illustrates a method for implementing a plan associated with an operation in accordance with various embodiments of the present application.

[0030] [Figure 14] 1 illustrates a method for implementing a plan associated with an operation in accordance with various embodiments of the present application.

[0031] [Figure 15] 1 illustrates a method for implementing a plan associated with an operation in accordance with various embodiments of the present application.

[0032] [Figure 16A] FIG. 1 illustrates a discrete representation in accordance with various embodiments of the present application.

[0033] [Figure 16B] 1 illustrates a discrete representation of geographic locations in accordance with various embodiments of the present application.

[0034] [Figure 16C] 1 illustrates a discrete representation of geographic locations in accordance with various embodiments of the present application.

[0035] [Figure 17A] 1 illustrates a discrete representation of geographic locations in accordance with various embodiments of the present application.

[0036] [Figure 17B] 1 illustrates a discrete representation of geographic locations in accordance with various embodiments of the present application.

[0037] [Figure 17C] 1 illustrates a discrete representation of geographic locations in accordance with various embodiments of the present application.

[0038] [Figure 17D] 1 illustrates a discrete representation of geographic locations in accordance with various embodiments of the present application.

[0039] [Figure 18] 1 illustrates a method for determining a plan for executing one or more tasks in accordance with various embodiments of the present application.

[0040] [Figure 19A] 1 illustrates a method for determining a flight plan in accordance with various embodiments of the present application.

[0041] [Figure 19B] 1 illustrates a method for determining a flight plan in accordance with various embodiments of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0042] The present invention may be embodied in various forms, including as a process, an apparatus, a system, a composition of matter, a computer program product embodied on a computer-readable storage medium, and / or a processor configured to execute instructions stored in and / or provided by a memory coupled to the processor. These embodiments or any other form the present invention may take may be referred to herein as techniques. In general, the order of steps of a disclosed process may be altered within the scope of the present invention. Unless otherwise noted, components such as a processor or memory described as configured to perform a task may be implemented as general components temporarily configured to perform the task at a given time, or as specific components manufactured to perform the task. As used herein, the term "processor" refers to one or more devices, circuits, and / or processing cores configured to process data, such as computer program instructions.

[0043] A detailed description of one or more embodiments of the present invention will now be given with reference to the drawings illustrating the principles of the present invention. The present invention will be described in connection with such embodiments, but is not limited to any of them. The scope of the present invention is limited only by the claims, and the present invention includes many alternatives, modifications, and equivalents. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. These details are for the purpose of example, and the present invention may be practiced according to the claims without some or all of these specific details. For simplicity, technical matters well known in the art related to the present invention have not been described in detail so as not to unnecessarily obscure the present invention.

[0044] According to various embodiments, a system for grouping assets is disclosed. The system may include a communication interface and one or more processors connected to the communication interface. The one or more processors may be configured to: (i) receive, via the communication interface, data associated with one or more tasks performed by the assets, the assets including a plurality of drones, the drones being at least semi-autonomous; (ii) determine, from among the plurality of assets, a group of assets for performing the one or more tasks; and (iii) communicate, via the communication interface, instructions to at least one drone in the group of assets, the instructions directing that the one or more tasks be at least partially completed by the at least one drone. Determining the group of assets for performing the one or more tasks may include determining one or more functions associated with one or more features of the one or more tasks, and determining the plurality of drones based at least in part on (i) the one or more functions associated with the one or more features of the one or more tasks and (ii) the one or more drone functions associated with each of the plurality of drones.

[0045] According to various embodiments, a drone is disclosed. The drone may be a semi-autonomous drone. The drone may include a communication interface and one or more processors connected to the communication interface. The one or more processors may be configured to: (i) receive an indication via the communication interface that the drone is part of a constellation of assets, the constellation being tasked with performing one or more elements of one or more tasks, the constellation including a plurality of drones; (ii) communicate information regarding the one or more elements via the communication interface; and (iii) communicate information regarding a plan for performing the one or more tasks via the communication interface, the information regarding the plan for performing the one or more tasks being communicated to at least one other drone in the constellation of assets. The information regarding the one or more elements may be communicated to at least one other drone in the constellation of assets. The information regarding the one or more elements may be based at least in part on information acquired by one or more sensors of the constellation of assets. The information regarding the one or more elements may be used in connection with determining a plan for performing the one or more tasks.

[0046] According to various embodiments, a drone is disclosed. The drone may be a semi-autonomous drone. The drone may include a communication interface and one or more processors connected to the communication interface. The one or more processors may be configured to (i) receive an indication via the communication interface that the drone is part of a constellation of assets, the constellation being tasked with performing one or more elements of one or more tasks, the constellation of assets including a plurality of drones, (ii) determine that at least one drone in the constellation of assets has experienced a malfunction, (iii) update a plan to perform the one or more tasks to an updated plan in response to determining that the at least one drone has experienced a malfunction, and (iv) communicate information regarding the updated plan via the communication interface, the information regarding the updated plan being communicated to at least one remaining drone in the constellation of assets. The one or more processors may be further configured to communicate information obtained during execution of at least one element of the updated plan via the communication interface, the information regarding the updated plan being communicated to at least one remaining asset of the set of drones.

[0047] According to various embodiments, a system for dynamically grouping assets is disclosed. The system may include a communication interface and one or more processors coupled to the communication interface. The one or more processors may be configured to: (i) obtain data associated with one or more tasks performed by the assets, the assets including a plurality of drones, the plurality of drones being at least semi-autonomous; (ii) determine to modify the assets based at least in part on the data associated with the one or more tasks; and (iii) communicate instructions to at least one drone in the assets via the communication interface, the instructions directing the modification to the assets. Determining to modify the assets based at least in part on the data associated with the one or more tasks may include determining one or more functions associated with the one or more remaining tasks, and determining the modification to the assets based at least in part on (i) the one or more functions associated with the one or more remaining tasks and (ii) one or more drone functions respectively associated with the assets in the set.

[0048] According to various embodiments, a system is disclosed. The system may include a communication interface and one or more processors connected to the communication interface. The one or more processors may be configured to: (i) display a first user interface, the first user interface including one or more selectable elements associated with characteristics of one or more tasks to be performed; (ii) receive one or more user selections via the first user interface regarding the characteristics of the one or more tasks to be performed; (iii) display a second user interface in response to receiving the one or more user selections entered into the first interface; (iv) receive one or more user selections via the second user interface regarding a group of one or more assets to be deployed to perform the operation; (v) determine an operation to be performed, the operation being determined based at least in part on (a) the one or more user selections regarding the characteristics of the one or more tasks to be performed and (b) the one or more user selections regarding the group of one or more assets to be deployed to perform the operation; and (vi) communicate information regarding the operation via the communication interface. The second user interface may be configured at least in part based on at least one of the one or more user selections entered into the user interface. The second user interface may comprise one or more selectable elements associated with a fleet of one or more assets performing a task, where the fleet of one or more assets includes one or more drones, and where the one or more drones are semi-autonomous. Information regarding the task is communicated to at least one drone in the fleet of assets. The information regarding the task causes the fleet of assets to be deployed to perform at least a portion of the task.

[0049] According to various embodiments, a system is disclosed. The system may include a communication interface and one or more processors connected to the communication interface. The one or more processors may be configured to: (i) obtain information associated with one or more tasks performed by the assets; (ii) determine a discrete representation of a geographic area, the discrete representation including a plurality of discrete elements each corresponding to a volume within the geographic area; (iii) annotate the discrete representation to create an annotated representation with one or more parameters related to the geographic area comprising at least a subset of the plurality of discrete elements based at least in part on determining that the one or more parameters relate to the geographic area; (iv) determine a plan for performing the one or more tasks, the plan based at least in part on the annotated representation; and (v) cause the one or more tasks to be performed based at least in part on the plan. The assets may include a plurality of drones, the plurality of drones being at least semi-autonomous. The data associated with the one or more tasks may include one or more parameters related to a geographic area in which at least one asset in the assets performs the one or more tasks.

[0050] According to various embodiments, a system is disclosed. The system may include a communication interface and one or more processors connected to the communication interface. The one or more processors may be configured to: (i) obtain data associated with one or more tasks performed by the assets; (ii) determine a discrete representation of the geographic area, where the discrete representation includes a plurality of discrete elements each corresponding to a volume within the geographic area; (iii) annotate the discrete representation with a flight plan of one or more other drones to create an annotated representation; (iv) determine a first flight plan for at least one drone of the plurality of drones, where the first flight plan is determined based at least in part on the annotated representation; and (v) communicate information regarding the first flight plan to at least one other asset in the assets. The assets may include a plurality of drones, where the plurality of drones may be at least semi-autonomous. The data associated with the one or more tasks may include a flight plan of the one or more other drones.

[0051] According to various embodiments, the system receives high-level instructions for performing a task using the drone swarm. The high-level instructions may include a high-level description or definition of the task. In response to receiving the high-level instructions, the system may determine one or more low-level instructions in association with performing the task using the drone swarm. For example, a semi-autonomous drone, such as a leader drone, may break down the high-level instructions into one or more tasks for the drone swarm, and the semi-autonomous drone may communicate the one or more tasks (or a plan for completing one or more tasks or task elements) to one or more drones (e.g., follower drones) in the drone swarm. In some embodiments, a user or a control system provides a high-level definition of the task, and one or more semi-autonomous drones autonomously determine the tasks or task elements associated with the task. The drone swarm (e.g., including one or more semi-autonomous drones) then performs the tasks or task elements associated with the task.

[0052] As used herein, a "semi-autonomous drone" refers to a drone that determines a plan for performing at least a portion of a task or task based on high-level instructions for performing the task or task without human intervention, receives feedback information regarding the status of the drone and / or the plan, and updates or determines a new plan to accommodate the feedback information. The updated or new plan may provide instructions for controlling the drone in a manner consistent with the high-level instructions or task. The high-level instructions may be provided to the drone by a user interface or by a server via a network connection. In response to receiving the high-level instructions, the drone determines a plan for performing the task or task and commands another asset to perform the plan or provide the task or task and provide feedback to the drone. The term "semi-autonomous drone" may be used interchangeably with "drone" herein. For example, the task of "scanning a polygon and tracking a moving target" may include planning how one or more drones will effectively scan the polygon, but in the middle of a scan pattern, a new target may appear and the semi-autonomous drone re-plans and begins tracking the new target.

[0053] The high-level instructions may include one or more of an indication of the type of work, the location where the work is to be performed, the target of the work, the time of day the work is to be performed, etc. The low-level instructions may provide more specific information or definition regarding the work. In some embodiments, the low-level instructions may indicate the specific assets to perform the tasks within the work, the manner in which the tasks are to be performed (e.g., speed, altitude, type of sensor, etc.). The low-level instructions may further be determined based on environmental or situational information within the area in which the work is to be performed. An example of a high-level instruction may be an instruction to perform the work of monitoring a particular road on a specific date. In contrast, a corresponding low-level instruction may include an instruction to an asset in a group of assets indicating a specific location (or range of locations) along the road where the specific asset will capture information. The leader drone may determine the specific follower drone to execute the low-level instruction. The low-level instruction may indicate the type of information to be captured, the speed at which the asset will travel, and the altitude at which the asset will travel or monitor. As another example, a high-level command might include "observe this target from several good vantage points," and the corresponding low-level command might be "fly drone 1 100 meters away from the target on a heading of 30 degrees, and fly drone 2 150 meters away on a heading of 180 degrees."

[0054] As used herein, an "asset" may correspond to a device, terminal, vehicle, and / or system that may receive information from a server (e.g., directly or via another asset), utilize sensors to acquire information, communicate the collected information to a leader drone / asset or server, deliver payloads, and / or execute a plan to perform a task or an element of a task. Examples of assets include, without limitation, drones, satellites, communication towers, observation stations, vehicles such as autonomous or semi-autonomous vehicles (e.g., planes, boats, cars, trucks, helicopters, etc.), drones (e.g., large drones, small drones, etc.), jet aircraft, small autonomous sensors (e.g., sensors, "dust" sensors, etc.). An asset has an associated identifier and a defined set of features, functions, characteristics, etc. (collectively also referred to herein as "capabilities"). The set of defined capabilities may indicate the set of functions that the asset may perform. For example, the set of capabilities may indicate a set of peripherals (cameras, sensors, payloads, payload deployment mechanisms, pre-loaded software, etc.) that are included with or otherwise operatively connected to the asset. As another example, the set of defined capabilities may indicate whether the asset includes a turret (e.g., a multi-axis pointing mechanism to which other functions are attached (e.g., a pan-tilt unit on a surveillance camera)). As another example, the set of defined capabilities may indicate whether the asset can send and receive data from other communication sources, such as disabling / enabling transmissions.

[0055] As used herein, "plan information" may correspond to information regarding a plan for performing tasks, elements, and / or activities. As an example, plan information may include parameters for tasks, elements, and / or activities. As an example, plan information may include instructions for operating assets.

[0056] As used herein, "capability information" may correspond to information regarding the capabilities of an asset (e.g., a drone). As an example, capability information may include an indication of a capability and / or set of capabilities of the drone.

[0057] As used herein, "sensor information" may correspond to information obtained by a sensor (such as a sensor on an asset). The sensor information may be processed locally by the asset or device, such as to convert raw sensor input into a predetermined format.

[0058] As used herein, "state information" may correspond to information regarding the status or state of a device, such as an asset (e.g., a drone). As an example, state information may include information indicating a state of a particular function or set of functions of a drone. As an example, state information may include information indicating a state of one or more tasks (e.g., the extent to which an asset has completed a task). As an example, state information may include information regarding the results of the execution of a task, element, and / or operation.

[0059] Various embodiments include a system for configuring a task to be performed by a group of assets, such as semi-autonomous drones. The system may include one or more servers, client terminals, and / or a group of one or more assets. The server, client terminal, and group of one or more assets may communicate through the one or more assets. The client terminal may provide a user interface through which the task may be configured (e.g., defined), the assets may be deployed to perform at least a portion of the task, and feedback regarding the status of the execution of the task may be provided. In connection with configuring the task, the assets performing at least a portion of the task may be determined. The portion of the task may be determined by a semi-autonomous asset, such as a leader drone. For example, the leader drone may analyze the task (e.g., high-level instructions for performing the task) and determine one or more tasks each corresponding to at least a portion of the task. For example, a first task may correspond to a first portion of the task, and a second task may correspond to a second portion of the task. The assets may at least semi-autonomously determine a plan for performing at least a portion of the task and execute the plan.

[0060] In some embodiments, the server may determine a set of assets to perform one or more tasks associated with the operation. The set of assets may be determined based at least in part on one or more functions associated with one or more characteristics of the one or more tasks. For example, the one or more functions associated with one or more characteristics of the one or more tasks may correspond to a function to be performed (e.g., for a surveillance task, the function may correspond to video capture, image capture, etc.; for a drone delivery, the function may correspond to a range, a mechanism for carrying a payload, a weight limit of the payload, etc.). The set of assets to perform the one or more tasks may be determined based at least in part on one or more drone functions respectively associated with the multiple drones. For example, in response to determining a configuration of the operation, the server may determine one or more tasks corresponding to the operation and determine functions associated with the tasks (e.g., based at least in part on a mapping of tasks to functions, etc.). The server may determine one or more assets that match the functions associated with the task (e.g., based at least in part on the mapping of assets to functions) and determine a group of assets to perform one or more tasks or to add an asset to a given group of assets to perform one or more tasks. In response to deploying the group of assets to perform the work or determining the group of assets, at least one asset (e.g., a leader drone) in the group of assets may provide information about the work (e.g., one or more characteristics or identifiers associated with the one or more tasks, an indication of the group of assets to perform the one or more tasks, etc.). In some embodiments, assets in the group of assets, or each asset in the group of assets, are provided with an indication that the particular asset is used in connection with performing one or more tasks. The group of assets may be determined at least in part based on the availability of one or more assets. For example, the ownership of the asset (e.g., lease agreement) may be used in connection with determining whether to include the asset in the group of assets to perform one or more tasks.

[0061] Various embodiments include a grouping of assets that perform at least a portion of a task (e.g., one or more tasks). The assets may include one or more semi-autonomous drones. In response to a task being configured (e.g., at a server), high-level instructions for the task (or a portion thereof, such as a task) are provided to at least one drone in the assets. The high-level instructions may include a high-level definition of the task, such as one or more characteristics of the task (e.g., the task to be performed, the location where the task is to be performed, the time the task is to be performed, the area in which the assets are permitted to operate, etc.). In response to receiving information about the task (e.g., an indication that the drone is included in the assets), the semi-autonomous drone may determine one or more elements or tasks of the task or tasks to be performed by the assets. The semi-autonomous drone may be a leader drone among the drones in the assets. In some embodiments, the leader drone may determine a plan for at least one drone in the assets to perform at least one element of the task or tasks. For example, a leader drone may execute (e.g., locally by a processor on the drone) a planning service that determines a plan for executing one or more tasks assigned to the fleet of assets. Determining a plan for executing one or more tasks may be based at least in part on one or more of: (i) the functionality to be performed, (ii) the environment in which the task is to be performed, (iii) functionality associated with the task to be performed, (iv) functionality of at least one drone in the fleet of assets, etc. The plan may be updated based on feedback information provided by one or more assets in the fleet (e.g., indications of asset failure, changes in the environment, location of targets, etc.).

[0062] According to various embodiments, assets in an asset group may communicate with each other. For example, a leader drone in an asset group may provide information regarding at least one element of a task (e.g., a plan for an asset to perform an element of a task or perform a task). As another example, another drone in the asset group or another asset in the asset group may send information to the leader drone regarding the status of the execution of the plan, the status of the asset (e.g., the asset's environment (weather, etc.), etc.), etc. In some embodiments, one or more assets (e.g., drones) may listen to information communicated between the leader drone and follower drones or other assets in the asset group. For example, a dormant leader may obtain information communicated between the assets group to enable a relatively seamless takeover of the leader responsibilities / roles in response to a determination that the leader drone has failed or in response to a determination that the dormant drone will become the leader of a partition of the asset group. The leader drone may also communicate with a control center, such as a server, that manages / coordinates the execution of the work. For example, the leader drone may provide feedback information regarding the work (e.g., the real-time status of the work, such as an indication of whether one or more tasks are completed, etc.). As another example, the leader drone may send a request for additional assets or an indication that some of the assets are to be relieved of work, such as when some of the assets have completed a corresponding element of one or more tasks and the functionality associated with some of the assets is no longer needed to perform one or more tasks.

[0063] According to various embodiments, the asset group may semi-autonomously respond to the failure of one or more assets in the asset group. For example, in response to a control center (e.g., a server) providing an indication that the asset group will perform a task or one or more tasks associated with the task, the asset group may break down the task or task to determine a plan to be implemented in connection with the execution of the task or task associated therewith. During the execution of the plan, an asset may fail due to a loss of power, a loss of communication with the asset group (leader drone), a crash, or an interaction with a third party. In response to a failure of an asset in the asset group, the remaining assets in the asset group may continue to execute the task or task associated therewith without human intervention. The remaining assets may determine that the asset has failed and update the plan for executing the task or task, for example, to reallocate elements or functions assigned to the failed asset. In response to a determination that the leader drone has failed, another drone in the asset group may be promoted to the role of leader drone among the remaining assets in the asset group. The drone promoted to leader drone may be based at least in part on a predefined ranking / priority, or the like. In some embodiments, a dormant leader is promoted to the role of leader drone upon failure of the current leader drone. The dormant leader may correspond to the highest ranking / priority drone among the assets (e.g., next to the leader drone), as indicated by a predefined ranking / priority, etc. The dormant leader may synchronize with the leader drone in real time or at predefined intervals (e.g., short intervals to ensure that the dormant leader can seamlessly assume the role of leader drone, if necessary).

[0064] In various embodiments, the assets determined to perform the work (or one or more tasks associated with the work) may be updated during the performance of the work. In some examples, the assets may be updated based on the features or functions of the tasks or elements of the tasks that have not yet been performed and / or the features or functions of the assets in the assets. For example, if an asset in the assets does not have the features / features of the remaining tasks to be performed, the asset may be released. As another example, if there are no assets in the assets that have features / features that match the features / features of the remaining tasks to be performed, or if there is an insufficient number of such assets in the assets, the assets may be updated to include one or more assets with features / features that match the features / features of the remaining tasks to be performed. Updates to the assets may be determined by a control center (e.g., a server) that manages or coordinates the work, and the updates to the assets may be communicated to at least a leader drone in the assets.

[0065] According to various embodiments, one or more user interfaces may be provided in connection with one or more of configuring the task, communicating (e.g., real-time updates) the status of the task, and / or updating the task (e.g., based on user input, etc.). In conventional systems for defining a task, a user inputs the parameters of the task, the specific assets to be used, and the plan to be implemented with a high degree of specificity. In contrast, various embodiments include conditional display of various user interfaces that allow a user to configure the task, where a high-level plan or definition of the task is provided to at least a leader drone, and the assets (e.g., the leader drone) determine a plan for executing the tasks or elements of the tasks decomposed from the high-level plan or definition of the task. Thus, various embodiments provide a user interface wizard for inputting one or more characteristics or requirements of the task. The server may cause a client terminal to display a user interface to a user. The terminal may be caused to display a first user interface, the first user interface comprising one or more selectable elements associated with the characteristics of one or more tasks to be performed, and in response to an input to the user interface, the terminal may be caused to display a second interface associated with the assets performing the task. The second user interface may be configured based on the input to the first user interface. In response to the input to the second user interface, an operation may be determined. The server may cause the terminal to sequentially configure and provide user interfaces based at least in part on user input to previous user interfaces, and the sequence of user interfaces may be used to collectively determine the operation to be performed. After the operation is determined and the assets are commanded to perform the operation, the server may cause the terminal to provide a user interface that includes the current status of the operation (e.g., a real-time feed or real-time updates to various parts of the operation), etc.In some embodiments, a user may enter one or more inputs into the user interface while a task is being performed in connection with pausing the task, updating the task (e.g., requesting faster completion of the task, such as by increasing the number of assets in the group of assets, requesting more fire capabilities, requesting faster coverage of a geographic area, etc.), canceling the task, etc.

[0066] In some embodiments, the system determines a discrete representation of a geographic location. The geographic location may correspond to a location where an operation is performed or a task (or an element of a task) of the operation is performed. The discrete representation of the geographic location may include multiple discrete elements, each corresponding to a volume at a geographic location. The system may use the discrete representation of the geographic location in connection with determining and / or communicating parameters associated with the operation (e.g., parameters associated with a task or an element of a task), determining a plan for performing the task, and determining flight paths (e.g., trajectories) for assets (e.g., drones) of the asset group determined to perform the operation (e.g., set of one or more tasks). According to various embodiments, multiple assets in the asset group are configured to determine the discrete representation. For example, an asset performing a planning service may be configured to determine a discrete representation of a location of a task or element assigned (e.g., allocated) to a location or asset corresponding to the operation. As an example, the discrete representation is determined locally at an asset in the asset group performing one or more tasks. In some embodiments, multiple assets determine their own respective versions of the discrete representation of the location based at least in part on the information communicated regarding one or more tasks.As an example, the leader drone and / or follower drone may each determine a discrete representation of a geographic location (e.g., a local version of the discrete representation) based at least in part on one or more of the following: (i) information about the one or more tasks, such as one or more parameters or characteristics associated with the one or more tasks received from a server (e.g., a server associated with the configuration of the related work) (e.g., keep-in / keep-out definitions for one or more tasks, target location, maximum altitude, etc.); (ii) information about the geographic location from a third-party service, such as a service for map information (e.g., geographic information), a service for weather information, or a service for information obtained by another asset or sensor (e.g., camera, satellite, etc.) at or near the geographic location; and (iii) information about the geographic location or one or more tasks from another asset in the group of assets assigned to the work (e.g., a status of execution of one or more tasks, information associated with communication line of sight, a flight plan for the asset, an indication that a particular location / volume / area is occupied, etc.).

[0067] According to various embodiments, determining the discrete representation of the geographic location includes determining (e.g., creating, generating, populating, etc.) a 3-D representation of the geographic location, the 3-D representation including a plurality of discrete elements each corresponding to a voxel. In some embodiments, the 3-D representation of the geographic location may be determined at least in part based on transforming the geographic location from a real-world representation into a plurality of discrete elements based at least in part on the curvature of the Earth, the downward direction in the grid of the 3-D representation corresponds to the direction of gravity. The dimensions of the 3-D representation may be configured to include a substantially larger number of discrete elements (e.g., boxes) in the x-axis and y-axis compared to the number of discrete elements in the z-axis. The z-axis may be an axis parallel to gravity. As an example, when an asset moves down the 3-D representation in the z-direction, the asset moves down towards the ground, and conversely, when the asset moves up the 3-D representation in the z-direction, the asset moves towards the sky. The length or dimension of the discrete representation in the z-axis may be configured based at least in part on a maximum elevation configuration set in association with the configuration of the task. For example, the system may have a default maximum elevation associated with the task (or associated with different types of tasks). As another example, a user may input a maximum elevation during task configuration.

[0068] According to various embodiments, a system (e.g., a server, a leader drone, a follower drone, etc.) generates a model of a geographic location associated with one or more tasks. Generating the model of the geographic location may include determining a discrete representation of the geographic location and annotating the discrete representation to create an annotated representation. For example, the annotated representation may correspond to the model of the geographic location. In some embodiments, the system updates the annotated representation based at least in part on received information regarding one or more tasks and / or geographic locations. For example, the system may repeatedly update the annotated representation periodically, such as when updated or new information regarding one or more tasks and / or geographic locations is received. The annotated representation may be updated until one or more tasks are completed (e.g., until a determination that the work is completed, stopped, paused, etc.). In some embodiments, annotating the discrete representation of the geographic location includes associating metadata with one or more discrete elements of the discrete representation, respectively. The metadata may include values ​​or suggestions associated with one or more tasks, geographic locations, etc.

[0069] Various embodiments for managing, coordinating, and / or executing a task increase efficiency with respect to utilization of resources (e.g., assets) assigned to execute the task and improve fault tolerance of the execution of the task by enabling dynamic grouping of assets and / or dynamic updating of plans for executing tasks / elements associated with the task. Various embodiments increase efficiency and effectiveness in defining a task, matching assets to execute the task, and determining how the task is to be executed. In related art methods, a human operator selects assets to utilize in executing the task, and a centralized service (e.g., running on a server) defines the task in more detail. The centralized service provides a detailed task plan to execute the task. Various embodiments include a system of user interfaces configured to intuitively allow a user to provide a high-level description of a desired task. Various embodiments provide an efficient method for grouping assets to execute the task. The grouping of assets may be based at least in part on automatic negotiation of ownership availability and capabilities associated with the task and the task of the assets. The grouping of assets is more efficient and provides better organized task execution. In the related art, systems for defining operations generally require a human operator to select assets to be used in performing the operation, and the pool of assets available for selection by the human operator may be further limited because ownership availability is not readily discernible. In some embodiments, the system for performing operations is more fault-tolerant. For example, utilizing a leader drone (e.g., a semi-autonomous drone) in connection with determining one or more tasks and to coordinate / manage the execution of the tasks by one or more assets (e.g., follower drones) in the pool of assets provides a more efficient way of dynamically updating plans for executing tasks or task elements associated with the operation.As another example, dynamic grouping of assets to perform tasks is more efficient because assets with capabilities that are no longer needed to perform the remaining set of tasks are freed from the asset pool (and made available for reassignment to another task). Dynamic grouping of assets is also more fault tolerant because additional assets can be assigned to the asset pool in response to asset failures or changes in the task's conditions. Dynamic management / coordination of task execution (e.g., by a leader drone) allows tasks to be quickly reassigned to other assets in the asset pool in response to changes in asset capabilities and / or changes in the task's conditions (e.g., environment), etc.

[0070] FIG. 1 illustrates a system for performing operations according to various embodiments of the present application. In the example illustrated in FIG. 1, the system 100 may include a server 105 and one or more assets 120, 125, and 130. The system 100 may further include a network 110, a network 115, and / or a client terminal 135. The server 105 and the assets 120, 125, and / or 130 may communicate with each other, such as via one or more networks (e.g., the network 110, the network 115, and / or any other suitable network), which may include a wired network and / or a wireless network (e.g., a cellular network, a wireless local area network (WLAN0), etc.). According to various embodiments, the server 105 may correspond to a single server or multiple servers. Similarly, the system 100 may include various other assets.

[0071] In some embodiments, the server 105 may configure the task, such as determining a group of assets to perform the task. The group of assets may include one or more of the assets 120, 125, and / or 130. The server 105 may communicate an indication that an asset will perform the task and / or an indication of the group of assets (e.g., a list of assets in the group, such as a list of asset identifiers) to at least one asset in the group of assets. In some embodiments, the indication that an asset will perform the task may include a high-level definition or description of the task, such as a high-level task to be performed (e.g., monitor an area or target, deliver a payload, etc.). At least one asset in the group of assets (e.g., a leader drone) may resolve the high-level definition or description of the task and autonomously determine a plan for at least some of the assets to perform / complete the task. The high-level definition or description of the task may further include several parameters of the task, such as a location, an indication of restricted areas where the assets should not enter / operate, an indication of areas / areas where the assets should operate, etc. After receiving the high-level definition or description of the task, at least one asset (e.g., a leader asset) may automatically determine a plan for at least some of the assets to perform / complete the task, and the at least one asset may cause the assets to execute the plan and communicate information about the task (e.g., the current state of the task, etc.) to the server 105 via the network 110 and / or the network 115. For example, the leader drone may break down the high-level definition or description of the task and determine a plan for one or more tasks corresponding to the task. In some embodiments, a discrete representation of a geographic location associated with the one or more tasks is generated. As an example, the discrete representation of the geographic location is generated at least in part based on the geographic location and / or information about the one or more tasks. The information is received from the server 105 and / or another asset in the asset fleet (e.g., a follower asset, a leader drone, etc.). In some embodiments, a plan for the one or more tasks is determined at least in part based on the discrete representation.For example, the discrete representation is annotated to create an annotated representation. As an example, annotating the discrete representation includes setting metadata or associating metadata with one or more discrete elements of the discrete representation. In some embodiments, an asset in the asset group (e.g., asset 120, asset 125, and / or asset 130) and / or server 105 uses the annotated representation to determine a plan, such as using information included in the metadata associated with the discrete elements (e.g., to determine a trajectory or flight path of the asset). In response to determining a plan for one or more tasks, the leader drone may transmit a plan for at least one task to another asset in the asset group (e.g., a follower asset). In response to receiving a plan for at least one task, the follower asset may execute the plan and may autonomously determine any further instructions to be executed locally to execute the at least one task. The assets performing the work may be dynamically updated (e.g., while the work is being performed) to add assets (e.g., when an asset with a particular functionality fails and is replaced with a new asset), to remove assets (e.g., for any task requiring a particular functionality, an asset is included in the assets to provide that functionality and the task is completed), and so on.

[0072] In various embodiments, the server 105 communicates with the client terminal 135 over a network (such as network 115). The server 105 may cause the client terminal 135 to display one or more user interfaces in connection with configuring a task, displaying the status of the task, and / or updating the task (e.g., pausing, canceling, or modifying the task). The client terminal 135 may receive one or more user inputs to the various user interfaces and communicate suggestions of the one or more user inputs to the server 105. The server 105 may use the one or more user inputs to determine the various user interfaces to be displayed on the client terminal 135, such as in connection with configuring the various user interfaces to create a wizard through which a user defines or inputs the characteristics of the task. During execution of the task, the server 105 may use the user inputs to change the status or parameters of the task, and in response to such changes, the server 105 may communicate the changes to at least the leader asset / drone (e.g., to have the leader drone implement the changes). For example, in response to changing the status or parameters of the task, the server 105 may update a high-level definition or description of the task, and the server 105 may provide corresponding high-level instructions or updates to a leader drone in the assets performing the task. In some embodiments, the server 105 receives information (e.g., feedback information) about the task from the assets (e.g., from a leader drone). The information about the task may indicate a status of the task, real-time images or videos of the task, and the like. In response to receiving the information about the task, the server 105 may provide a status update, real-time task status, to the client terminal 135. In some embodiments, the server 105 may determine one or more recommendations or options for updating the task, and provide at least one of the recommendations or options to the user via a user interface displayed on the client terminal 135.As an example, the one or more recommendations or options may include a suggestion of the option to increase the number of assets assigned to the asset group to speed up the execution of the work (e.g., may indicate that adding a certain number of assets may reduce the time to completion by a calculated estimated time).

[0073] FIG. 2 is a block diagram illustrating a device for configuring or controlling a task according to various embodiments of the present application. In the example illustrated in FIG. 2, the device 200 may include a communication interface 202 and / or one or more processors 205. The device 200 may correspond to the server 105 of FIG. 1, the control center 460 of FIG. 4A-4C, and / or the control center of FIG. 5A-5C. The device 200 may perform the process 700 of FIG. 7A, the process 720 of FIG. 7B, the process 1000 of FIG. 10A, and / or the process 1020 of FIG. 10B, and the process 1100 of FIG. 11. According to various embodiments, the one or more processors 205 may include or perform one or more of the communication module 210, the asset ownership module 220, the function-to-asset mapping 230, the task definition module 240, the grouping module 250, the pre-planning module 260, the feedback information module 270, and the user interface module 280.

[0074] Device 200 may implement one or more modules in connection with determining a task, such as one or more characteristics of the task, determining a group of assets to perform the task, and communicating a suggestion of the task (e.g., a high-level description of the task) to one or more assets (e.g., a leader drone) in the group of assets. During the execution of the task, device 200 may implement one or more modules in connection with communicating information with at least a leader drone to receive information about a current status of the task or to send instructions for changes to the task, and / or communicating information with a client terminal to provide updates about the status of the task or to receive changes to the task.

[0075] The device 200 may use the communications module 210 to communicate with assets, a client terminal, another server or terminal, etc. For example, the communications module may provide to the communications interface 202 to be communicated. As another example, the communications interface 202 may provide information received by the device 200 to the communications module 210.

[0076] According to various embodiments, the determination of the assets to perform the task is based at least in part on the ownership of one or more assets in the assets. The device 200 may determine to include in the assets only those assets for which the device 200 has ownership (e.g., the organization to which the device 200 belongs owns or manages the asset) or for which the device 200 may obtain at least temporary ownership of the asset. Ownership of an asset may include the ability to control the asset, such as providing instructions to the asset to perform a particular function (e.g., a task, an element of a task, etc.). As an example, ownership of an asset may include a lease agreement in which the owner has the ability to control the asset. The lease agreement may be for a fixed / predetermined period or may be associated with the completion of a particular task. In some embodiments, the transfer of ownership (e.g., associated with the ownership negotiation to determine the assets) is a permanent transfer of ownership or is for an otherwise indefinite period or until the device 200 or its organization relinquishes ownership of the asset. In some examples, the asset group is configured to have a leader drone have a longer ownership period (e.g., lease length, permanent ownership) than the ownership periods of other assets (e.g., follower drones) in the asset group. In some examples, if it is determined that the task is relatively likely that the follower drone may lose communication while performing the task (e.g., based on the topography of the area in which the task is performed, etc.), device 200 may configure the asset group to have a longer ownership period for the follower drone. The longer ownership period may ensure that ownership of the follower drone continues throughout periods when the follower drone is not in communication with the leader drone and / or device 200, etc. In some embodiments, the ownership of the asset includes a failover entity to which the ownership of the asset is transferred in the event of a failure of the current entity having the ownership.Information regarding asset ownership may be stored in one or more of: (i) locally on the asset; (ii) a centralized ownership service, such as a service hosted by one or more servers providing services across multiple organizations; (iii) the device 200; and (iv) the owner of the asset. Information regarding asset ownership may include one or more of an indication of the current owner, an indication of the length of the current ownership, availability of ownership transfer (e.g., the current owner's willingness to transfer ownership), such as an indication of the ability of another organization to acquire ownership, a current failover owner, an indication of a procedure for negotiating an ownership transfer, etc. In some embodiments, asset ownership may include automatic takeover ownership, whereby a device or organization requesting ownership (e.g., with proper authorization) may automatically take over ownership of the asset, such as upon a request for ownership.

[0077] In some embodiments, the device 200 may utilize the asset ownership module 220 in connection with determining ownership of a particular asset and / or to negotiate acquisition of ownership of an asset. The asset ownership module 220 may obtain information regarding the ownership of an asset, and the device 200 may use such information in connection with determining a group of assets. For example, the device 200 may use the information regarding the ownership of an asset to determine whether a particular asset is available for allocation / assignment to a group of assets to perform work. The asset ownership module 220 may negotiate with an entity a transfer of ownership of an asset for use associated with the work. For example, the asset ownership module 220 may request a lease for a predetermined period of time. The predetermined period of time may be the expected length of the work, the expected length of time that the asset is required to perform a particular task or element of a task (e.g., the period of time that the functionality of the asset is required to perform the work), etc. The asset ownership module 220 may request information regarding ownership from a particular asset, or the particular asset may communicate information regarding the ownership of such asset. In some embodiments, device 200 may transmit a request / command to take ownership of an asset in response to a determination of a group of assets used in connection with performing a task. For example, device 200 may transmit a request to take ownership of an asset in connection with (e.g., as part of, or along with, etc.) providing an asset with an indication of the group of assets, such as an instruction indication indicating that the asset is included in the group of assets. As a further example, device 200 may transmit a request to take ownership of one or more assets in connection with providing the indication of the group of assets to a leader, who can then communicate with one or more assets in the group of assets in connection with taking applicable ownership of the one or more assets (e.g., to perform at least one task in the task).

[0078] According to various embodiments, the set of assets used in connection with performing the task is determined and / or updated based at least in part on the capabilities of the corresponding assets included in the set of assets. In connection with configuring the task, the device 200 may determine a set of capabilities associated with performing the task (e.g., capabilities required to perform various functions included in performing the task). The device 200 may use the set of capabilities associated with performing the task in connection with determining the set of assets. For example, the device 200 may match assets in a superset of assets having capabilities matching the set of capabilities associated with performing the task. The device 200 may query a mapping of capabilities to assets (or asset identifiers) in connection with determining a superset of assets having capabilities matching at least one function in the set of capabilities associated with performing the task. In response to determining that the functionality of the task does not have a matching asset (e.g., an asset with a corresponding functionality) in the superset of available assets, the device 200 may determine whether the functionality can be collectively achieved using two or more assets or reconfigure the task. For example, device 200 may provide an indication to the client terminal / user that the work feature does not have a matching asset and / or prompt the user with instructions to reconfigure the work (e.g., based on further user input) or cancel the work.

[0079] In some embodiments, device 200 may use asset-capability mapping 230 in connection with determining the capabilities of assets. Asset-capability mapping 230 may store such capability mapping locally or may communicate with a remote service that stores the mapping. Asset-capability mapping 230 may query a mapping with a particular identifier of an asset to look up capabilities for such assets. Similarly, asset-capability mapping 230 may query a particular capability mapping to look up assets having such capabilities. In some examples, in response to receiving one or more parameters of an operation, device 200 may determine one or more capabilities associated with the operation and use asset-capability mapping 230 to determine assets having matching capabilities. For example, device 200 may determine assets to perform an operation based at least in part on determining that the assets collectively have capabilities matching capabilities associated with performing the operation. Device 200 may determine assets such that at least one asset in the assets has at least one capability matching capabilities associated with performing the operation. The asset-to-capability mapping 230 may be stored locally on the device 200, or a module for obtaining a particular mapping of capabilities may be used to query a remote service that provides the asset-to-capability mapping 230. The asset-to-capability mapping may be updated in response to a determination that an asset's capabilities have changed. For example, an asset may advertise a new capability, a loss of capability (e.g., in response to detecting a peripheral failure), and / or an update to a capability. In some embodiments, the device 200 may update the asset-to-capability mapping.In some examples, a remote service (e.g., a service hosted by a server) updates the functions mapped to particular assets, and device 200 may periodically synchronize updates to the mapping of functions to assets, or device 200 may query the mapping of functions to assets as needed.

[0080] In some embodiments, an asset stores an indication of its capabilities. For example, an asset may store a definition of the capabilities locally, and the asset may update the definition of the capabilities based on the asset's current capabilities in response to failure of a module that provided the capabilities (e.g., a sensor, camera, or peripheral such as an added gimbal), addition of a new module (e.g., addition of a new peripheral, new software loaded on the asset, updates to existing software, etc.), etc. The definition of the capabilities may be a list of the asset's capabilities, or an indication of the capabilities according to a predefined format / protocol. In some examples, an asset may advertise its capabilities by transmitting a definition of the capabilities over one or more networks, to a leader drone during execution of a plan / task, etc., or the asset may provide an indication of its capabilities in response to a query from a server (e.g., a server configuring the task), a centralized capabilities server, a leader drone, etc. In some examples, device 200 (e.g., capability-to-asset mapping 230), or a remote service managing the capability-to-asset mapping, may ping one or more assets for updates regarding the current capabilities of the one or more assets or may otherwise provide pings for updates on the health of the assets, and the corresponding capability-to-asset mapping may be updated (e.g., to reflect the current set of capabilities and / or capability state for the one or more assets).

[0081] According to various embodiments, device 200 may define one or more tasks to be performed by one or more assets. For example, the definition of the task may include at least one of setting one or more objectives associated with the task, setting a location where the task is to be performed, setting a target for the task, setting one or more restrictions on the performance of the task (e.g., rules of engagement for engaging the target, restricted or permitted airspace, etc.), setting a date for the task, etc. The one or more tasks may be defined at least in part based on one or more user inputs, such as user selections entered into a user interface provided by user interface module 280. In some embodiments, device 200 defines one or more tasks using task definition module 240. For example, task definition module 240 may receive one or more user inputs by a user from communication module 210 and / or user interface module 280 into the user input, and task definition module 240 may determine one or more characteristics / parameters associated with the task in connection with defining the task. An example of a task definition may include a task to find or track a specific target (e.g., person XYZ, truck ABC, etc.) in Los Angeles from January 1, 2021 to January 3, 2021, with the assets not leaving 100 miles from Los Angeles or entering within 2 miles of a permanently prohibited zone or airport. In some embodiments, one or more user inputs associated with the task may include a suggestion of one or more types of assets to be used in connection with performing the task. Continuing with the previous example, the task definition may further include an indication that the task is to be performed using a drone and / or a fixed-wing aircraft.

[0082] In some embodiments, a task may be decomposed into a set of one or more associated functions or features. For example, a definition of a task (e.g., a task defined using a task definition module) may be analyzed, and one or more functions associated with the task may be determined based on the analysis. The set of one or more functions associated with the task may be determined based at least in part on querying a mapping of task parameters to functions. The parameters associated with the task may be determined based on the task definition. Examples of operation parameters include type operation (e.g., scanning a road network, finding, determining, tracking, loitering flight, scanning along a route, etc.), operation classification (e.g., attack, defense, surveillance, etc.), location, the path to be followed when the task is performed (e.g., path for tracking / scanning), type of asset (e.g., drone, fixed wing, helicopter, boat, etc.), target (e.g., personal, airplane, land vehicle, target of unknown type, etc.), range to perform the operation, location of the operation, rule set for performing the operation, indication of what can / cannot be performed (e.g., permitted actions), survivability during the operation (e.g., how likely is it that the asset will be shot down or otherwise terminated), communication requirements (e.g., whether the asset will engage in communications silent mode or not), and the like. , an indication of when a link may be lost, an indication of the type of communication to be utilized, or other parameters or constraints regarding communication during the task, etc.), target behavior (e.g., an indication of expected target movement, such as whether the target is stationary or moving along a road, movement patterns, etc.), potential threats (e.g., locations of sites that can destroy or disable drones, such as SAMs (surface-to-air missile launchers), RF jamming sites, etc.), unlikely target zones (e.g., areas that are not predicted to contain any targets of interest specified by the user, etc.), maximum altitude (e.g., above sea level) that the asset will travel to, keep-in areas (e.g., areas where the asset will remain), keep-out areas (e.g., areas where the asset is prevented / prohibited from entering), etc. In some examples, the task definition module 240 determines one or more functions associated with the task.In another embodiment, the grouping module 250 determines one or more functions associated with the task.

[0083] According to various embodiments, the device 200 determines a group of assets (e.g., a team of assets) to perform the task. The device 200 may automatically determine a group of assets to perform the task in response to a task being defined (e.g., based on one or more user inputs). The group of assets may be determined based at least in part on one or more of: (i) task parameters, (ii) one or more capabilities associated with the task, (iii) capabilities of one or more assets, and (iv) ownership of one or more assets. Various other factors / variables may be used in connection with determining a group of assets to perform the task. In some embodiments, the device 200 determines a group of assets based on a best fit based at least in part on one or more factors / variables associated with the assets and / or the task. For example, the device 200 may determine the group of assets using a cost function. For example, the cost function may include weightings for various factors used in determining the group of assets. The device 200 may determine the group of assets based on minimizing an overall cost value determined by the cost function for a combination of assets in the group of assets. In some embodiments, the assets may be determined such that their corresponding total cost values ​​using the cost function are less than a cost value threshold. The cost value threshold may be configurable (e.g., by an administrator, a user, etc.) in conjunction with the definition of the job, etc. In some embodiments, the cost value threshold is determined to correspond to a predefined percentile of all possible sets of assets (e.g., such that the assets performing the job are in the top 10 percent of all possible sets of assets, or some other configurable percentile). One example of a cost function in the context of pairing assets to road assignments may include weighting a) distance from the road, and b) the probability that the target is on the road. Using the above cost function, if there are assets that are far from the road, those assets will not be selected because the distance component of the cost function would exclude those assets from being desirable assets.Similarly, if there are less likely roads, the less likely roads are scanned in the future and higher priority roads take precedence.

[0084] In some embodiments, determining the asset group may include determining a number of assets to include in the asset group. The number of assets to include in the asset group may be determined based at least in part on the task (e.g., functionality associated with the task) and the functionality of the assets. For example, determining the number of assets may be based on a cost value of the grouping of assets and / or a set of functionality provided by the grouping of assets with respect to functionality associated with the task. As discussed further below, the number of assets to be included in the asset group may be based at least in part on a decision to include redundancy in the asset group with respect to one or more functionality between the assets in the asset group and / or the degree to which redundancy is included between the assets in the asset group.

[0085] The device 200 may determine the asset groups using the grouping module 250. The grouping module 250 may determine the asset groups based at least in part on a definition of the task. For example, the asset groups may be determined at least in part on a set of one or more functions associated with the task and / or one or more task parameters. The grouping module 250 may determine a group of one or more assets having functionality matching the set of one or more functions associated with the task. As an example, the grouping module 250 may determine the asset groups such that for each functionality in the set of one or more functions associated with the task, the asset group includes at least one asset with the matching functionality.

[0086] In some embodiments, device 200 determines a pre-plan for the task in connection with determining a set of assets to perform the task. Device 200 may use pre-plan module 260 to determine the pre-plan for the task. A pre-plan for a task may be a high-level description or decomposition of the task (e.g., into a set of tasks or elements of tasks to be performed) to ascertain what will be done during the execution of the task. Generating a pre-plan in connection with determining a set of assets to be used during the execution of the task enables grouping module 250 to evaluate the functionality associated with the task (e.g., various functions required to perform the task) and / or the characteristics or features of assets required or desired to perform the task (e.g., a set of tasks or elements of tasks to be performed during the task). In some embodiments, in response to determining a pre-plan for the task, device 200 determines assets having one or more capabilities that match the functionality or requirements of the task.

[0087] The device 200 may determine to include redundancy in the asset group for at least one of one or more functions associated with the operation. For example, the device 200 may use the grouping module 250 to determine to include redundancy in the asset group based at least in part on the type of operation, the operation classification, the location of the operation, etc. (e.g., if the type / operation classification / location of the operation is associated with a corresponding likelihood of asset failure that exceeds a likelihood threshold or is otherwise associated with an increased risk of failure). If the operation is an offensive operation or is in a hostile environment, the likelihood of an asset failing may be increased. The degree of redundancy that the device 200 builds into the asset group may be based on the degree of likelihood of the asset failing. For example, multiple thresholds may be defined, each threshold including a corresponding likelihood of asset failure, and the multiple thresholds may be used in connection with determining the degree of redundancy to be built into the asset group (e.g., the degree of redundancy may be mapped to a particular threshold corresponding to the likelihood of asset failure, and the mapping may be queried to determine the degree of redundancy to be applied to the asset group). The degree of redundancy may indicate the number of assets included in an asset group that each have a particular function (e.g., a function that is mapped to an elevated risk of asset failure or a function of an asset that has been identified as having an elevated risk of asset failure).

[0088] The device 200 may communicate the asset suggestion in response to determining the assets to be used in connection with performing the task. For example, in response to the grouping module 250 determining the assets, the device 200 communicates the asset suggestion using the communication module 210. In some embodiments, the device 200 transmits to at least a leader (e.g., a leader drone) in the asset group. The leader drone may then provide a suggestion to other assets in the asset group that the other assets are in the asset group. For example, the leader drone may notify each asset that it is included in a team (e.g., assets) of assets for the task. As another example, the leader drone provides a suggestion of the asset in the asset to each asset in the asset group so that each member knows, for example, the entire group of assets in the team for performing the task. The suggestion provided to the leader drone or to one or more other assets in the asset group may be communicated together with or separately from the task or plan associated with performing the task.

[0089] According to various embodiments, a pre-plan for performing the task may be determined via a remote service and then provided to the assets (e.g., at least the leader drone). The remote service for determining the pre-plan may be performed by the device 200, such as using the pre-planning module 260 or via one or more third-party pre-planning modules with which the pre-planning module 260 or the device 200 can communicate. For example, the pre-planning module 260 may include an application programming interface (API) for interfacing with the device 200 in providing a pre-plan associated with the task. The pre-planning module 260 and / or the third-party pre-planning module may obtain a definition of the task and / or one or more parameters associated with the task. In some embodiments, determining a pre-plan for the task is an iterative process of determining a pre-plan, providing information about the pre-plan to a user (e.g., displaying the pre-plan on a client terminal) via the user interface module 280 or the like, receiving feedback information about the pre-plan (e.g., one or more inputs about the pre-plan or parameters of the associated task), and updating the pre-plan based on the feedback information. The updated pre-plan may then be provided to a user, and feedback information regarding the pre-plan may be received for further refinement. Providing the pre-plan and receiving feedback information may be performed iteratively, for example, until a user provides instructions that the operation be initiated. As an example, device 200 may determine a pre-plan indicating that the assets will track a target along a path or otherwise move along a particular path, and device 200 may cause the path to be displayed to a user at a client terminal. The user may provide one or more inputs that modify the path of the assets, and device 200 may update the pre-plan accordingly. As another example, device 200 may determine a pre-plan indicating how the assets will fly when scanning a number of areas.For example, when performing or analyzing a lawn service, it may be useful for the task to see the back and forth mowing pattern on the polygon and potentially how to fly when scanning the area if the original polygon shape does not lead to a less than optimal scan. For example, if a user performed a vertical mowing but constructed a long horizontal polygon to scan / analyze the mowing, performing the scan according to the original long horizontal polygon may lead to significant overshooting and back and forth of the asset, so the user or drone may adapt to a horizontal scan pattern in this case.

[0090] In some embodiments, the device 200 may transmit a pre-plan for the assets to follow when performing the task, and the leader asset may determine a plan for the assets to perform the task based on the pre-plan. As an example, the device may determine and transmit a pre-plan when a task is defined to ensure that the assets performing the task follow the plan framework. The pre-plan may be a high-level definition or description of what is to be accomplished by the assets or a portion thereof, and the leader drone may determine a lower-level plan that further specifies how the assets perform the task (or its tasks). The leader drone may then provide a plan for performing the task to another asset (e.g., a follower drone) in the asset group, which may then determine a more specific plan for executing the plan provided by the leader drone. As an example, in the context of a surveillance task, the device 200 (e.g., a server) may determine a pre-plan that indicates a path or boundary for the assets to travel to obtain surveillance information about a particular location, and the device 200 may provide the pre-plan to a leader asset (e.g., a leader drone) in the asset group. In response to receiving the pre-plan, the first asset may break down the perimeter or path within the pre-plan into smaller segments, assign such segments to various follower assets (e.g., follower drones), and correspondingly provide instructions to the various follower assets to travel to the segments assigned to the follower assets to capture surveillance information regarding the segments. Continuing with this example, the various follower assets may receive instructions to capture surveillance information regarding particular segments, and the various follower assets may determine the specific manner in which to travel to the segments and capture surveillance information (e.g., the follower drones may plan a route to avoid obstacles (such as trees) along the route).

[0091] According to various embodiments, the device 200 may obtain feedback information regarding the definition of the task, the pre-planning for the task, and / or the status or result of the task execution. The device 200 may obtain the feedback information using the feedback information module 270, such as via the communication module 210. In some embodiments, the feedback information module 270 may obtain information regarding the task definition (such as one or more characteristics or parameters of the task input into a user interface displayed on the client terminal). In response to receiving information regarding the task definition, the feedback information module 270 may provide such information to the task definition module 240, etc. In some embodiments, the feedback information module 270 may obtain information regarding the pre-planning for the task (e.g., input to change or update the pre-planning), such as via input into a user interface provided by the client terminal. In response to receiving information regarding the pre-planning, the feedback information module 270 may provide information regarding the pre-planning to the pre-planning module 260, which may then update the pre-planning based on such information. In some embodiments, the feedback information module 270 may obtain information regarding the status or result of the task execution. For example, the feedback information module 270 may receive real-time operation information (e.g., indications of asset failure, live video feeds, etc.) from the assets (e.g., a leader drone) while a task is being performed. The feedback information module 270 may provide the real-time operation information to the user interface module 280 in connection with displaying the status of the task. As another example, the feedback information module 270 may receive an indication that a task is completed.In response to receiving an indication that the work is complete, the feedback information module 270 may provide a suggestion or corresponding instruction to one or more modules, such as (i) to the asset ownership module 220 to provide an indication that the work is complete, or (ii) to the asset ownership module 220 to relinquish or return ownership of one or more assets in the asset group.

[0092] According to various embodiments, the device 200 may cause one or more user interfaces to be displayed on one or more client terminals. The user interfaces may be displayed in connection with configuring or defining a task, providing and updating pre-planning, and / or providing the status of the task. The device 200 may use the user interface module 280 to configure the user interfaces displayed on the client terminals. The user interface module 280 may configure the user interfaces based at least in part on one or more templates. Additionally, the user interface module 280 may provide a wizard or workflow for generating and providing multiple user interfaces to a user for configuring a task based on one or more inputs to the multiple user interfaces. In some embodiments, user input to one user interface when configuring a task is used in connection with generating a next user interface used to configure the task. In this manner, the user interface module 280 may provide one or more user interfaces (or pages of user interfaces) to the client terminals to guide the user through the task configuration process. In contrast, a task is currently defined within a single composite interface that requires all task features or parameters to be entered into a single page. The use of multiple user interfaces that are logically connected through user input to a previous user interface simplifies the organization of tasks, thereby lowering the skill level of human operators and reducing the risk of errors or inaccurate inputs being made.

[0093] FIG. 3 is a block diagram illustrating a device for performing at least a portion of the operations according to various embodiments of the present application. In the example illustrated in FIG. 3, the device 300 may include a communication interface 302 and / or one or more processors 304. The device 300 may correspond to the asset 120, the asset 125, and / or the asset 130 of FIG. 1, the asset 405, the asset 410, the asset 420, the asset 430, the asset 440 (e.g., a satellite), and / or the asset 450 (e.g., a tower) of FIG. 4A-4C, and / or the asset 505, the asset 510, the asset 515, the asset 520, the asset 525, the asset 530, the asset 535, the asset 540 (e.g., a satellite), and / or the asset 545 (e.g., a tower) of FIG. 5A-5C. Device 200 may perform process 800 of FIG. 8A, process 830 of FIG. 8B, process 830 of FIG. 8C, process 900 of FIG. 9A, process 930 of FIG. 9B, process 950 of FIG. 9C, process 951-2 of FIG. 9C, process 950 of FIG. 9D, process 1000 of FIG. 10A, process 1020 of FIG. 10B, process 1200 of FIG. 12A, process 1215 of FIG. 12B, process 1215 of FIG. 12C, process 1300 of FIG. 13, process 1400 of FIG. 14, process 1500 of FIG. 15, process 1800 of FIG. 18, process 1900 of FIG. 19A, and / or process 1940 of FIG. 19B.

[0094] According to various embodiments, the one or more processors 305 may comprise or execute one or more of a communications module 310, a planner services module 320, a mapping of capabilities to assets 330, a dynamic grouping module 340, an ownership module 350, a partitioning module 350, a feedback information module 370, and an asset failure module 380. The device 300 may implement one or more modules associated with performing a task, such as determining a plan for the assets to perform a task (or elements of a task) of the task, dynamically modifying the assets, communicating with one or more other assets in the assets, updating a plan for performing a task based on information received from another asset in the assets, etc., communicating with one or more other assets in the assets assigned to perform the task, etc.

[0095] According to various embodiments, the device 300 corresponds to a semi-autonomous drone. The device 300 may receive a high-level plan or instructions to perform a task (or a task or element thereof), and the device 300 may autonomously determine how to perform the task (or a task or element thereof). In some embodiments, the device 300 corresponds to a leader drone in a group of assets assigned to perform a task. In some embodiments, the device 300 corresponds to a follower drone in a group of assets assigned to perform a task. The device 300 may include the same set of modules regardless of whether the device is a leader drone or a follower drone. For example, multiple assets in the group of assets may include the same modules as the leader drone, such as to provide redundancy in case of a failure of the leader drone, or when the group of assets is partitioned to provide different aspects of the task (or task of the task) and multiple partitions include a leader drone for the partition. As another example, the group of assets may include a dormant leader drone that synchronizes with or backs up the leader drone to provide redundancy in case of a failure of the leader drone. In some examples, certain types of assets may have the same set of modules or leader capabilities as a leader drone, while other types of assets may have some of the modules or leader capabilities, for example, a semi-autonomous drone in a fleet may have the same set of modules or capabilities as a leader drone, while an autonomous tower, sensor, or satellite in the fleet may have a different set of assets and may not have sufficient capabilities to be a leader in the fleet.

[0096] The device 300 may use the communication module 310 to communicate with assets, another server or terminal (e.g., the server 105 of the system 100 of FIG. 1), etc. For example, the communication module 310 may provide the communication interface 302 to be communicated. As another example, the communication interface 302 may provide information received by the device 300 to the communication module 310. In response to receiving information from the communication interface 302, the communication module 310 may provide such information to a corresponding module. For example, if the device 300 receives a suggestion to perform a task, the communication module 310 may provide such a suggestion to the planner service module 320. As another example, if feedback information is received from one or more follower drones during the execution of the task, the communication module 310 may provide the feedback information to the feedback information module 370 and / or the planner service module 320, etc.

[0097] According to various embodiments, planning of how a job, task, or element is to be performed is done through various entities in the system. A server configuring a job may determine job parameters and provide such parameters to a leader drone and / or determine a high-level pre-plan for the assets to perform the job and provide such pre-plan to the leader drone. The leader drone may then determine a lower level plan than the parameters or pre-plan for the job received from the server. The plan determined by the leader drone may autonomously fill in some gaps in the information received from the server related to performing the job. The leader drone may provide a plan to one or more follower drones for performing the task or elements of the task. In response to receiving a plan from the leader drone, one or more follower drones may each determine a way to execute the plan, such as autonomously filling in gaps in the plan received from the leader. The foregoing framework enables a system (such as system 100 of FIG. 1) to be a modular system that supports hierarchical planning, and such corresponding planning by the leader drone and / or follower drones may be autonomous. In some embodiments, assets in an asset group may advertise the level of planning or tasking they support. The leader drone may determine plans corresponding to the assets in the asset group based on the level of planning or tasking supported by the follower assets. For example, a follower drone with a more robust planning service may not require as detailed a plan as an asset with a rudimentary planning service (or no planning service at all). In some embodiments, the leader drone may store a mapping of planning or tasking capabilities to assets and use such mapping in connection with determining respective plans for the follower assets.According to various embodiments, at least one follower drone has the capability to perform a robust planning service that is the same or substantially the same as the planning service performed by the leader drone.

[0098] According to various embodiments, in response to receiving an indication that device 300 is included in a fleet of assets (e.g., from a server (e.g., a server providing an operation control service)), device 300 autonomously determines (e.g., generates) a plan for performing one or more tasks associated with the operation, and device 300 communicates at least a portion of the plan to one or more other assets. Device 300 may be a leader asset (e.g., a leader drone) in the fleet of assets for performing one or more tasks associated with the operation. Device 300 may receive information regarding the operation to be performed by the fleet of assets. The information regarding the operation may include a high-level definition or description of the operation. As an example, the high-level definition or description of the operation may include a high-level task to be performed (e.g., monitor an area or target, deliver a payload, etc.). Examples of information included in a high-level definition or description of an operation include one or more of: (i) asset suggestions; (ii) some parameters of the operation, such as location, suggested restricted areas where the assets should not enter / operate, suggested areas / regions where the assets should operate; (iii) functionality associated with the operation; and / or (iv) a pre-plan for performing the operation (or its tasks). In some embodiments, information about the operation is communicated only to a leader asset (or leader asset and a dormant leader) in the asset fleet. In response to receiving information about the operation, the leader asset may communicate the asset suggestions to other assets in the asset fleet (e.g., to follower assets).

[0099] In some embodiments, information regarding the operation is sent only to the leader asset (e.g., the leader asset) of the asset group. Information regarding the operation may also be sent to a dormant leader (e.g., which may be referred to herein as a second leader) of the asset group, which may be a backup of the leader asset that synchronizes operation information (e.g., current operation status, plans for the execution of various tasks, etc.). In this manner, the dormant leader may have a full backup of the operation information stored / managed in the leader asset. In some embodiments, the dormant leader is selected based on negotiations between multiple assets in the asset group (e.g., negotiations between semi-autonomous drones in the asset group). An asset may be selected as a dormant leader for an operation based at least in part on a determination of one or more capabilities that the asset possesses and / or the availability of ownership (e.g., lease or ownership is available for the duration of the operation). For example, an asset may be selected as a dormant leader based at least in part on a determination that the dormant leader has one or more predefined capabilities (e.g., a planning service module), etc. As another example, an asset may be selected as a dormant leader based at least in part on a determination that the asset has a particular set of capabilities that match the set of capabilities of the leader asset (e.g., the dormant leader has all of the capabilities of the leader asset, the dormant leader has the planning or work management capabilities of the leader asset, etc.) As another example, an asset may be selected as a dormant leader based at least in part on a determination that the asset is the best fit as a dormant leader among other assets in the asset pool.

[0100] In some embodiments, multiple assets in the asset group (e.g., each of the assets, each of the drones in the asset group, etc.) may receive the asset group suggestion. The asset group suggestion may include information identifying the assets in the asset group. The suggestion may further include suggestions of various functions of the assets in the asset group. In some examples, each asset in the asset group (e.g., each member of a team performing an operation) may have knowledge of all other assets in the asset group. A follower asset may receive the asset group suggestion from a leader asset for the asset group, or a follower asset may receive the asset group suggestion from a server (e.g., a server providing operation control services). Based on predefined leader rankings of the assets, negotiations among at least some of the assets, etc., one asset may be determined as a leader asset (e.g., leader drone) for the asset group. In some examples, the predefined leader rankings are input by a user, such as during configuration of the operation. In other examples, the leader rankings are negotiated (or the predefined leader rankings are updated), such as by at least some of the assets.

[0101] According to various embodiments, in response to receiving receipt of information regarding the operation, the leader asset (e.g., a leader drone) autonomously determines a plan for executing a task (or an element of a task of the operation) associated with the operation. The leader asset may communicate at least a portion of the plan for executing the task or element of the task to at least one follower asset (e.g., a follower asset determined to execute the task or element of the task). The leader asset may determine a plan for executing a task or element of a task on a per-task, per-element, or per-follower asset (e.g., per-follower drone) basis. In some embodiments, the leader asset determines an individual plan for the follower drone that the follower drone utilizes in connection with executing the corresponding task or element of a task. The leader asset may determine an individualized plan for each follower drone in the group of assets, each follower drone that has a currently assigned task or element of a task, etc.

[0102] In some embodiments, the device 300 uses the planner service module 320 in connection with determining a plan for performing a task or an element of a task. If the device 300 is a leader drone, the planner service module 320 may determine a plan for the assets to perform an operation (e.g., one or more tasks or elements of a task associated with the operation). For example, the planner service module 320 may determine a plan for one or more follower drones to implement in connection with performing the task or element of a task. The plan may be determined based on one or more parameters of the operation and the assets. In some embodiments, the plan may be determined based at least in part on capabilities associated with the corresponding task or element of a task and capabilities of the follower drones. For example, the planner service may determine one or more follower drones that have capabilities matching capabilities associated with the corresponding task or element of a task and determine at least one of such follower drones to be assigned to perform at least a portion of the task or element. In some embodiments, the plan determined by the device 300 is a lower-level plan than the pre-plan (if any) received from the server (e.g., along with instructions to perform the operation). For example, the device 300 may decide to perform a more specific task or element than is provided in the corresponding pre-plan. As an example, in the context of road search (e.g., a task to search a particular set of roads), a high-level task may search a polygon that includes a set of roads and a set of preferred target locations (where targets were believed to be located at a certain time). A leader drone may evaluate the set of roads included in the polygon, spread the probability of preferred targets across the roads, and then assign assets (e.g., follower drones) to the highest priority and closest roads (e.g., to perform a search of those roads). The low-level plan provided to the follower drones includes the roads along which the follower drones will scan and the order in which to scan those roads.

[0103] In some embodiments, the device 300 uses the planner service module 320 in connection with determining a method for executing the received plan. If the device 300 is a follower drone, in response to receiving a plan for executing a task or an element of a task, the device 300 may use the planner service module 320 to plan the execution of the plan. For example, the planner service module 320 may determine an even lower level plan for executing the plan received from the leader drone. The planner service module 320 may determine a particular method for controlling the device 300 to execute the assigned task or element of the task. As an example, if the device 300 receives a plan for performing monitoring of a defined area, the device 300 may obtain information about the area (such as the location of various obstacles or objects) and determine a path for moving the device 300 within the area while avoiding the obstacles or objects. As another example, even if the plan received from the leader drone includes a defined path along which the device 300 moves, the device 300 may perform dynamic monitoring of the area to perform collision avoidance during the execution of the plan. In some embodiments, the follower drone receives a detailed plan from the leader drone that effectively identifies different steps to be taken by the follower drone. The follower drone may execute the steps defined by the leader drone while dynamically monitoring or updating the plan to prevent asset failure or to keep the follower drone on course to execute the plan.

[0104] According to various embodiments, the device 300 employs the planner service module 320 in connection with determining a discrete representation of a geographic location associated with the operation or where at least one asset in the asset group performs one or more tasks. As one example, the discrete representation includes a number of discrete elements each corresponding to a volume in the geographic location. As another example, the planner service module 320 determines the discrete representation in connection with determining a plan for the device 300 or a plan for an asset in the asset group performing one or more tasks.

[0105] In some embodiments, after the discrete representation is determined (e.g., in response to determining the discrete representation of the geographic location), the device 300 annotates the discrete representation with the planner service module 320, such as to create an annotated representation. As an example, the planner service module 320 annotates the discrete representation based at least in part on received information regarding one or more tasks and / or geographic locations. The planner service module 320 annotates the discrete representation based at least in part on current information regarding one or more tasks and / or geographic locations. The current information corresponds to information stored or accessible by the device 300 at the time the annotated representation is created. In some embodiments, the system updates the annotated representation based at least in part on received information regarding one or more tasks and / or geographic locations. For example, the system may repeatedly update the annotated representation periodically, such as when updated or new information regarding one or more tasks and / or geographic locations is received. The annotated representation may be updated until one or more tasks are completed (e.g., until a decision is made to complete, stop, pause work, etc.). In some embodiments, annotating the discrete representation of the geographic location includes associating metadata with one or more discrete elements of the discrete representation, respectively. The metadata may include values ​​or suggestions associated with one or more tasks, geographic locations, etc.

[0106] One example of annotating the discrete representation includes setting metadata associated with at least a portion of the plurality of discrete elements to set keep-in areas (e.g., areas where the device 300 or assets in the group of assets are kept) or keep-out areas (e.g., areas where the device 300 or assets in the group of assets are excluded / prevented from entering). For example, each discrete element in the group of discrete elements has a metadata field corresponding to a keep-in indicator and / or a keep-out indicator. As an example, the keep-in and / or keep-out areas are configured in conjunction with a configuration of operations (e.g., a user inputs the keep-in and / or keep-out areas, certain predefined keep-in and keep-out areas are set based on restricted airspace, etc., a third-party service provides input to set the keep-in and / or keep-out areas, etc.).

[0107] Another example of annotating the discrete representation includes setting metadata associated with at least a portion of the discrete elements to set a location of one or more targets of the operation. In the case of an operation to perform surveillance of a particular target, the configuration of the operation includes obtaining a current location of the target (predicted location, last known location, etc.). As an example, a user sets a current location of a particular target in association with configuring the operation. The planner service module 320 determines a particular discrete element that corresponds to a current location of the particular target and sets a metadata field associated with the particular discrete element to indicate that the target is located at / within the discrete element. As another example, in the case of an operation to perform surveillance of a particular target (such as a building or road), the planner service module 320 determines a discrete element that corresponds to (or includes) the particular target (such as a building or road) and sets a metadata field corresponding to the discrete element to indicate that the particular target is included in the discrete element.

[0108] Another example of annotating the discrete representation includes setting metadata associated with at least some of the discrete elements to define a communication line of sight between assets, between assets and a leader drone, between a leader drone and a ground control station, etc. As an example, setting the metadata to define a communication line of sight includes indicating for a particular discrete element whether a communication line of sight is obtainable when an asset is located at a geographic location corresponding to the particular discrete element.

[0109] Another example of annotating the discrete representation includes setting metadata associated with at least some of the discrete elements to indicate whether the discrete elements are occupied or unoccupied. As an example, a discrete element is considered occupied if an asset (e.g., an asset from the asset pool, another friendly asset not in the asset pool, etc.) is included in the discrete element. As another example, a discrete element is considered occupied if an asset cannot move into the discrete element, such as because there is insufficient space within the discrete element to support the asset without risk of collision (e.g., risk of collision above a threshold risk) with another asset or object (e.g., a building, tree, mountain, etc.). In some embodiments, the planner services module 320 sets metadata associated with the discrete elements to indicate whether the discrete elements are occupied or unoccupied in connection with registering or storing a flight plan. For example, the device 300 determines a flight plan for an asset in the asset group or receives information about the flight plan from an asset in the asset group, and the planner services module 320 determines discrete elements affected by the flight plan (e.g., a set of discrete elements that intersect with the flight plan) and sets metadata for such discrete elements to indicate that such discrete elements are occupied. Setting the discrete elements corresponding to the flight plan as occupied may ensure that another flight plan that intersects with the flight plan is not created or registered. For example, the device 300 and / or other assets to which the annotated representation or model (or information from which the annotated representation may be created) is published utilizes the annotated representation in connection with determining a new flight plan, and the annotated representation includes the discrete elements that are set as occupied for the existing flight plan (e.g., at least during the forecast period corresponding to the flight plan). In some embodiments, the planner services module 320 further sets a threshold number of adjacent discrete elements adjacent to the set of discrete elements corresponding to the flight plan. The adjacent discrete elements may act as a buffer to provide additional clearance between the existing flight plan and any subsequently created flight plans, and so on.The threshold of adjacent discrete elements may be configurable. As an example, for any particular discrete element corresponding to a flight plan, the planner services module 320 sets N discrete elements in all directions of the particular discrete element as occupancy, where N is a positive integer.

[0110] In some embodiments, the device 300 uses the planner service module 320 to update the annotated representation based at least in part on received information regarding one or more tasks and / or geographic locations. For example, the planner service module 320 repeatedly updates the annotated representation, e.g., periodically (e.g., at default intervals, configurable intervals, etc.), such as after updated or new information regarding one or more tasks and / or geographic locations is received. As an example, the planner service module 320 updates the annotated representation until one or more tasks are completed (e.g., until a decision is made to complete, stop, pause, etc. the work). In some embodiments, annotating the discrete representation of the geographic location includes associating metadata with one or more discrete elements of the discrete representation, respectively. The metadata may include values ​​or suggestions associated with one or more tasks, geographic locations, etc. The repeated or continuous updates of the annotated representation refine the model of the geographic location or work. As additional information is obtained regarding geographic location, such as via an asset (e.g., a follower drone) deployed to a particular location, the resolution of a certain amount or type of information included in the model is improved.

[0111] According to various embodiments, the planner service module 320 uses the annotated representation in connection with determining a plan for performing one or more tasks. For example, the planner service module 320 utilizes metadata associated with discrete elements in the discrete representation to determine a destination for one or more assets in the fleet of assets, determine a route for one or more assets, determine elements to be performed by a particular asset, etc. In some embodiments, the resolution of the geographic location / task model improves, such as by receiving feedback information (e.g., updated information about a task or an element of a task, or information about a portion of the geographic location of an asset) from an asset (e.g., a follower drone). As the resolution of the geographic location / task model improves, the plan for performing one or more tasks is updated (e.g., refined).

[0112] According to various embodiments, the device 300 uses the asset-capability mapping in connection with assigning tasks or elements of the work and / or to determine a plan for a particular follower asset to perform the task or element. If the device 300 is a leader drone, the device 300 may store the asset-capability mapping 330 locally on the device 300 and query the asset-capability mapping to assign tasks or elements or to determine a plan. At the start of the work (e.g., in response to receiving an instruction to perform the work), the leader drone may receive a current set of capabilities for various assets in the asset group. For example, the leader drone may cause the assets group to provide their current capabilities to the leader drone (e.g., send a request to the follower drone to register the corresponding capabilities with the leader drone). As another example, the leader drone may receive a set of capabilities for various assets in the asset group from a server. The leader drone may update the asset-capability mapping 330 during the execution of the work. For example, the leader drone may ping / request the follower drones to provide updates about their current set of capabilities, and the leader drone may update the mapping accordingly. As another example, the leader drone may determine updates to capabilities of a particular follower drone based on feedback information communicated by the follower drones to the leader drone during the execution of a task (e.g., execution of a plan by the follower drone). The device 300 may determine from the feedback information that a particular sensor or camera from the follower drone is not functioning properly or has failed (e.g., the feedback information may not have a clear signal associated with the normal operation of such sensor / camera).

[0113] According to various embodiments, device 300 may utilize a mapping of capabilities to assets in connection with the execution of a plan and / or to maintain a current state of the capabilities of the assets. If device 300 is a follower drone, device 300 may provide the current set of capabilities of the follower drone or the status of various capabilities in the set of capabilities associated with the follower drone (e.g., an indication that a particular capability is fully operational, partially operational, or inoperable, etc.) to the corresponding leader drone. Device 300 may provide such information to the leader drone at predetermined intervals, upon request from the leader drone, or along with feedback information communicated to the leader drone during execution of a plan.

[0114] According to various embodiments, the assets assigned to perform a task may be dynamically updated during the execution of the task. For example, the assets in the asset group may be modified / updated in response to: (i) a user request to reassign an asset in the asset group to another task, (ii) loss of ownership of an asset in the asset group, (iii) loss of ownership of an asset in the asset group (e.g., if a functionality required to complete a task is lost during the execution of a task, a new / additional asset with such functionality may be assigned to the asset group), (iv) user input that changes the rate at which the task is expected to be completed (e.g., additional assets are added to speed up completion or assets are removed from the asset group if the user decides to slow down completion), (v) a cost function associated with the execution of multiple tasks indicates that a more optimal completion of multiple tasks includes the utilization of additional assets or reallocation of assets from the asset group, (vi) a determination that an asset in the asset group has failed, (vii) a condition defined within the task is met (e.g., under a surveillance task, the condition may include detection of a target that another or different asset is assigned to intercept, etc.). The assets may be dynamically updated in response to various other conditions. The assets may be dynamically updated by the server, the leader drone, or both.

[0115] In some embodiments, the dynamic grouping module 340 may be used in connection with dynamically updating a group of assets. The device 300 may use the dynamic grouping module 340 to determine whether another asset is to be added to the group of assets and / or whether an asset is to be removed from the group of assets. In response to the dynamic grouping module 340 determining to update the group of assets, the dynamic grouping module may provide such suggestions to various other modules (e.g., to the planner services module 320 to update or create a new plan for the performance of a task or element using the updated group of assets, to the asset to function mapping 330 to update the function mapping for the updated group of assets, to the communications module 310 to notify or request the server of the update, or to the ownership module 350 to request / update the ownership of the asset to be added to or removed from the group of assets).

[0116] According to various embodiments, the leader drone may manage the ownership of assets in the asset fleet. In response to receiving an indication that the leader drone is among the assets performing work, the leader drone may obtain ownership authorization and / or ownership status (e.g., transfer of ownership of the corresponding assets to the organization for which the work is being performed, if applicable) corresponding to the assets in the asset fleet. The ownership transfer may correspond to the transfer of ownership of the assets for (i) a portion of the work, such as during the performance of a particular task or element, (ii) the duration of the work, or (iii) an indefinite period (e.g., permanent transfer of ownership until the transferring organization relinquishes ownership). The ownership authorization may include automatic handover (e.g., allowing ownership to be transferred to the transferring organization / leader drone upon request), negotiation of approval / denial of asset handover (e.g., conditional transfer pending an emerging need of the transferring organization, a defined period / lease period for which ownership is transferred, a defined task / element for which ownership is transferred, etc.). The ownership state may indicate whether an asset is available for transfer to the leader drone (or the organization for which the leader drone is performing the operation), whether the asset is unavailable for transfer, or a set of conditions under which the asset may or may not be transferred. In some embodiments, the leader drone may obtain the ownership permission and / or ownership state based on querying an ownership service (e.g., a third-party service or a centralized service where various organizations can register / transfer ownership of assets), querying a mapping of assets to ownership rights stored locally or remotely on a server managing the operation (e.g., a server that configured the operation), and / or querying various assets in the asset group (e.g., each asset may store its corresponding ownership permission and / or ownership state and / or a protocol under which the leader drone can request an ownership transfer). The leader drone may use the ownership module 350 to manage / update the ownership of assets in the asset group and / or negotiate or request transfer of assets to / from the asset group.For example, in response to an indication that an asset is being added to an asset group or that ownership of an asset within an asset is being requested or updated, the ownership module 350 may be called upon to request / arrange a transfer of ownership of the asset.

[0117] In the course of performing the work, the leader drone may determine that the work (or tasks / elements of the work) can be performed more effectively or efficiently by one or more portions of the assets, and the leader drone may divide the assets. The leader drone may determine how to divide the assets. For example, various partitions may be determined based on the functions of the tasks / elements performed by various partitions and / or functions of the assets in the assets. In some embodiments, the partitions are assigned a corresponding leader asset. For example, each partition has a corresponding leader drone. Thus, a follower drone may be promoted to the position of leader drone for a particular partition. The leader drone of the assets may be a primary leader, and various partition leaders may report feedback information (such as the status of assigned tasks / elements of the partition). The primary leader may then aggregate feedback information received from the partition leader and / or any assets that report directly to the primary leader (e.g., assets in the same partition as the primary leader) and provide the feedback information to the server and / or provide instructions to the partition leader, such as modifications to tasks or elements assigned to the partition. In response to splitting the assets, the primary leader may provide instructions to the partitions (e.g., partition leaders) to execute tasks or elements that the primary leader assigned to the partitions. In response to a partition receiving an indication to execute a task or element, the partition leader may determine a plan for the corresponding split assets. As an example, the partition leader may play a role for the partition such that the role is similar to that played by the primary leader for the assets (e.g., prior to the split). Similarly, a partition may have one or more follower assets that (i) receive plans from the partition leader, (ii) implement the plans received from the partition leader, and / or (iii) communicate feedback information to the partition leader.According to various embodiments, in response to a determination that a partition has completed the tasks / elements assigned to it, the divided assets may be (i) reintegrated into the assets (e.g., with the partition corresponding to the primary leader), (ii) reassigned to other partitions to assist in the completion of the tasks / elements assigned to such other partitions, or (iii) released from the assets (e.g., released to perform other work for another leader or organization). In some embodiments, when a partition is created, a set of constraints within which the partition will operate may be determined. The primary leader drone may provide the set of constraints to the corresponding partition leader.

[0118] According to various embodiments, the device 300 employs the partitioning module 360 ​​in connection with determining whether to partition the assets and how the assets are partitioned. The partitioning module 360 ​​may determine the assets to be included in various partitions. In response to determining that a partition has completed the tasks / work assigned to it, if an asset in the partition is to be assigned to another partition or reintegrated into the assets, the partitioning module 360 ​​may determine the partition in which such asset is to be included. In some embodiments, a partition is redeployed to perform another task / work after completing the tasks / work assigned to the partition (e.g., the primary leader or a planner service module on the primary leader decides to assign another task / work).

[0119] The device 300 may send / receive feedback information regarding the execution of the task. If the device 300 is a leader drone, the device 300 may receive feedback information from one or more follower drones, such as information acquired by the sensors / cameras of the follower drones, the status of tasks / elements assigned to the follower drones, the status of tasks / elements assigned to the follower drones, the status of the drones, updates to the status or capabilities of the drones' functions, etc. Additionally or alternatively, the device 300 may send feedback information to a server (such as a server managing (or configuring) the task). For example, the leader drone may send the status of the task, a live feed of information acquired by one or more sensors in the assets, a request for future instructions regarding the task, etc. If the device 300 is a follower drone, the device 300 may send feedback information to the leader drone, such as information acquired by the sensors / cameras of the follower drones, the status of tasks / elements assigned to the follower drones, the status of the drones, updates to the status or capabilities of the drones' functions, etc. The device 300 may communicate the feedback information using the feedback information module 370. The feedback information module 370 may receive the feedback information, determine various information contained in the feedback information, and forward the information to a corresponding module. As an example, if the feedback information includes an indication that a sensor on the follower device has failed, the feedback information module 370 may determine that such information relates to asset functionality and provide such information to the functionality to asset mapping 330 so that the functionality to asset mapping in the asset group is updated.As another example, if a follower drone communicates sensor information (e.g., images, live feeds, weather conditions, etc.) to the leader drone, the feedback information module 370 may determine such sensor information in the feedback information and provide such information to the planner services module so that the leader drone can utilize such information in connection with determining the status of the operation and / or deciding whether to update a plan for the follower drone (or other follower drones).

[0120] According to various embodiments, the team of assets is dynamically managed throughout the execution of the task, such as from the time the task is configured to the time the task is completed. Assets may be added / removed from the pool of assets assigned to perform the task during the execution of the task. The decision to add / remove an asset may be performed at the server. In some examples, the decision to add / remove an asset may be performed by the leader drone of the pool of assets. Dynamic management of the team provides an extensible framework that can adapt to changes in the pool of assets, unexpected or conditional events that occur during the execution of the task, or changes in the configuration of the task (e.g., user-driven changes to the task). Examples of changes in the pool of assets may include asset failure, failure / loss of functionality of an asset, and partitioning of the pool of assets (e.g., an asset pool may have only one specific functionality required by multiple partitions, resulting in a requirement to add a specific functionality to an asset). Examples of unexpected or conditional events occurring during the execution of a task may include a new target appearing during monitoring of an area, weather or environmental changes that require or may enhance certain capabilities (e.g., a larger drone to withstand extreme weather such as heavy winds, rain, etc.), a task that extends into the night to enhance the cost of having night vision capabilities, etc. Examples of changes in the configuration of a task may include a user request to complete the task faster (e.g., speeding up execution by the asset fleet), a user request or other decision to prioritize another task, a user request to pause a task, etc. In some embodiments, in response to determining that a new asset is to be added to the asset fleet, the server sends an indication to the leader that the new asset has been included in the asset fleet. The leader drone may then responsively determine a plan for executing the task and / or manage ownership of the new asset (e.g., taking appropriate ownership of the new asset). In some embodiments, in response to a follower drone being removed from the asset fleet, an indication is sent to the follower drone indicating that the follower drone is no longer part of the asset fleet.The server and / or leader drone may send such instructions to the follower drones.

[0121] According to various embodiments, in response to determining that a follower drone has failed during the execution of a task, the leader drone may update the plan for executing the task. For example, the leader drone may determine the status of the task / element assigned to the failed follower drone and reallocate the remaining portion of the task / element (or the entire task / element) to another asset in the asset group. In response to the failure of the follower drone, the leader drone may update the capabilities of the asset group, and the leader drone may use the updated capabilities to reallocate / reallocate elements of the plan to other follower drones. In some embodiments, in response to determining that a follower drone has failed, the leader drone may add additional assets to the asset group or send a request (e.g., to a server) to add additional assets. For example, the leader drone may identify one or more capabilities of the failed follower drone and attempt to compensate for those capabilities (or capabilities for which the asset group has no redundancy or insufficient redundancy). The leader drone may determine that the follower drone has failed in response to a determination that the leader drone has not received feedback information from the follower drone within a predetermined period of time and / or by an estimated time (e.g., the estimated time may be based on a schedule currently assigned to the follower drone, such as when the follower drone is predicted not to communicate for a period of time). In some embodiments, if the leader drone does not receive feedback information from the follower drone within a predetermined period of time or by the estimated time, the leader drone may send a ping to the follower drone for the follower drone's health status, and in response to the follower drone failing to respond to the ping (e.g., within a threshold period of time), the leader drone may determine that the follower drone has failed. In some embodiments, the server may determine that the follower drone has failed in response to at least a determination that the server has not received any information or ping from the follower within a predetermined period of time and / or by the estimated time, and the server may notify the leader drone that the follower drone has failed.

[0122] According to various embodiments, in response to determining that the leader drone has failed during the execution of the task, a remaining asset in the asset group is promoted to a leader position, and such asset becomes the new leader drone for the execution of the task. The leader drone may be determined to have failed if the leader drone has failed or lost communication with some or all of the assets. The remaining assets in the asset group may negotiate to determine a new leader in the asset group. The negotiation to determine a new leader may be based at least in part on the leader rankings of the assets in the asset group (or a prioritized list of the assets). The leader rankings may be a predefined ranking determined during the configuration of the task and provided to all assets in the asset group, or a predefined ranking provided to at least one asset other than the initial leader asset, and the asset with the highest ranking among the remaining assets is promoted to the new leader drone. In some examples, the remaining assets negotiate a new leader ranking of the assets, and the asset with the highest ranking among the remaining assets is promoted to the new leader drone. In some embodiments, when a dormant drone is included in a group of assets (the dormant drone is among the remaining assets), the dormant leader is automatically promoted to the new leader drone.

[0123] According to various embodiments, partitions may be formed based on network or communication failures. For example, if a portion of the assets loses communication with a leader drone, the portion of the assets may determine (individually or collectively) that the leader drone has failed. In response to determining that the leader drone has failed, the portion of the assets may form a group of assets (e.g., a partition), and the portion of the assets may negotiate for a partition leader. If a partition leader is determined, the partition leader may determine a plan for the partition to continue executing the work (or collective tasks / elements assigned to the portion of the assets). The partition leader may attempt to establish communication with a server and / or another partition associated with the same work. In response to establishing communication with a server, the partition leader may provide feedback information regarding the execution of the work (or a portion of the work performed by the partition). In response to establishing communication with a partition leader of another partition, the partition leader may request or otherwise attempt grouping of the two partitions, or the partition leader may provide feedback information regarding the work to the other partition leader. In some embodiments, different partition leaders may negotiate the division of tasks / elements associated with a work, such that different partitions perform different parts of the work. Partitions may have predefined or negotiated constraints under which they operate. Examples of constraints include geographic constraints, time restrictions, restrictions on certain actions (e.g., restrictions on engaging with targets), etc.

[0124] In some embodiments, if a partition leader fails, the remaining assets in the partition fail over to a partition or group of assets (e.g., the partition with the leader drone corresponding to the highest ranking leader in a given leader ranking). The remaining assets in the partition may fail over to a partition or group of assets if a new partition leader cannot be determined.

[0125] According to various embodiments, each operation has a default plan that is implemented by the assets in the fleet if a new leader cannot be determined. For example, the default plan may be a plan for each asset to return to its home in response to a determination that a new leader cannot or will not be determined (e.g., within a predetermined period of time).

[0126] The device 300 may use the asset failure module 380 in connection with determining that an asset has failed and to determine a failover protocol to implement if an asset is determined to have failed. For example, if the failed asset was a leader drone, the asset failure module 380 may determine to negotiate with the remaining assets to determine a new leader drone. As another example, if the failed asset was a follower drone, the asset failure module 380 may determine to reallocate a portion of the plan assigned to the failed follower drone among the remaining assets.

[0127] According to various embodiments, in addition to determining a plan for executing a task or element, the leader drone determines a method for communicating the plan to the follower drone. The leader drone may determine a method for communicating the plan based on the communication link between the leader drone and the follower drone, the quality of the communication link, etc. In some embodiments, the leader drone may determine a method for communicating the plan based on an estimated quality of communication between the leader drone and the follower drone at a future time (e.g., a particular time in the execution of the task / element). For example, the leader drone may determine that the follower drone will move over a mountain in connection with executing the task, that the follower drone may lose communication with the leader drone, or that the communication link may have a relatively low quality of service. In response to such a determination, the leader drone may provide a larger range of plans for the follower drone to execute the task / task. As another example, if the leader drone determines that the current communication link is relatively weak (e.g., low speed / low bandwidth), in response to such a determination, the leader drone may determine to transmit only a relatively small portion of the plan for the task to the follower drone and transmit a larger portion of the plan at a future time when the communication link is or is predicted to be good. According to various embodiments, the leader drone may determine the granularity at which the plan is divided and communicated to the follower drones based on network conditions and / or network constraints. The network conditions and / or network constraints used in such determination may be current conditions / constraints or predicted conditions / constraints at a particular future time. Examples of network constraints may be one or more values ​​related to one or more of network topology (e.g., areas where the network is accessible by the leader drone or follower drones), communication bandwidth, link quality, etc. The determination of the granularity at which the plan is divided / communicated to the follower drones and / or the manner in which the plan is communicated to the follower drones may be part of the planning process performed by the leader drone.For example, the leader drone may plan the operation in multiple stages and sequentially transmit portions of the current plan or portions of the plan for multiple stages of the operation. Similarly, in some embodiments, the server managing / configuring the operation may determine how / to what extent information about the operation (e.g., operation parameters) is communicated to the leader drone based at least in part on network conditions and / or network constraints and / or the planning service capabilities of the leader drone.

[0128] FIG. 4A illustrates a system for performing at least a portion of a task, according to various embodiments of the present application. In the example illustrated in FIG. 4A, the system 400 may include a group of assets (e.g., assets 405, 410, 420, 430, 440, and / or 450) and a control center 460 (e.g., a server providing task control services). The system 400 may be implemented by the system 100 of FIG. 1. The control center 460 may be implemented by the device 200 of FIG. 2. One or more assets in the group of assets may be implemented by the device 300 of FIG. 3. For example, the asset 405, the asset 410, the asset 420, and / or the asset 430 may be implemented by the device 300 of FIG. 3. In some embodiments, the asset 405, the asset 410, the asset 420, and / or the asset 430 are semi-autonomous drones. Asset 440 may be a satellite that communicates with control center 460 and / or another asset (such as asset 405). Asset 450 may be a tower (such as a semi-autonomous tower). Asset 450 may communicate with control center 460 and / or another asset (such as asset 405).

[0129] In the example shown in FIG. 4A , the assets may be instructed to perform a task. For example, the task may be tracking vehicle 470 and / or monitoring a road (as well as tracking a particular vehicle or any vehicle moving along the road). Control center 460 may configure the task and provide task parameters to the assets (e.g., asset 405). Asset 405 may be determined to be a leader asset, and the other assets (e.g., asset 410, asset 420, and / or asset 430) may be determined to be follower assets. In response to the leader asset receiving the task parameters and pre-planning (if any) from control center 460, the leader asset may determine a plan for the other follower assets to perform tasks or elements of the task. Continuing with the above example, the leader asset may determine a plan for follower asset 2 (asset 410) to track vehicle 470, a plan for follower asset 4 (asset 430) to perform monitoring of a first portion of the road, and a plan for follower asset 3 (asset 420) to perform monitoring of a second portion of the road. In response to determining the plans for the follower assets, the leader asset may transmit the plans to the respective follower assets accordingly. The follower assets may implement the plans in response to receiving the plans from the leader drone, and the follower assets may fill in any gaps in the plan provided by the leader drone. The leader asset may determine that an asset 440 (e.g., a satellite) will provide monitoring of the road and assistance in tracking the vehicles along the road. The leader asset may determine a plan for asset 450 to capture information about the area / environment using one or more sensors. The captured information may be provided to the leader asset as feedback information, which may be utilized by the leader asset to update the plans for various assets in the asset fleet.

[0130] 4B illustrates a system for performing at least a portion of the operations according to various embodiments of the present application. In the example system 400 illustrated in FIG. 4B, a leader asset (asset 405) is determined to have failed. In response to determining that the leader asset has failed, a determination of a new leader for the assets is made among the remaining assets. For example, asset 410, asset 420, and asset 430 may negotiate a new leader asset.

[0131] According to various embodiments, determining a new leader asset includes determining a new leader asset based on a leader ranking. For example, a leader ranking 480 may be obtained by one or more of the remaining assets. In some examples, each asset stores a predefined leader ranking. In some examples, the follower assets obtain a leader ranking by negotiating a leader ranking based on the respective capabilities of the remaining assets, or the like. In the example shown in FIG. 4B, the leader ranking 480 is predefined, such as configured by the control center 460 simultaneously with the configuration of the operation. Asset 1 in the leader ranking 480 is the highest ranking asset and asset 4 is the next highest ranking asset. Since asset 1 has been determined to have failed, asset 4 is determined to be the highest ranking asset among the remaining assets. Thus, asset 4 is determined to be the next leader asset for the group of assets. In some embodiments, in response to asset 4 being determined as the new leader asset, asset 4 determines a state of the operation and determines a plan for the execution of at least the remaining portion of the operation. Asset 4 may send the corresponding plan to the remaining follower assets to which the follower assets have been assigned.

[0132] FIG. 4C illustrates a system for performing at least a portion of a task, according to various embodiments of the present application. In the example of the system 400 illustrated in FIG. 4C, a leader asset (asset 420) is determined to have failed. In response to determining that the leader asset has failed, the leader asset may determine a status of the task and / or determine an updated plan for the remaining assets (e.g., asset 410 and asset 430). According to various embodiments, in response to determining that the follower asset has failed, the leader asset may reallocate tasks / elements assigned to the failed follower asset (or the remaining tasks / elements) among the remaining assets. The leader asset may update the plan for the remaining assets based at least in part on the determined reallocation of tasks / elements. In some embodiments, the leader drone and / or asset 450 may determine to add one or more assets to the asset group (e.g., to compensate for functionality lost due to the failure of asset 430, to expedite the execution of the task, etc.). For example, a list of additional assets 490 may be determined. The leader drone and / or assets 450 (e.g., a tower with a computer system) may dynamically update the assets, such as by adding one or more of the additional assets in the list of additional assets 490. In some embodiments, the list of additional assets 490 may be determined at least in part based on the capabilities of the corresponding assets, such that the assets on the list of additional assets 490 have at least one capability that matches the capability of the remaining tasks to be completed. In some embodiments, the list of additional assets 490 may be determined at least in part based on the ownership availability of the assets on the list of additional assets (e.g., the assets may be currently available for work or may be reassigned to perform work after submitting a request to take over ownership or lease the assets).

[0133] In some embodiments, the control center (e.g., control center 460 of FIG. 4A, FIG. 4B, or FIG. 4C) is one of multiple control centers that control a system including semi-autonomous drones. In various embodiments, the control center includes a ground control center, an airborne control center, a surface control center, or any other suitable control center. In various embodiments, a single control center is in control, each control center controls a portion of the system assets, control is rotated among multiple control centers for coordinated takeover upon a change of control, or any other suitable allocation of control to control centers is made.

[0134] FIG. 5A illustrates a system for performing at least a portion of an operation, according to various embodiments of the present application. In the example illustrated in FIG. 5A, the system 500 may include a group of assets (e.g., assets 505, 510, 515, 520, 525, 530, 535, 540, and / or 545) and a control center 550 (e.g., a server providing operation control services). The system 500 may be implemented by the system 100 of FIG. 1. The control center 550 may be implemented by the device 200 of FIG. 2. One or more assets in the group of assets may be implemented by the device 300 of FIG. 3. For example, the assets 505, 510, 515, 520, 525, 530, and / or 535 may be implemented by the device 300 of FIG. 3. In some embodiments, asset 505, asset 510, asset 515, asset 520, asset 525, asset 530, and / or asset 535 are semi-autonomous drones. Asset 540 may be a satellite that communicates with control center 550 and / or another asset (such as asset 505). Asset 545 may be a tower (such as a semi-autonomous tower). Asset 545 may communicate with control center 550 and / or another asset (such as asset 505).

[0135] In the example shown in FIG. 5A , the group of assets may be instructed to perform a task. For example, the task may be tracking a vehicle 555 and / or monitoring a road (as well as tracking a particular vehicle or any vehicle moving along the road). A control center 550 may configure the task and provide task parameters to the group of assets (e.g., asset 505). Asset 505 may be determined to be a leader asset, and the other assets (e.g., asset 505, asset 510, asset 515, asset 520, asset 525, asset 530, and / or asset 535) may be determined to be follower assets. In response to the leader asset receiving the task parameters and pre-planning (if any) from the control center 550, the leader asset may determine a plan for the other follower assets to perform tasks or elements of the task. Continuing with the above example, the leader asset may determine a plan for the follower asset 525 to track the vehicle 555, a plan for the follower asset 510 to perform monitoring of a first portion of the road, and a plan for the follower asset 515 to perform monitoring of a second portion of the road. In response to determining the plans for the follower assets, the leader asset may transmit the plans to the respective follower assets accordingly. In response to receiving the plan from the leader drone, the follower assets may implement the plan, and the follower assets may fill in any gaps in the plan provided by the leader drone. The leader asset may determine that an asset 540 (e.g., a satellite) will provide monitoring of the road and assistance in tracking the vehicle along the road. The leader asset may determine a plan for the asset 545 to capture information about the area / environment using one or more sensors. The captured information may be provided to the leader asset as feedback information, which may be utilized by the leader asset to update the plans for various assets in the asset fleet.

[0136] FIG. 5B illustrates a system for performing at least a portion of an operation, according to various embodiments of the present application. In the example of the system 500 illustrated in FIG. 5B, an additional vehicle 560 appears and moves on a road monitored by the assets. One or more assets in the assets may capture an image of the vehicle 560. For example, a satellite may identify the vehicle 560 moving on the road. As another example, the asset 510, the asset 520, and / or the asset 515 may provide feedback information to a leader asset (asset 505), which may process the feedback information to determine that the vehicle 560 is within a geographic area established by the operation parameters and / or meets the conditions for tracking the vehicle moving on the road.

[0137] In response to detecting vehicle 560, the leader asset may determine to split the assets. For example, the leader asset may determine that an operation (e.g., including tracking vehicle 555 and vehicle 560) is likely to be performed more effectively if the assets are split into two partitions. In response to determining to split the assets, the leader asset may instruct corresponding follower assets in the assets, such as to provide the follower assets with an indication of which partition the follower assets belong to. Assets 505, 510, 515, 520, 55, 530, and 535 may be split into two partitions (first partition 570 and second partition 580). In some embodiments, the leader asset is in the first partition 570, so the leader asset is determined as the partition leader of the first partition 570. In some embodiments, the second partition 580 includes assets that were originally follower assets in the assets, so at least one of the assets in the second partition 580 is promoted to a partition leader of the second partition 580. The assets in the second partition 580 may negotiate to determine the leadership of the second partition 580. For example, the partition leader may be determined with a predefined leadership ranking, such as a leadership ranking determined contemporaneously with the configuration of the work. The partition leader may be determined to be the asset with the highest corresponding ranking on the leadership ranking among the assets in the second partition 580. In the illustrated example, asset 525 may be promoted to partition leader.

[0138] In the illustrated example, the task of tracking vehicle 560 may be assigned to a first partition, the task of tracking vehicle 555 may be assigned to a second partition 580, the task of providing road surveillance may be assigned to asset 540, and the task of monitoring the environment may be provided to asset 545 (e.g., a tower). In some embodiments, a leader asset in first partition 570 may determine a respective plan for asset 510, asset 515, and asset 520 to execute corresponding elements of the task of tracking vehicle 560. In response to receiving the plan for executing the corresponding elements, asset 510, asset 515, and asset 520 may implement the plan and fill in any gaps in the plan (e.g., provide collision avoidance, such as avoiding hills or trees, determine the altitude of the flight, determine when to capture images, etc.). Asset 510, asset 515, and asset 520 may provide feedback information to the leader asset, the feedback information relating to the status of execution of an element of the task of tracking vehicle 560 and / or information captured by one or more sensors (e.g., live video feeds, images such as license plate images, etc.). The leader asset may provide feedback information to the control center 550, such as information regarding vehicle 560.

[0139] In some embodiments, the partition leader in the second partition 580 may determine respective plans for the assets 530 and 535 to execute the corresponding elements of the task of tracking the vehicle 555. In response to receiving the plan for executing the corresponding elements, the assets 530 and 535 may implement the plan and fill in any gaps in the plan (e.g., provide collision avoidance such as avoiding hills or trees, determine the altitude of flight, determine when to capture images, etc.). The assets 530 and 535 may provide feedback information to the partition leader, the feedback information relating to the status of execution of the elements of the task of tracking the vehicle 555 and / or information captured by one or more sensors (e.g., live video feeds, images such as images of license plates, etc.). In some embodiments, the partition leader may provide the feedback information to the leader asset (asset 505), which may then provide the feedback information (or information including such feedback information aggregated or processed / analyzed with the feedback information from the first partition 570). The leader asset may then provide feedback information to the control center 550, such as information regarding the vehicle 560. In some embodiments, the partition leader may provide feedback information directly to the control center 560 (e.g., provide information to the control center without providing such information to the leader asset).

[0140] FIG. 5C illustrates a system for performing at least a portion of an operation, according to various embodiments of the present application. In the example illustrated in FIG. 5C, the leader asset may determine that tracking of the vehicle 555 is no longer performed based at least in part on feedback information (e.g., feedback information from asset 510, asset 515, and / or asset 520). In some embodiments, the leader asset may determine that the state of the vehicle 555 no longer meets the operation parameters for tracking the vehicle moving on the monitored road. For example, based on the feedback information, the leader asset may determine that the vehicle 555 has left (is no longer within) the geographic area being monitored or tracked based at least in part on the operation parameters and / or constraints. As another example, the leader asset may determine that the vehicle 555 has entered an area that is a restricted area according to the operation parameters, and that the assets will not enter such area.

[0141] In some embodiments, the system 500 may decide to dynamically add an asset to the asset fleet. For example, the leader asset and / or the control center 550 may decide to add the asset 585 and / or the asset 590 to the asset fleet. In response to a determination that better coverage of a geographic area / road is desirable for monitoring the road (e.g., based on user input to a user interface operatively provided by the control center 550, etc.), an asset may be added to the asset fleet. In response to a decision to add the asset 585 and / or the asset 590 to the asset fleet, the leader asset may update the mapping of capabilities to the asset and update a plan for performing the work. The leader asset may determine a plan for the asset 585 to perform a task based at least in part on the capabilities of the asset 585, etc.

[0142] In some embodiments, the control center (e.g., control center 550 of FIG. 5A, FIG. 5B, or FIG. 5C) is one of multiple control centers that control a system including semi-autonomous drones. In various embodiments, the control center includes a ground control center, an air control center, a surface control center, or any other suitable control center. In various embodiments, a single control center is in control, each control center controls a portion of the system assets, control is rotated among multiple control centers for coordinated takeover upon a change of control, or any other suitable allocation of control to control centers is made.

[0143] FIG. 6 illustrates a user interface for configuring, monitoring, and / or controlling operations in accordance with various embodiments of the present application.

[0144] According to various embodiments, the server may configure one or more user interfaces in connection with configuring a task. For example, the server may generate and configure a set of user interfaces to create a wizard that allows the configuration process to efficiently and intuitively define a task. In various embodiments, the user interfaces are configured based on user input to a preceding user interface (e.g., an immediately preceding user interface).

[0145] As shown in FIG. 6, a set of user interfaces 610, 630, and 650 may be displayed to a user to allow the user to systematically define parameters for an operation.

[0146] In interface 610, the user interface includes elements for a user to (i) enter a name and description of the operation, (ii) select a type of operation (e.g., scan road network, scan along a route, loiter fly and track, find, confirm, and track a target), and (iii) select a type of target (e.g., monitor or track target) to be operated on. User interface 630 may be configured in response to the inputs to interface 610.

[0147] In user interface 630, the user interface provides selectable elements for a user to select parameters associated with the type of asset and / or the manner in which the asset moves (e.g., air, ground, sea). The various selectable elements on user interface 630 may be determined in response to inputs to interface 610.

[0148] In the user interface 650, the user interface provides selectable elements for the user to select one or more parameters associated with the task. For example, the user may define a terrain or boundary within which the task is to be performed. As another example, the user may define restricted areas within which assets performing the task may not enter, etc. As another example, the user may define a date and time within which the task is to be performed, or a date or time within which the task is restricted. The user interface 650 may provide selectable elements for causing the server to generate the task. For example, the server may generate the task based on a user-defined task configuration / definition.

[0149] In response to generating the task, the server may cause a user interface to display a live task screen. The live task screen may include selectable elements for the user to select to activate the task, deactivate the task, or abandon the task. In some embodiments, the live task screen includes current status information regarding the status of execution of the task. For example, the live task screen may indicate a percentage of the task completed. As another example, the live task screen may include a live stream of video captured by at least one asset in the group of assets. In some embodiments, the live task screen may include selectable elements through which the user can invoke a user interface (or a wizard in the user interface) to modify the task. For example, the user may have the server add one or more assets to the group of assets. As another example, the user interface may display options (e.g., server-generated options such as possible modifications) that allow the user to select to modify the task according to suggested modifications. The server may cause the user interface to display improvements to the task by adding a particular asset, adding an asset with a particular functionality, or adding one or more assets. The user may input a selection to modify the task according to suggested modifications presented by the server. In some examples, the server may cause a user interface to display the cost associated with performing an operation with the assets and / or the cost difference or cost impact when an asset is added to or removed from the assets.

[0150] 7A is a diagram illustrating a method for configuring an operation according to various embodiments of the present application. In some embodiments, the process 700 of FIG. 7A is performed by the server 105 of the system 100 of FIG. 1 and / or the device 200 of FIG.

[0151] At step 710, data associated with one or more tasks to be performed is received. In some embodiments, a server receives the data associated with one or more tasks in connection with configuring an operation. The data associated with the one or more tasks may be received based at least in part on one or more user inputs provided to a user interface presented at the client terminal. According to various embodiments, the data associated with the one or more tasks includes one or more operation parameters and / or operation constraints. The one or more tasks to be performed may correspond to an operation to be performed.

[0152] At step 720, a group of assets is determined. In some embodiments, the server determines a group of assets to collectively perform one or more tasks or operations that include one or more tasks. The group of assets may be determined based on one or more functions associated with the one or more tasks / operations and / or the capabilities of the assets. For example, the server may determine a group of assets that have capabilities that match capabilities associated with the one or more tasks / operations.

[0153] According to various embodiments, the assets may be determined based at least in part on the availability of the assets and / or the ownership (or availability for transfer of ownership) of the assets. For example, the server may determine a superset of the assets (e.g., all assets across multiple organizations) and query the assets or a third-party service for an indication of whether the assets are available and / or available for transfer of ownership for use in performing work. The third-party service may be an ownership service that manages the ownership of assets across multiple organizations and / or the transfer or licensing of ownership according to negotiated terms (e.g., fixed-term lease, lease for a specific task, lease for a specific work, permanent transfer, etc.).

[0154] In response to determining that the assets will perform one or more tasks (or corresponding operations), process 700 proceeds to step 730 where instructions are communicated to at least one drone in the assets. In some embodiments, the server determines a leader drone in the assets, and the server communicates instructions to the leader drone. The instructions may include, or may be communicated in conjunction with, an indication that a leader drone is in the assets. In some embodiments, the instructions communicated to the leader drone include operation parameters. The instructions may also include a pre-plan if the server or other service has generated a high-level pre-plan for the operation.

[0155] In response to communicating the command to the at least one drone, process 700 proceeds to step 740 where a determination is made as to whether the process is complete. As an example, the process may be determined to be complete based at least in part on user input (such as an input to cancel or pause the task). As another example, the process may be determined to be complete in response to a user selecting to begin performing the task. If the process is deemed complete, process 700 ends. Otherwise, process 700 returns to step 710 where further data associated with one or more tasks is received. In some embodiments, process 700 may be deemed incomplete at step 740 in response to user input to further refine / configure the task. As an example, prior to starting the task, a user may input a request to edit the task into a user interface. As another example, during the performance of the task by the assets, a user may input a selection to modify the task, such as dynamically adding assets to the assets to accelerate completion of the task and removing assets based on reprioritizing the task to another task.

[0156] If the assets are not determined at step 720, process 700 may proceed to step 740. The assets cannot be determined if there are no assets available, if ownership of assets matching the functionality of one or more tasks or operations is not available, if no assets are found whose functionality collectively matches the functionality of the operation, etc. At step 740, a prompt may be provided to the user indicating that the assets cannot be determined, requesting the user to modify the configuration of the operation, or requesting whether the user requests to cancel the operation.

[0157] FIG 7B illustrates a method for configuring an operation according to various embodiments of the present application. In some embodiments, process 720 of FIG 7B is performed by server 105 of system 100 of FIG 1 and / or device 200 of FIG 2. Process 720 of FIG 7B may correspond to step 720 of FIG 7A.

[0158] One or more characteristics of one or more tasks are obtained in step 721. In some embodiments, the server may obtain the one or more characteristics based on a definition of the task (e.g., task parameters).

[0159] At step 722, one or more capabilities of the asset are obtained. In some embodiments, the server may obtain one or more capabilities of the asset based on a mapping of capabilities to the asset. The mapping of capabilities of the asset may be determined based on querying the assets for an indication of their capabilities (or their current capabilities), based on the assets registering / advertising their capabilities, etc.

[0160] At step 723, a determination is made as to whether any assets match the characteristics of the task. In some embodiments, the server determines one or more capabilities that correspond to the characteristics of the task and determines assets that have capabilities that match the capabilities that correspond to the characteristics of the task. In response to determining that an asset or assets have capabilities that match the characteristics of the task, process 720 may proceed to step 727.

[0161] At step 724, an asset group is determined. In some embodiments, the server determines the asset group from among the assets having functionality matching the characteristics of one or more tasks. The server may select assets to be included in the asset group based on asset availability, asset ownership, availability of transfer of asset ownership, a determination that the asset group has redundancy with respect to functionality, etc. In some embodiments, the server uses a cost function in connection with selecting assets for inclusion in the asset group. For example, the server may use the cost function to optimize the cost of the asset group or select assets having a cost below a threshold cost value.

[0162] In response to the determination of the assets, process 720 proceeds to step 725 where a determination is made as to whether the determination of the assets is complete. For example, the server may prompt the user to confirm the assets or may provide a selectable element through which the user may choose to improve / modify one or more tasks or one or more features of the tasks. In response to determining that the determination of the assets is not complete, process 720 returns to step 721. Otherwise, process 720 proceeds to step 726. If in step 723 no assets were found that match the characteristics of the task (or if for each characteristic of the task no assets were found that have the corresponding functionality), process 720 may proceed to step 725.

[0163] At step 726, suggestions for asset groups and / or that asset groups are possible (eg, assets may be grouped together in relation to performing an operation) are provided.

[0164] FIG 8A illustrates a method for performing at least one task of an operation, according to various embodiments herein. In some embodiments, process 800 of FIG 8A is performed by asset 120, asset 125, and / or asset 130 of system 100 of FIG 1 and / or device 300 of FIG 3. According to various embodiments, process 800 is performed by a semi-autonomous drone.

[0165] In some embodiments, process 800 is performed by a leader drone in a fleet of assets to perform a task. In some embodiments, process 800 is performed by a follower drone in a fleet of assets to perform a task.

[0166] An indication that the drone is part of a constellation of assets performing an element of a task is received at step 810. The drone may receive an indication that the drone is part of a constellation of assets associated with configuring a task.

[0167] According to various embodiments, when process 800 is performed by a leader drone, the leader drone may receive an indication from a server (e.g., a server configuring the operation) that the drone is part of a fleet of assets, along with operation parameters, etc. associated with the operation for which the task elements are being performed.

[0168] According to various embodiments, when process 800 is performed by a follower drone, the follower drone may receive an indication from a leader drone that the drone is part of a constellation of assets.

[0169] Information regarding one or more elements of one or more tasks is communicated at step 820. The information regarding the one or more elements may include parameters of the elements and / or constraints for executing the elements.

[0170] According to various embodiments, when process 800 is performed by a leader drone, the leader drone may communicate a plan for executing one or more elements to one or more follower drones assigned to execute the one or more elements.

[0171] According to various embodiments, when process 800 is performed by a follower drone, the follower drone may receive a plan for executing one or more elements. The follower drone may receive the plan from the leader drone. The plan may be determined based on the capabilities of the follower drone, capabilities associated with the elements, the status of the follower drone (e.g., availability, location, etc.), etc.

[0172] At step 840, a determination is made as to whether process 800 is complete. In response to determining that process 800 is complete, process 800 ends. In response to determining that process 800 is not complete, process 800 may return to step 810. According to various embodiments, process 800 is determined to be complete in response to one or more elements being completed or a determination that an operation is completed. Process 800 may also be determined to be complete if an operation is paused or canceled / aborted.

[0173] FIG. 8B illustrates a method for performing at least one task of a task, according to various embodiments herein. In some embodiments, process 830 of FIG. 8B is performed by asset 120, asset 125, and / or asset 130 of system 100 of FIG. 1 and / or device 300 of FIG. 3. According to various embodiments, process 830 is performed by a semi-autonomous drone. Process 830 of FIG. 8B may correspond to step 830 of process 800 of FIG. 8A. Process 830 of FIG. 8B may be performed by a follower drone in a group of assets assigned to perform the task.

[0174] In step 831, data from one or more sensors of the drone is received. In some embodiments, during execution of a plan (such as a plan provided by a leader drone), a follower drone acquires information about the element / task. For example, a follower drone may capture information such as images, live stream video, weather information, etc.

[0175] At step 832, a determination is made as to whether the data is relevant to the plan. In some embodiments, in response to receiving data from a sensor on the drone, the drone determines whether such data is relevant to the plan to be implemented. As an example, in the case of a plan to track a target, the data received from the sensor may include an image. The drone may determine whether the image includes a target that the drone is tracking. For example, the drone may perform image analysis to determine that the target is in the image (or that the likelihood that the image includes the target exceeds a predetermined threshold), etc.

[0176] In response to determining that the data is associated with the plan, process 830 may proceed to step 833 where information regarding the plan is transmitted to the leader drone. In some embodiments, the follower drone may determine that the captured information is associated with the plan (e.g., indicates the status of elements / tasks, includes targets / subjects of the plan, includes information regarding the geographic area in which the plan is to be implemented, etc.).

[0177] In response to transmitting the information regarding the plan to the leader drone, process 830 proceeds to step 834 where a determination is made as to whether process 830 is complete. In some embodiments, process 830 is determined to be complete in response to a determination that the task is completed, stopped, and / or paused. In response to a determination that process 830 is complete, process 830 ends. In response to a determination that process 830 is not complete, process 830 may return to step 831 where the drone continues to monitor data / information captured by its sensors and repeatedly transmit associated information to the leader drone. As an example, process 830 is determined to be not complete in response to a determination that the task is not complete (e.g., if a drone is tasked with scanning a road and the drone is unable to maintain a series of sharp turns in the road and misses scanning a portion of the road, the task is deemed not complete and the drone may turn back and rescan the missed portion of the road).

[0178] If at step 832 it is determined that the data is not associated with the plan, the process 830 proceeds to step 834, which is performed as described above.

[0179] FIG. 8C illustrates a method for performing at least one task of a task, according to various embodiments herein. In some embodiments, process 830 of FIG. 8C is performed by asset 120, asset 125, and / or asset 130 of system 100 of FIG. 1 and / or device 300 of FIG. 3. According to various embodiments, process 830 is performed by a semi-autonomous drone. Process 830 of FIG. 8C may correspond to step 830 of process 800 of FIG. 8A. Process 830 of FIG. 8C may be performed by a leader drone in a group of assets assigned to perform the task.

[0180] Feedback information or control information is received from one or more follower drones at step 835. According to various embodiments, the leader drone receives feedback information from the follower drones contemporaneously with execution of a plan for a task or element of a task associated with the operation.

[0181] The feedback information may include current / contemporaneous information such as current location, live video stream, current status of the operation (e.g., in the case of a surveillance operation, the status may be that no target is present within the defined area of ​​the operation).

[0182] The control information may include information provided from the server to the leader drone. For example, the control information may include indications that the operation has been updated, operation parameters, operation constraints, instructions / commands for modifying the assets, etc.

[0183] At step 836, a decision is made as to whether to update the plan. According to various embodiments, the leader drone autonomously decides whether to update the plan based on the feedback information or control information. For example, if the feedback information indicates completion of an element or task of the work, the leader drone may update the plan for the corresponding follower drone so that the follower drone performs a new task / element. As another example, in the case of surveillance within a defined area, in response to determining that the feedback information indicates that the target is within the defined area, the leader drone may decide to update the plan to move closer to the target in order to obtain specific details about the target, or the leader drone may decide to update the plan to instruct the follower drone to switch to a tracking task to track the target).

[0184] In response to determining at step 836 that the plan is updated, process 830 may proceed to step 837, where the updated plan is transmitted to one or more follower drones, and process 830 proceeds to step 838. Conversely, in response to determining at step 836 that the plan is not updated, process 830 proceeds to step 838.

[0185] At step 838, a determination is made as to whether process 830 is complete. In some embodiments, process 830 is determined to be complete in response to determining that the operation is completed, stopped, and / or paused. In response to determining that process 830 is complete, process 830 ends. In response to determining that process 830 is not complete, process 830 may return to step 835 where the drone continues to monitor for feedback or control information regarding the operation.

[0186] FIG 9A illustrates a method for executing a plan associated with an operation according to various embodiments of the present application. In some embodiments, process 900 of FIG 9A is performed by asset 120, asset 125, and / or asset 130 of system 100 of FIG 1 and / or device 300 of FIG 3. According to various embodiments, process 800 is performed by a semi-autonomous drone.

[0187] An indication that the drone is part of a constellation of assets performing an element of a task is received at operation 910. The drone may receive an indication that the drone is part of a constellation of assets associated with configuring a task.

[0188] According to various embodiments, when process 900 is performed by a leader drone, the leader drone may receive an indication from a server (e.g., a server configuring the operation) that the drone is part of a fleet of assets, along with operation parameters, etc. associated with the operation for which the task elements are being performed.

[0189] According to various embodiments, when process 900 is performed by a follower drone, the follower drone may receive an indication from a leader drone that the drone is part of a constellation of assets.

[0190] The plan is executed at step 920. In some embodiments, the drone obtains a plan associated with performing a task and executes the plan.

[0191] According to various embodiments, when process 900 is performed by a leader drone, the leader drone may receive control information from a server (such as a server that configured the task). The control information may include task parameters, constraints for performing the task, pre-planning, any combination thereof, etc. The leader drone may determine a plan for the execution of the task based on high-level information for the task. For example, the leader drone may determine a plan for one or more follower drones to perform tasks or elements of the task of the task and command the follower drones to implement the plan for performing the tasks / elements.

[0192] According to various embodiments, when process 900 is performed by a follower drone, the follower drone may implement a plan for executing the tasks / elements. The follower drone may receive the plan from the leader drone. In some embodiments, the follower drone may update the plan to fill in gaps in the plan received from the leader drone. For example, the follower drone modifies the plan to have finer granularity than the plan received from the leader drone.

[0193] At step 930, a determination is made as to whether the asset has failed. According to various embodiments, the determination of whether the asset has failed may be based on the time since the asset last communicated with the drone or the lack of a response to a ping for the asset's health status. In response to determining that the asset has failed, process 900 proceeds to step 940.

[0194] The plan is updated at step 940. According to various embodiments, the plan is updated to take into account the determination that the asset has failed.

[0195] According to various embodiments, when process 900 is performed by a leader drone, the leader drone may determine a status of the work, such as a status of a task / element assigned to an asset determined to have failed. Based on the status of the task / element, the leader drone may update the plan to reallocate any remaining assignments (e.g., uncompleted portions of a task / element assigned to an asset) to one or more remaining assets in the group of assets assigned to perform the work.

[0196] According to various embodiments, when the process 900 is performed by a follower drone, the follower drone may update the plan in response to determining that the leader drone has failed. In response to determining that the leader drone has failed, the follower drone may negotiate with one or more other remaining assets in the group of assets to determine a new leader. In response to the leader drone being promoted to the new leader drone, the new leader drone may update the plan based on its status as the new leader drone. For example, the new leader drone may determine one or more plans for the remaining assets. The new leader drone may determine the plan based on the capabilities of each of the remaining assets and the capabilities or features associated with the tasks / elements that have not been completed in the execution of the work.

[0197] Information regarding the updated plan is communicated at step 950. In some embodiments, the updated plan is communicated to the remaining assets.

[0198] At step 960, information is communicated during implementation of the updated plan.

[0199] According to various embodiments, when process 900 is performed by a leader drone, the leader drone may receive information from a follower drone / asset or a server. As an example, the leader drone may receive control information from a server (such as a server that configured the operation). The control information may include updated operation parameters, constraints for performing the operation, pre-planning, any combination thereof, etc. As another example, the leader drone may receive feedback information from the follower drone regarding the operation, such as feedback information obtained by the follower drone during execution of the updated plan.

[0200] According to various embodiments, when the process 900 is performed by a follower drone, the follower drone sends feedback information to the leader drone.

[0201] At step 970, a determination is made as to whether process 900 is complete. In response to a determination that process 900 is complete, process 900 ends. In response to a determination that process 900 is not complete, process 90 may return to step 920. According to various embodiments, process 900 is determined to be complete in response to a determination that one or more elements have been completed or that an operation has been completed. Process 900 may also be determined to be complete if an operation has been paused or canceled / aborted. Process 900 may proceed to step 970 in response to a determination at step 930 that no asset failure has occurred.

[0202] FIG 9B illustrates a method for executing a plan associated with an operation according to various embodiments of the present application. In some embodiments, process 930 of FIG 9A is performed by asset 120, asset 125, and / or asset 130 of system 100 of FIG 1 and / or device 300 of FIG 3. Process 930 of FIG 9B may correspond to step 930 of process 900 of FIG 9A.

[0203] At step 931, feedback information is received from one or more follower drones or control information is received. In some embodiments, the leader drone may receive feedback information from the follower drones at the same time the follower drones execute a plan to perform a task / operation. In some embodiments, the leader drone may receive control information (e.g., updated control information) from a server. For example, the control information may include an indication that the operation has been updated, operation parameters, operation constraints, instructions / commands to modify the assets, etc.

[0204] At step 932, a determination is made as to whether feedback has been received from an asset in the asset group. In some embodiments, the leader drone performs process 930 of FIG. 9B for each asset in the asset group. For example, the leader drone may determine whether feedback information has been received from a particular follower drone. In some embodiments, the determination of whether feedback information has been received from a particular asset may be based on the time that has elapsed since the particular asset last communicated feedback information. In some embodiments, a determination of whether feedback information has been received from a particular asset within a predetermined threshold time. In some embodiments, a determination of whether feedback information has been received from a particular asset by a predetermined time (e.g., an estimated time (e.g., the estimated time may be determined at least in part based on a plan being implemented by the asset, the location of the asset, a communication link with the asset, etc.)).

[0205] In response to a determination at step 932 that feedback information is received from an asset in the assets, process 930 proceeds to step 936. In step 936, a determination is made as to whether process 930 is complete. In response to a determination that process 930 is complete, process 930 proceeds to step 937. In response to a determination that process 930 is not complete, process 930 may return to step 931. According to various embodiments, process 930 is determined to be complete in response to a determination that one or more elements are completed or that the task is completed. Process 930 may also be determined to be complete if the task is paused or canceled / aborted. In step 937, an indication of whether an asset has failed is provided. In some embodiments, the module that determines whether an asset has failed provides an indication of whether an asset has failed to a planning service of the leader drone (e.g., a service that determines / updates a plan for executing one or more tasks associated with the task).

[0206] In response to determining at step 932 that feedback information has not been received from an asset in the asset pool, process 930 proceeds to step 933. At step 933, a ping is sent to the asset. A leader drone may send the ping as a health check on the asset.

[0207] At step 934, a determination is made as to whether a response has been received from the asset. In some embodiments, the leader drone determines whether a response has been received based on a determination of whether the asset responded to the ping within a threshold period. In response to a determination that a response (to the ping) has been received from the asset, process 930 proceeds to step 936. Conversely, in response to a determination that a response (to the ping) has not been received from the asset, process 930 proceeds to step 935 where it is determined that the asset has failed. Process 930 then proceeds to step 936.

[0208] FIG 9C illustrates a method for executing a plan associated with an operation according to various embodiments of the present application. In some embodiments, process 950 of FIG 9C is performed by asset 120, asset 125, and / or asset 130 of system 100 of FIG 1 and / or device 300 of FIG 3. Process 950 of FIG 9C may correspond to step 950 of FIG 9A. In some embodiments, process 950 is performed by a semi-autonomous leader drone.

[0209] At step 951-1, feedback information regarding the execution of the plan is received. According to various embodiments, the leader drone receives feedback information from the follower drones contemporaneously with the execution of the plan regarding a task or element of a task associated with the operation. The feedback information may include current / contemporaneous information such as current location, live video stream, current status of the operation (e.g., for a surveillance operation, the status may be that no target is present within a defined area of ​​the operation).

[0210] At step 951-2, a determination is made as to whether the plan is to be updated. In some embodiments, the leader drone determines whether to update the plan (e.g., a plan for a particular follower drone or plans for multiple follower drones) based at least in part on the feedback information.

[0211] In response to a determination that the plan will not be updated, process 950 proceeds to step 951-7 where a determination is made as to whether process 950 is complete. In response to a determination that process 950 is complete, process 950 ends. In response to a determination that process 950 is not complete, process 950 may return to step 951-1. According to various embodiments, process 950 is determined to be complete in response to one or more elements being completed or a determination that the task is complete. Process 950 may also be determined to be complete if the task is paused or canceled / aborted.

[0212] In response to a determination that the plan is to be updated, process 950 proceeds to step 951-3 where a determination is made as to whether a plan update is possible. In some embodiments, the leader drone determines whether a plan update is possible based on parameters of the operation, the status of the operation, the capabilities of one or more assets in the asset fleet, etc. For example, if it is determined that the operation is to be updated or the capabilities required to perform the operation have changed or been newly determined, the leader drone may determine whether the asset fleet includes an asset having the required capabilities.

[0213] In response to determining at step 951-3 that updating the plan is not possible, process 950 proceeds to step 951-5 of indicating that updating the plan is not possible. In some embodiments, the leader drone may provide a prompt to the user at a user interface or to the server (which may result in a prompt at a client terminal) that the plan cannot be updated. The prompt may include elements that allow the user to cancel the operation, modify the operation, etc. Process 950 then proceeds to step 951-7.

[0214] In response to determining at step 951-3 that a plan update is possible, process 950 proceeds to step 951-4 where the plan is updated. In some embodiments, the leader terminal updates the plan accordingly (e.g., based at least in part on the feedback information). The plan may be updated based at least in part on the capabilities of the follower drones.

[0215] At step 951-6, the updated plan is transmitted to the follower drone. The process 950 may then proceed to step 951-7.

[0216] FIG 9D illustrates a method for executing a plan associated with an operation according to various embodiments of the present application. In some embodiments, operation 951-2 of FIG 9D is performed by asset 120, asset 125, and / or asset 130 of system 100 of FIG 1 and / or device 300 of FIG 3. Operation 951-2 of FIG 9D may correspond to operation 951-2 of FIG 9C. In some embodiments, operation 951-2 is performed by a semi-autonomous leader drone.

[0217] At step 951-A, a determination is made as to whether an asset failure has occurred. According to various embodiments, the determination of whether an asset has failed may be based on the time since the asset last communicated with the drone or the lack of a response to a ping for the asset's health status.

[0218] In response to determining the asset, process 951-2 proceeds to step 951-B where a determination is made as to whether the failure of the asset affects the performance of the work. The leader drone may determine whether the failed asset had any outstanding tasks or elements that affected the work.

[0219] In response to a determination that the asset failure does not affect the execution of the task, process 951-2 may proceed to step 951-G where a decision is made not to update the plan. Process 951-2 may then proceed to step 951-H where a determination is made as to whether process 951-2 is complete. In response to a determination that process 951-2 is complete, process 951-2 ends. In response to a determination that process 951-2 is not complete, process 951-2 may return to step 951-A. According to various embodiments, process 951-2 is determined to be complete in response to a determination that one or more elements have been completed or that the task is complete. Process 951-2 may also be determined to be complete if the task is paused or canceled / aborted.

[0220] In response to a determination that the asset failure will not affect the performance of the work, process 951-2 may proceed to step 951-C where a decision is made to update the plan. Process 951-2 may then proceed to step 951-H where a determination is made as to whether process 951-2 is complete. In response to a determination that process 951-2 is complete, process 951-2 ends.

[0221] In response to determining that the asset has not failed, process 951-2 proceeds to step 951-D. At step 951-D, a determination is made as to whether the plan has failed. In some embodiments, the leader drone may determine whether the plan has failed based on a determination of whether the asset has failed. For example, the leader drone may determine that the plan has failed if the plan had a corresponding capability (e.g., a required capability) and if the failed asset had a capability for which the asset group did not have redundancy in at least one other asset (and it was determined that a new asset / capability cannot be acquired).

[0222] In response to determining that the plan has failed, process 951-2 proceeds to step 951-E where a determination is made as to whether the work is to be stopped (or paused). The leader drone may determine to stop the work in response to determining that the plan is a necessary component of the work (including tasks / elements required by the work). The leader drone may determine to stop the work based at least in part on the work having a corresponding function (e.g., a required function) and the failed asset had a function for which the asset group does not have redundancy in at least one other asset (and it has been determined that a new asset / function cannot be acquired).

[0223] In response to determining at step 951-E to stop working, process 951-2 proceeds to step 951-H.

[0224] In response to a determination at step 951-D that no plan failure occurred or a determination at step 951-E that work will not be aborted, process 951-2 proceeds to step 951-F where a decision is made that the plan will be updated.

[0225] FIG 9E illustrates a method for executing a plan associated with an operation according to various embodiments of the present application. In some embodiments, process 950 of FIG 9E is performed by asset 120, asset 125, and / or asset 130 of system 100 of FIG 1 and / or device 300 of FIG 3. Process 950 of FIG 9E may correspond to step 950 of FIG 9A. In some embodiments, process 950 is performed by a semi-autonomous follower drone.

[0226] At step 952-1, data from one or more sensors of the drone is received. In some embodiments, during execution of a plan (such as a plan provided by a leader drone), a follower drone acquires information about the element / task. For example, a follower drone may capture information such as images, live stream video, weather information, etc.

[0227] At step 952-3, a determination is made as to whether the data is relevant to the plan. In some embodiments, in response to receiving data from a sensor on the drone, the drone determines whether such data is relevant to the plan to be implemented. As an example, in the case of a plan to track a target, the data received from the sensor may include an image. The drone may determine whether the image includes a target that the drone is tracking. For example, the drone may perform image analysis to determine that the target is in the image (or that the likelihood that the image includes the target exceeds a predetermined threshold), etc.

[0228] In response to determining that the data is related to the plan, process 950 may proceed to step 952.5, where information regarding the plan is transmitted to the leader drone. In some embodiments, the follower drone may determine that the captured information is associated with the plan (e.g., indicates the status of elements / tasks, includes targets / subjects of the plan, includes information regarding the geographic area in which the plan is to be implemented, etc.).

[0229] In response to transmitting information regarding the plan to the leader drone, process 950 proceeds to step 952-7 where a determination is made as to whether process 950 is complete. In some embodiments, process 950 is determined to be complete in response to determining that the operation is completed, stopped, and / or paused. In response to determining that process 950 is complete, process 950 ends. In response to determining that process 950 is not complete, process 950 may return to step 952-1 where the drone continues to monitor data / information captured by its sensors and repeatedly transmit associated information to the leader drone.

[0230] If, at step 952-3, it is determined that the data is not related to the plan, then process 950 proceeds to step 952-1, which is performed as described above.

[0231] FIG 10A illustrates a method for executing a plan associated with a task according to various embodiments of the present application. In some embodiments, process 1000 of FIG 10A is performed by server 105, asset 120, asset 125, and / or asset 130 of system 100 of FIG 1, device 200 of FIG 2, and / or device 300 of FIG 3. According to various embodiments, process 1000 is performed by a semi-autonomous drone or a server managing the task.

[0232] At step 1010, data associated with one or more tasks being performed by the assets is obtained. According to various embodiments, the data associated with one or more tasks may include information regarding updates to the configuration / definition of the task or feedback information obtained and communicated during the execution of the task or elements of the task. The updates to the configuration / definition of the task may be based at least in part on one or more user inputs corresponding to a user request to modify the task (e.g., to accelerate completion of the task, to change the priority of the task relative to at least one other task, etc.). The feedback information may include a change in the status of the task, such as a condition being met that triggers one or more other tasks. As a specific example, in the context of a surveillance task in a defined geographic area, the task may be defined to trigger a tracking task if a target object / vehicle is detected within the geographic area. Thus, during a surveillance task by one or more follower drones, when a target is found within the geographic area, the follower drone may provide feedback information to the leader drone to indicate the sighting. As another specific example, the feedback information may include an indication that the follower drone has malfunctioned or been destroyed. The failure or destruction of a follower drone may degrade the functionality of the asset constellation or the redundancy of the asset constellation's functionality.

[0233] According to various embodiments, when process 1000 is performed by a leader drone, the leader drone may receive data associated with one or more tasks from a server, such as via control information, or from one or more follower drones, such as via feedback information transmitted simultaneously as the follower drone executes a plan associated with performing an operation (e.g., performing one or more tasks).

[0234] According to various embodiments, when process 1000 is performed by a server, the server may receive data associated with one or more tasks from the server, such as via user input to a user interface. A user may enter commands or requests into the user interface that may cause an operation to be modified or parameters of an operation to be modified.

[0235] At step 1020, a decision is made as to whether the assets are to be modified. According to various embodiments, the decision to modify the assets may be based at least in part on data associated with one or more tasks being performed by the assets (e.g., operation data related to a current state or characteristics of the operation). The decision to modify the assets may include a decision to add one or more assets to the assets. Similarly, the decision to modify the assets may include a decision to remove one or more assets from the assets. For example, the device may determine that a new task is to be performed based at least in part on the data. As another example, the new task may include or require a function for which the current assets do not include an asset with the corresponding functionality. As another example, the device may determine that the assets have lost a functionality or redundancy in a functionality due to an asset failure. Redundancy in a functionality may allow multiple tasks requiring the functionality to be performed (e.g., in parallel) by multiple assets having the functionality. Loss of functional redundancy may reduce the number of tasks that can be performed in parallel or reduce the flexibility with which a leader drone can allocate tasks or make planning decisions across a fleet of assets.

[0236] According to various embodiments, the decision to modify the assets may be based on a decision to add functionality to the assets and / or a determination that the functionality of the assets is no longer required (or is predicted to be no longer required) for the current task. The assets may be determined to be modified to add one or more assets, or to remove one or more assets, or to both add one or more assets and remove one or more assets. In some embodiments, the decision whether to modify the assets may be based on a cost (e.g., a cost determined by a predetermined cost function) associated with using the assets to perform the task or the remainder of the task. Determining whether to modify the assets while the task is being performed allows for dynamic resource management that may ensure, for example, that appropriate functionality is assigned to the assets. Additionally, dynamic resource management may allow for optimization or improvement of costs associated with performing the task.

[0237] In response to a decision to modify the group of assets at step 1020, process 1000 proceeds to step 1030 where instructions are communicated indicating the modification of the group of assets. The instructions may be provided to the affected assets (e.g., assets being added to or removed from the group) and / or to the leader asset of the corresponding group of assets.

[0238] According to various embodiments, when process 1000 is performed by a leader drone, the leader drone may send instructions to modify the assets to the assets affected by the modification and / or to a server (such as a server managing or configuring the work). In some embodiments, the leader drone may send the modification instructions or suggestions to an ownership service that manages ownership of assets across multiple organizations. For example, instructions may be sent to an ownership service in connection with negotiating ownership of assets affected by the modification.

[0239] According to various embodiments, when process 1000 is performed by a server, the server communicates instructions to modify the assets to a leader drone and / or to assets affected by the modification, and the server may provide suggestions to a user via a user interface on a client terminal, where the suggestions inform the user that the assets are to be modified.

[0240] In response to communicating the indication indicating the modification of the assets, process 1000 proceeds to step 1040 where a determination is made as to whether process 1000 is complete. In some embodiments, process 1000 is determined to be complete in response to determining that the work is completed, stopped, and / or paused. In response to determining that process 1000 is complete, process 1000 ends. In response to determining that process 1000 is not complete, process 1000 may return to step 1010 where the drone continues to monitor data / information captured by its sensors and repeatedly transmit associated information to the leader drone. Additionally, in response to a determination at step 1020 that the assets are not modified, process 1000 may proceed to step 1040.

[0241] FIG 10B illustrates a method for executing a plan associated with an operation, according to various embodiments of the present application. In some embodiments, process 1020 of FIG 10B is performed by server 105, asset 120, asset 125, and / or asset 130 of system 100 of FIG 1, device 200 of FIG 2, and / or device 300 of FIG 3. Process 1020 of FIG 10B may correspond to step 1020 of FIG 10A.

[0242] In step 1021, one or more features of the one or more remaining tasks are obtained. In some embodiments, a device (e.g., a server or a drone, such as a leader drone) may determine the remaining tasks for the operation and determine one or more features associated with the remaining tasks. The one or more features associated with the remaining tasks may be determined based on a mapping of tasks to features. For example, the device may use the remaining tasks to query a mapping of tasks to features.

[0243] At step 1022, one or more capabilities of the asset are obtained. In some embodiments, the device may obtain one or more capabilities of the asset based on a mapping of capabilities to the asset. The mapping of asset capabilities may be determined based on querying the assets for an indication of their capabilities (or their current capabilities), based on the assets registering / advertising their capabilities, etc.

[0244] At step 1023, a determination is made as to whether any assets match the characteristics of the remaining task. In some embodiments, the server determines one or more capabilities that correspond to the characteristics of the task and determines assets that have capabilities that match the capabilities that correspond to the characteristics of the task. In response to determining that an asset or assets have capabilities that match the characteristics of the remaining task, process 1020 may proceed to step 1024.

[0245] At step 1024, modifications to the asset group are determined. In some embodiments, the device determines the asset group from among the assets having functionality that matches the characteristics of one or more remaining tasks. In some embodiments, the device determines the modifications to the asset group based on assets in the asset group having a set of functionality that does not match the characteristics of one or more remaining tasks (e.g., there is no overlap between the functionality of the excluded asset and the features / functionality of the remaining tasks). The device may select assets to be added to the asset group based on asset availability, asset ownership, availability of transfer of asset ownership, a determination that the asset group has redundancy with respect to functionality, etc. In some embodiments, the device uses a cost function in connection with selecting assets to include in or exclude from the asset group. For example, the server may use the cost function to optimize the cost of the asset group or select assets having a cost lower than a threshold cost value.

[0246] In response to the determination of the assets, the process 1020 proceeds to step 1025 where a determination is made as to whether the determination of the assets is complete. For example, the device may prompt the user to confirm the assets or may provide a selectable element where the user can choose to improve / modify one or more tasks or one or more features of the tasks. In response to determining that the determination of the assets is not complete, the process 1020 returns to step 1021. Otherwise, the process 1020 proceeds to step 1026. If at step 1023 no assets were found that match the characteristics of the remaining tasks (or if for each characteristic of the task no assets were found that have corresponding features), the process 1020 may proceed to step 1025. At step 1026, a suggestion is provided as to whether to modify the assets. In some embodiments, the suggestion as to whether to modify the assets is provided to a planning service running on the leader drone. In some embodiments, the suggestion as to whether to modify the assets is provided to a control service, such as on a server.

[0247] FIG. 11 illustrates a method for configuring a task according to various embodiments of the present application. In some embodiments, process 1000 of FIG. 10A is performed by server 105, asset 120, asset 125, and / or asset 130 of system 100 of FIG. 1, and / or device 200 or FIG. 2. According to various embodiments, process 1100 is performed by a server (e.g., a server that configures and / or manages tasks). Process 1000 may be performed by a server in conjunction with a client terminal.

[0248] At step 1105, a first user interface is displayed. In some embodiments, the server causes the client terminal to display the first user interface. The first user interface may correspond to an interface through which one or more features or parameters associated with the operation are input by a user. For example, the first user interface may be used in connection with defining an operation. The user may input one or more features or parameters via one or more selectable elements provided on the user interface.

[0249] A user selection regarding a feature of the task to be performed is received at step 1110. The user selection may be made to a client terminal, which may communicate the user selection to a server over one or more networks.

[0250] At step 1115, a configuration for the alternative user interface is determined based at least in part on the user selection. In some embodiments, the server determines one or more features, parameters, or constraints that are mapped to the user selection (e.g., mapped to the features of the task). For example, if the user selection into the first user interface is a selection of a type of work (e.g., surveillance), the server may determine the alternative user interface to include a definition of a surveillance context, such as a time, date, geographic location, a type of target (e.g., moving target, stationary target, etc.). As another example, for example, if the user selection into the first user interface is a selection of a type of work and / or a context of the work (e.g., surveillance), the server may determine the alternative user interface to include a definition of one or more features related to assets used in connection with performing the work.

[0251] The alternative user interface is displayed at step 1120. In some embodiments, in response to determining the configuration of the alternative interface, the server causes the client terminal to display the alternative user interface.

[0252] At step 1125, a user selection is received regarding assets to be deployed to perform the operation. The user selection may be a selection of specific assets or types of assets to be deployed to perform the operation. In some embodiments, the user selection regarding assets to be deployed to perform the operation includes a user selection for one or more parameters of the operation that are used by the server to determine the assets or types of assets to deploy. The user selection may be made to a client terminal, which may communicate the user selection to the server over one or more networks.

[0253] At step 1130, a decision is made whether to provide another user interface for configuring the task. The decision to provide another user interface may be based at least in part on task and / or asset parameters obtained in the first user interface and the other user interfaces. In some embodiments, the decision to provide another user interface is based on user input, such as for canceling the task or starting the task. For example, the server may fill gaps in the task definition and deploy corresponding assets. As an example, the server may determine any remaining parameters of the task based at least in part on one or more of: (i) historical information, such as information of similar / past tasks; (ii) a best fit associated with the user-defined parameters of the task; and (iii) a cost of performing the task determined using a cost function.

[0254] In response to determining that an alternative user interface is to be provided for configuring the work, process 1100 proceeds to step 1135, where the alternative interface is displayed. The alternative interface may be configured based at least in part on user selections regarding characteristics of the task to be performed and / or user selections regarding assets to be deployed.

[0255] At step 1140, a user selection for another user interface is received. The user selection may be further input regarding the definition or configuration of the task. The user selection may be made to a client terminal, which may communicate the user selection to a server over one or more networks. Process 1100 may then proceed to step 1160.

[0256] At step 1160, a determination is made as to whether the determination of the configuration of the task is complete. For example, the device may prompt the user to review the assets and review the configuration of the task, or may provide selectable elements through which the user may select to improve / modify one or more tasks or one or more features of a task. In response to determining that the configuration of the task is not complete, process 1100 returns to step 1130 where the process iteratively configures the user interface based on a previous user input to the user interface or user inputs to one or more previous user interfaces. Otherwise, process 1100 ends.

[0257] In response to a determination at step 1130 that a user interface associated with configuring the task is not provided, process 1100 may proceed to step 1145 where the task to be performed is determined (defined) based at least in part on the user selection. Other variables may be used in connection with determining the task, such as historical information (e.g., for similar tasks), cost of performing the task, etc. (e.g., a task may be defined to minimize cost or reduce the cost of performing the task below a defined threshold cost).

[0258] At step 1150, information regarding the task is communicated. In some embodiments, the server communicates task suggestions to one or more assets. For example, the server may communicate task suggestions to a leader drone of the assets determined to perform the task. The task suggestions may be instructions for the leader drone or the assets to perform the task. In some embodiments, the task suggestions may include one or more of task parameters, task constraints, asset suggestions, and the like.

[0259] At step 1155, a user interface for performing the work is displayed. In some embodiments, the server causes the client terminal to display a user interface for displaying the performance of the work. The user interface for performing the work may provide information about the current state of the work, such as a live video stream, an indication of completed work tasks, work tasks that have not yet been performed, an indication of currently being performed tasks, etc. In some embodiments, the user interface for performing the work includes information about the assets. For example, the user may select to drill down to see more granular information about the assets, such as the status of the asset (e.g., offline, broken, online, operating normally, partially operating normally, etc.), a list of the capabilities of the asset or assets, the current set of capabilities of the asset or assets. In some embodiments, the user interface for performing the work comprises one or more selectable elements through which the user can enter user input. Examples of user input may include pausing the work, canceling the work, modifying the work (e.g., modifying parameters of the work), modifying the assets, etc.

[0260] According to various embodiments, the server configures and provides a set of user interfaces, at least some of which are logically connected based at least in part on user input in another user interface. For example, a set of user interfaces may be configured and provided to a user to prompt the user to input information about the job being configured. The information entered by the user may include parameters or characteristics of the job, the type of job, the type of assets utilized, etc. In some embodiments, a user interface in the set of user interfaces used to configure a job is based at least in part on one or more user inputs to a previous user interface in the set of user interfaces.

[0261] FIG 12A illustrates a method for executing a plan associated with an operation according to various embodiments of the present application. In some embodiments, process 1200 of FIG 12A is performed by asset 120, asset 125, and / or asset 130 of system 100 of FIG 1 and / or device 300 of FIG 3. According to various embodiments, process 1000 is performed by a semi-autonomous drone.

[0262] At step 1205, information about the task is received. In some embodiments, an asset (e.g., a leader drone) obtains information about the task from a server (such as a server configuring the task). The information about the task may include suggestions for assets to perform the task and / or one or more parameters or constraints associated with the task. In some embodiments, the information about the task may include a pre-plan, which may be a high-level plan for the leader drone to perform the task. The high-level plan may have a coarser-grained or less specific definition of the tasks to be performed and / or how the tasks will be performed than the plan determined by the leader drone.

[0263] In step 1210, the asset determines to operate as a leader drone based at least in part on the information regarding the operation.

[0264] In step 1215, the leader drone determines a plan to complete at least one task to perform the task. In some embodiments, the leader drone determines one or more tasks of the task and a plan for performing the one or more tasks. As an example, the leader drone may decompose the task into smaller constituent tasks, and the leader drone may determine a plan for performing the tasks.

[0265] In step 1220, the leader drone determines at least one asset to execute at least a portion of the plan. For example, the leader drone may assign the plan or task to an asset in the group of assets to perform the work.

[0266] In step 1225, the leader drone communicates at least a portion of the plan to at least one asset. For example, the leader drone may communicate at least a portion of the plan to assets assigned to the plan and / or task.

[0267] At step 1230, the leader drone determines whether to communicate the portion of the plan to another asset. In response to determining the other portion of the plan to the assets, process 1200 may proceed to step 1225, where the leader drone repeatedly transmits the portion of the plan to different assets each assigned to a portion of the plan or task. The leader drone may repeatedly transmit the portion of the plan to different assets until the leader drone determines at step 1230 that the portion of the plan will not be communicated to another asset.

[0268] At step 1235, the leader drone determines whether to determine a plan for another task and whether to transmit the plan to the asset, etc. In response to the leader drone determining to determine and transmit a plan for another task, process 1200 proceeds to step 1215 where the leader drone repeatedly determines a plan for the task and transmits at least a portion of the plan to one or more assets until the leader drone determines that no future plans need to be determined or transmitted to the asset. In response to determining at step 1235 that a plan for another task does not need to be determined and transmitted, process 1200 proceeds to step 1240.

[0269] At step 1240, the leader drone acquires information during execution of at least a portion of the plan. In some embodiments, the leader drone receives feedback information from one or more follower drones during execution of the corresponding plan. The leader drone may also receive control information from the server regarding the operation. For example, the control information may relate to modifications to parameters of the operation, modifications to the assets, etc.

[0270] In step 1245, the leader drone transmits information regarding the status of the operation. In some embodiments, the leader drone may provide update information regarding the status of the operation to the server. For example, in response to receiving feedback information from one or more follower drones, the leader drone may aggregate feedback from various follower drones and / or determine the status of the operation. The leader drone may transmit the status of the operation and / or information acquired by one or more drones in the assets (e.g., live stream video, images, current location, current status of the implementation of the corresponding plan, etc.) to the server. In some embodiments, the leader drone may transmit information regarding the status of the operation to one or more follower drones. For example, the leader drone may provide update information regarding the status or plan of the operation.

[0271] In step 1250, the leader drone determines whether the task is complete. The leader drone may determine that the task is complete in response to determining that all tasks associated with the task have been completed by the assets. The leader drone may also determine that the task is complete in response to receiving a notification from the server that the task has been canceled or paused. As an example, the task may be canceled or paused in response to a user input to a user interface.

[0272] In response to determining at step 1250 that the task is not completed, process 1200 may proceed to step 1260, where the leader drone determines whether to update the plan. For example, the leader drone may determine to update the plan based on feedback information received from the follower drones and / or control information received from the server, etc. As another example, the leader drone may determine to update the plan based on changes in the task status (e.g., detection of targets, status of delivery of payloads, changes in environmental factors, etc.). In response to determining that the plan is updated, process 1200 may proceed to step 1215. In response to determining that the plan is not updated, process 1200 may proceed to step 1240.

[0273] In response to determining at step 1250 that the work is complete, process 1200 may proceed to step 1255 where status is provided to a server and / or a controlling terminal (eg, a client terminal).

[0274] FIG 12B illustrates a method for executing a plan associated with an operation according to various embodiments of the present application. In some embodiments, the process 1215 of FIG 12B is performed by the server 105, the asset 120, the asset 125, and / or the asset 130 of the system 100 of FIG 1, and / or the device 300 of FIG 3. The process 1215 of FIG 12B may correspond to the step 1215 of FIG 12A.

[0275] At step 1215a, the leader drone obtains information regarding the operation. At step 1215b, the leader drone determines a location where the plan will be performed. At step 1215c, the leader drone determines a task or element for the follower drone to perform at the location. At step 1215d, the leader drone determines one or more characteristics of the environment of the location. At step 1215e, the leader drone determines a model of the area in which at least one task will be performed. According to various embodiments, the model may include a model of the environment in which the assets operate. For example, the model may be a 2D or 3D model of the world in which the assets are operating or performing the operation. At least a portion of the plan is determined at least in part based on the model at least in part.

[0276] According to various embodiments, a planner service implemented in the leader drone may iteratively refine the plan. For example, as the leader drone receives more information (e.g., more specific / detailed information about the task, such as the conditions in which the assets are operating), the leader drone may update its model of the task and the corresponding plan for performing the task based on the current model. As an example, early in the execution of the task, the leader drone may have a simple understanding of the topography of the area in which the assets are operating. As time and the task progresses, the assets can monitor and capture information using various sensors or cameras on the assets of the fleet, and the information may be returned as feedback information to the leader drone.

[0277] FIG 12C illustrates a method for executing a plan associated with an operation according to various embodiments of the present application. In some embodiments, the process 1215 of FIG 12C is performed by the server 105, the asset 120, the asset 125, and / or the asset 130 of the system 100 of FIG 1, and / or the device 300 of FIG 3. The process 1215 of FIG 12B may correspond to the step 1215 of FIG 12A.

[0278] In step 1215g, the leader drone obtains a high-level pre-defined plan (e.g., pre-plan) for the task. For example, the leader drone may receive the pre-plan from a server. In step 1215h, the leader drone obtains information received from one or more assets regarding the implementation of at least a portion of the plan. As an example, the leader drone may receive feedback information obtained by the follower drone during the implementation of the corresponding plan. In step 1215i, the leader drone determines at least one element of the task to be performed by the asset. The leader drone may break down the task into components and determine the functionality required to perform the element (e.g., the functionality mapped to the element). The leader drone may determine an asset with functionality matching the functionality required to perform the element and may assign the element to the asset. In step 1215j, the leader drone determines a model for the implementation of at least that element of the task. In some embodiments, the leader drone determines a model of the environment in which the asset will perform the element. As an example, if the element is to track a car on a road, the leader drone may generate a model of the environment including the roads, the topology of the area, the relative positions of buildings in the area, the relative positions of trees and other objects in the area, and the position of the car in the area. At step 1215k, the leader drone determines a lower level plan for the task. The lower level plan may include more detail or finer granularity than the pre-plan received from the server. For example, the pre-plan for the task may indicate that a car will be tracked in a given area, and the lower level plan may include elements for implementing the plan, such as (i) tracking the car as it travels on roads in the given area, (ii) moving a given distance away from the vehicle being tracked, (iii) capturing images of the car, (iv) sending updates about the status of the tracking of the car at a particular time period, and (v) sending an alert if the car stops at a certain location and providing a suggestion and timestamp of the stop.

[0279] FIG 13 illustrates a method for executing a plan associated with an operation, according to various embodiments of the present application. In some embodiments, process 1300 of FIG 13 is performed by asset 120, asset 125, and / or asset 130 of system 100 of FIG 1 and / or device 300 of FIG 3. According to various embodiments, process 1000 is performed by a semi-autonomous drone. Process 1300 may be performed by a leader drone.

[0280] At step 1305, the drone obtains information regarding a higher level task or operation to be performed. At step 1310, the drone may obtain information regarding an environment in which at least a portion of the task or operation will be performed. At step 1315, the drone determines a set of assets that will perform at least a portion of the task. At step 1320, it determines whether to split up the set of assets (e.g., a group or team of assets) that will perform a portion of the task.

[0281] In response to the drone determining at step 1320 that the assets will not be split, process 1300 may proceed to step 1325. At step 1325, the drone sends instructions to the assets to perform at least a portion of the task. At step 1330, the drone receives feedback information. For example, the leader drone may receive feedback information from one or more follower drones. At step 1335, the leader drone determines whether to update the plan. For example, the leader drone may determine whether to update the plan based at least in part on the feedback information.

[0282] In response to determining to update the plan, process 1300 proceeds to step 1340 where the plan is updated. Updating the plan may include updating a model of the environment in which the follower drones are operating and determining whether their ability to perform the task or elements of the task has changed based on the model update or changes in the context of the operation. At step 1345, the leader drone transmits the updated plan. As an example, the leader drone may transmit the updated plan to follower drones assigned to the previous version of the plan (or the task corresponding to the plan). As another example, if the plan update includes assigning new, different, or additional assets to the task, the leader drone may transmit the updated plan to such assets. Process 1300 then returns to step 1330.

[0283] In response to a determination at step 1335 that the plan is not updated, process 1300 may proceed to step 1350 where the leader drone determines whether the task is complete. In response to a determination that the task is not complete, process 1300 may return to step 1305 and process 1300 may be repeated again. In response to a determination that the task is complete, process 1300 may end.

[0284] In response to a determination at step 1320 to split the assets to perform the portions of the task, process 1300 may proceed to step 1335, where a leader of the partition (e.g., a partition leader) is determined. At step 1360, the leader drone may send an instruction to the partition leader to perform the corresponding portion of the task. Process 1300 may then proceed to step 1330. In response to the partition leader receiving the instruction to perform the corresponding portion of the task, the partition leader may perform process 1300 as the leader drone for the partition.

[0285] Figure 14 illustrates a method for executing a plan associated with an operation, according to various embodiments of the present application. In some embodiments, process 1400 of Figure 14 is performed by asset 120, asset 125, and / or asset 130 of system 100 of Figure 1 and / or device 300 of Figure 3. According to various embodiments, process 1000 is performed by a semi-autonomous drone.

[0286] At operation 1405, a drone receives information regarding the operation. In some embodiments, the drone receives a plan for performing a portion of the operation (e.g., a task or an element of a task). The drone may receive instructions to perform the portion of the operation as a partition that includes a portion of the assets assigned to perform the operation. According to various embodiments, the information regarding the operation includes an indication that the drone is assigned to be a partition leader for the partition.

[0287] At step 1410, the drone determines to act as a leader drone for the partition (e.g., a partition leader) based at least in part on the information.

[0288] In step 1415, the partition leader determines a plan for completing at least one task of the work. In some embodiments, the partition leader determines one or more plans for completing one or more tasks assigned to the partition. The partition leader may determine the plan based at least in part on one or more of parameters and / or constraints of the work (or tasks assigned to the partition), capabilities or features associated with the at least one task, capabilities of one or more assets in the partition, etc. According to various embodiments, the partition leader implements a planning service similar or the same as the planner service implemented by a leader drone of the group of assets assigned to perform the work.

[0289] In step 1420, the partition leader determines at least one asset to which at least a portion of the plan is to be communicated. In some embodiments, the partition leader assigns at least a portion of the plan to an asset in the partition. The partition leader may determine an asset in the partition to which at least a portion of the plan is to be assigned. For example, the asset may be determined based on a function of the asset and / or a function associated with a task or element that corresponds to at least a portion of the plan.

[0290] In step 1425, the partition leader communicates at least a portion of the plan to the at least one asset. In some embodiments, in response to determining an asset in the partition to which the portion of the plan is assigned, the partition leader sends the portion of the plan to the at least one asset.

[0291] At step 1430, the partition leader determines whether to communicate the portion of the plan to another asset in the partition. In response to determining that the other asset of the plan is assigned a portion of the plan, process 1400 proceeds to step 1420, where the partition leader repeatedly performs steps 1420, 1425, and 1430 until the partition leader determines not to communicate the portion of the plan to another asset.

[0292] At step 1435, the partition leader determines whether to determine a plan for another task and whether to transmit the plan to assets, etc. In response to the partition leader determining to determine and transmit a plan for another task, process 1400 proceeds to step 1415 where the leader drone repeatedly determines a plan for the task and transmits at least a portion of the plan to one or more assets until the partition leader determines that no future plans need to be determined or transmitted to assets. In response to determining at step 1435 that a plan for another task does not need to be determined and transmitted, process 1400 proceeds to step 1440.

[0293] At step 1440, the partition leader obtains information during execution of at least a portion of the plan. In some embodiments, the partition leader receives feedback information from one or more follower drones (in the partition) during execution of the corresponding plan. The partition leader may also receive control information regarding the operation from the server (or leaders of assets assigned to the operation). For example, the control information may relate to modifications to parameters of the operation, modifications to assets, etc.

[0294] In step 1445, the partition leader transmits information about the status of the work. In some embodiments, the partition leader may provide an update about the status of the work to the server. For example, in response to receiving feedback information from one or more follower drones, the partition leader may aggregate the feedback from the various follower drones and / or determine the status of the work. The partition leader may transmit the status of the work (or its tasks) and / or information acquired by one or more drones in the assets (e.g., live stream video, images, current location, current status of the implementation of the corresponding plan, etc.) to the server. In some embodiments, the partition leader may transmit information about the status of the work to one or more follower drones. For example, the leader drone may provide an update about the status or plan of the work. In some embodiments, the partition leader provides information about the status of the work to a leader drone (e.g., the leader of the assets that were initially assigned to perform the work), which in turn provides information about the status of the work to the server.

[0295] In step 1450, the partition leader determines whether the portion of work corresponding to the partition is complete. The leader drone may determine that the portion of work is complete in response to determining that all tasks associated with the portion of work have been completed by the assets. The partition leader may also determine that the portion of work is complete in response to receiving a notification from the server (or leader drone) that the work has been canceled or paused. As an example, the work may be canceled or paused in response to a user input to a user interface.

[0296] In response to determining at step 1450 that the task is not completed, process 1440 may proceed to step 1460 where the leader drone determines whether to update the plan. For example, the partition leader may decide to update the plan based on feedback information received from the follower drones and / or control information received from the server, etc. As another example, the partition leader may decide to update the plan based on changes in the status of the task (e.g., detection of targets, status of delivery of payloads, changes in environmental factors, etc.). In response to determining that the plan is updated, process 1400 may proceed to step 1415. In response to determining that the plan is not updated, process 1400 may proceed to step 1240.

[0297] In response to determining at step 1450 that the work is complete, process 1400 may proceed to step 1455 where status is provided to a server and / or a controlling terminal (eg, a client terminal).

[0298] FIG 15 illustrates a method for executing a plan associated with an operation, according to various embodiments of the present application. In some embodiments, process 1500 of FIG 15 is performed by asset 120, asset 125, and / or asset 130 of system 100 of FIG 1 and / or device 300 of FIG 3. According to various embodiments, process 1000 is performed by a semi-autonomous drone.

[0299] At operation 1505, the drone receives information about the operation. In some embodiments, the drone receives a plan for performing the operation (e.g., a task or an element of a task). In some embodiments, the drone receives information about the operation from a leader drone of the assets. The information about the operation may include a suggestion of the assets to perform the operation. The information about the operation may include a pre-defined leader ranking.

[0300] At step 1510, the drone determines to operate as a dormant leader drone based at least in part on the information about the task. The drone may determine to operate as a dormant leader based on a pre-determined leader ranking. For example, the dormant leader may be the drone with the second highest ranking (e.g., among active assets). In some embodiments, the information about the task is an instruction for the drone to act as a dormant leader.

[0301] In operation 1515, the dormant leader drone communicates at least a portion of the plan with at least one asset. In some embodiments, the dormant leader drone receives information about the plan from the leader drone. The dormant leader drone may synchronize information about the operation stored by the leader drone. For example, the dormant leader may have a corresponding shadow copy of the leader drone.

[0302] In step 1520, the dormant leader drone executes the corresponding portion of the plan. In some embodiments, the dormant leader drone executes one or more tasks or elements in connection with performing the work. In some examples, the dormant leader drone does not determine or execute the plan. For example, the dormant leader drone may function as a backup for the leader drone.

[0303] In step 1525, the dormant leader drone acquires information during execution of at least a portion of the plan. In some embodiments, the information acquired during execution of at least a portion of the plan includes feedback information provided by the follower drone, synchronization information provided by the leader drone, and / or information acquired by one or a sensor of the dormant leader drone. In addition to sending feedback information to the leader drone, the follower drone may send feedback information to the dormant leader drone.

[0304] At step 1530, the dormant leader drone determines whether to operate as a leader. In some embodiments, the dormant leader drone determines to operate as a leader in response to determining that the leader drone has failed. As an example, the leader drone may be deemed to have failed if it has lost communication with the dormant leader drone or assets for a predetermined threshold period, etc. As another example, the leader drone may be deemed to have failed if the leader drone has a communication link with the assets and the communication link falls below a predetermined quality of service threshold. If the leader drone is unable to communicate with the dormant leader drone or assets for a predetermined threshold period, the dormant drone or follower drone may send a ping to check the health of the leader drone. If the leader drone fails to respond to the health check within a set period, the leader drone may be deemed to have failed.

[0305] In response to the dormant leader drone determining not to act as a leader, process 1500 proceeds to step 1535 where information regarding the status of the operation is transmitted to the leader drone.

[0306] In step 1540, the dormant leader drone determines whether an indication that the task is complete has been received. In response to determining an indication that the task is complete, the process 1500 ends. Conversely, in response to determining that an indication that the task is complete has not been received, the process 1500 proceeds to step 1545 where the dormant leader drone determines whether an updated plan has been received. In response to determining that an updated plan has been received, the process 1500 proceeds to step 1520. In response to determining that an updated plan has not been received, the process 1500 proceeds to step 1525.

[0307] If at step 1530 it is determined that the dormant leader drone will act as the leader, process 1500 proceeds to step 1550 where the dormant leader drone loads information about the task as the leader. In some embodiments, the dormant leader drone loads a synchronized copy of the leader information about the task. For example, the dormant leader drone may load a backup copy of the information about the task that was backed up from a local copy at the leader drone.

[0308] In step 1555, the leader drone (e.g., a previously dormant leader drone) determines a plan for completing at least a portion of the task.

[0309] At step 1560, the leader drone determines which assets to complete at least a portion of the plan.

[0310] At step 1565, the leader drone communicates at least a portion of the plan to the asset.

[0311] At step 1570, the leader drone determines whether to communicate a portion of the plan to another asset. In response to determining that a portion of the plan is to be communicated to another asset, the process 1500 returns to step 1560. Conversely, in response to determining that a portion of the plan is not to be communicated to another asset, the process 1500 proceeds to step 1570 where the leader drone determines whether there is another task (e.g., a plan is to be determined and assigned) at step 1575. In response to determining at step 1575 that there is another task, the process 1500 returns to step 1555. In response to determining at step 1575 that there is no another task, the process 1500 proceeds to step 1580 where the leader drone obtains information during the execution of at least a portion of the plan. For example, the leader drone may receive feedback information from the follower drone.

[0312] In step 1585, the leader drone transmits information about the status of the work. The leader drone may transmit information about the status of the work to a server (e.g., a control center) and / or a client terminal. The leader drone may transmit information about the status of the work to one or more follower drones.

[0313] At step 1590, the leader drone determines whether the task is complete. In response to determining at step 1590 that the task is complete, process 1500 ends. Conversely, in response to determining at step 1590 that the task is not complete, process 1500 proceeds to step 1595, where the leader drone determines whether the plan is updated. As an example, the task is determined to be not complete in response to determining that the task is not complete (e.g., if a drone is tasked with scanning a road and the drone is unable to maintain a series of sharp turns in the road and misses scanning a portion of the road, the task is deemed not complete and the drone may turn back and rescan the missed portion of the road). In response to the determination at step 1595 to update the plan, process 1500 proceeds to step 1555. Conversely, in response to determining that the plan is not updated, process 1500 proceeds to step 1580.

[0314] 16A is a diagram illustrating a discrete representation according to various embodiments of the present application. In the example shown in FIG. 16A, the discrete representation 1600 includes a plurality of discrete elements corresponding to voxels. The discrete representation 1600 may be implemented by the system 100 of FIG. 1, the device 300 of FIG. 3, etc.

[0315] As used herein, the term "voxel" corresponds to a value on a regular grid in three-dimensional space, e.g., a voxel corresponds to a discrete element of a discrete representation.

[0316] According to various embodiments, each discrete element (e.g., cell) in the 3-D representation corresponds to a different portion of the geographic location. For example, the portion of the geographic location is mapped to a corresponding discrete element in the 3-D representation in connection with determining the discrete representation 1600. As shown in FIG. 16A, the 3-D dimensions of the discrete representation 1600 correspond to an x-axis, a y-axis, and a z-axis. The z-axis corresponds to a direction parallel to gravity. As an example, the number of discrete elements provided on the z-axis is determined at least in part based on a maximum altitude or above sea level associated with the configuration of the operation (e.g., parameters entered by a user during configuration of the operation, a maximum altitude capability of an asset in the group of assets selected to perform the operation, a default maximum altitude or above sea level, etc.).

[0317] 16A, sets of information 1610 are each mapped or associated with a set of discrete elements of discrete representation 1600. In some embodiments, each discrete element in discrete representation 1600 has information mapped to it. For example, a discrete element has one or more metadata fields that each correspond to a different type of information mapped to it. Examples of the types of information that may be mapped or associated with a discrete element include an indication of whether the discrete element corresponds to a location identified as a keep-in area, an indication of whether the discrete element corresponds to a location identified as a keep-out area, an indication of whether the discrete element is occupied, an indication of whether the discrete element is available or unoccupied, an indication of whether an asset within the discrete element has a clear line of sight (e.g., to the current location of a leader drone, tower, or ground station), an indication of whether a target is included in the discrete element, an identifier of a target included in the discrete element, an indication of whether the discrete element has or is experiencing inclement weather, or the type of weather (e.g., sunny, rainy, wind, temperature, wind direction and speed, etc.).

[0318] In general, the discretization of the terrain breaks down as distance increases and the curvature of the Earth becomes significant or important to the calculation. For example, if each discrete element in the 3-D representation corresponds to a volume of the same size, the curvature of the Earth will cause the bottom of the 3-D representation to correspond to a plane that is tangent to the Earth at a particular intersection point, and as the 3-D representation extends further in the x or y direction, the 3-D representation will deviate more and more from the Earth's surface. In some embodiments, the discrete representation 1600 is generated based at least in part on the curvature of the Earth. For example, the system takes the curvature of the Earth into account when generating the discrete representation 1600 in a way that transforms real space to the 3-D representation such that the z-axis corresponds to the direction of gravity (or a direction that is perpendicular to the Earth's surface at the location corresponding to the discrete element). As another example, the z-axis corresponds to the direction of gravity, regardless of how far the 3-D representation is transversely shifted from the origin in the x or y direction.

[0319] According to various embodiments, the discrete representation is used in connection with modeling one or more tasks. For example, the system references (e.g., searches) the discrete representation and related information (e.g., metadata including fields of geographic location and / or information related to one or more tasks). As an example, the system determines a first discrete element corresponding to a first point, determines a second discrete element corresponding to a second point, and connects the first and second discrete elements to determine a path from the first point to the second point through a set of contiguous (e.g., adjacent) discrete elements through which the asset can pass (e.g., the discrete elements are unoccupied, etc.). To determine the set of sets of contiguous (e.g., adjacent) discrete elements, the system searches across the discrete representation to find / determine a set of contiguous assets. In some embodiments, discrete elements that are shown to be unoccupied are searchable in connection with determining a plan (e.g., flight path) and discrete elements that are shown to be occupied are not searchable. As an example, the plan is determined based on the occupancy of one or more discrete elements of the discrete representation.

[0320] An operation (e.g., one or more tasks performed by a fleet of one or more assets) typically extends over a distance in at least the x and y directions. Thus, a model of a geographic location corresponding to an operation (e.g., one or more tasks performed by a fleet of one or more assets) includes a large discrete representation and a corresponding large number of discrete elements. According to various embodiments, to aid in rapid processing of the discrete representation 1600 in connection with determining a plan (e.g., a plan for orienting follower drones, flight paths of assets, etc.), the system creates a discrete representation that limits the extent to which the discrete elements extend in the z direction. For example, because altitude / above sea level is typically limited according to a configured operation (e.g., user input, asset capabilities, and / or default settings), the discrete representation 1600 may be similarly limited to limit the number of cells processed. In some embodiments, the discrete representation 1600 comprises a 400×400×100 grid segment. As an example, discrete representation 1600 comprises 400 discrete elements in a dimension along the x-direction, 400 discrete elements in a dimension along the y-direction, and 100 discrete elements in a dimension along the z-direction. Discrete representations 1600 of various other dimensions may be implemented. For example, the number of discrete elements in discrete representation 1600 (e.g., in all dimensions or in a single dimension) is configurable according to user input, etc.

[0321] In some embodiments, modeling of the geographic location / environment associated with determining a plan for one or more tasks uses a discrete representation of the geographic location environment having the same dimensions, such as the same number of discrete elements in one or more directions, regardless of the size of the geographic location (e.g., user-defined geographic location) corresponding to the task or one or more tasks. Thus, the larger the geographic location, the coarser the granularity of the discrete representation. Conversely, the smaller the geographic location, the finer the granularity of the discrete representation. The resolution of the modeling (e.g., discrete representation) increases as the geographic location being modeled becomes smaller. Thus, in some embodiments, when a leader drone is assigned a particular portion of a task / element of a task and a follower drone is deployed to perform a portion of the task / element, the follower drone generates a model of the geographic location corresponding to the particular portion of the task / element assigned to the follower drone. As an example, a model of geographic location for a particular subset of tasks / elements assigned to a follower drone (e.g., discrete representation 1600, such as the discrete representation generated by the follower drone) has higher resolution than a model of geographic location for the work (e.g., a superset of one or more tasks that a leader drone manages and directs follower drones to perform).

[0322] 16B is a diagram illustrating a discrete representation of a geographic location according to various embodiments of the present application. In the example shown in FIG. 16B, the discrete representation 1650 is populated with geographic information. For example, the discrete representation 1650 includes representations of trees, roads, cars, hills, and a ground control station 1620. Similarly, the discrete representation 1650 is annotated with one or more parameters related to the geographic location with at least a portion of the discrete elements. For example, the set of information 1610 includes a set of information fields, and the set of information 1610 is populated with information related to the geographic location and mapped to the discrete representation 1650. As an example, as shown in FIG. 16B , the set of information 1610 may include (i) information indicating whether one or more discrete elements correspond to a keep-in area, (ii) information indicating whether one or more discrete elements are occupied or unoccupied, (iii) information indicating whether one or more discrete elements correspond to a clear line of sight (e.g., a clear communication / radio line), (iv) information indicating whether one or more discrete elements include a target, (v) information indicating whether one or more discrete elements are exposed to inclement weather, and the like.

[0323] FIG. 16C illustrates a diagram of a discrete representation of geographic locations, according to various embodiments of the present application. As shown, a group of drones (e.g., D1 1630 and D2 1640, etc.) are deployed at geographic locations corresponding to the discrete representation 1675. As an example, a task includes tracking / monitoring a target, for example, at the discrete element (x=2, y=4, z=0). According to various embodiments, the leader drone D1 1630 provides a plan to the follower drone D2 1640 for tracking / monitoring the target 1670. As an example, the leader drone D1 1630 provides a flight plan 1680 to the follower drone D2 1640. As another example, the leader drone D2 1630 provides a suggestion for the follower drone D2 1640 to perform the task of tracking / monitoring the target 1670, and the follower drone D2 1640 determines a flight plan 1680 for moving to a position where the follower drone D2 1640 has line of sight to the target.

[0324] In the example shown in FIG. 16C, the target 1670 is not in the line of sight of the ground control station 1620 or the leader drone D1 1630. For example, a hill / mountain is located between the ground control station 1620 and the target 1670. In some embodiments, to perform the task of monitoring the target 1670, the follower drone D2 1640 is deployed to move to a position with line of sight to the target 1670. The system determines a flight plan 1680 for the follower drone D2 1640 to move along. As shown in FIG. 16C, the flight plan 1680 is configured to direct the follower drone D2 1640 to fly over trees and around the hills. In some embodiments, a cost function associated with moving the drone along a path indicates that moving the drone vertically to an altitude sufficient to fly over the hills is more costly than moving the drone laterally in the x and / or y directions to get around the hills. As an example, moving in the x and / or y directions is more efficient for the drone than moving in the z direction (e.g., increasing altitude). Thus, longer flight plans that involve travel in the x and / or y directions are more efficient (e.g., less costly) than shorter flight plans that require travel in the z direction.

[0325] 17A is a diagram illustrating a discrete representation of a geographic location according to various embodiments of the present application. In the example shown in FIG. 17A, the discrete representation 1700 includes a plurality of discrete elements corresponding to voxels. The discrete representation 1700 may be implemented by the system 100 of FIG. 1, the device 300 of FIG. 3, etc.

[0326] According to various embodiments, the system repeatedly updates the discrete representation 1700 during the life of the task (e.g., the time until the task is paused, completed, or terminated). As an example, the discrete representation 1700 is generated at the start of the task. When the task is started, such as at the ground control station 1710, information regarding the geographic location and / or one or more tasks associated with the task is added to the discrete representation 1700 and associated set of information 1705 (e.g., metadata associated with one or more discrete elements of the discrete representation 1700). In some embodiments, the discrete representation 1700 is annotated with the set of information 1705.

[0327] According to various embodiments, the information included in set of information 1705 is obtained via one or more pre-stored configurations (e.g., definitions) of operations on a server or a local service (e.g., a map or geographic service, a service that manages various deployed assets) and / or a third-party service (e.g., a weather service, a map service, etc.).

[0328] 17A, only a portion of the discrete representation 1700 is filled with corresponding information, e.g., only the portion 1715 that is visible from ground control station 1710 is provided. A hill 1720 blocks the view to the feature / information discrete element on the other side of the hill 1720.

[0329] FIG. 17B illustrates a discrete representation of geographic locations, according to various embodiments of the present application. In the example illustrated in FIG. 17B, a leader drone D1 1730 and a follower drone D2 1735 are deployed. For example, the leader drone D1 1730 and the follower drone D2 1735 are deployed in association with the initiation of a task or one or more tasks associated with the task. In some embodiments, the leader drone D1 1730 commands the follower drone D2 1740 to move to a position 1745 to observe / monitor an area opposite 1720 (e.g., to identify a target, etc.). A flight plan 1740 for the follower drone D2 1735 is determined, and the follower drone D2 1735 moves along the path of the flight plan 1740 (e.g., to (x=0, y=0, z=3)). In some embodiments, the set of information 1705 associated with the discrete representation 1725 is updated in response to (or during) the follower drone D2 1735 moving along the path of the flight plan 1740. For example, the follower drone D2 1735 communicates (e.g., to the leader drone D1 1730) feedback information regarding one or more tasks and / or geographic location. As shown in FIG. 17B, the discrete representation 1725 is updated to include more information regarding the road 1715. Additionally, the set of information 1705 is updated to include information associated with a discrete element located at (x=0, y=0, z=3). In some embodiments, the annotated representation is updated locally in each of the follower drone D2 1735 and the leader drone D1 1730.

[0330] 17C is a diagram illustrating a discrete representation of a geographic location according to various embodiments of the present application. As shown in FIG. 17C, when the follower drone D2 1735 moves to a discrete location located at (x=0, y=2, z=3), the discrete representation 1750 and / or the set of information 1705 are updated (e.g., the annotated representation is updated). For example, when the follower drone D2 1735 moves to (x=0, y=2, z=3), the follower drone D2 has line of sight to more roads 1715 and trees 1755. The follower drone D2 1735 updates the annotated representation locally at the follower drone D2 1735 and / or communicates feedback information to the leader drone D1 1730, which correspondingly updates the locally stored discrete representation 1750 and / or the set of information 1705 (e.g., the annotated representation stored in the leader drone D1 1730).

[0331] 17D is a diagram illustrating a discrete representation of a geographic location according to various embodiments of the present application. As shown in FIG. 17D, when the follower drone D2 1735 moves to a discrete location located at (x=0, y=3, z=4), the discrete representation 1775 and / or the set of information 1705 are updated (e.g., the annotated representation is updated). For example, when the follower drone D2 1735 moves to (x=0, y=3, z=4), the follower drone D2 has line of sight to more of the road 1715, and the follower drone D2 1735 identifies the target 1780 moving on the road 1715. Follower drone D2 1735 updates the annotated representation locally at follower drone D2 1735 and / or communicates feedback information to leader drone D1 1730, which correspondingly updates the locally stored discrete representation 1775 and / or set of information 1705 (e.g., the annotated representation stored in leader drone D1 1730). As an example, the set of information 1705 annotating the discrete representation 1775 includes an entry identifying that a discrete element located at (x=2, y=5, z=0) includes a target (e.g., target 1780).

[0332] FIG. 18 illustrates a method for determining a plan for performing one or more tasks, according to various embodiments of the present application. According to various embodiments, process 1800 is performed at least in part by system 100 of FIG. 1 and / or device 300 of FIG. 300. In an embodiment, process 1800 is performed in connection with determining or updating a plan. Process 1800 may be performed during the execution of an operation. Process 1800 is performed by a leader drone and / or a follower drone of a group of assets assigned to perform one or more tasks.

[0333] At step 1810, data associated with the one or more tasks is obtained. According to various embodiments, obtaining the data associated with the one or more tasks includes one or more of step 710 of process 700 of FIG. 7A, step 835 of process 835 of FIG. 8C, step 931 of process 930 of FIG. 9B, step 950 of process 900 of FIG. 9A, step 952-1 of process 950 of FIG. 9E, step 1010 of process 1000 of FIG. 10A, step 1021 of process 1020 of FIG. 10B, step 1215a of process 1215 of FIG. 12B, step 1305 and / or step 1310 of process 1300 of FIG. 13, step 1405 and / or step 1440 of process 1400 of FIG. 14, and / or step 1505 and / or step 1525 of process 1500 of FIG. 15.

[0334] In some embodiments, the data associated with the one or more tasks is associated with the operation or the geographic location where the one or more tasks are performed, and / or one or more characteristics or parameters associated with the operation. As an example, the data associated with the one or more tasks is received from a server (e.g., a server providing an operation control service) along with the configuration of the operation (e.g., based on user input, etc.). As another example, the data associated with the one or more tasks is received from another asset in the fleet of assets performing the operation. In the case of a leader drone, the leader drone receives the data associated with the one or more tasks from a follower drone, such as feedback information regarding updates to the status of the execution of the one or more tasks. In the case of a follower drone, the follower drone receives the data associated with the one or more tasks from the leader drone and / or another asset in the fleet of assets. As another example, the data associated with the one or more tasks is received from a third party service, such as a server providing services / information regarding the operation (e.g., weather services), information regarding other assets deployed in the geographic location, mapping or geographic services, etc.

[0335] At step 1820, a discrete representation of the geographic location is determined. According to various embodiments, the discrete representation corresponds to discrete representation 1600 of FIG. 16A and / or discrete representation 1700 of FIG. 17A. The discrete representation is generated at least in part based on at least a portion of the data associated with one or more tasks. In some embodiments, determining the discrete representation includes converting the real-world representation of the geographic location into a predetermined number / dimensions of discrete elements. For example, the volume or amount of space represented by a particular discrete element is based at least in part on the size of the geographic location for which the discrete representation is generated.

[0336] At step 1830, the discrete representation is annotated. In some embodiments, the discrete representation is annotated based at least in part on (e.g., to include) one or more parameters related to the geographic location and / or information related to one or more tasks. As an example, the discrete representation is annotated in response to a determination that one or more parameters relate to the geographic location. In some embodiments, annotating the discrete representation includes setting / adding information regarding at least some of the discrete elements in the discrete representation (e.g., setting the set of information 1610 of FIG. 16A and / or the set of information 1705 of FIG. 17A, etc.).

[0337] According to various embodiments, the discrete representation is annotated in connection with creating an annotated representation (e.g., an annotated representation of a geographic location). As an example, annotating the discrete representation includes setting or associating metadata with one or more discrete elements of the discrete representation.

[0338] At step 1840, a plan for performing the one or more tasks is determined. According to various embodiments, the plan for performing the one or more tasks is determined based at least in part on the annotated representation. The determination of the plan is based on the one or more tasks to be performed (e.g., one or more features associated with the one or more tasks) as well as one or more parameters associated with the geographic location. As an example, the one or more parameters associated with the geographic location are included in the annotated representation.

[0339] At step 1850, information regarding the plan is communicated. In some embodiments, information regarding the plan is transmitted to other assets in the constellation of assets. As an example, in the case of a leader drone, the leader drone transmits information regarding the plan to a ground control station and / or one or more follower drones, and the leader drone receives information, such as feedback information, from the ground control station and / or one or more follower drones. As another example, in the case of a follower drone, the follower drone transmits information regarding the plan to the leader drone and / or one or more other assets in the constellation of assets performing one or more tasks, and the follower drone receives update information from the leader drone and / or one or more other assets in the constellation of assets.

[0340] At step 1860, a determination is made as to whether updated information has been received. According to various embodiments, the drone may receive updated information from another asset in the fleet or from a server.

[0341] In some embodiments, the updated information is associated with a geographic location where the operation or one or more tasks are performed, and / or one or more features or parameters associated with one or more of the operations or tasks. As an example, the updated information associated with one or more tasks is received from a server (e.g., a server providing an operation control service) along with changes to the operation (e.g., based on user input, etc.). As another example, the data associated with the one or more tasks is received from another asset in the fleet of assets performing the operation. In the case of a leader drone, the leader drone receives updated information from the follower drone, such as feedback information regarding updates to the status of the execution of the one or more tasks, or information of a portion of the geographic location detected by a sensor or camera of the follower drone. In the case of a follower drone, the follower drone receives data associated with the one or more tasks from the leader drone and / or another asset in the fleet of assets. As another example, the data associated with the one or more tasks is received from a third party service, such as a server providing services / information regarding the operation (e.g., weather services), information about other assets deployed at the geographic location, mapping or geographic services, flight plans of other assets, etc.

[0342] In response to determining at step 1860 that updated information has been received, process 1800 returns to step 1830. In some embodiments, the annotated representation is repeatedly updated with the updated information in response to determining that updated information has been received.

[0343] In response to determining at step 1860 that updated information has not been received, process 1800 proceeds to step 1870 where a determination is made as to whether the process is complete. As an example, process 1800 may be determined to be complete based at least in part on user input (such as an input to cancel or pause the task). As another example, process 1800 may be determined to be complete in response to a user selecting to end the task. As another example, process 1800 may be determined to be complete in response to completion of the task (e.g., completion of one or more tasks associated with the task). If the process is deemed complete, process 1800 ends. Otherwise, process 1800 returns to step 1860 where the process checks / monitors for updated information.

[0344] FIG. 19A illustrates a method for determining a flight plan, according to various embodiments of the present application. According to various embodiments, process 1900 is performed at least in part by system 100 of FIG. 1 and / or device 300 of FIG. 300. In an embodiment, process 1900 is performed in connection with determining or updating a plan (e.g., a flight plan). Process 1900 may be performed during the execution of an operation. Process 1900 is performed by a leader drone and / or a follower drone of a group of assets assigned to perform one or more tasks.

[0345] At step 1910, data associated with the one or more tasks is obtained. According to various embodiments, obtaining the data associated with the one or more tasks includes one or more of step 710 of process 700 of FIG. 7A, step 835 of process 835 of FIG. 8C, step 931 of process 930 of FIG. 9B, step 950 of process 900 of FIG. 9A, step 952-1 of process 950 of FIG. 9E, step 1010 of process 1000 of FIG. 10A, step 1021 of process 1020 of FIG. 10B, step 1215a of process 1215 of FIG. 12B, step 1305 and / or step 1310 of process 1300 of FIG. 13, step 1405 and / or step 1440 of process 1400 of FIG. 14, and / or step 1505 and / or step 1525 of process 1500 of FIG. 15. Acquiring data along with one or more tasks may be similar to step 1810 of process 1800 of FIG.

[0346] At step 1920, a discrete representation of the geographic location is determined. According to various embodiments, the discrete representation corresponds to the discrete representation 1600 of Figure 16A and / or the discrete representation 1700 of Figure 17A. In some embodiments, the discrete representation is determined in a manner similar to step 1820 of process 1800 of Figure 18.

[0347] At step 1930, the discrete representation is annotated. In some embodiments, the discrete representation is annotated based at least in part on (e.g., to include) one or more parameters related to the geographic location and / or information related to one or more tasks. As an example, the discrete representation is annotated in response to a determination that one or more parameters relate to the geographic location. In some embodiments, annotating the discrete representation includes setting / adding information about at least some of the discrete elements in the discrete representation (e.g., setting the set of information 1610 of FIG. 16A and / or the set of information 1705 of FIG. 17A, etc.). In some embodiments, the discrete representation is annotated in a manner similar to step 1830 of process 1800 of FIG. 18.

[0348] According to various embodiments, annotating the discrete representation includes registering one or more flight plans associated with another asset in the fleet. For example, in response to receiving a flight plan from a drone in the fleet, the discrete representation is annotated to associate the flight plan with discrete elements of the discrete representation that are intersected / occupied by the flight plan. As another example, the discrete representation is annotated to set discrete elements that are affected by the discrete representation (e.g., discrete elements that are intersected / occupied by the flight plan) to set an indication that such discrete elements were occupied for at least the time period associated with the flight plan or until the time that the drone associated with the flight plan passed through the particular discrete element. The set of discrete elements affected by the flight plan is updated as the corresponding drone executes the flight plan (e.g., to set discrete elements that the drone passed through as unoccupied, to release discrete elements for use by another asset, etc.).

[0349] At step 1940, a flight plan for the drone is determined. According to various embodiments, the flight plan is determined based at least in part on the annotations of the discrete representation (e.g., the annotated representation). In some embodiments, the drone determines the flight plan based on a geographic location and / or parameters associated with one or more tasks. For example, the drone determines the flight plan based on a current location (e.g., a discrete element corresponding to the current location), a destination location (e.g., a discrete element corresponding to the destination location), and parameters associated with one or more tasks (e.g., a type of task assigned to the drone). As another example, the flight plan is further based on one or more parameters of a set of discrete elements in the discrete representation (e.g., a discrete element between the current location and the destination location, a series of discrete elements along the flight plan connecting the current location and the destination location, etc.).

[0350] In some embodiments, the leader drone determines multiple flight plans, such as plans for different follower drones in the asset constellation, the leader drone plans multiple flight plans to avoid each other (e.g., to ensure collision avoidance among the asset constellation), and so on.

[0351] At step 1950, information associated with the flight plan is communicated. According to various embodiments, in response to determining the flight plan, the flight plan is communicated to one or other assets in the asset group. As an example, the flight plan is published to the other assets in the asset group (e.g., on an information feed / channel for information regarding the flight plan or one or more tasks). The information associated with the flight plan is communicated to other assets in the asset group (e.g., the leader drone, the follower drone, etc., in the case where the follower drone communicates the information) to enable the other assets to store such information locally and / or update their respective annotated representations of the geographic locations to indicate that the discrete elements corresponding to the flight plan are occupied during the time that the flight plan is predicted to affect the corresponding discrete elements or until the next communication indicates updated information about the drone's flight status and provides clearing of the discrete elements corresponding to the previous portion of the flight plan.

[0352] A determination is made as to whether updated information has been received at step 1960. According to various embodiments, the drone may receive updated information from another asset in the fleet or from a server.

[0353] In some embodiments, the updated information is associated with a geographic location where the operation or one or more tasks are performed, and / or one or more features or parameters associated with one or more of the operations or tasks. As an example, the updated information associated with one or more tasks is received from a server (e.g., a server providing an operation control service) along with changes to the operation (e.g., based on user input, etc.). As another example, the data associated with the one or more tasks is received from another asset in the fleet of assets performing the operation. In the case of a leader drone, the leader drone receives updated information from the follower drone, such as feedback information regarding updates to the status of the execution of the one or more tasks, or information of a portion of the geographic location detected by a sensor or camera of the follower drone. In the case of a follower drone, the follower drone receives data associated with the one or more tasks from the leader drone and / or another asset in the fleet of assets. As another example, the data associated with the one or more tasks is received from a third party service, such as a server providing services / information regarding the operation (e.g., weather services), information about other assets deployed at the geographic location, mapping or geographic services, flight plans of other assets, etc.

[0354] In response to determining at step 1960 that updated information has been received, process 1900 returns to step 1930. In some embodiments, the annotated representation is iteratively updated with the updated information in response to determining that updated information has been received. In response to the iterative updates of the annotated representation, the flight plan is iteratively determined / updated (e.g., a decision is made as to whether to keep the flight plan the same or whether to update the flight plan, etc.).

[0355] In response to determining at step 1960 that updated information has not been received, process 1900 proceeds to step 1970 where a determination is made as to whether the process is complete. As an example, process 1900 may be determined to be complete based at least in part on user input (such as an input to cancel or pause the task). As another example, process 1900 may be determined to be complete in response to a user selecting to end the task. As another example, process 1900 may be determined to be complete in response to completion of the task (e.g., completion of one or more tasks associated with the task). If the process is deemed complete, process 1900 ends. Otherwise, process 1900 returns to step 1960 where the process checks / monitors for updated information.

[0356] FIG. 19B illustrates a method for determining a flight plan, according to various embodiments of the present application. According to various embodiments, process 1940 of FIG. 19B is performed in association with step 1940 of process 1900 of FIG. 19A. According to various embodiments, process 1940 is performed at least in part by system 100 of FIG. 1 and / or device 300 of FIG. 300. In an embodiment, process 1940 is performed in association with determining or updating a plan (e.g., a flight plan). Process 1940 may be performed during the execution of an operation. Process 1940 is performed by a leader drone and / or a follower drone of a group of assets assigned to perform one or more tasks.

[0357] In operation 1941, a current location is determined. In some embodiments, the current location is associated with the asset for which the flight plan is being determined. The current location indicates a discrete element of the discrete representation in which the asset is located. In some embodiments, a GPS location of the asset is determined and a discrete element corresponding to the GPS location is determined. As one example, the current location is determined based at least in part on feedback information received from the asset. As another example, the current location of the asset is determined based at least in part on a locally stored annotated representation of a geographic location. For example, the annotated representation is queried for a discrete element in which metadata indicates that the asset is located.

[0358] At step 1942, a target location (or destination location) is determined. The target location indicates a discrete element of the discrete representation to which the asset will move. The target location is determined at least in part based on a plan or task associated with the item. As an example, if the leader drone determines that the task for the follower drone is to perform surveillance of a road or building, the target location is determined to a location corresponding to the road or building. As another example, if the leader drone determines that the task for the follower drone is to intercept a target or monitor a target, the target location is determined at least in part based on the location of the target. For example, the location of the target is determined by querying the annotated representation for discrete elements that include the target to be intercepted / monitored.

[0359] At step 1943, a set of contiguous or adjacent discrete elements from the current location to the target location is determined. The set of contiguous or adjacent discrete elements is determined based at least in part on the annotated representation. For example, the set of contiguous or adjacent discrete elements corresponds to a path along which the asset may travel from the current location to the target location. In some embodiments, the set of contiguous or adjacent discrete elements includes only discrete elements that are not indicated as occupied or are indicated as unoccupied.

[0360] At step 1944, a value of a cost function is determined for the set of continuous discrete elements determined at step 1943. In some embodiments, a cost associated with moving an asset from a current location to a target location via a flight plan corresponding to the set of continuous discrete elements is determined. According to various embodiments, the cost function includes one or more variables. In some embodiments, each of the one or more variables has a corresponding weighting (e.g., between 0 and 1, etc.). The variables in the cost function may relate to the amount of effort to move the asset from the current location to the target location, the risk of loss, the time to travel the flight path, etc. Examples of cost functions include (i) the length of the flight plan, (ii) the extent to which the flight plan includes a vertical climb (e.g., moving the asset to a higher altitude), (iii) the extent to which the flight plan maintains line of sight (e.g., radio line of sight) with the leader drone or other assets or control stations, (iv) the extent to which the flight plan includes adverse weather conditions, (v) the extent to which the flight plan exposes the asset to risk of loss (e.g., risk of loss exceeding a threshold probability of loss), (vi) the time required for the asset to move from a current location to a target location along a flight path that is a continuous set of discrete elements, etc. A variety of other variables may be implemented in conjunction with the cost function.

[0361] In step 1945, a decision is made as to whether to determine another set of contiguous discrete elements.

[0362] In some embodiments, the system iteratively determines a set of successive discrete elements until the successive discrete elements meet a cost threshold (e.g., a predetermined threshold, a configurable threshold, etc.) For example, the system may utilize a "good enough" determination method to determine a flight plan such that if a flight plan is determined that has a cost less than the cost threshold, process 1940 proceeds to step 1946, otherwise, a further set of successive discrete elements is determined.

[0363] In some embodiments, the system determines a predetermined number of sets of contiguous discrete elements from which a selected set of contiguous discrete elements is selected to correspond to the flight plan. By way of example, the predetermined number of sets of contiguous discrete elements determined is configurable, such as by a user or administrator.

[0364] In some embodiments, the system determines the set of consecutive discrete elements over a predetermined period of time, for example, the system may allocate a particular time for determining the set of consecutive discrete elements, and the system may determine the set of consecutive discrete elements until such predetermined period has elapsed.

[0365] In response to a determination at step 1945 that another set of contiguous discrete elements is determined, the process 1940 returns to step 1943. Conversely, in response to a determination at step 1945 that another set of contiguous elements is not determined, the process 1940 proceeds to step 1946.

[0366] At step 1946, a set of contiguous discrete elements is selected. According to various embodiments, the set of contiguous discrete elements is selected from among a set (e.g., multiple sets), such as a set determined based on repeatedly performing steps 1943-1945 of process 1940. In some embodiments, the set of contiguous discrete elements is selected as the set of contiguous discrete elements corresponding to a flight plan for a particular asset (e.g., a current location, a target location, and a selected set of contiguous discrete elements corresponding to the flight plan).

[0367] According to various embodiments, the selected set of continuous discrete elements is selected based at least in part on the value of the cost function determined for the selected set of continuous discrete elements. For example, the system determines an optimal set of continuous discrete elements from among the set of continuous discrete elements (e.g., determined based on repeatedly performing steps 1943-1945 of process 1940) based at least in part on the cost function. For example, the system selects the set of continuous discrete elements having the lowest cost. As another example, the system selects multiple flight plans for the multiple assets based on an optimal aggregate cost associated with the multiple assets moving from their respective current locations to their respective target locations.

[0368] In step 1947, information related to the flight plan is provided. For example, the information related to the flight plan includes a selected set of continuous discrete elements.

[0369] At step 1948, a determination is made as to whether the process is complete. As one example, the process 1940 may be determined to be complete based at least in part on a user input (such as an input to cancel or pause the task). As another example, the process may be determined to be complete in response to a user selecting to end the task. As another example, the process may be determined to be complete in response to a determination that there are no further flight plans (e.g., one or more tasks associated with the completion of the task) to be determined. If the process is deemed to be complete, the process 1940 ends. Otherwise, the process 1900 returns to step 1941 where another flight plan is determined by repeatedly performing steps 1941-1947.

[0370] Various examples of the embodiments described herein are described with reference to flow charts. Although the examples may include certain steps performed in a particular order, various steps may be performed in various orders according to various embodiments. In some embodiments, some steps may be combined or excluded from the examples discussed herein.

[0371] Although the above embodiments have been described in some detail for ease of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and are not intended to be limiting.

Claims

1. 1. A system comprising: A communication interface; one or more processors connected to the communication interface; Equipped with The one or more processors: Obtaining data associated with one or more tasks performed by the assets; the assets include a plurality of drones, the plurality of drones being at least semi-autonomous; the data associated with the one or more tasks includes flight plans for one or more other drones; determining a discrete representation of the geographic location, the discrete representation including a plurality of discrete elements each corresponding to a volume associated with the geographic location; annotating the discrete representation with flight plans of the one or more other drones to create an annotated representation; determining a first flight plan for at least one drone of the plurality of drones, the first flight plan being determined based at least in part on the annotated representation; The system is configured to communicate information regarding the first flight plan to at least one other asset in the fleet.

2. The system of claim 1 , wherein determining the discrete representation of the geographic location includes determining a 3-D representation associated with the geographic location.

3. 3. The system of claim 2, determining the 3-D representation associated with the geographic location includes transforming the geographic location from a real-world representation to a 3-D representation based at least in part on a curvature of the Earth; A system in which the downward direction within a grid of the 3-D representation corresponds to the direction of gravity.

4. 2. The system of claim 1, annotating the discrete representation with flight plans of the one or more other drones to create an annotated representation, determining a set of one or more discrete elements of the flight plans of the one or more other drones that intersect with a second flight plan; setting one or more metadata fields associated with the set of one or more discrete elements to indicate that the corresponding discrete elements are occupied.

5. 5. The system of claim 4, wherein the one or more processors are further configured to update the one or more metadata fields associated with the set of one or more discrete elements to be set as unoccupied based at least in part on (i) a current time and (ii) a time when a first drone corresponding to the second flight is predicted to intersect with the set of one or more discrete elements.

6. 6. The system of claim 5, wherein the second flight plan indicates a time when the first drone is predicted to be located at a particular point in time, and the time when the first drone is predicted to intersect with the set of one or more discrete elements is based at least in part on the second flight plan.

7. 5. The system of claim 4, wherein the one or more processors further comprise: receiving an updated position or a second flight plan associated with the first drone; determining whether to update the one or more metadata fields associated with the set of one or more discrete elements based at least in part on the updated position or the second flight plan; in response to a determination to update the one or more metadata fields associated with the set of one or more discrete elements, the system is configured to update at least a portion of the one or more metadata fields associated with the set of one or more discrete elements.

8. 2. The system of claim 1, wherein determining the first flight plan of the at least one flight plan comprises determining a set of contiguous discrete elements between a position of the first drone and a target destination of the first drone.

9. 2. The system of claim 1, wherein determining the first flight plan of the at least one flight plan comprises determining a set of contiguous discrete elements between a position of the first drone at a first time and a target destination of the first drone at a second time.

10. 2. The system of claim 1, wherein determining the first flight plan of the at least one flight plan comprises: determining a plurality of sets of continuous discrete elements between a position of the first drone at a first time and a target destination; determining, from among the plurality of sets of continuous discrete elements, a particular set of continuous discrete elements that is optimal with respect to a cost function.

11. 11. The system of claim 10, wherein the cost function is an indication of a cost associated with the at least one drone traveling from the position of the first drone at the first time to the target destination.

12. 11. The system of claim 10, The cost function is also based at least in part on one or more variables; The one or more variables include one or more of: (i) a change in altitude along a flight path corresponding to a set of continuous discrete elements; (ii) a change in position in a plane perpendicular to the change in altitude along the flight path corresponding to the set of continuous discrete elements; and (iii).

13. 13. The system of claim 12, wherein the one or more variables include a number or percentage of the contiguous discrete elements in the set of contiguous discrete elements that have a clear line of communication with a leader drone or a ground control service.

14. 2. The system of claim 1, wherein determining the first flight plan of the at least one flight plan comprises determining at least one set of continuous discrete elements between a position of the first drone at a first time and a target destination; the first flight plan corresponds to one set of the at least one set of contiguous discrete elements between the position of the first drone at the first time and the target destination; The first flight plan satisfies a predetermined threshold associated with communication line of sight.

15. 15. The system of claim 14, wherein the predetermined threshold associated with communications line of sight includes a requirement that a predetermined number or a predetermined percentage of elements in the set of contiguous discrete elements have associated metadata indicating the corresponding discrete element has a clear communications line of sight with a leader drone or ground control service.

16. 16. The system of claim 15, wherein the number or percentage of elements in the set of contiguous discrete elements having associated metadata indicating the corresponding discrete element has a clear communications line of sight is such that all discrete elements in the set of contiguous discrete elements of the first flight plan have the clear communications line of sight.

17. 2. The system of claim 1, wherein the one or more processors further comprise: determining to update the first flight plan; updating the first flight plan to generate an updated first flight plan; In response to determining that the first flight plan has been updated, the system is configured to communicate information regarding the updated first flight plan.

18. 2. The system of claim 1, wherein the one or more processors: monitoring communications regarding the one or more tasks from at least one other drone or ground control service in the fleet of assets; determining that the communication has been received regarding the one or more tasks from the at least one other drone in a fleet of assets or the ground control service; In response to receiving a communication regarding the one or more tasks from the at least one other drone in the fleet of assets or the ground control service, determine whether to update a plan for performing at least one element of the one or more tasks; responsive to determining to update the plan for performing at least one element of the one or more tasks, updating the plan to generate an updated plan; The system is configured to execute the updated plan in connection with performing the elements of the one or more tasks.

19. 20. The system of claim 18, wherein the communications regarding the one or more tasks from the at least one other drone in the fleet of assets or the ground control service include information regarding one or more of terrain, weather, targets, and characteristics of discrete elements of the discrete representation of the geographic location.

20. 1. A method comprising: Obtaining data associated with one or more tasks performed by the assets; the assets include a plurality of drones, the plurality of drones being at least semi-autonomous; the data associated with the one or more tasks includes flight plans for one or more other drones; determining a discrete representation of the geographic location, the discrete representation including a plurality of discrete elements each corresponding to a volume associated with the geographic location; annotating the discrete representation with flight plans of the one or more other drones to create an annotated representation; determining a first flight plan for at least one drone of the plurality of drones, the first flight plan being determined based at least in part on the annotated representation; communicating information regarding the first flight plan to at least one other asset in the fleet; A method comprising:

21. A computer program product embodied in a non-transitory computer-readable medium, computer instructions for obtaining data associated with one or more tasks performed by the assets; the assets include a plurality of drones, the plurality of drones being at least semi-autonomous; the data associated with the one or more tasks includes flight plans for one or more other drones; computer instructions for determining a discrete representation of the geographic location, the discrete representation including a plurality of discrete elements each corresponding to a volume associated with the geographic location; computer instructions for annotating the discrete representation with a flight plan of the one or more other drones to create an annotated representation; computer instructions for determining a flight plan for at least one drone of the plurality of drones, the flight plan being determined based at least in part on the annotated representation; computer instructions for communicating information regarding the flight plan to at least one other asset in the fleet; A computer program product comprising:

Citation Information

Patent Citations

  • Control device for drone, control method for drone and control program for drone

    JP2018165931A

  • Three-dimensional map data and control device

    JP2018180359A

  • Unmanned aircraft management device, unmanned aircraft management method, and program

    JP2019038479A

  • Information processing device

    JP2019091288A

  • Route determination device, program, route determination method, and system

    JP2021124380A