Method and system for controlling a swarm of flying objects

Abstract

A method for controlling a swarm of flying objects comprising at least a plurality of unmanned flying objects (10), the flying objects of the swarm flying in a swarm configuration, the method comprising the following steps: - detecting at least one relative flight information item independently by each individual flying object in relation to at least one of its neighboring flying objects by means of a sensor system (11) arranged on the respective flying object and having at least one sensor, - generating flight control commands independently by each individual unmanned aerial vehicle (10) for the flight control (14) of the respective unmanned aerial vehicle (10) as a function of a cost function related to the swarm configuration and the at least one acquired item of relative flight information by means of a flight control device (12), wherein the cost function is continuously optimized taking external influences into account, - Providing the generated flight control commands by each individual unmanned aerial vehicle (10) at a control command input interface (13) of the respective unmanned aerial vehicle (10) in order to imprint the generated flight control commands on the control elements of the respective unmanned aerial vehicle (10).

Description

[0001] The invention relates to a method for controlling a swarm of flying objects comprising at least a plurality of unmanned flying objects, wherein the flying objects of the swarm fly in a swarm configuration. The invention also relates to a system of unmanned flying objects for this purpose.

[0002] The support of manned aircraft with additional unmanned aerial vehicles (UAVs) is currently an active field of research. These unmanned aerial vehicles are intended to assist the manned aircraft in performing specific functions while the manned aircraft are in flight, thus expanding the overall functional scope of the fleet. For example, it is conceivable that manned aircraft could be supported by unmanned drones in carrying out missions such as aerial refueling, reconnaissance for intelligence gathering, material transport, and / or the engagement of enemy targets. This could increase the effectiveness of manned aircraft without requiring additional personnel.This combination of manned and unmanned aircraft creates an effective alliance of aircraft while simultaneously reducing costs and risks for manned aircraft.

[0003] For example, helicopter pilots on rescue missions may in future have access to additional drones, enabling them to effectively search a larger area cost-effectively, such as during offshore operations. Another example is the deployment of military fighter jet pilots who are to be assisted by drones. In this case, drones can minimize the risk to the pilot by taking over the "dull, dirty, and dangerous" part of the mission.

[0004] Such flight tasks are usually carried out through formation flight, in which the aircraft of the formation fly in a fixed swarm formation, which defines the spatial positioning of the aircraft relative to each other during the flight. Such a formation often follows a geometric pattern in a flight plane, for example, in the shape of a triangle or a diamond. The attitude of the aircraft, their orientation, and their position relative to each other are determined by the flight formation and are essentially maintained during the formation flight.

[0005] To control a swarm (also called a fleet) flying in a swarm formation and consisting of manned and unmanned aircraft, a control concept is necessary to steer the aircraft in the swarm and specifically influence their flight characteristics. Otherwise, this could lead to disruptions or even crashes of the manned and, in some cases, unmanned aircraft if the swarm is not controlled in a coordinated manner in close cooperation with the pilots of the manned aircraft.

[0006] US 10 719 076 B1 discloses a system for controlling a fleet of unmanned aerial vehicles, comprising a lead vehicle and a follower vehicle. The lead vehicle transmits corresponding control commands to the follower vehicle via a data link, with the control commands being based on sensor detection of the surroundings.

[0007] From US 7 469 183 B2 a method for navigating unmanned aircraft in a formation is known, wherein, depending on the formation and the associated geometric shape of the formation, a waypoint is determined for each unmanned aircraft to which the respective unmanned aircraft is to fly.

[0008] US 2019 / 0 130 782 A1 discloses a flight system in which the pilot of a manned aircraft can teleport into the cockpit of an unmanned aircraft via virtual reality (VR) in order to control the unmanned aircraft. The disadvantage of this is that the pilot relinquishes control of his own aircraft to control another aircraft instead.

[0009] From US 10 114 384 B2 a method for formation flight of unmanned aircraft is known, in which the lead vehicle receives corresponding control commands from a ground station, which are then transmitted from the lead vehicle to the other following vehicles in the formation.

[0010] From the subsequently published DE 10 2023 118 284.6, a method for controlling a swarm of flying objects comprising a plurality of unmanned flying objects flying in a swarm configuration is known. Manual control inputs on a manned flying object are used to generate formation control commands, which are then transmitted to the remaining flying objects in the swarm to control the flying objects according to the swarm configuration.

[0011] DE 10 2020 203 054 A1 discloses a method for controlling a formation of a cooperating swarm of unmanned mobile units.

[0012] US 2019 / 0 080 621 A1 reveals a swarm consisting of a large number of light drones.

[0013] DE 10 2020 000 663 A1 discloses a method for improving navigation quality by actively shaping a movement formation of a collaborative swarm of mobile unmanned units.

[0014] The main disadvantage of controlling the aircraft via a ground station is that the aircraft are controlled by external operators using high-level commands. These can be waypoints or observation targets, for example. The flight path of each individual drone in the fleet is precisely planned in advance. However, this requires that each drone can be controlled centrally, which quickly makes the necessary infrastructure complex. Even if the formation is controlled decentrally, responding to unforeseen events is cumbersome and requires repeated commands from the ground station.

[0015] For flights in conjunction with manned aircraft, prior consultation with the crew is required. Furthermore, operators / ground stations are usually located in remote areas, so communication with the drones often takes place via satellite technology, which increases the signal delay between the unmanned aircraft and the operator, making precise control of the drones more difficult.

[0016] In some situations, however, more direct control of unmanned aircraft is desirable. Difficult situations can sometimes only be successfully resolved through the experience and / or decision-making power of a pilot. The concept presented above, in which the pilot teleports into the unmanned aircraft, solves the problems mentioned. A local network is established with the unmanned aircraft, thus avoiding the cumbersome communication via a ground station, and at the same time, the pilot takes control from a mobile seat. While this allows for precise control of the UAV, it has at least two disadvantages: First, the pilot must relinquish control of his own aircraft while controlling the UAV. Second, this approach allows only a single UAV to be controlled, not an entire group.

[0017] Against this background, it is the object of the present invention to provide an improved method and an improved system for formation flight, wherein the formation consists of at least a plurality of unmanned aircraft.

[0018] The object is achieved according to the invention with the method according to claim 1 and the system according to claim 11. Advantageous embodiments of the invention can then be found in the corresponding subclaims.

[0019] According to claim 1, a method for controlling a swarm of flying objects is proposed, wherein the swarm comprises a plurality of unmanned flying objects. The flying objects of the swarm fly in a swarm configuration that defines a spatial positioning and / or spatial behavior of the flying objects in the swarm during flight. From the set of flying objects in the swarm, one flying object can be designated as the swarm leader, while the other flying objects are follower flying objects.

[0020] A procedure may be provided that includes the following steps: - Recording of at least one relative flight information item independently by each individual flying object in relation to at least one of its neighbouring flying objects by means of a sensor system arranged on the respective flying object and having at least one sensor, - generating flight control commands independently by each individual flying object for the flight control of the respective flying object depending on a cost function related to the swarm configuration and the at least one recorded relative flight information by means of a flight control device, and - Providing the generated flight control commands by each individual flying object to a control command input interface of the respective flying object in order to imprint the generated flight control commands on the control elements of the respective flying object.

[0021] It can be intended that the flying objects of the swarm organize themselves within their formation or swarm configuration and act autonomously. For this purpose, each individual flying object is equipped with a sensor system that allows it to record relative flight information in relation to neighboring flying objects in the swarm. Such a sensor system has at least one sensor that is set up to record such relative flight information in relation to a neighboring flying object. This can be, for example, radar and / or lidar sensors that determine the distance and direction of neighboring flying objects from the own flying object. Camera-based systems that record and analyze image data and then use this to determine relative flight information to neighboring flying objects are also conceivable. It is conceivable that such relative flight information is determined for each neighboring flying object.Such relative flight information can include, in particular, the direction in which the neighboring flying object is currently traveling. Other information, such as distance and / or altitude, is also conceivable.

[0022] Relative flight information is thus understood to mean, in particular, relative position, speed, and / or attitude information relative to at least one neighboring flying object. The at least one piece of flight information is, in particular, information about the relative location of the flying object within the swarm relative to at least one neighboring flying object. It can also be referred to as a position within the swarm.

[0023] Neighboring flying objects are those flying objects that are, for example, within a certain radius of your own flying object or that are closest to your own flying object. Neighboring flying objects can also be those flying objects that are the only flying objects contained in a given attitude segment.

[0024] Each of these flying objects then independently generates flight control commands using a flight control device to steer the flying object. The flight control commands are generated based on a cost function related to the swarm configuration and at least one piece of acquired relative flight information.

[0025] For this purpose, for example, an optimization process can be carried out in which a cost function is optimized. The cost function is based in particular on the underlying swarm configuration and the relative flight information to neighboring flying objects. The flight control commands are then derived from the cost function, which requires the relative flight information. The flight control commands are then used to control the flying object in such a way that the respective flying object assumes its appropriate position and / or its appropriate flight behavior within the swarm configuration with respect to the other, particularly neighboring, flying objects. In order to generate the flight control commands, a cost function is optimized which describes the behavior and / or relative location of the individual flying objects, whereby the recorded relative flight information is taken into account.

[0026] The flight control commands are derived from a cost function. Various conditions can be incorporated into this cost function, such as the desired (geometric) formation of the controlled swarm, but also the communication infrastructure within the swarm, i.e., which flying objects receive which information (e.g., via sensors) or can or cannot exchange information with each other. In conjunction with the flight information, which allows the relative positioning of the flying object within the swarm, it is possible to generate flight control commands that optimize the underlying cost function, thereby determining the individual behavior of the flying objects and thus bringing about the desired collective swarm behavior. It is not the optimized cost function, but the optimization of the cost function (i.e., the process, not the result) that determines the swarm behavior.During formation flight, this optimization process continues continuously, allowing the swarm to react flexibly to external influences, such as the failure of an aircraft, and to reconfigure itself. The optimization of the cost function describes the desired swarm configuration and ensures it during flight, thus determining the swarm's behavior.

[0027] It can therefore be provided that each individual flying object independently generates flight control commands for controlling the respective flying object using a flight control device. An optimization process is carried out in which, taking into account at least one piece of relative flight information of the respective flying object, a cost function related to the swarm configuration is optimized to determine a position and / or flight behavior within the swarm. The optimization process can be carried out on each flying object or globally by a remote device.

[0028] Each flying object furthermore has a control command input interface, via which corresponding flight control commands can be provided to the flight control system of the flying object in order to control the flying object. The flight control system converts the flight control commands into corresponding control signals for controlling the control elements of the respective flying object. This imposes the generated flight control commands on the control elements of the flying object and thus enables control of the flying object. In particular, the speed, direction, attitude, and / or altitude of the flying object are changed.

[0029] This can be illustrated with an example. Given N flying objects, if each flying object (i) minimizes the distance to its neighbor (i+1), then all flying objects meet at a single point. The local law here is "Minimize the distance to the nearest neighbor." The resulting global effect is then "All flying objects meet at a single point." If there is also a swarm leader that does not change its position itself or is given a position from outside, then all flying objects meet at the position of the swarm leader. The global optimization problem "Minimize the distances of all N flying objects to position XY" or "Steer all N flying objects to point XY" can be solved in this way, even though the individual flying objects do not need to explicitly know the position of the swarm leader.

[0030] With the help of the present invention, a type of local control of the flying objects in the swarm is established, with the flying objects organizing themselves. The formation or flight task as well as the information and communication structure of the swarm allow the formulation of a cost function from which flight control laws can be derived for each individual flying object. These allow each individual flying object to autonomously and independently generate flight control commands that move the flying object according to the specified swarm configuration and position it in the correct position and / or define the correct flight behavior.

[0031] The key advantage here is that a self-organizing swarm is achieved, in which a command object can be defined, but this is not mandatory. In such a self-organizing swarm, no active communication channel from a swarm leader aircraft or a ground station is necessary to control the individual aircraft according to the swarm configuration. If a participant leaves the swarm or a participant is added to the swarm, the swarm reorganizes itself independently and autonomously, since the continuous optimization of the cost function allows an adapted swarm configuration to be found and adopted through appropriate flight control commands.

[0032] According to one embodiment, it is provided that the swarm comprises at least one manned flying object, wherein control command inputs are generated by manual control inputs at a control command input device of the manned flying object and the control command inputs are provided at a control command input interface of the manned flying object in order to impose the control command inputs on the control elements of the manned flying object.

[0033] In this embodiment, the swarm includes at least one other manned flying object, meaning the flying object is manually controlled by a pilot (including operation of the autopilot). The pilot inputs manual control inputs to a control command input device, and corresponding control command inputs are generated based on these inputs. These are then provided to a control command input interface so that they can be used to control the control elements of the manned flying object and thus fly the flying object.

[0034] In this way, the swarm can be manually controlled by the pilot controlling the manned aircraft. The remaining unmanned aircraft would then generate appropriate flight control commands, causing the unmanned aircraft to fly accordingly within their swarm configuration.

[0035] According to one embodiment, it is provided that a swarm leader flying object is defined from the number of flying objects of the swarm, wherein only the swarm leader flying object continues to generate its flight control commands depending on a flight destination.

[0036] Such a swarm leader aircraft knows the corresponding flight destination, and the flight control commands are then generated based on this flight destination. Such a swarm leader aircraft can be a manned aircraft.

[0037] According to one embodiment, it is provided that the flight control commands are also generated by each individual flying object depending on a flight destination.

[0038] In this embodiment, each individual flying object knows the corresponding flight destination, for example, a waypoint, whereby the flight control commands are then generated by optimizing the cost function, also depending on this flight destination. The flight destination can specify a corresponding flight direction, whereby this flight direction can be included in the cost function, for example, as an additional condition. This is also conceivable, for example, if only a single (manned or unmanned) flying object knows the actual flight destination, and the other flying objects follow accordingly.

[0039] According to one embodiment, it is provided that the flight control commands are also generated by each individual flying object depending on topographical environmental information.

[0040] Such topographical environmental information can, for example, be maps containing mountains and valleys, or even tall buildings or landmarks. Flight control commands are also generated based on this relevant environmental information to prevent collisions with such topographically relevant elements.

[0041] According to one embodiment, it is provided that the cost function is further optimized by each individual flying object, taking into account a number of flying objects assigned to the swarm.

[0042] According to one embodiment, it can be provided that at least one additional flying object is dynamically added to the swarm or that an existing flying object is removed.

[0043] According to one embodiment, it is provided that environment-related image data are recorded by one, several or all flying objects of the swarm by means of a camera of the sensor system, wherein for the respective flying object in the environment-related image data at least one neighboring flying object of the swarm is recognized by means of an image recognition device and the at least one relative flight information in relation to the recognized neighboring flying object is determined from the environment-related image data.

[0044] The remaining flying object records its surroundings with a camera or camera system and generates environment-related image data. An image recognition device then identifies corresponding neighboring flying objects in this image data, and then determines relative flight information relative to the detected neighboring flying object from the image data. This can be done, for example, by positioning the flying object within the image data in order to locate the flying object within the swarm. It is also conceivable, however, that the distance is estimated based on its size within the image data, for example.

[0045] According to one embodiment, the swarm configuration is selected from a plurality of predefined swarm configurations stored in a data memory based on a selection signal.

[0046] Here, various spatial positioning and / or spatial behavior for each swarm configuration are stored in a data storage device, which can be manually selected by the pilot, for example, via an input device in the manned aircraft. If a corresponding selection is made, a corresponding selection signal is generated based on this selection input, which then selects and sets the corresponding swarm configuration from the data storage. The swarm configuration can be changed during formation flight.

[0047] The swarm configuration can define a fixed spatial positioning for each flying object in the swarm. A fixed spatial positioning means that each flying object is located relative to another flying object in the swarm and maintains this spatial positioning. Parameters such as the size of the swarm (i.e., the distance between the flying objects), as well as the altitude, speed, and / or direction of the swarm can be controlled.

[0048] However, the swarm configuration can also define the spatial behavior of the flying objects in the swarm, for example, such that the unmanned flying objects orbit the swarm leader or the manned flying object during flight. Defining the swarm configuration in terms of the spatial behavior of the flying objects creates a non-rigid formation defined by the behavior of the flying objects.

[0049] The problem is also solved with the system, which has a swarm with a plurality of flying objects, all of which are configured to carry out the method described above.

[0050] The invention is explained by way of example with reference to the accompanying figures. They show: Fig. 1 schematic representation of an unmanned aerial vehicle; Fig. 2 schematic representation of a swarm in formation; Fig. 3 schematic representation of the information flow between the flying objects; Fig. 4 schematic representation of a dynamic formation; Fig. 5 schematic representation of a formation depending on the sensor range.

[0051] Fig. 1 shows, in a highly simplified schematic representation, an unmanned flying object 10 suitable for flying in a swarm according to the method according to the invention. For this purpose, the flying object 10 initially has a sensor system 11 with which the existence of other, neighboring flying objects 10 can be detected and relative position information can be determined. Such a sensor system 11 can, for example, have a camera that records the environment around the flying object 10 and thereby provides corresponding environment-related image data. From this environment-related image data, the neighboring flying objects are then identified, and the relative position information with respect to the own flying object 10 is determined, so that corresponding relative position information can be determined for every other flying object in the environment.

[0052] This relative position information determined in this way as well as the knowledge of other flying objects in the surrounding area are then transmitted to a flight control device 12 in order to generate corresponding flight control commands for controlling the flying object 10.

[0053] The flight control device 12 is configured to generate the flight control commands based on the defined swarm configuration as well as the relative position information of the flying objects in the vicinity and a cost function in order to find the best possible position and / or the best possible behavior of the flying object within the swarm configuration. The cost function is optimized, which can be done by the flight control device 12 or by a central facility (not included in Fig. 1 shown).

[0054] Flight control commands are generated that are intended to move the flying object 10 according to the swarm configuration. This moves the flying object 10 to the position specified in the swarm configuration, further optimizing the cost function until the flying object 10 has assumed the desired position.

[0055] The flight control commands thus generated are then transferred to a control command input interface 13, which represents the interface to the flight control system 14. The flight control system 14 then generates signals for controlling the control surfaces 15 of the flying object 10, so that the flying object is controlled accordingly.

[0056] Fig. Figure 2 shows a highly simplified schematic representation of a swarm formation in which a swarm leader aircraft 20 is provided, followed by a series of follower aircraft 21 in a specific swarm configuration. The follower aircraft 21 regularly monitor their neighboring aircraft using sensors, thus generating relative position information.

[0057] If the swarm leader flying object 20 moves to the left or right, the own position of the following flying objects 21 is no longer within the desired swarm configuration, so that the cost function is no longer minimized and corresponding flight control commands are generated, which lead to a position correction within the swarm configuration.

[0058] Fig. Figure 3 shows an example of an information flow within a swarm configuration. There is a swarm leader aircraft 30, which is complemented by a plurality of follower aircraft 31 to 35 within the swarm. The follower aircraft 31 to 35 follow the swarm leader aircraft 30 in terms of movement, ie, in particular, in terms of speed, altitude, flight direction, and, if applicable, flight attitude.

[0059] In the first row of follower flying objects 31 and 32, the relative position information regarding the swarm leader flying object 30 is determined, in particular. This is because the swarm leader flying object 30 is a neighboring flying object for the first row of follower flying objects 31 and 32. Furthermore, for example, the follower flying object 33 located in the second row is considered a neighboring flying object by the follower flying object 31.

[0060] By determining the relative position information of neighboring flying objects, each flying object within the swarm knows approximately where it is located within the swarm and thus knows its relative position to the other flying objects. An exchange of data between them is unnecessary. By optimizing the cost function, the location of the flying object within the swarm can now be determined based on the swarm configuration, allowing appropriate flight control commands to be generated to maintain the respective flying object at its position within the swarm configuration or to return it to the optimal position within the swarm configuration.

[0061] The follower flying objects therefore follow the swarm leader flying object and align themselves independently within the swarm according to the swarm configuration.

[0062] It is conceivable that a swarm leader aircraft can be dispensed with, allowing the aircraft to align themselves intelligently within the swarm.

[0063] Fig. Figure 4 shows, using the example of two swarm leader aircraft 41 and 42, that their relative distance from each other is crucial for the spatial extent of the swarm. The follower aircraft 43 to 45 move closer to each other even if the swarm leader aircraft 41 and 42 reduce their relative distance from each other. This spatial dynamic is Fig. 4, where a larger extent can be seen on the left side, while on the right side the spatial extent of the swarm becomes smaller.

[0064] The withdrawal or addition of individual participants can be more easily compensated because the swarm is organized in a decentralized manner and the participants independently create new connections or reorganize their position within the swarm.

[0065] Fig. Figure 5 finally shows the self-organization of three swarm participants without a swarm leader aircraft based on their sensor range, for example, to achieve optimal area coverage. List of reference symbols 10 flying object 11 Sensor technology 12 Flight control device 13 Control command input interface 14 Flight control 15 control surfaces 20, 30, 41, 42 Swarm Leader Flying Object 21, 31-35, 43-45 follow-on flying objects

Claims

[1] A method for controlling a swarm of flying objects comprising at least a plurality of unmanned flying objects (10), the flying objects of the swarm flying in a swarm configuration, the method comprising the following steps: - detecting at least one relative flight information item independently by each individual flying object in relation to at least one of its neighboring flying objects by means of a sensor system (11) arranged on the respective flying object and having at least one sensor, - generating flight control commands independently by each individual unmanned aerial vehicle (10) for the flight control (14) of the respective unmanned aerial vehicle (10) as a function of a cost function related to the swarm configuration and the at least one acquired item of relative flight information by means of a flight control device (12), wherein the cost function is continuously optimized taking external influences into account, - Providing the generated flight control commands by each individual unmanned aerial vehicle (10) at a control command input interface (13) of the respective unmanned aerial vehicle (10) in order to imprint the generated flight control commands on the control elements of the respective unmanned aerial vehicle (10). [2] Method according to claim 1, characterized bythat the swarm comprises at least one manned flying object, wherein control command inputs are generated by manual control inputs at a control command input device of the manned flying object and the control command inputs are provided at a control command input interface of the manned flying object in order to impress the control command inputs on the control elements of the manned flying object (10). [3] Method according to claim 1, characterized by that a swarm leader flying object (20, 30, 41, 42) is determined from the number of unmanned flying objects (10) of the swarm, wherein only the swarm leader flying object (20, 30, 41, 42) continues to generate its flight control commands depending on a flight target. [4] Method according to claim 2, characterized by that the manned flying object is a swarm leader flying object (20, 30, 41 ,42). [5] Method according to one of claims 1 and 2, characterized bythat a swarm leader flying object (20, 30, 41, 42) is determined from the number of unmanned flying objects (10) of the swarm, wherein the swarm leader flying object (20, 30, 41, 42) continues to generate its flight control commands depending on a flight destination and each individual unmanned flying object (10) continues to generate the flight control commands depending on a flight destination. [6] Method according to one of the preceding claims, characterized by that the flight control commands continue to be generated by each individual flying object (10) depending on topographical environmental information. [7] Method according to one of the preceding claims, characterized by that each individual unmanned aerial vehicle (10) further optimizes the respective cost function taking into account a number of aerial vehicles assigned to the swarm. [8] Method according to one of the preceding claims, characterized bythat environment-related image data are recorded by one, several or all flying objects of the swarm by means of a camera of the sensor system (11), wherein for the respective flying object in the environment-related image data at least one neighboring flying object of the swarm is recognized by means of an image recognition device and the at least one relative flight information item in relation to the recognized neighboring flying object is determined from the environment-related image data. [9] Method according to one of the preceding claims, characterized by that at least one additional flying object is dynamically added to the swarm or an existing flying object is removed. [10] Method according to one of the preceding claims, characterized by that the swarm configuration is selected from a plurality of predefined swarm configurations stored in a data memory based on a selection signal. [11] System comprising a swarm of flying objects comprising a plurality of unmanned flying objects (10) configured to carry out the method according to one of the preceding claims.

Images