Method and system for controlling a swarm of flying objects

Abstract

The invention relates to a method for controlling a swarm of flying objects comprising at least a plurality of unmanned flying objects, the flying objects of the swarm flying in a swarm configuration, the method comprising the following steps: - Detecting at least one relative flight information item independently by each individual flying object in relation to at least one of its neighboring flying objects by means of a sensor system arranged on the respective flying object and having at least one sensor, - Generation of flight control commands independently by each individual flying object for the flight control of the respective flying object depending on a cost function related to the swarm configuration and the at least one recorded relative flight information by means of a flight control device, - Providing the generated flight control commands by each individual flying object to a control command input interface of the respective flying object in order to imprint the generated flight control commands on the control elements of the respective flying object.

Description

The invention relates to a method for controlling a swarm of flying objects, which comprises at least a plurality of unmanned flying objects, wherein the flying objects of the swarm fly in a swarm configuration. The invention also relates to a system of unmanned aerial objects for this purpose.It is currently an active research field to support manned aircraft by additional unmanned aircraft (UAV). In this case, these unmanned aircraft should stand aside the manned aircraft for carrying out specific functions during the flight of the manned aircraft and thus increase the overall range of functions of the fleet. For example, it is conceivable that manned aircraft are supported by unmanned drones when executing missions, e.g., in air refueling, in information retrieval explanation, in material transport, and / or in fighting hostile targets. This allows the effectiveness of manned aircraft to be increased without additional additional expenditure on personnel being necessary. This combination of manned and unmanned aerial vehicles provides an effective dressing of aircraft, while at the same time reducing costs and risk for manned aerial vehicles.For example, it is possible to use a switch. Helicopter pilots in rescue operations will have further drones in the future, so that a larger area can be effectively searched for at low cost, for example, in offshore operations. Another example is the use of pilots of military combat jets to which drones are to be put aside. Drones may minimize the risk for the pilot by taking over the part of the mission that is "ull, dirty and dangerous" (obtuse, drunk, and dangerous).Such flight tasks are usually implemented by a formation flight in which the flying objects of the formation fly in a defined swarm formation which defines the spatial positioning of the flying objects relative to one another during the flight. Often, such a formation follows a geometric pattern in a flight plane, for example in the form of a triangle or a diamond. The position of the flying objects, their orientation and their position relative to one another is predefined by the flight formation and substantially retained during the formation flight.For the control of a swarm (also called fleet) which flies in a swarm formation and consists of mannedent and unmanned aircraft, a control concept is necessary in order to control the aircraft of the swarm and to influence their flight properties in a targeted manner. Otherwise, the manned aircraft and, if appropriate, the unmanned aircraft may be hindered as far as falling if the swarm is not controlled in a coordinated manner in close coordination with the pilots of the manned aircraft.U.S. Pat. No. 10,719,076 B1 discloses a system for controlling a fleet of unmanned aircraft, wherein a leading vehicle and a trailing vehicle are fixed. The lead vehicle transmits corresponding control commands to the following vehicle via a data link for controlling the following vehicle, wherein the control commands are based on the sensory detection of the environment.U.S. Pat. No. 7,469,183 B2 discloses a method for navigating unmanned aircraft in a formation, wherein here a path point which the respective unmanned aircraft is intended to fly is determined for each unmanned aircraft depending on the formation and the associated geometric shape of the formation.US 2019 / 0130782 A1 discloses a flight system in which the pilot of a manned aircraft can teleport into the cockpit of an unmanned aircraft via a virtual reality (VR) in order to control the unmanned aircraft. A disadvantage here is that the pilot gives control over his own aircraft in order to control another aircraft instead.U.S. Pat. No. 10,114,384 B2 discloses a method for the formation flight of unmanned aircraft, in which the lead vehicle receives corresponding control commands from a ground station, which are then transmitted from the lead vehicle to the remaining following vehicles of the formation.From post-published DE 10 2023 118 284.6 a method for controlling a swarm of flying objects is known, which comprises a plurality of unmanned flying objects that fly in a swarm configuration. Manual control inputs on a manned flying object are used to generate formation control commands which are then transmitted to the remaining flying objects of the swarm in order to control the flying objects in accordance with the swarm configuration.A disadvantage of controlling via a ground station is, above all, that the flying objects are commanded by external operators by means of "high-level" commands. These can be, for example, waypoints or observation targets. In this case, the flight path of each individual drone in the fleet is scheduled exactly in advance. However, this requires that each drone must be able to be controlled centrally, which makes the infrastructure required for this quickly complex. Even if the formation is controlled in a decentralized manner, it is thus possible to react to unpredictable events only with difficulty by means of new commands from the ground station.In flights combined with manned aircraft, pronunciation must be held with the crew beforehand. Furthermore, the operators / ground stations are usually located in widely remote areas, so that the communication with the drones frequently takes place via satellite technology, whereby the signal transit time between the unmanned aerial vehicle and the operator is increased and thus a precise-fitting control of the drones is made more difficult.However, in some situations, more direct control of the unmanned aerial vehicles is desirable. Difficult situations may be successfully resolved, in part, only by a pilot's experience and / or human decision force. The above-presented concept of the pilot teleporting into the unmanned aerial vehicle solves the problems addressed. A local network is set up with the unmanned aerial vehicle, thus bypassing the awkward communication via a ground station, and at the same time the pilot takes over the control in a moving seat. This allows precise control of the UAV, but has at least two disadvantages: First, as long as the pilot controls the UAV, the pilot must give control over his own aircraft. Secondly, only a single UAV can be controlled in this way, but not an entire cluster.Against this background, it is an object of the present invention to specify an improved method and an improved system for a formation flight, wherein the formation consists of at least a plurality of unmanned aerial vehicles.The object is achieved according to the invention by the method according to claim 1 and by the system according to claim 11. Advantageous embodiments of the invention are then found in the corresponding dependent claims.According to claim 1, a method for controlling a swarm of flying objects is proposed, wherein the swarm comprises a plurality of unmanned flying objects. The flying objects of the swarm fly in a swarm configuration which defines a spatial positioning and / or a spatial behavior of the flying objects in the swarm during the flight. From the set of flying objects of the swarm, one flying object can be defined as a swarm driver flying object, while the other flying objects are follow-up flying objects.According to the invention, the method comprises the following steps:acquiring at least one relative piece of flight information independently by each individual flying object in relation to at least one of its adjacent flying objects by means of a sensor system arranged on the respective flying object and having at least one sensor,generating flight control commands independently by each individual flying object for flight control of the respective flying object as a function of a cost function related to the swarm configuration and the at least one detected relative flight information by means of a flight control device, andproviding the generated flight control commands by each individual flying object at a control command input interface of the respective flying object in order to apply the generated flight control commands to the control elements of the respective flying object.According to the invention, it is provided that the flying objects of the swarm organize themselves within their formation or swarm configuration and act autonomously in the process. For this purpose, each individual flying object is equipped with a sensor system which makes it possible to record relative flight information with respect to adjacent flying objects of the swarm. Such a sensor system has at least one sensor which is configured to record such relative flight information with respect to an adjacent flight object. These can be radar and / or lidar sensors, for example, which determine the distance and direction of adjacent flying objects to their own flying object. However, camera-based systems are also conceivable which record image data and analyze it and then determine relative flight information with respect to adjacent flight objects therefrom. It is conceivable that such relative flight information is determined for each adjacent flying object. Such relative flight information can in this case contain in particular the direction in which the adjacent flying object is currently located. Further information such as distance and / or height level is also conceivable.Relative flight information is thus understood to mean, in particular, relative position, speed and / or position information in relation to at least one adjacent flight object. The at least one piece of flight information is in particular an information about the relative location of the flying object within the swarm in relation to at least one adjacent flying object. It can also be referred to as a position in the swarm.Adjacent flying objects are those flying objects which are located, for example, within a specific radius around their own flying object or which are closest to their own flying object. Adjacent flying objects can also be those flying objects which are contained in a specific location segment as the single flying object.Each of these flying objects now generates flight control commands independently with the aid of a flight control device in order to control the flying object. The flight control commands are generated in this case as a function of a cost function related to the swarm configuration and the at least one detected relative flight information.For this purpose, for example, an optimization method can be carried out in which a cost function is optimized. The cost function is based in particular on the underlying swarm configuration and the relative flight information with respect to adjacent flight objects. The flight control commands are then derived from the cost function, for which purpose the relative flight information is required. The flight control commands are then used to actuate the flying object in such a way that the respective flying object assumes its corresponding position and / or its corresponding flight behavior within the swarm configuration with respect to the remaining, in particular adjacent, flying objects. In order to generate the flight control commands, a cost function is thus optimized, which describes the behavior and / or the relative location of the individual flying objects, the detected relative flight information being taken into account for this.The flight control commands are derived from a cost function. Various conditions can be incorporated into this cost function, for example the desired (geometric) formation of the controlled swarm, but also the communication infrastructure within the swarm, that is to say which flying objects can receive which information (for example by sensor technology) or possibly even exchange it with one another or not. In conjunction with the flight information which allows the flying object to be located relatively in the swarm, it is possible to generate flight control commands which optimize the cost function on which it is based, in order that the individual behavior of the flying objects is established and the desired, collective swarm behavior is thus brought about. Not the optimized cost function, but the optimization of the cost function (i.e. the process, not the result) determines the swarm behavior. During formation flight, this optimization process continues to run permanently and thus allows the swarm to respond flexibly to external influences, such as the failure of a flight object, and to reconfigure itself. The cost function optimization describes the desired swarm configuration and ensures it during flight, thereby determining the swarm behavior.Accordingly, it can be provided that flight control commands for flight control of the respective flying object are independently generated by each individual flying object by means of a flight control device, in that an optimization method is carried out in which a cost function relating to the swarm configuration is optimized for determining a position and / or a flight behavior within the swarm, taking into account the at least one relative flight information item of the respective flying object. The execution of the optimization method can be carried out on each flying object or can be carried out globally by a remote device.Each flying object furthermore has a control command input interface, via which flight control commands corresponding to the flight control of the flying object can be provided in order to control the flying object. The flight control commands are converted by the flight controller into corresponding control signals for controlling the control elements of the respective flying object in order to apply the generated flight control commands to the control elements of the flying object and thus to enable the control of the flying object. In this case, in particular the speed, the direction, the attitude and / or the height of the flying object are changed.This can be illustrated by way of example. There are N flying objects. If each flying object (i) minimizes the distance to its neighbor (i+1), then all flying objects meet at one point. The local law here is "Minimiere the distance to the nearest neighbor". The resulting global effect is then "All flying objects meet at one point.". If there is additionally a swarm driver, which does not change its position itself or is specified from outside, then all flying objects meet at the position of the swarm driver. The global optimization problem "Minimize the distances of all N flying objects to the position XY" or "Control all N flying objects into the point XY" can be solved in this way, although the position of the swarm driver as such does not have to be explicitly known to the individual flying objects.With the aid of the present invention, a type of local control of flying objects of the swarm is thus set up, wherein the flying objects organize themselves. The formation or flight task and the information and communication structure of the swarm allow the formulation of a cost function from which flight control laws can be derived for each individual flying object, which flight control laws allow each individual flying object to generate flight control commands independently and independently, which move the flying object according to the predefined swarm configuration and position it at the correct position and / or define the correct flight behavior.The decisive advantage here is that a self-organizing swarm is achieved, in which a guide object can be defined, but does not necessarily have to take place. In such a self-organizing swarm, no active communication channel is necessary from a swarm driver flying object or from a ground station in order to control the individual flying objects according to the swarm configuration. If a subscriber leaves the swarm or a subscriber is added to the swarm, the latter reorganizes itself independently and autonomously, since, as a result of the continuous optimization of the cost function, an adapted swarm configuration can then be found and assumed by corresponding flight control commands.According to one embodiment, it is provided that the swarm has at least one manned flying object, wherein control command inputs are generated by manual control inputs at a control command input device of the manned flying object and the control command inputs are provided at a control command input interface of the manned flying object in order to apply the control command inputs to the control elements of the manned flying object.In this embodiment, the swarm has at least one further flying object that is manned, i.e. the flying object is manually controlled by a pilot (also comprises the operation of the autopilot). Manual control inputs are input by the pilot at a control command input device and corresponding control command inputs are generated on the basis thereof. These are then provided at a control command input interface, so that these can be used to control the control elements of the manned flying object and thus to fly the flying object.In this way, the swarm can be manually controlled by the manned flying object by the pilot controlling the manned flying object. The remaining unmanned aerial objects would then generate corresponding flight control commands which lead to the unmanned aerial objects correspondingly flying within their swarm configuration.According to one embodiment, it is provided that a swarm driver flight object is defined from the set of flight objects of the swarm, wherein only the swarm driver flight object continues to generate its flight control commands also as a function of a flight destination.Such a swarm driver flying object knows the corresponding flight destination, wherein the flight control commands are then also further generated depending on this flight destination. Such a swarm driver flying object can be the manned flying object.According to one specific embodiment, it is provided that the flight control commands are also generated by each individual flying object as a function of a flight destination.In this embodiment, each individual flying object knows the corresponding flight destination, for example a path point, wherein the flight control commands are then also generated by optimizing the cost function as a function of this flight destination. The flight destination can indicate a corresponding flight direction, wherein this flight direction can be included in the cost function as an additional condition, for example. This is also conceivable, for example, if only a single (manned or unmanned) flying object knows the actual flight destination and the other flying objects follow accordingly.According to one specific embodiment, it is provided that the flight control commands are also generated by each individual flight object as a function of topographical surroundings information.Such topographical environmental information can be, for example, maps which contain mountains and valleys or also tall buildings or landmarks. The flight control commands are also generated in this case on the basis of this corresponding environmental information in order to prevent collisions with such topographically relevant elements.According to one specific embodiment, it is provided that the cost function is furthermore optimized by each individual flying object, taking into account a number of flying objects assigned to the swarm.In this case, according to one embodiment, provision can be made for at least one further flying object to be dynamically added to the swarm or for an existing flying object to be removed.According to one specific embodiment, it is provided that environment-related image data are recorded by one, several or all flying objects of the swarm using a camera of the sensor system, at least one adjacent flying object of the swarm being recognized for the respective flying object in the environment-related image data using an image recognition device, and the at least one relative piece of flight information in relation to the recognized adjacent flying object being ascertained from the environment-related image data.The remaining flying object records the environment around itself with a camera or with a camera system and generates environment-related image data. In this image data, corresponding adjacent flying objects are now identified with the aid of an image recognition device and then relative flight information in relation to the recognized adjacent flying object is determined from the image data. This can be done, for example, by positioning the flying object within the image data, in order to locate the flying object in the swarm. However, it is also conceivable that the distance is estimated, for example, on the basis of the size within the image data.According to one embodiment, it is provided that the swarm configuration is selected from a plurality of predefined swarm configurations stored in a data memory based on a selection signal.In this case, different spatial positionings and / or spatial behaviors for each swarm configuration are stored in a data memory, which spatial positionings and / or spatial behaviors can be manually selected by the pilot, for example, via an input device in the manned flying object. If a corresponding selection is made, a corresponding selection signal is generated based on this selection input, which then selects and defines the corresponding swarm configuration from the data memory. During the flight of the formation, the swarm configuration can be changed.The swarm configuration can define a fixed spatial positioning for each flying object in the swarm. A fixed spatial positioning is understood here to mean that each flying object is located in the swarm relative to another flying object and retains this spatial positioning. In this case, for example, it is possible to use a metal coating. In this case, parameters such as the extent of the swarm, i.e. the distance of the flying objects from one another, and the height, speed and / or direction of the swarm can be controlled.However, the swarm configuration can also define a spatial behavior of the flying objects of the swarm, for example in such a way that the unmanned flying objects circle around the swarm driver flying object or the manned flying object during the flight. Defining the swarm configuration in the form of a spatial behavior of the flying objects produces a non-rigid formation which is defined by the behavior of the flying objects.The object is also achieved with the system, wherein a swarm has a plurality of flying objects, all of which are configured to carry out the method described above.The invention is explained by way of example with reference to the appended figures. The following are shown: FIG. 1 shows a schematic illustration of an unmanned aerial object; FIG. 2 is a schematic illustration of a swarm in formation; FIG. 3 shows a schematic representation of the information flow between the flying objects; FIG. 4 is a schematic illustration of a dynamic formation; FIG. 5 shows a schematic representation of a formation as a function of the sensor range.FIG. 1 shows a schematically greatly simplified illustration of an unmanned flying object 10 which is suitable for flying in a swarm according to the method according to the invention. For this purpose, the flying object 10 first has a sensor system 11 with which the existence of other, adjacent flying objects 10 can be recognized and a relative position information can be determined. Such a sensor system 11 can have, for example, a camera that records the environment around the flying object 10 and supplies corresponding environment-related image data. From this environment-related image data, the adjacent flying objects are then identified and the relative position information with respect to the own flying object 10 is determined, so that a corresponding relative position information can be determined for every other flying object in the environment.This relative position information thus determined and the knowledge of other flying objects located in the environment are then transmitted to a flight control device 12 in order to generate corresponding flight control commands for controlling the flying object 10.The flight control device 12 is configured to generate the flight control commands on the basis of the established swarm configuration and the relative position information of flight objects located in the environment and a cost function in order to find the best possible position and / or the best possible behavior of the flight object within the swarm configuration. In this case, the cost function is optimized, which can pass through the flight control device 12 or through a central device (not shown in FIG. 1 ).In this case, flight control commands are generated which are intended to move the flying object 10 accordingly in accordance with the swarm configuration. As a result, the flying object 10 is moved into the position provided according to the swarm configuration, whereby the cost function is further optimized until the flying object 10 has assumed the desired position.The flight control commands generated in this way are now transferred to a control command input interface 13 which represents the interface to the flight control 14. The flight controller 14 now generates signals for controlling the control surfaces 15 of the flying object 10, so that the flying object is controlled accordingly.FIG. 2 shows a schematically greatly simplified representation of a swarm formation in which a swarm guide flying object 20 is provided, followed by a series of follow-up flying objects 21 in a specific swarm configuration. The following flying objects 21 usually take their neighboring flying objects through the sensor system and thus generate a relative position information.If the swarm guide flying object 20 moves to the left or right, the own position of the following flying objects 21 is no longer within the desired swarm configuration, so that the cost function is no longer minimized and corresponding flight control commands are generated, which lead to a position correction within the swarm configuration.FIG. 3 shows the example of an information flow within a swarm configuration. There is a swarm operator's flying object 30 complemented by a plurality of follow-up flying objects 31 to 35 in the swarm. The following flying objects 31 to 35 follow the swarm driver flying object 30 with regard to the movement, i.e. in particular with regard to the speed, the height and the flight direction and optionally the attitude.In the first row of follow-up flying objects 31 and 32, in particular the relative position information with respect to the swarm guide flying object 30 is determined. This is because the swarm driver flying object 30 is an adjacent flying object for the first row of the following flying objects 31 and 32. Furthermore, for example, of the following flying object 31, the following flying object 33 located in the second row is regarded as an adjacent flying object.By ascertaining the relative position information with respect to the adjacent flying objects, each flying object within the swarm approximately knows where is located within the swarm and thus knows its relative position with respect to the other flying objects. An exchange of data with one another is not necessary. By optimizing the cost function, it is now possible to determine, based on the swarm configuration, where the flying object is to be located within the swarm, such that corresponding flight control commands can be generated in order to maintain the respective flying object at its position within the swarm configuration or in order to bring it back into the optimum position within the swarm configuration.The following flying objects accordingly follow the swarm guide flying object and align themselves independently within the swarm according to the swarm configuration.It is conceivable that a swarm driver's flying object can be dispensed with, as a result of which the flying objects are automatically aligned quasi intelligently within the swarm.FIG. 4 shows in the example in which 2 swarm guide flying objects 41 and 42 are provided that their relative distance to one another is decisive for the spatial extent of the swarm. The following flying objects 43 to 45 move closer to each other, although the swarm guide flying objects 41 and 42 also reduce their relative distance from each other. This spatial dynamics is shown in FIG. 4, wherein a larger extent can be seen on the left side, while the spatial extent of the swarm becomes smaller on the right side.In this case, discarding or adding individual participants can be compensated more easily, since the swarm is organized in a decentralized manner and the participants independently create new connections or reorganize their position within the swarm.Finally, FIG. 5 shows the self-organization of 3 swarm participants without swarm driver flying object based on their sensor range in order to achieve optimal area coverage, for example.List of reference characters10 Flying object 11 Sensor system 12 Flight control device 13 Control command input interface 14 Flight control 15 Control surfaces 20, 30, 41, 42 Swarm operator flying object 21, 31-35, 43-45 Subsequent flying objectsReferences included in the specificationThis list of documents cited by the applicant has been produced in an automated manner and is only included for the better information of the reader. The list is not part of the German patent application or utility model application. The DPMA does not take any adhesion for any faults or omissions.Patent Literature citedU.S. Pat. No. 10,719,076 B1

[0006] U.S. Pat. No. 7,469,183 B2

[0007] US 2019 / 0130782 A1

[0008] U.S. Pat. No. 10,114,384 B2

[0009] DE 10 2023 118 284.6

[0010]

Claims

Method for controlling a swarm of flying objects (10) which has at least a plurality of unmanned flying objects (10), wherein the flying objects (10) of the swarm fly in a swarm configuration, wherein the method comprises the following steps: - recording at least one relative piece of flight information independently by each individual flying object (10) in relation to at least one of its adjacent flying objects (10) by means of a sensor system (11) arranged on the respective flying object (10) and having at least one sensor, - generating flight control commands independently by each individual flying object (10) to the flight control (14) of the respective flying object (10) on the basis of a cost function related to the swarm configuration and the at least one recorded relative piece of flight information by means of a flight control device (12), providing the generated flight control commands by each individual flying object (10) at a control command input interface (13) of the respective flying object (10) in order to apply the generated flight control commands to the control elements of the respective flying object (10).Method according to claim 1, characterised in that the swarm has at least one manned flying object (10), wherein control command inputs are generated by manual control inputs at a control command input device of the manned flying object (10) and the control command inputs are provided at a control command input interface of the manned flying object (10) in order to apply the control command inputs to the control elements of the manned flying object (10).Method according to claim 1 or 2, characterised in that a swarm guide flying object (20, 30, 41, 42) is defined from the set of flying objects (10) of the swarm, wherein only the swarm guide flying object (20, 30, 41, 42) continues to generate its flight control commands also as a function of a flight destination.Method according to Claims 2 and 3, characterized in that the swarm driver flying object (20, 30, 41, 42) is the manned flying object (10).Method according to one of the preceding claims, characterized in that the flight control commands are also generated by each individual flying object (10) on the basis of a flight destination.Method according to one of the preceding claims, characterized in that the flight control commands are also generated by each individual flying object (10) on the basis of topographical environmental information.Method according to one of the preceding claims, characterized in that, by means of each individual flying object (10), the respective cost function is furthermore optimized taking into account a number of flying objects (10) assigned to the swarm.Method according to one of the preceding claims, characterized in that environment-related image data are recorded by one, more or all flying objects (10) of the swarm by means of a camera of the sensor system (11), wherein for the respective flying object (10) in the environment-related image data at least one adjacent flying object (10) of the swarm is recognized by means of an image recognition device and the at least one relative flight information in relation to the recognized adjacent flying object (10) is determined from the environment-related image data.Method according to one of the preceding claims, characterized in that at least one further flying object (10) is dynamically added to the swarm or an existing flying object (10) is removed.Method according to one of the preceding claims, characterized in that the swarm configuration is selected from a plurality of predefined swarm configurations stored in a data memory on the basis of a selection signal.A system comprising a swarm of flying objects (10) comprising a plurality of unmanned flying objects (10) configured to perform the method according to any one of the preceding claims.

Images