Multi-agent system

Abstract

A system for carrying out a task in respect of a target in physical environment 200, the system comprising a fleet of autonomous vehicles 202 such as unmanned aerial vehicles or drones or the like, each autonomous vehicle configured for operation under the control of a control neural network 204, 206. Each control neural network comprises a different respective instance of the same trained neural network. The trained neural network is trained to provide a series of control outputs 218 for navigating a corresponding autonomous vehicle of the fleet towards the target, and for controlling the corresponding autonomous vehicle to provide at least a contribution to carrying out the task. Each control output is respectively generated, by the trained neural network, based on a plurality of input vectors. The plurality of input vectors includes a first input vector, that represents a first set of observation inputs 208 for the corresponding autonomous vehicle, and at least one further input vector, each further input vector respectively represents a further set of observation inputs for a different autonomous vehicle of the fleet.

Description

The present invention relates to a multi-agent system, for example, a multi-agent system in which the behaviour of a group of autonomous agents is controlled to achieve a predefined goal or carry out a predefined task at a remote target location. The present invention has particular, but not exclusive, relevance to methods and apparatus for providing collaborative coordination of: the journeys of a set of agents, in the form of autonomous vehicles, from a base location, to a target location; and the performance of a specific pre-defined task by the agents at the target location. There are many potential applications for coordinated groups of two or more artificial intelligence (Al) controlled autonomous agents, such as vehicles or drones (e.g., terrestrial, aerial, marine surface, submersible, and / or space borne vehicles). Depending on context, such multi-agent groups may be referred to using many different collective terms such as, for example, swarms, flocks, fleets, platoons, squadrons, and / or the like. For example, in addition to traditional tasks requiring the delivery of cargo to, and / or the collection of cargo from, a destination (e.g., in an automated warehouse or the like), such autonomous groups may be used for reconnaissance, surveillance, surveying, and / or inspection missions, for exploration or investigation of remote target areas or objects, that may be difficult, costly and / or hazardous for a human to carry out. A sensor and / or camera payload of an agent may be used to collect data, for example from inspection of linear assets. In this context a multi-agent system provides a remote (e.g. arial) data collection platform. The data can be streamed or held in the agent and provides the basis for many digital services to deliver value in lots of applications. Whilst the use of an individual, remotely piloted, vehicle such as a drone might be used to carry out some such exploration or investigation, the use of groups or fleets of autonomous vehicles has the potential to provide a number of benefits including, for example: • Increased resilience, because even if a member of the fleet is lost, or otherwise rendered incapable of contributing to a mission, the mission can still be completed by the other member(s) of the group; • Greater scalability, since a relatively large autonomous fleet can be subject to oversight / supervision by one person (e.g., acting as a ‘human-on-the-loop’). This offers significant commercial benefits for unmanned aerial, terrestrial, or marine system operations, as it avoids the inherent limitations associated with “one drone to one human” ( / .e., one human-in-the-loop to one vehicle). • Increased capability to conduct higher value, more complex missions, since members of the group / fleet can collaborate with one another to take a different role that contributes to completion of the task; and • Greater efficiency, since a group / fleet of autonomous vehicles or drones can cover a larger area, and / or complete a more complex task more quickly than individual human piloted drones. To complete such exploration or investigation tasks successfully, it is typically necessary for each autonomous vehicle of the group to respectively travel from a base location to a target location, to carry out its own individual role in completing the task / mission successfully, and to return to the base or some other specified location. This will typically involve appropriate navigation to and from the target, collision avoidance (both with other members of the group and with other static or dynamic hazards), and appropriate coordination between the autonomous vehicles of the group to complete the task / mission efficiently, when in the vicinity of that target region I object. This coordination will typically involve, for example, coordinating the use of appropriate sensors from suitable positions and angles to provide the required coverage of, and / or inspection data for, the target region I object whilst minimising overlap between the information gathered by each member vehicle. However, neither autonomous navigation to a specified location nor providing the necessary coordination between different agents is trivial - especially as the number of agents increases. For example, the use of a conventional approaches to multi-agent path finding to determine optimal collision-free paths for a group of agents, from their current locations, to an assigned target becomes increasingly unfeasible with environment size and the number of agents involved. Accordingly, for applications in which multi-agent path finding is involved, there are inevitable trade-offs between efficiency and effectiveness. Similarly, conventional approaches to multi-agent planning to coordinate the resources and activities of multiple agents, to achieve a predefined goal, in an optimal way becomes more difficult as the number of agents increases. The invention aims to provide at least one method and / or apparatus that overcomes or at least partially ameliorates the above issues. In one aspect the invention provides a system for carrying out a task in respect of a target in a physical environment, the system comprising: an agent group comprising a plurality of autonomous agents, each autonomous agent comprising a vehicle configured for operation under the control of a control neural network; wherein each control neural network comprises a different respective instance of the same trained neural network, and the trained neural network is trained to provide: a series of control outputs for navigating a corresponding autonomous agent of the agent group towards the target, and for controlling the corresponding autonomous agent to provide at least a contribution to carrying out the task, wherein each control output is respectively generated, by the trained neural network, based on a plurality of input vectors, the plurality of input vectors include a first input vector that represents a first set of observation inputs for the corresponding autonomous agent, and at least one further input vector, and each further input vector respectively represents a further set of observation inputs for a different autonomous agent of the agent group. The trained neural network may comprise a first stage for generating the first input vector based on the first set of observation inputs. The first stage may be configured to perform compression and / or encoding of the first set of observation inputs to generate the first input vector. The first input vector may be combined with each further input vector to generate a combined vector. The first input vector may be combined with each further input vector to generate the combined vector by concatenating a sum of each further input vector with the first input vector. The trained neural network may comprise at least one further stage for generating each control output based on the combined vector. Each further input vector may be respectively received from the control neural network of the corresponding different autonomous agent. Each set of observation inputs may be respectively based on a coordinate reference frame centred on the autonomous agent to which that set of observation inputs relates. Each autonomous agent may be respectively configured for transforming an initial set of observation inputs for that autonomous agent into the coordinate reference frame centred on that autonomous agent to generate the set of observation inputs for provision to the control neural network of that autonomous agent. Each set of observation inputs may be respectively based on one or more of the following: a position of the autonomous agent to which that set of observation inputs relates; at least one movement characteristic (e.g., velocity) of the autonomous agent to which that set of observation inputs relates; a position of the target; and / or at least one measurement representing distance to at least one object. In one aspect the invention provides an autonomous agent, for providing at least a contribution to carrying out a task in respect of a target in a physical environment as part of an agent group comprising a plurality of such autonomous agents, the autonomous agent comprising: a vehicle configured for operation under the control of a control neural network; wherein the control neural network comprises an instance of a trained neural network, and the trained neural network is trained to provide: a series of control outputs for navigating the autonomous agent towards the target, and for controlling the autonomous agent to provide at least a contribution to carrying out the task, wherein each control output is respectively generated, by the trained neural network, based on a plurality of input vectors, the plurality of input vectors include a first input vector that represents a first set of observation inputs for the autonomous agent, and at least one further input vector, and each further input vector respectively represents a further set of observation inputs for a different autonomous agent of the agent group. In one aspect the invention provides a training architecture for training over a series of training iterations an actor neural network to become a trained neural network for deployment in the autonomous agents of the above system or in the above autonomous agent, the training architecture comprising: a respective instance of the actor neural network, and a respective instance of a critic network, for each training agent of a plurality of training agents, wherein each training agent represents a respective autonomous agent of the system, the actor neural network is configured for generating, at each training iteration, at least one control output for controlling a corresponding training agent based on a first set of neural network parameters, each control output being respectively generated based on a plurality of input vectors, the plurality of input vectors including a first input vector that represents a first set of observation inputs for the corresponding training agent, and at least one further input vector, each further input vector respectively representing a further set of observation inputs for a different training agent of the plurality of training agents, and the critic neural network is configured for generating, at each training iteration, at least one value of an environment state for the corresponding training agent based on a second set of neural network parameters, each value being generated based on a critic input vector that represents a critic set of observation inputs received by the critic neural network; a reward function for respectively generating, at each training iteration, at least one reward for each training agent; and a policy optimisation function for optimising, at each training iteration, the first set of neural network parameters and the second set of neural network parameters, based on the at least one control output respectively generated for each training agent by a corresponding instance of the actor neural network, the at least one value respectively generated for each training agent by a corresponding instance of the critic neural network, and the at least one reward respectively generated for each training agent by the reward function. The actor neural network may comprise a first stage for generating the first input vector based on the first set of observation inputs. The first stage may be configured to perform compression and / or encoding of the first set of observation inputs to generate the first input vector. The first input vector may be combined with each further input vector to generate a combined vector. The first input vector may be combined with each further input vector to generate the combined vector by concatenating a sum of each further input vector with the first input vector. The actor neural network may comprise at least one further stage for generating each control output based on the combined vector. Each of the first and further sets of observation inputs may be respectively based on a coordinate reference frame centred on the training agent to which that set of observation inputs relates. The critic set of observation inputs may be based on a coordinate reference frame centred on the target. Each set of observation inputs may be respectively based on one or more of the following: a position of the autonomous agent to which that set of observation inputs relates; at least one movement characteristic (e.g., velocity) of the autonomous agent to which that set of observation inputs relates; a position of the target; and / or at least one measurement representing distance to at least one object. The reward function may be configured to generate the at least one reward based on at least one agent specific reward. The at least one agent specific reward may include at least one of the following: a progress reward for positively rewarding progress towards a goal or negatively rewarding progress away from the goal; a time penalty incurred for each step taken towards reaching a goal; a collision penalty incurred fora collision; and / or a proximity based penalty. The reward function may be configured to generate the at least one reward based on at least one coordination reward that is shared between training agents. The at least one coordination reward may include at least one of the following: an inspection target progress reward for rewarding progress of all training agents towards a goal or negatively rewarding progress away from the goal; a reward that represents a proportion of a target that is being covered or inspected at a given time; a cumulative reward for rewarding coverage or inspection of part of the target that is newly covered or inspected; a completion reward for rewarding task completion. Embodiments of the invention will now be described, by way of example, with reference to the accompanying drawings in which: Figure 1 is a simplified illustration of a multi-agent scenario; Figure 2 is a simplified schematic illustrating a network architecture for the multi-agent system during training; Figure 3 shows a coordinate reference frame, following coordinate transformation to an agent centred reference frame, for each of a plurality of physical agents shown in the multi-agent scenario of Figure 1; Figure 4 shows a coordinate reference frame, following coordinate transformation to a target centred reference frame, for the multi-agent scenario of Figure 1; Figure 5 is a simplified schematic illustrating the network architecture for the multiagent system fordeployment / inference; and Figure 6 is a simplified block schematic of a physical agent that may be used in the multi-agent scenario of Figure 1. Overview A system for training and deployment of autonomous, artificial intelligence (Al) controlled, agents of a multi-agent system will now be described by way of example only. Figure 1 is a simplified illustration of a multi-agent scenario, which the described multiagent system may be used for. As seen in Figure 1, a set of autonomous agents 102-1, 102-2, 102-3 and 102-4 are deployed in an environment 100. In the illustrated example, the environment 100 is a physical region of space within which the agents 102 operate and the agents 102 comprise ‘physical’ agents in the form of unmanned aerial vehicles or‘drones’. It will, nevertheless, be appreciated that the environment 100 may be any type of environment (e.g., terrestrial, marine, or nonterrestrial) or may be a mixed environment. Similarly, the set of agents 102 may include physical agents of any suitable type (e.g., terrestrial, aerial, marine surface, submersible, and / or space borne vehicles) and may include agents of different types depending on requirements. It will also be appreciated that while the illustrated set of agents 102 includes four agents, in a real system there may be any suitable number of agents. The agents 102 are tasked with collaborating with one another to navigate, from a base 110 to a target region or object 112 (as illustrated by the arrow labelled Abt), in as efficient manner as possible. The agents 102 are configured to do this by balancing the need to traverse the shortest distance to the target 112 that is reasonably possible and the need to avoid collision with any static and dynamic obstacles (including each other) which could cause damage to the agents 112 and reduce the effectiveness of the wider system. Beneficially, the agents 102 are able to navigate to the target 112 successfully without an ‘optimum’ path or route needing to be predetermined and set centrally by the system during training. The agents 102 are configured, on reaching the target region or object 112, to collaborate with one another to inspect, or provide coverage of, as much of the target 112 as possible in a minimum time (or at least below an acceptable maximum amount of time) - if necessary, by moving relative to the target 112 as shown illustratively by the arrows labelled B. In this example, the task is complete when the entirety of the target region or object 112 has been inspected / covered by the set of agents 102 as a whole (even though no individual agent may have inspected / covered the entire target region or object 112). It will, nevertheless, be appreciated that task completion for other types of task may be defined in a different way (for example, specific agents visiting different specified parts of a target, or one or more agents visiting multiple different target locations). The inspection / coverage is carried out by means of one or more on-board sensors and / or cameras respectively carried by each agent 102. On arrival within sensor / camera range of the target 112, the agent 102 orients itself (and / or the onboard sensor(s) / camera(s)) appropriately (if necessary) to bring a part of the target 112 requiring inspection / coverage into the field of view 120-1, 120-2, 120-3 and 120-4 of the corresponding sensor / camera. It will be appreciated that different agents 102 may be provided with a different payload. For example, the set of agents 102 may include one or more agents carrying a camera, one or more other agents may have a specific type of sensor, still one or more other agents may carry a different sensor type or some other tool that can contribute to completion of the ultimate task. This has the potential to allow more complex, higher value, tasks / mission to be carried out with different subsets of one or more drones providing different sensor / camera data. In the example of Figure 1, on completion of the task the agents 102 return to the base 110 (as illustrated by the arrow labelled Atb), although the agents 112 may, alternatively, be configured to travel to another specified location (e.g., another target region / object) and / or to complete another inspection / coverage task at that location. As will be described in more detail later, the agents 102 are each controlled by a neural network based artificial intelligence (Al) that decides the action to be performed by each agent 102 at a given time instance. The inputs to the neural network may comprise a location of the target 112 (e.g., coordinates of a centre point of the target region / object 112) and sensor and / or odometry readings for each agent 102. It will, nevertheless, be appreciated that other potential inputs may be used, such as data defining a map of the environment, or a target region of the environment requiring mapping. The neural network based Al is trained using a reinforcement learning (RL) method that provides, as its output, emergent behaviour for the agents 102 whereby the agents 102 are effectively configured to decide amongst themselves how best to achieve a given task. This contrasts with other approaches in which a method to achieve a task is prespecified. The agents 102 are trained to perform in a stochastic environment. The environment 100 itself is not fixed, so that the agents 102 are able to learn to generalise to different environments. During training, through feedback from how well they achieve a given task, the system iteratively improves how efficiently it performs a given task. Beneficially, the system utilises an actor-critic reinforcement learning framework in a particularly advantageous way. The reinforcement learning training method used (in this example Proximal Policy Optimisation (PPO)) has been configured (and in particular the set of inputs to the network have been defined) to make particularly efficient use of the raw observation data from the environment 100 that is available from the agents 102 in a manner that makes successful training possible. Moreover, the network architecture is configured to process the input data to provide processed information that is particularly relevant to an individual agent 102 and to all the agents 102 involved in the collaboration. Beneficially, the processed information is encoded and compressed for sharing between all the agents 102 involved in the collaboration (or at least between nearby agents) in a particularly efficient manner to encourage collaborative behaviour to emerge during training whilst also limiting the total amount of data that needs to be shared at each time instance. Beneficially, the collaborative group of agents 102 are trained in parallel, with the respective experiences of each individual agent 102 being aggregated, to train a unified control policy. This unified control policy is deployed in a decentralised manner to each agent 102 in the collaboration so that, at inference, the agents 102 all share the same control policy (i.e., sharing the same neural network weights). By sharing weights between the agent control policies the agents are, effectively, trained in parallel and hence, advantageously, there is no divergence between the agents during the learning process. Accordingly, this avoids potential issues arising from high sensitivity to any divergence in learning by any one of the individual agents 102 and hence helps to increase the stability to the system. It will be appreciated that, whilst the agents 102 are trained in parallel, the agents 102 operate individually during deployment / inference. The agents 102 will respond differently to each other due their different relative locations, odometry, and sensor from the regions covered by their sensor(s). Moreover, it can be seen that the emergent behaviour arising from the training is not representative of a predefined set of rules but, in effect, the emergent behaviour involves the agents 102 deciding how to act based on: the observation(s) and / or objective(s) for all the agents; and how the agents 102 interact with one another. In this way the agents 102 are trained both to locate and reach the specified target 112 and, upon arrival, to efficiently divide amongst themselves the responsibility of inspecting, or providing coverage of, the full spatial extent of the target region in one or more different ways. The system provides effective control not from the perspective of an individual centralised control or lead intelligence, but rather from a decentralised intelligence provided for each agent. The group of agents 102 does not follow a predefined flight path or organise in some pre-ordained fleet arrangement. Instead, the agents effectively navigate based on an interpretation of the mission provided by the decentralised lead intelligence and sensing of the environment. Unlike other implementations that make use of reinforcement learning, the way in which reinforcement learning is used in the multi-agent system described herein provides benefits in terms of improved scalability from relatively few to multiple (which could include hundreds or even more) autonomous agents. Furthermore, the loss of an agent has less impact on the effectiveness of the resulting system and the system is thus robust to such potential losses. Such highly scaled systems would have significant benefits in some use cases, for example responding to natural disasters or in operational cases such as defence perimeter control, or for experience impact in entertainment. Al Network Architecture The Al network architecture for the multi-agent system will now be described in more detail with reference to Figures 2 to 5. Training: Figure 2 is a simplified schematic illustrating a network architecture for the multi-agent system during training. As seen in Figure 2, the training network architecture comprises an RL framework comprising an actor-critic network which, in operation, optimises the unified control policy using a simulated training environment 200, and a plurality of ‘virtual’ agents 202-1,202-I, 202-N that simulate the behaviour of the physical agents 102 in the real-world environment 100. It will be appreciated that once training is completed, the optimised unified control policy can be shared between the physical agents 102 by deploying the trained actor-critic network (or at least the actor portion of it) to each physical agent 102 in a decentralised manner. Training is carried out over multiple (typically thousands) of procedurally generated episodes with stochastic initialisation, during which the virtual agents 202 attempt to reach a simulated target region / object to carry out the predefined task (and possibly return to base or move on to another specified location). The unified control policy is gradually optimised over the course of the episodes until the virtual agents 202 are able to navigate to the simulated target region / object efficiently, to carry out the task successfully, and then return (or move on to the next location) as necessary. The actor-critic network respectively comprises an actor neural network portion 204 and a critic neural network portion 206. A respective instance of the actor neural network portion 204, and the critic neural network portion 206, is provided for each virtual agent 202. The actor neural network portion 204 of the actor-critic network for a given virtual agent 202 determines what action that agent 202-1 should take based on a current control policy and a given environment observation 208 provided by the training environment 200. The critic neural network portion 206 provides an estimate 214 of a value of an environment state that is used for optimising the parameters of both the actor neural network 204 and the critic neural network 206 during training. During training the system is optimised for coordination between agents as they navigate towards a target. Training is based on a reward scheme made up of set of numerical rewards designed to encourage desirable emergent behaviour and discourage undesirable behaviour. The respective observation for determining what action each agent 202 should take is provided as a set of environment observation inputs. These observation inputs are optimised for successful and efficient training (and hence successful navigation and task completion at deployment / inference) and typically include inputs indicating, for example: • the agent’s position xa = (xa,ya) • one or more characteristics of the agent’s movement such as velocity (xa = (xa,ya); • the target’s position xt = (xt,yty and / or • a set of range measurements specifying a minimum distance to any other static or dynamic objects or surfaces in the vicinity. These may, for example, be based on a set of equally spaced non-overlapping sectors (e.g., circular, or spherical), spaced around the agent 202, for example around a perimeter defined around the agent (such as the circumference of a circle centred on the agent). It will be appreciated that the set of inputs may be refined for a particular multi-agent scenario, may include a subset of the inputs, and / or may include additional inputs appropriate. For example, whilst the exemplary inputs are defined in a two-dimensional frame of reference, they may be provided in a three dimensional frame of reference where appropriate. Moreover, the ‘target’ position need not be fixed but may change as the virtual agents 202 approach a final target position. For example, the target position at a given time may be an intermediate target position corresponding to the position of an intermediate destination or waypoint that a virtual agent 202 can head towards (but not necessarily reach) on route to the final target. The target position may then change to a different intermediate destination, or the final target location, at an appropriate juncture. Similarly, the intermediate target position may change steadily over time, generally following a course towards the final target. A similar approach could also be used to provide a position of one or more follow-up targets following task completion in respect of the initial, or an earlier, target. Before being processed by the actor neural network portion 204 the observation inputs are subject to a coordinate transformation 210a to transform the inputs into a local coordinate reference frame that is centred on that agent’s location (xa). This is illustrated, by way of example only, in Figure 3, which shows a coordinate reference frame, following coordinate transformation to an agent centred reference frame, when applied in respect of each of the physical agents 102 shown in the exemplary scenario of Figure 1. The coordinate transformation 210a may also rotate the coordinate reference frame for alignment with the target position (e.g., along the x-axis), in scenarios in which it is beneficial to do so i.e., such that the target has the coordinates (xt - xa,0). The observation inputs may also be normalised, by a nominal velocity / distance divisor. It will be appreciated that the transformation into a relative coordinate system beneficially allows the system to generalise for efficient navigation in an arbitrary absolute cartesian coordinate systems. In some scenarios, the coordinate system rotation has the potential to simplify the size and complexity of the observation space. This is because equivalent experiences / observations from different perspectives effectively reduce to essentially the same transformed observation and thus limits the number of experiences the control policy has to learn to handle in a tractable training period, with modest neural network complexity. Nevertheless, in other scenarios omission of the coordinate system rotation can enhance performance, for example when the agent needs to know its position relative to the target and not just its radial distance. Normalisation of the inputs helps to keep the data within a working range of the neural network. The inputs corresponding to the observations 208, for each of the virtual agents 202, are also provided to the corresponding critic neural network instance 206 following a coordinate transformation 210-2. In this case, however, the observation inputs 208 for each agent 202 are effectively ‘inverted’, by transforming them into a coordinate reference frame that is centred on the target (i.e., based on the locations of each agent 202, at each timestep, from a reference frame that is centred on the target region). This is illustrated, by way of example only, in Figure 4, which shows a coordinate reference frame, following coordinate transformation to a target centred reference frame, for the multi-agent scenario of Figure 1. Transforming the inputs provided to the critic neural network 206 in this way beneficially helps the virtual agents 202 in the training environment 202 to be trained efficiently to reach a simulated target region relatively quickly (in a minimum, or near minimum, amount of time) whilst reducing the risk of random dithering. This effectively aids the training of the virtual agents 202 (and hence operation of the physical agents 102 atdeployment / inference) to orient themselves relative to the simulated target (and hence the real-world target 112 at deployment / inference). The critic neural network 206 is able to discern, for example, the position, relative to the simulated target region, of each virtual agent 202 (i.e., which ‘side’ of the simulated target the virtual agent is on) and hence from which direction that virtual agent 202 is approaching. Moreover, beneficially, this training approach allows optimisation of the control policy to control the virtual agents 202 (and hence the physical agents 102 at deployment / inference) to be able to adjust their behaviours based on their distance from the target. More specifically, part of the emergent behaviour arising from this training is the way in which the agents 202 position themselves relative to one another, and relative to the target. Ultimately, following training, this advantageously adapts the physical agents 102 (on which the trained network is deployed) to retain, in general, a natural formation all the way from the base 110 until they reach the target 112. The transformed observation inputs to the actor neural network 204 and the critic neural network 206 are respectively compressed, by a first stage (comprising one or more layers) of the corresponding neural network 204, 206, into an encoded latent feature vector ( / .e., a vector that represents one or more features latent in - or inherent to, but not directly observable from - the corresponding observation). It will be appreciated that the way in which the observation inputs are respectively compressed (encoded) in each neural network 204, 206, is, itself, the subject of optimisation during training and so is effectively learned during the training stage. The latent feature vector respectively produced by each instance of the actor neural network 204 for a corresponding virtual agent 202, is shared with the instances of the actor neural network 204 for all the other virtual agents 202 completing the same task (or at least the nearby agents) as seen at 216. This effectively gives each virtual agent 202 (and hence each physical agent 102 following deployment) knowledge of the state of its collaborating partners. It will be appreciated that training for very large systems could be further optimised by only sharing the latent vectors with agents that are nearby and actively participating in the same task. Other agents that come close could be treated as obstacles to avoid. This allows the system to be extended to very large fleets. As the compression / encoding scheme is learned during the training stage the latent feature vector is optimised to have a relatively small memory overhead (e.g., in comparison to the original raw input observation) and so can be shared, in a resource efficient manner, using a relatively limited communication bandwidth. Moreover, the resulting compression itself is relatively efficient and can be computed with a relatively low latency. The latent feature vectors are also combinable through addition, enabling the system to scale with the number of agents in the collaboration (without placing any fixed requirements on the number of agents operating and sharing encoded observations at any one time). For each instance of the actor neural network 204, the shared latent feature vectors for the different virtual agents 202 are combined to produce an agent specific combined latent feature vector that is processed by further layers of the actor neural network 204 to infer an appropriate action 218. Specifically, for each observation update at each specific virtual agent 202 of a given collaborative group of agents, the latent feature vectors produced by the first stages of the actor neural network instances 204 of other agents 202 of the collaborative group, are summed (i.e., by adding the latent vectors from the other agents 202 together) and concatenated with that specific virtual agent’s latent feature vector. Hence, this results in an agent specific latent feature vector for each specific virtual agent 202 of the collaborative group. Each agent specific latent feature vector is respectively processed by a second stage of the actor neural network instance 204 of the corresponding virtual agent 202, to infer an action 218 (e.g., represented by one or more control outputs) for that corresponding virtual agent 202. For example, at each timestep, control outputs corresponding to a new set of coordinates and / or a displacement vector may be generated, to allow each virtual agent 200 to step towards the target and, on arrival in the vicinity of the target, to carry out the inspection / coverage of the target. A point on a boundary (e.g., circumference) or bounding surface of the target region / object may be considered as inspected or covered when it is determined to have passed within a field of view and depth of field of an onboard sensor and / or camera of the virtual agent 202. During training the system is optimised for coordination between the virtual agents 202 as they navigate towards the target. Training is based on a reward scheme made up of a set of numerical rewards 222 designed to encourage desirable emergent behaviour and discourage undesirable behaviour. For example, as training progresses, the control policy is developed based on a set of ‘agent’ rewards and penalties 222 (or negative rewards) that are provided by the training environment 200 (e.g., by an appropriately configured reward function within the environment 200), for each agent 202. These rewards typically include, but are not limited to: • A conservative agent progress reward that is used to positively reward progress towards a goal, and to negatively reward progress away from that goal, and so encourage movement towards the goal without specifying an optimum route. This beneficially provides a non-sparse / rich reward during training, that contributes to making the policy train faster and more stably. The ‘conservative’ nature of this reward means that the agent is effectively ‘encouraged’ to move towards the goal, as opposed to being directed towards a direct path to the goal; • A constant time penalty (negative reward) that is incurred for each step taken towards reaching a goal (e.g., at each time step). This encourages task completion in the shortest period of time to minimise the cumulative time penalty; • A significant (relatively large) collision penalty (negative reward) that is incurred in the event of a collision; and / or • A proximity based ‘discomfort reward’, which gradually increases to increasingly penalise (negatively reward) agents as they get closer to one another. Whist this may be in addition to the collision penalty that is applied in the event of a collision, the discomfort reward can further reduce the risk of collision and help to avoid a sudden large fluctuation in reward value that might otherwise arise from a collision during training. Moreover, to foster coordination between agents, a number of ‘shared’ or ‘coordination’ rewards may be provided by the training environment 200 which are averaged across the agents 202 and therefore encourage coordination / collaboration at a group level. These coordination rewards typically include, but are not limited to: • A conservative inspection target progress reward that is similar to the conservative agent progress reward but is focused on the target, for all agents of the collaborative group; • A reward for the proportion of the target which is currently being inspected / covered by an agent. This is given by the length / area of the target’s external boundary which is currently being inspected / covered by an agent, divided by the total length / area of the target’s external boundary. This may be used, for example, to promote maximum cumulative coverage by all agents 202 in a given timestep (e.g., by encouraging different agents to cover different parts of the target); • A cumulative reward, which accounts for every new part of the target’s inspectable / coverable external boundary that has been inspected / covered by the agents and therefore gradually increases as more of the external boundary has been inspected / covered. This effectively rewards progress towards task completion; and / or • A completion reward that rewards the agents when the task is deemed to be complete. During training, a policy optimisation scheme 224 is provided that is configured for generating parameter updates 226 for modifying (e.g., for adjusting the weights applied in) both the actor and the critic neural networks 204, 206, based on: • The rewards 222 determined for each agent 202 that are provided by the simulated training environment 200; • The value of each environment state 214, determined for the agents 202 collectively, that are provided by the critic neural network 206; and / or • The respective actions (control output(s)), determined for each agent 202, that are provided by the corresponding actor neural network instance 206. It will be appreciated that, in the context of real world physical agents 102 (i.e., individual hardware systems that interact with one another) safety is an important consideration that needs to be ensured not only for the agents themselves but also for users, and other people who may be in the vicinity while the system is operating. This is particularly important in the context of unmanned aerial vehicles due to the constraining nature of current aviation regulations. Current regulations are, for example, restricted to having a pilot-in-the-loop primarily due to wider societal concerns regarding the safety of autonomous systems. The approach described has the potential to lead to demonstrable safety improvements that can support a relaxation of regularity constraints and hence allow more efficient ways of operating drone fleets. As well as providing benefits in terms of efficiency, reliability, and effectiveness (in terms of how the agents move to and inspect a specific region collectively in the shortest time reasonably possible), the reward scheme used in training helps to provide enhanced safety by ensuring that when the trained actor network is deployed on physical agents the agents will not collide with one another or with any other obstacles in their path. Deployment: Figure 5 is a simplified schematic illustrating the network architecture for the multiagent system for deployment / inference. As seen in Figure 5, the deployed network architecture in the illustrated example comprises a respective instance of the trained actor neural network 504-1,504-2, 504-i and 504-N, for each of a plurality of agents 102-1, 102-2, 102-i, and 102-N (e.g., the physical agents shown in Figure 1). The instances of the trained actor neural network 504-1, 504-2, 504-i and 504-N may, for example, be instances of the actor neural network 206 shown in Figure 2 following training completion. Each instance of the actor neural network 504 respectively determines what action the corresponding agent 102 should perform based on a control policy and a given environment observation 508 based on inputs provided by that agent 102 when deployed in the physical environment 100 (e.g., from sensors and / or a positioning system carried the agent) and any other relevant input (e.g., target position 526). The respective environment observation for determining what action each agent 102 should take is provided as a set of real-world environment observation inputs corresponding to those used during training - i.e., including inputs indicating, for example: • the agent’s position xa = (xa,ya), as indicated, for example, by a positioning system or the like; • one or more characteristics of the agent’s movement such as velocity (xa = (xa,ya) . as indicated, for example, by motion sensors (such as an accelerometer) or the like; • the target’s position xt = (xt,yty and / or • the set of range measurements specifying a minimum distance to any other static or dynamic objects or surfaces in the vicinity of the agent. The range measurements may, for example, be based on appropriate proximity sensor measurements (e.g., light detection and ranging, LiDAR or ultrasonic range measurements). Before being processed by the actor neural network portion 204 the observation inputs are subject to the coordinate transformation 510a (corresponding to the coordinate transformation 210a used during training) to transform the inputs into the local coordinate reference frame that is centred on that agent’s location (xa). The coordinate transformation 510a may also rotate the coordinate reference frame for alignment with the target position (e.g., along the x-axis), in scenarios in which it is beneficial to doso / .e., such that the target has the coordinates (xt - xa, 0). The observation inputs may also be normalised, where appropriate, by a nominal velocity I distance divisor. The transformed observation inputs to each actor neural network instance 504 are respectively compressed, by a first stage 530a-1, 530a-2, 530a-i, and 530a-N (comprising one or more layers), of that actor neural network instance 504 into an encoded latent feature vector. The latent feature vector respectively produced by the first stage 530 each instance of the actor neural network 504 for a corresponding agent 102, is shared with the instances of the actor neural network 504 for all the other virtual agents 102 completing the same task (or at least the nearby agents) as seen generally at 516. This effectively gives each agent 102 knowledge of the state of its collaborating partners. The latent feature vector produced by each instance of the actor neural network 504 for a corresponding agent 102, is shared with the instances of the actor neural network 504 for all the other virtual agents 102 completing the same task (or at least the nearby agents) as seen generally at 516. This effectively gives each agent 102 knowledge of the state of its collaborating partners. For each instance of the actor neural network 504, the shared latent feature vectors for the different virtual agents 102 are combined, by a further stage 532a-1, 532a-2, 532a-i, and 532a-N (comprising one or more further layers), of that actor neural network 504, to produce an agent specific combined latent feature vector. Each combined latent feature vector is passed to a final stage 534a-1, 534a-2, 534a-i, and 534a-N (comprising one or more further layers) of the corresponding the actor neural network instance 504 to determine an appropriate action 518 (one or more control outputs) that is then applied to control the corresponding agent 102 in an appropriate manner. Agent An autonomous agent 102 for use in the multi-agent system will now be described, by way of example only, with reference to Figure 6, which is a schematic block diagram illustrating the main components of the agent 102. It will be appreciated that while an unmanned aerial vehicle is shown for illustrative purposes, the agent 102 may be of any suitable type (e.g., a terrestrial, aerial, marine surface, submersible, and / or space borne vehicle). As seen in Figure 6, the agent 102 has a transceiver circuit 610 for transmitting signals to and for receiving signals from other agents and / or other entities (such as an agent controller and / or a control centre) via one or more antenna 612. The agent 614 will typically include one or more external interfaces 614 for allowing a user to connect to and / or otherwise interact with the agent (e g., to install and / or update software and / or apply appropriate configuration settings). The agent 614 has a controller 616 for controlling the operation of the agent 614. The controller 616 is associated with a memory 618. Software may be pre-installed in the memory 618, may be downloaded via a communications network, and / or may be installed from a removable data storage device (RMD) (e.g., connected via an appropriate external interface 614). The controller 616 is configured to control the overall operation of the agent 102 by, in this example, program instructions or software instructions stored within memory 618. As shown, these software instructions include, among other things, an operating system 620, a communications control module 622, an agent control module 624 and, in the illustrated example, an Al module 626 comprising a trained actor instance 628. The agent 102 also includes one or more output devices 630 that are controlled by the controller 616. The output devices 630 include one or more propulsion devices (such as motors and / or the like) for moving the agent 102 as it navigates to the target. The output devices 630 may also include one or more other output devices where appropriate (e.g., lighting devices 634 such as light emitting diodes (LEDs) or other light sources, and / or audio devices 636 for providing sound). The agent 102 also includes input devices 638 (e.g., sensors) for providing feedback related to the agent 102 (e.g., to support the environment observations). The input devices 638 include one or more positioning devices 640 (such as a satellite positioning module of a global positioning system (GPS) or similar) for providing information identifying a location of the device, and one or more motion sensors 642 (such as accelerometers gyroscopes and or the like). The input devices 644 may also include one or more other input devices where appropriate. The input device may, for example, include: one or more pressure sensors 644 (e.g., a barometric pressure sensor for providing information indicating an altitude); one or more optical sensors 646; one or more thermal sensors 648; one or more proximity sensors 650 (e.g., based on LIDAR); one or more ultrasonic sensors 652 (e.g., for helping provide improved altitude stability); one or more cameras 654 (e.g., still image, moving image, visible light, infrared, cameras for providing appropriate imaging of the target); one or more ground penetrating radar devices 656 (e.g., for providing subsurface inspection data); and / or one or more magnetometers (e.g., for providing information identifying a heading of the agent 102). It will be appreciated that there are many different sensors / input devices that may be deployed on the agent 102 depending on requirements. The communications control module 622 is operable to control the communication between the agent 102, via the transceiver circuitry 610, with other agents and / or other entities (such as an agent controller and / or a control centre). The communications control module 622 controls, for example, the communication of acquired input data from the input devices 638 for supporting the sharing of environment observations. The agent control module 624 provides agent side control of the agent 102 for example by controlling the agent’s movement (e.g., via the agent’s propulsion device(s) 632) and acquisition of data from the input devices 638. The Al module 626 provides the instructions to the agent control module 624 for controlling the overall operation of the agent 102 as it navigates towards a target. The trained actor instance 628, forming part of the Al module 626, comprises an instance of the trained actor neural network described with reference to Figures 2 and 5. It will be appreciated that the Al module 626 may also include the coordinate transformation function(s) and also an instance of the critic neural network described with reference to Figures 2 and 5. It will be appreciated that whilst the Al module 626 including the trained actor instance 628 are, in the illustrated example, deployed on the agent 102 they may be deployed elsewhere in the system for example on a separate agent controller for providing control to the agent (e.g., in a similar manner to the way in which a human operator would control the drone). Moreover, he Al module 626 including the trained actor instance 628 may be deployed at a central control server (e.g., as part of centralised controller that provides control inputs for all agents). Modifications and Aiternatives A detailed embodiment has been described above. As those skilled in the art will appreciate, a number of modifications and alternatives can be made to the above embodiments whilst still benefiting from the inventions embodied therein. For example, whilst the neural network architecture has been described in the context of a single network it will be appreciated that two networks may be used. One network may, for example, be responsible for setting intermediate destination positions (or ‘waypoints’) for the agents on the way to the final target region / object (which may be referred to as a coordination network). Another network may be responsible for progressing the agents towards the target based on the destination waypoints provided by the coordination network (which may be referred to as refer to as an agent network). The agent network may be similarto that described with reference to Figures 2 and 3 albeit that at a given time point, the ‘target position’ may correspond to an intermediate destination provided as an input from the coordination network. The coordination network may also be trained using a reinforcement learning framework, based on an actor-critic method, in which the actor neural network provides outputs corresponding to the target position. In this context the agent network may be trained based on the set of agent rewards and penalties described with reference to Figure 2, or a subset of those rewards / penalties. Similarly the coordination network may be trained based on the set of coordination rewards and penalties described with reference to Figure 2, or a subset of those rewards / penalties. It will also be appreciated that whilst training using a simulated training environment provides a particularly efficient and safe way to train the actor neural network for deployment in a physical agent, training could be carried out in a real-world physical environment. Various other modifications will be apparent to those skilled in the art and will not be described in further detail here.

Claims

1. A system for carrying out a task in respect of a target in a physical environment, the system comprising:an agent group comprising a plurality of autonomous agents, each autonomous agent comprising a vehicle configured for operation under the control of a control neural network;wherein each control neural network comprises a different respective instance of the same trained neural network, and the trained neural network is trained to provide:a series of control outputs for navigating a corresponding autonomous agent of the agent group towards the target, and for controlling the corresponding autonomous agent to provide at least a contribution to carrying out the task, whereineach control output is respectively generated, by the trained neural network, based on a plurality of input vectors,the plurality of input vectors include a first input vector that represents a first set of observation inputs for the corresponding autonomous agent, and at least one further input vector, andeach further input vector respectively represents a further set of observation inputs for a different autonomous agent of the agent group.

2. A system as claimed in claim 1, wherein the trained neural network comprises a first stage for generating the first input vector based on the first set of observation inputs.

3. A system as claimed in claim 2, wherein the first stage is configured to perform compression and / or encoding of the first set of observation inputs to generate the first input vector.

4. A system as claimed in any preceding claim, wherein the first input vector is combined with each further input vector to generate a combined vector.

5. A system as claimed in claim 4, wherein the first input vector is combined with each further input vector to generate the combined vector by concatenating a sum of each further input vector with the first input vector.

6. A system as claimed in claim 4, or 5, wherein the trained neural network comprises at least one further stage for generating each control output based on the combined vector.

7. A system as claimed in any preceding claim, wherein each further input vector is respectively received from the control neural network of the corresponding different autonomous agent.

8. A system as claimed in any preceding claim, wherein each set of observation inputs is respectively based on a coordinate reference frame centred on the autonomous agent to which that set of observation inputs relates.

9. A system as claimed in claim 8, wherein each autonomous agent is respectively configured for transforming an initial set of observation inputs for that autonomous agent into the coordinate reference frame centred on that autonomous agent to generate the set of observation inputs for provision to the control neural network of that autonomous agent.

10. A system as claimed in any preceding claim, wherein each set of observation inputs is respectively based on one or more of the following:a position of the autonomous agent to which that set of observation inputs relates;at least one movement characteristic (e.g., velocity) of the autonomous agent to which that set of observation inputs relates;a position of the target; and / orat least one measurement representing distance to at least one object.

11. An autonomous agent, for providing at least a contribution to carrying out a task in respect of a target in a physical environment as part of an agent group comprising a plurality of such autonomous agents, the autonomous agent comprising:a vehicle configured for operation under the control of a control neural network;wherein the control neural network comprises an instance of a trained neural network, and the trained neural network is trained to provide:a series of control outputs for navigating the autonomous agent towards the target, and for controlling the autonomous agent to provide at least a contribution to carrying out the task, whereineach control output is respectively generated, by the trained neural network, based on a plurality of input vectors,the plurality of input vectors include a first input vector that represents a first set of observation inputs for the autonomous agent, and at least one further input vector, andeach further input vector respectively represents a further set of observation inputs for a different autonomous agent of the agent group.

12. A training architecture for training over a series of training iterations an actor neural network to become a trained neural network for deployment in the autonomous agents of the system of any of claims 1 to 10, or in the autonomous agent of claim 11, the training architecture comprising:a respective instance of the actor neural network, and a respective instance of a critic network, for each training agent of a plurality of training agents, whereineach training agent represents a respective autonomous agent of the system,the actor neural network is configured for generating, at each training iteration, at least one control output for controlling a corresponding training agent based on a first set of neural network parameters, each control outputbeing respectively generated based on a plurality of input vectors, the plurality of input vectors including a first input vector that represents a first set of observation inputs for the corresponding training agent, and at least one further input vector, each further input vector respectively representing a further set of observation inputs for a different training agent of the plurality of training agents, andthe critic neural network is configured for generating, at each training iteration, at least one value of an environment state for the corresponding training agent based on a second set of neural network parameters, each value being generated based on a critic input vector that represents a critic set of observation inputs received by the critic neural network;a reward function for respectively generating, at each training iteration, at least one reward for each training agent; anda policy optimisation function for optimising, at each training iteration, the first set of neural network parameters and the second set of neural network parameters, based on the at least one control output respectively generated for each training agent by a corresponding instance of the actor neural network, the at least one value respectively generated for each training agent by a corresponding instance of the critic neural network, and the at least one reward respectively generated for each training agent by the reward function.

13. A training architecture as claimed in claim 12, wherein the actor neural network comprises a first stage for generating the first input vector based on the first set of observation inputs.

14. A training architecture as claimed in claim 13, wherein the first stage is configured to perform compression and / or encoding of the first set of observation inputs to generate the first input vector.

15. A training architecture as claimed in any of claims 12 to 14, wherein the first input vector is combined with each further input vector to generate a combined vector.

16. A training architecture as claimed in claim 15, wherein the first input vector is combined with each further input vector to generate the combined vector by concatenating a sum of each further input vector with the first input vector.

17. A training architecture as claimed in claim 15, or 16, wherein the actor neural network comprises at least one further stage for generating each control output based on the combined vector.

18. A training architecture as claimed in any of claims 12 to 17, wherein each of the first and further sets of observation inputs is respectively based on a coordinate reference frame centred on the training agent to which that set of observation inputs relates.

19. A training architecture as claimed in any of claims 12 to 18, wherein the critic set of observation inputs is based on a coordinate reference frame centred on the target.

20. A training architecture as claimed in any of claims 12 to 19, wherein each set of observation inputs is respectively based on one or more of the following:a position of the autonomous agent to which that set of observation inputs relates;at least one movement characteristic (e.g., velocity) of the autonomous agent to which that set of observation inputs relates;a position of the target; and / orat least one measurement representing distance to at least one object.

21. A training architecture as claimed in any of claims 12 to 20, wherein the reward function is configured to generate the at least one reward based on at least one agent specific reward.

22. A training architecture as claimed in claim 21, wherein the at least one agent specific reward includes at least one of the following:a progress reward for positively rewarding progress towards a goal or negatively rewarding progress away from the goal;a time penalty incurred for each step taken towards reaching a goal;a collision penalty incurred for a collision; and / ora proximity based penalty.

23. A training architecture as claimed in any of claims 12 to 22, wherein the reward function is configured to generate the at least one reward based on at least one coordination reward that is shared between training agents.

24. A training architecture as claimed in claim 23, wherein the at least one coordination reward includes at least one of the following:an inspection target progress reward for rewarding progress of all training agents towards a goal or negatively rewarding progress away from the goal;a reward that represents a proportion of a target that is being covered or inspected at a given time;a cumulative reward for rewarding coverage or inspection of part of the target that is newly covered or inspected;a completion reward for rewarding task completion.

25. A method for training over a series of training iterations an actor neural network to become a trained neural network for deployment in the autonomous agents of the system of any of claims 1 to 10, or in the autonomous agent of claim 11, using the training architecture of any of claims 12 to 24, the method comprising:each instance of the actor neural network of the training architecture generating, at each training iteration, at least one control output for controlling a corresponding training agent based on a first set of neural network parameters, each control output being respectively generated based on a plurality of input vectors, the plurality of input vectors including a first input vector that represents a first set ofobservation inputs for the corresponding training agent, and at least one further input vector, each further input vector respectively representing a further set of observation inputs for a different training agent of the plurality of training agents,each instance of the critic neural network of the training architecture 5 generating, at each training iteration, at least one value of an environment state for the corresponding training agent based on a second set of neural network parameters, each value being generated based on a critic input vector that represents a critic set of observation inputs received by the critic neural network;the reward function of the training architecture respectively generating, at each 10 training iteration, at least one reward for each training agent; andthe policy optimisation function of the training architecture optimising, at each training iteration, the first set of neural network parameters and the second set of neural network parameters, based on the at least one control output respectively generated for each training agent by a corresponding instance of the actor neural 15 network, the at least one value respectively generated for each training agent by a corresponding instance of the critic neural network, and the at least one reward respectively generated for each training agent by the reward function.

Images