Systems and methods for controlling remotely operable machines

Abstract

A method for controlling a machine, such as a drone, that is remotely operable to perform a task. The method comprises: i) Creating node graph 900 for the task. The node graph includes a plurality of

Description

[0002] Remotely operable machines (ROMs) are being increasingly used in many fields of technology to accomplish tasks at a distance. There are many different types of ROMs, e.g. stationary or mobile, which have varying functionality and can be controlled via internal hardware systems. In addition to hardware configured to move the ROM itself, a ROM is often equipped with various sensors, such as imaging or environmental sensors, to collect data that can be subsequently analysed and acted upon.

[0003] Some examples of ROMs include vehicles such as drones or unmanned ground vehicles which may be used to perform tasks, such as parcel delivery, road monitoring (e.g. to provide traffic updates, or collision reports), mapping and / or reconnaissance tasks. Mobile ROMs may need to be controlled both independently and co-operatively to complete any given task.

[0004] ROMs are also widely used in the energy industry, for example wind turbines comprise hardware systems that are responsible for the operation and security of the wind turbines. These hardware systems have different functionality to the hardware systems comprised in the vehicles described above, thus requiring different input control signals. The wind turbine hardware systems may include functionality for internal security or system updates but may also communicate with environmental sensors comprised in the wind turbine. Furthermore, in addition to ensuring the safe operation of a single turbine, it is desirable for users to be able to simultaneously ensure the safe operation of a plurality of wind turbines located on a single wind farm. Therefore, there is high motivation in multiple technical fields to use multiple hardware systems co-operatively and safely. SUMMARY OF THE DISCLOSURE

[0005] Different types of ROMs are made by different manufacturers, using different hardware that is controlled using different commands. Existing systems to control remotely operable machines are often hard-coded and specific to particular types of machines or manufacturers. Furthermore, existing approaches for completing tasks through the collaboration of two or machines tend to focus on enabling co-operation between hardware systems which have been built by a single manufacturer. As hardware systems built by different manufacturers often require different input control signals, each type of hardware system requires a different hard-coded solution. Enabling collaboration between hardware systems is often an arduous and inefficient process requiring bespoke re-writing of large sections of code.

[0006] Furthermore, existing systems tend to focus on controlling single ROMs to perform a task, whereas increasingly, it is desirable for two or more ROMs to collaborate together to perform a task.

[0007] Although there exist data models that aim to enable collaboration between a plurality of hardware systems, they only enable a small, limited quantity of hardware systems to co-operate. As mentioned above, this is often only achievable for hardware systems built by the same manufacturer.

[0008] As a result, existing systems to control ROMs have a number of limitations. Firstly, they are not scalable because they are hard-coded solutions for performing particular tasks that cannot easily be generalised to different types of ROMs or to multiple ROMs.

[0009] Another limitation of existing systems is that they have type safety limits resulting in potentially unsafe task execution. It is complex to integrate new ROMs or tasks into existing systems in a typesafe manner due to frequent incompatibility of the underlying data and data types and this can result in tasks being executed unsafely or failing to execute at all.

[0010] Moreover, when a data model is modified to control two or more hardware systems collaboratively, the combination of two or more hard-coded data models for each separate system may result in errors or failure modes of the systems. There is therefore a need for type safety checks to be carried prior to running the data models, to ensure that errors do not arise when the machines comprising the systems perform a task.

[0011] In summary, problems arise when a data model underlying a co-operative human-machine or machine-machine teaming system is either limited in its ability to scale (scale limits), limited in its compatibility with multiple manufacturers (agnosticism limits), or potentially unsafe in its task execution (type safety limits). It is an objective of embodiments herein to address at least some of the limitations described above.

[0012] Thus, according to an aspect herein, there is provided a system for controlling a first machine that is remotely operable to perform a first task, the system comprising one or more processors collectively configured to: i) create a first node graph for the first task, the first node graph comprising a plurality of execution nodes, each execution node defining execution logic (which may e.g. be verb logic) that is to be performed on data obtained from the first machine, to process said data, and wherein the output of the first node graph is an action to be performed by the first machine to perform the first task; ii) propagate the data received from the first machine in real-time, through the first node graph to determine a first action to be performed by the first machine; and iii) send a control message to the first machine to instruct the first machine to perform the first action.

[0013] The aforementioned system has a number of advantages as a result of the data model used. The structure of the node graph is inherently modular in nature which results in a scalable data model whereby new node graphs can easily be defined for new scenarios.

[0014] As each execution node defines verb logic that is to be performed on the data, each execution node only knows about the verb therein and is not concerned with the other nodes comprised in the node graph (as such, each node is agnostic to the nodes that precede it and / or follow it in the node graph). As a result of this structure, subsequent execution nodes can easily be added for controlling the first machine and modifying the actions performed to complete the first task.

[0015] Furthermore, the structure of the node graphs facilitate robust type-safe procedures as each node can independently be checked to ensure it is type safe. Moreover, using this structure, the entirety of the node graph can be mathematically checked to ensure that the result is type-safe.

[0016] Moreover, the system allows dynamic control of the first machine as the data received from the first machine is propagated through the node graph in real-time to determine an action to be performed by the first machine.

[0017] In some embodiments, the system may comprise a plurality of execution nodes wherein each execution node may have the following structure: a verb input interface defining one or more classes for the input parameters to the respective node; a verb output interface defining one or more classes of output parameters of the respective node; and a verb core comprising the execution logic (e.g. verb logic) for the respective node, wherein the verb logic corresponds to the execution logic for the respective execution node.

[0018] The above-described structure of an execution node results in well-defined and rigorously enforced input and output data types. Further, by defining the classes for the input parameters, data can be input to each execution node from multiple other nodes. These features result in a data model that is easily scalable.

[0019] In some embodiments, the data obtained from the first machine is a data stream and the verb core is configured to iteratively execute the execution logic in a loop, enabling data points in the data stream to be processed in real-time.

[0020] An advantage of processing the data stream in real-time is that the control message sent to the machine can be dynamically updated in real-time and in response to the data stream, regardless of the frequency and manner that the data is input to the node graph.

[0021] Furthermore, the real-time processing of the data stream enables more granular data processing than in data models of the prior art, thereby giving the user more control over the data.

[0022] In some embodiments, the verb core may be configured to iteratively execute until a stop condition defined in the verb core is met, the stop condition being a condition that is met when the first task is complete.

[0023] The iterative execution of the verb core leads to extremely quick and efficient data processing as each iteration of input data (each data point) is traversed through the entirety of the node graph in real-time. Therefore, different data points can be processed simultaneously in different nodes.

[0024] In some embodiments, the first node graph is multi-threaded and / or multi-processed and the one or more processors are configured to perform simultaneous, looped, multi-threaded or multiprocessed execution of the verb cores of each of the plurality of execution nodes in the first node graph. [002S] The multi-threaded / multi-processed node graph structure advantageously enables ail verb cores to execute simultaneously instead of sequentially, thereby enabling data processing parallelisation such that data is traversed through the node graph quickly and efficiently. This results in a highly efficient processing system for processing real-time streams of data enabling the system to be highly reactive to changes / updates in the data.

[0026] Additionally, in a multi-threaded / multi-processed structure, one or more threads can be prioritised over other threads, and the execution of non-priority threads can be optionally stopped to increase computational efficiency.

[0027] In some embodiments, the first node graph has a directed acyclic graph (DAG) structure.

[0028] A DAG node graph structure ensures that data only flows in one direction through the node graph, thereby eliminating redundant computation.

[0029] In some embodiments, each execution node is of one of the following types: a data node comprising a verb input interface acting as the entry point of the data into the first node graph, and a verb output interface for providing the data for ongoing propagation into the first node graph; an intelligence node configured to perform a transform on the data to obtain one or processed data items for use in determining the first action; and an action node configured to output the determined first action from the first node graph.

[0030] Pre-defined features of different types of execution nodes, facilitates users to easily create and / or modify node graphs by treating each execution node as a well-defined “building block”. These features result in a scalable data model, wherein additional nodes can easily be added to perform additional actions or additional tasks.

[0031] In some embodiments, at least one pair of nodes in the plurality of execution nodes are separated from one another in the first node graph by an intermediary node, the intermediary node acting as a holding node for storing a first output of a first node of the pair of nodes, and for providing said first output as input to a second node of the pair of nodes.

[0032] Advantageously, a holding node ensures robustness and auditability of the system. An intermediary node also ensures that each execution node does not have to be configured to output the appropriate data type for the subsequent execution node(s) as the intermediary nodes decouple one node from another.

[0033] In some embodiments, the intermediary node sits on an edge between the first and second nodes of the first node graph.

[0034] This chaining of nodes enables the first and second execution nodes to be agnostic to execution nodes more “upstream” in the node graph, providing a modular and therefore easily scalable node graph.

[0035] In some embodiments, the intermediary node is configured to enable the first output to be individually provided to the second node, separately from one or more other outputs of the first node.

[0036] Advantageously, the intermediary node can split up multiple data outputs of the first node into individual data sources such that the individual data sources can be individually provided to the second node, separately from the other outputs of the first node. Thus, any combination of data can be provided to any combination of nodes and ensures that each execution node is agnostic of the antecedent execution nodes.

[0037] In some embodiments, the intermediary node is further configured to provide the first output to one or more other execution nodes in the first node graph.

[0038] The intermediary node can therefore be provided with different types of data from the first node, split the data into individual data sources and provide them to the appropriate subsequent execution nodes. Therefore, each execution node does not have to be hard-coded to output an appropriate data type for the subsequent execution node,

[0039] In some embodiments, the determined action is agnostic to the hardware configuration of the first machine. In such embodiments, between steps ii) and Hi), the system may be further configured to send the action to an asset management layer of the system, for conversion of the action into the control message in a format that can be interpreted by the hardware of the first machine.

[0040] The provision of a node graph that is agnostic to the hardware configuration of a machine ensures that the data model used for controlling the machine is agnostic to the manufacturer of the hardware systems comprised in the machine.

[0041] In some embodiments, the one or more processors are further configured to control the first machine to perform a second task, concurrently to the first task, by creating a second node graph for the second task. In such embodiments, a third execution node in the first node graph may be configured to send one or more second outputs to a fourth execution node in the second node graph; or a third execution node in the first node graph may be configured to receive one or more second outputs of a fourth execution node in the second node graph.

[0042] Communication between multiple node graphs to control a machine enables multiple tasks to be carried out simultaneously and efficiently by the machine.

[0043] The inherent type-safety of the node graph structure also ensures that the multiple tasks performed by a machine are error-free and without failure modes.

[0044] In some embodiments, the one or more second outputs facilitate data sharing between the first node graph and the second node graph and / or timing information to co-ordinate processes between the first node graph and the second node graph.

[0045] Data sharing between multiple node graphs to control a machine enables multiple tasks to be carried out collaboratively.

[0046] In some embodiments, the one or more processors are further configured to control a second machine that is remotely operable to perform a third task, concurrently to the first task, by creating a third node graph for the third task. In such embodiments, a fifth execution node in the first node graph may be configured to send one or more third outputs to a sixth execution node in the third node graph; and / or a fifth execution node in the first node graph may configured to receive one or more third outputs of a sixth execution node in the third node graph.

[0047] The above-described configuration of controlling multiple machines, enables co-operative and collaborative task execution irrespective of the type and manufacturer of the hardware systems comprised in each machine.

[0048] Additionally, the communication between node graphs that control different machines provides users with a high degree of flexibility in creating complex cooperative tasks between different machines, quickly and easily.

[0049] In some embodiments, the one or more third outputs facilitate data sharing between the first task being performed by the first machine and the third task being performed by the second machine, and / or timing information to co-ordinate processes between the first node graph and the third node graph.

[0050] Advantageously, data sharing between node graphs that control multiple machines to perform different tasks facilitates co-operative task execution.

[0051] In some embodiments, each execution node in the first node graph is configured to perform a type-safe check on compilation.

[0052] The type-safe checks on compilation ensure that the entirety of the node graph is type safe and therefore the task will be performed error-free and without failure modes in the hardware systems of the machine.

[0053] Type-safety of each execution node on compilation also ensure that user modification to a node graph has not inadvertently introduced errors.

[0054] Further, type-safe checks on compilation ensure that when a node graph is modified to control two or more machines collaboratively, the task is executed safely and successfully.

[0055] In some embodiments, the first machine is an unmanned aerial vehicle, unmanned marine vehicle, or unmanned ground vehicle.

[0056] In some embodiments, the first task to be performed by the first machine is: to map an area; to survey an area with a particular sensor; to detect an object; to detect a landmine; to remove a landmine.

[0057] In some embodiments, the data obtained from the first machine is telemetry data from a sensor on the first machine.

[0058] In a second aspect herein, there is provided a method for controlling a first machine that is remotely operable to perform a first task, the method comprising: i) creating a first node graph for the first task, the first node graph comprising a plurality of execution nodes, each execution node defining execution logic that is to be performed on data obtained from the first machine, to process said data, and wherein the output of the first node graph is an action to be performed by the first machine to perform the first task; II) propagating the data received from the first machine in real-time, through the first node graph to determine a first action to be performed by the first machine; and ill) sending a control message to the first machine to instruct the first machine to perform the first action.

[0059] In a third aspect herein there is a computer program comprising instructions which when executed on at least one processor, cause the at least one processor to carry out the method of the second aspect.

[0060] In a fourth aspect herein there is a carrier containing a computer program according to the third aspect, wherein the carrier comprises one of an electronic signal, optical signal, radio signal or computer readable storage medium.

[0061] In a fifth aspect herein there is a computer program product comprising non-transitory computer readable media having stored thereon a computer program according to the third aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] For a better understanding and to show more clearly how embodiments herein may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings, in which:

[0063] Fig. 1a shows an example of a computer system according to some embodiments herein;

[0064] Fig. 1 b shows another example of a computer system according to some embodiments herein;

[0065] Fig. 2 shows an example method for controlling a remotely operable machine according to some embodiments herein;

[0066] Fig. 3 is a block diagram showing the structure of an intermediary node according to some embodiments herein;

[0067] Fig. 4 is a block diagram showing an example of chaining a plurality of nodes in a node graph according to some embodiments herein;

[0068] Fig. 5 is a block diagram showing an example of the overall system that controls a remotely operable machine according to some embodiments herein;

[0069] Fig. 6 is a schematic showing an unmanned aerial vehicle that can be controlled within an area defined by a polygon according to the method of Fig. 2;

[0070] Fig. 7 shows an example of a node graph according to embodiments herein that is used to control the unmanned aerial vehicle shown in Fig. 6;

[0071] Fig. 8 is a schematic showing an unmanned aerial vehicle and an unmanned ground vehicle that can be co-operatively controlled within an area defined by a polygon according to the methods of some embodiments herein;

[0072] Fig. 9 shows an example of a node graph according to embodiments herein that is used to control the unmanned aerial vehicle and the unmanned ground vehicle shown in Fig. 8. DETAILED DESCRIPTION

[0073] As described above, existing state of the art approaches for controlling ROMs tend to focus on enabling co-operation between autonomous hardware systems which have been built by a single manufacturer, enabling only a single autonomous system, or they enable only a small, limited quantity of autonomous systems to co-operate.

[0074] Problems arise when the data model underlying a co-operative human-machine or machinemachine teaming system is either limited in its ability to scale (scale limits), limited in its compatibility with multiple manufacturers (agnosticism limits), or potentially unsafe in its task execution (type safety limits).

[0075] The systems described herein enable multiple autonomous systems to be able to collaboratively execute pre-configured tasks in safety-critical environments, when those systems are built by different manufacturers to different standards. This is achieved through the use of multithreaded node graphs that take in data from a ROM and processes the data through a plurality of execution nodes that are networked together in a graph structure. In some embodiments, each execution node has a verb logic in its verb core that processes the data in some way to determine the next action that should be performed by the ROM. The verb cores of the nodes may execute continuously (e.g. only limited by the flow of data available) and can be multi-threaded so that the data is processed particularly efficiently, enabling the system to be reactive and adaptive to changes in the data being received from the ROM.

[0076] The execution nodes may be separated in the graph structure by intermediary nodes that act as temporary data stores for outputs of the nodes. The outputs of each node are split in the intermediary nodes so that they can be passed on individually to downstream nodes in the node graph. This makes each node agnostic to the nodes further upstream of it, enabling the nodes to be fully modular in nature, permitting new tasks made from new combinations of nodes to be easily and quickly assembled without new hard-coding. This provides scalability and versatility of the solutions herein. The modular nature of the node graphs, along with the intermediary nodes, makes it easier to enforce rigid type-safety across the full system, and to track down any errors at compilation.

[0077] Furthermore, the nodes may define processing actions that are independent of any particular hardware or manufacturer. The determined actions can then be converted into appropriate (e.g. machine and / or manufacturer specific) commands in a subsequent layer of programming which can be configured when a new type of machine is set up or added to the system. This makes the system more agnostic to manufacturer and model of machine, compared to previous systems.

[0078] Furthermore, the nodes can be made to be adaptive, and to dynamically adjust their processing power and resource allocation based on task complexity and urgency. This provides a system for chaining any number of systems together and enabling the sensors on those systems to - in combination - produce customised workflows (node graphs) for users.

[0079] Thus, the systems herein provide a mechanism for chaining any number of autonomous systems together and enabling the sensors on those systems to - in combination - produce novel workflows for users.

[0080] Turning now to Fig. 1a, embodiments of the disclosure herein are generally computer implemented and are performed by computer systems. Fig. 1a shows a system (e.g. a computer system) that may form part of some embodiments herein. Fig. 1a shows a computer system 100 comprising one or more processors 102. The one or more of processors 102 are configured to perform any of the methods herein, such as the method 200 described below. In another example shown in Fig. 1b, the system 100 may comprise a processor 102, a memory 104 and set of instructions 106. The memory holds instruction data (e.g. such as compiled code) representing set of instructions 106. The processor may be configured to communicate with the memory and to execute the set of instructions. The set of instructions, when executed by the processor, may cause the processor to perform any of the methods herein, such as the method 200 described below.

[0081] Processor (e.g. processing circuitry or logic) 102 may be any type of processor, such as, for example, a central processing unit (CPU), a Graphics Processing Unit (GPU), a Neural Processing Unit (NPU), or any other type of processing unit. Processor 102 may comprise one or more sub-processors, processing units, multi-core processors or modules that are configured to work together in a distributed manner to control the node in the manner described herein.

[0082] The system 100 may comprise a memory 104. In some embodiments, the memory 104 of the system 100 can be configured to store program code or instructions that can be executed by the processor(s) 102 of the system 100 to perform the functionality described herein. The memory 104 of the node 100, may be configured to store any data or information referred to herein, such as for example, requests, resources, information, data, signals, or similarthat are described herein. The one or more processors 102 of the system 100 may be configured to control the memory 104 of the system 100 to store such information.

[0083] In some embodiments, the system 100 may be a virtual system, e.g. such as a virtual machine or any other containerised computer system. In such embodiments, the one or more processors 102 and the memory 104 may be portions of larger processing and memory resources respectively.

[0084] It will be appreciated that a computer system 100 may comprise other components to those illustrated in Figs. 1a and 1b. For example, the system 100 may comprise a power supply (e.g. mains or battery power supply). The system 100 may further comprise a wireless transmitter and / or wireless receiver to communicate wirelessly with the ROMs or other computer systems described herein. In some embodiments, the system 100 may have a wired connection with which to communicate with the ROMs or other computer systems. In some embodiments, the system 100 may further comprise a user input device such as a mouse, keyboard, or touch pad, for receiving input user data such as for example, user input data indicating a ROM that is to be controlled, or a task to be performed by the indicated ROM. In some embodiments, the system 100 may further comprise a display for displaying any of the data described herein, such as for example, any of the outputs (or intermediate data products) of any of the methods described herein, such as, for example sensor data collected by a ROM being controlled by the system.

[0085] Fig. 2 shows a computer implemented method 200 for controlling a first machine to perform a first task. The first machine is remotely operable. The method 200 may be performed by a computer system such as the system 100 illustrated in Figs 1a and 1b described above, in brief, in a first step, the method 200 comprises: i) creating 202 a first node graph for the first task, the first node graph comprising a plurality of execution nodes, each execution node defining verb logic that is to be performed on data obtained from the first machine, to process said data, and wherein the output of the first node graph is an action to be performed by the first machine to perform the first task. In a second step 204, the method comprises ii) propagating the data received from the first machine in real-time, through the first node graph to determine a first action to be performed by the first machine. In a third step 206, the method 200 comprises iii) sending a control message to the first machine to instruct the first machine to perform the first action.

[0086] The present disclosure therefore relates to remotely operable machines, e.g. machines that are partly or fully remote controlled, for example, e.g. via wireless communication.

[0087] The machines herein can be software-driven machinery. In other examples, the machines herein can be software-driven robotic machinery (e.g. sophisticated machines such as the machines for remote surgery, or the vehicles described below).

[0088] The machines herein can also be basic machines, e.g. such as passive sensors (cameras, microphones, photometers, vibrometers etc) that can be remotely operated / controlled.

[0089] Generally, a machine that is remotely operable (e.g. a remotely operable machine, or ROM) can be a vehicle such as for example, an unmanned aerial vehicle (UAV), unmanned marine vehicle (UMV), or unmanned ground vehicle (UGV). Examples of such vehicles, include but are not limited to drones, aeroplanes, robotic multi-terrain vehicles, remote operated submersible vehicles, self-driving vehicles (e.g. such as cars, lorries or vans that are self- driving) and / or any other vehicle that can be remotely operated.

[0090] In other examples, a remotely operable machine can be mechanical equipment, such as, for example, manufacturing machinery. In other examples, a remotely operable machine can be part of a renewable energy system, e.g, a wind-turbine, solar panel, hydro-electric power-plant, or any other renewable energy system.

[0091] In other examples, a remotely operable machine could be for use in medicine, for example, remote-operated surgical equipment.

[0092] It will be appreciated that the machines listed above are merely examples and that there are many other different types of remotely operable machines and equipment to which the methods described herein may equally be applied to.

[0093] The method 200 relates to controlling the first remotely operated machine to perform a task. The task can generally be any type of task that a remotely operated machine can be configured to perform. Some non-limiting examples are as follows.

[0094] The first machine may be controlled to perform a mapping or surveying task. E.g. the method 200 may be used to control the first machine to obtain imaging or other sensor data of an area or region. In examples where the first machine is a UAV such as a drone, the first task may be an aerial mapping task or surveying task. The first machine may be controlled to perform the mapping or surveying tasks over a predefined area (e.g. such as a user-defined area; see the example below with respect to Fig. 6). [009S] The first machine may be controlled to survey the area using any type of sensor, examples, include but are not limited to a heat-sensor, magnetometer, or any other type of sensor.

[0096] The method 200 may further comprise steps of detecting an object in the sensor data. For example image data and / or sensor data obtained by the first machine may be processed to locate particular types of objects. As an example, the method 200 may be used to detect landmines, particular types of buildings, roads or other infrastructure, or any other type of man-made or natural land mark.

[0097] In some embodiments, the first machine may be controlled to detect and / or remove one or more landmines. As such the method 200 can be used in landmine detection and removal. In some embodiments, the method 200 may be used to control two or more remotely operable machines to this end. For example, a first remotely operable machine may be used to perform the first task of detecting a landmine and a second remotely operable machine may be used to perform a second task of removing, deactivating or detonating the landmine as part of a controlled detonation, in such examples, the first remotely operable machine may be e.g. a drone and the second remotely operable machine may be a second drone, or a UGV. The skilled person will appreciate however that these are merely examples and that tasks such as landmine detection and removal may be performed by many different combinations of remotely operable machines.

[0098] In step 202, the method 200 comprises I) creating a first node graph for the first task. The creation of the first node graph (which may otherwise be referred to herein as the first “task graph”) may be initiated by a user. For example, the system may receive a user input instructing the system to control the first machine to perform the first task. Although a ‘user’ may refer to a human operative, it will be appreciated that a user in this sense could equally be another computing system, external to the system 100. In response to such an instruction, the first node graph is created.

[0099] The node graphs herein are task-specific in the sense that they are defined with respect to a particular type of task, and in some embodiments, can be defined in a Specification File, As noted above, the nodes in each node graph are designed to be independent of one another so that a user can define new tasks by selecting new combinations of nodes. Thus, the Specification File contains an indication of which nodes have been selected to perform the particular type of task, and how they are to be networked together at run-time.

[0100] The skilled person will be familiar with the concept of a node graph or “graph-structure” which is a software design structure. Graphs are discussed in the paper entitled: “Applications of Graph Theory in Computer Science” by Riaz &All (2011) Third International Conference on Computational Intelligence, Communication Systems and Networks, Bali, Indonesia, 2011, pp. 142-145, doi: 10.1109 / C!CSyN.2011.40.

[0101] The node graphs herein may have a directed acyclic graph, DAG, structure. In this sense the node graphs may be uni-directional, without any reverse edges. As such, data flows through the node graphs in a single direction.

[0102] Generally, the first node graph comprises a plurality of execution nodes, each execution node defining verb logic (e.g. a single verb) that is to be performed on data input to the respective node. (In use, the verb is executed on the data obtained or derived from the first object, e.g. the machine that is performing the task), to process said data.

[0103] The skilled person will be familiar with the concept of the class in the context of computer coding. We define a verb in the context of the node graph as a type of object / ciass in the object-oriented programming paradigm that defines and guarantees a certain method. As an example, the verbs can be implemented as Python classes inheriting the abstract base Verb class with a guaranteed method that allows them to be executed by their parent node if compatible (e.g. the Action Node attempts to run run_effect on its child verb, the Intelligence Node attempts to run the command ‘mnjntelligence’ on its child verb). For reference, object-oriented programming is described in the paper by Peter Wegner, 1990 entitled: “Concepts and paradigms of object-oriented programming’''. SIGPLAN OOPS Mess. 1, 1 (Aug. 1990), 7-87. https: / / doi.org / 10.1145 / 382192.383004,

[0104] The execution nodes herein therefore each comprise a verb object, e.g, an action or method. Verb methods perform operations on objects. The execution nodes herein may hold repeatable execution logic (e.g. that can be executed in a loop). In other words, the verbs may be run in a looped manner. The execution nodes sit on the nodes (or vertices) of the graph structure. The edges of the graph structure can hold other nodes, such as the intermediary nodes (which may otherwise be referred to as signal nodes), described below.

[0105] In some embodiments, the execution nodes 300 have the structure illustrated in Fig. 3. For example, they may have a verb input interface 302 that defines one or more classes for the input parameters to the respective node. The execution node receives inputs from one or more other nodes 304 that are “upstream” of the respective execution node in the node graph, at the verb input interface 302. In some embodiments herein, the verb input interfaces 302 may take input from one or more intermediary nodes, described in more detail below. As an example, where the execution node takes inputs from nodes labelled ”signai_node_r and “signal_node_2” which have Area and Boolean types respectively, the verb interface may be defined as: signal_node_1: SignalNode[Area] signal„node„2: SignalNodefbool]

[0106] The execution node 300 may further comprise a verb core 306. The verb core contains the execution logic (otherwise referred to herein as verb logic 308) for the respective node. The verb core 306 expects to have the Inputs and Outputs available. Verb core logic may in some embodiments run continuously in a loop - which enables dynamic data updates. The verb core logic may expect to run indefinitely, and define its own conditions for stopping when its task is complete. In some embodiments, the verb core is configured to iteratively execute the execution logic in a loop, enabling data points in the data stream to be processed in real-time. In some embodiments, the iterations are performed in a continuous loop, e.g. in response to new input data being received by the node. The verb core 306 may be configured to iteratively execute the execution logic until a stop condition defined in the verb core 306 is met. The stop condition may be a condition that is met (e.g. associated with) the task being complete).

[0107] The verb core 306 may further interact with one or more controls 310 that lie outside of the node graph. For example, the verb core may make a call (e.g. send a query or instruction) to a machine learning model to cause the machine learning model to process part of the data. Other examples of outside controls 310 include but are not limited to image or video processing processes, and asset control functions for controlling the machine.

[0108] The execution node may then have a verb output interface 312 defining one or more classes of output parameters of the respective node. The execution node sends its outputs to one or more other nodes 314 that are “downstream” of the respective execution node, at the verb output interface. In some embodiments herein, the verb output interfaces may send their outputs to one or more intermediary nodes, as described below. In an example where the execution node outputs an area type variable to the node “signal_nodeJ3”, the verb output may specifiy: signal„node„3@SignalNode[Area], The Input and output Interfaces 302; 312 are strongly-typed object definitions that pair to a specific verb.

[0109] The execution nodes are self-contained in that each node executes the code within that node. In some embodiments, each execution node 300 is responsible for passing the objects to the Verb Interfaces on instantiation. In embodiments where the first node graph is multi-threaded, the execution node may also be configured to initiate the threaded calling and to govern the verb core logic. The execution nodes can, in some embodiments, also contain metadata (such as, e.g. one or more human-readable labels for the respective node) that can be used to annotate a visual representation of the first node graph and make the first node graph easier to understand for users inspecting it in the user interface.

[0110] The individual nodes are type-safe and this makes the system as a whole highly scalable. Each execution node is also agnostic to the number and type of other nodes it may be chained with. This means that tew tasks can be quickly and easily be created from existing nodes and / or new nodes can easily be added to existing chains of tasks. This functionality is enabled by means of the intermediary nodes described in more detail below.

[0111] Each execution node may generally be one of the following types:

[0112] Data Node - The type of node responsible for handling an ‘Information Verb’ type which pairs with a ‘Data Source’, which is responsible for bringing information into the Task Graph. Generally, data type nodes comprise a verb input interface acting as the entry point of the data into the first node graph, and a verb output interface for providing the data for ongoing propagation into the first node graph.

[0113] Intelligence Node - The type of node responsible for handing an 'Analysis Verb’ type. Generally, Intelligence nodes may be configured to perform a transform on the data to obtain one or processed data items for use in determining the first action. As an example, an intelligence node may interact with one or more outside controls 310 (e.g. such as by making a call to a machine learning model or other algorithm) as illustrated in Fig.3.

[0114] Action Node - The type of node responsible for handling an 'Effect Verb’ type. Generally, action nodes are configured to output the determined first action from the first node graph.

[0115] The execution nodes are networked together into a graph structure. In some embodiments, at least one pair of nodes in the first node graph are separated from one another by an intermediary node. In some embodiments, there are intermediary nodes (otherwise referred to herein as signal nodes) between each pair of nodes in the graph structure. This is illustrated in Fig. 4 which shows a portion of a first node graph 400 comprising an intermediary node 402a that outputs its data to an execution node 300a. The execution node 300a may be any of the node types described above, e.g. a Data, Action or Intelligence node. The execution node 300a processes the data in some manner to produce an output that is held by intermediary node 402b. Intermediary node 402b in turn passes the data that it holds to another execution node 300b further down (e.g. ‘downstream’) in the node graph structure. Execution node 300b may be any of the node types described above, e.g. a Data, Action or Intelligence node, and may be the same or different type to the execution node 300a.

[0116] The intermediary nodes 402a; 402b act as “holding nodes” or short-term repositories for data output by an execution node lying upstream of the respective signal node. In some embodiments, the intermediary nodes therefore lie on the edges of the node graphs. The intermediary nodes are used to separate the outputs of each execution node into individual data sources, so that they may be called individually as inputs to execution nodes further downstream in the structure of the first node graph. Thus, in an example where a first node in the first node outputs a first output and a second output (e.g. outputs values for two or more parameters), an intermediary node can be configured to enable the first output to be individually provided to a second node lying downstream of the first node, and this can be provided separately from one or more other outputs of the first node.

[0117] Thus, the intermediary nodes are intermediary data handlers. This helps with robustness and audit-ability, as it captures state, and enables the Verb Interface structure in Fig. 3. Furthermore, the intermediary nodes provide a mechanism for splitting multiple outputs up into individual data sources so that they can separately be provided to any nodes that need them. As a result, any combination of data computed by any combination of nodes, can be provided as input to any other combination of nodes in the node graph. Furthermore, the use of intermediary nodes in this way allows the other data / intelligence / action nodes to be agnostic of the antecedent data / intelligence / action nodes in the node graph. Thus, the execution nodes 300 act as independent blocks. This results in the aforementioned scalability and versatility of the solution herein and enables new tasks to be quickly built up from existing nodes (e.g. by inserting appropriate intermediary nodes between them).

[0118] With respect to the execution nodes described above, the verb input interfaces 302 of the execution nodes (as shown in Fig. 3) can provide a ciass definition specifying the required intermediary node inputs, attributing them semantic meaning and using type Generics to rigorously enforce type expectations, in a complementary manner, the verb output interfaces 312 of the execution nodes provide a class definition for data leaving the respective execution node.

[0119] Turning back to the method 200, in step ii of the method 200, the data received from the first machine is propagated in real-time, through the first node graph to determine a first action to be performed by the first machine. The propagation may be multi-threaded and / or multi-processed.

[0120] Multi-threaded computer programs have two or more parts that can run concurrently, performing different functions. In embodiments herein, the execution nodes may be multi-threaded and / or multi-processed, allowing more than one node in the node graph to be executed simultaneously. The graphs herein may be multi-threaded and / or multi-processed so that the execution nodes (and potentially all of the execution nodes) can be executed simultaneously in the node graph. The skilled person will be familiar with multi-threaded applications which are built-in to various coding languages. For example, in the programming language Rust, this would be the standard library std::thread; for Python this would be the standard library threading.

[0121] In step ii), the method 200 may thus comprise performing simultaneous, looped, multi-threaded execution of the verb cores of each of the plurality of execution nodes in the first node graph. In this way the data flows through the node-graph in a continuous, fluid manner, which may be visualised as a “waterfall” of information flowing through the node graph. In some embodiments, the verb logic of the execution nodes may handle race conditions and wait for valid data before executing, this ensures the data is processed consistently through the node graph e.g. in scenarios where the node graph branches,

[0122] Having the execution nodes simultaneously executing their execution logic on a sequence of real-time data or near real-time data (e.g. in a datastream) has the advantage of enabling many data points being processed simultaneously e.g. the processing on the next data point in the sequence can happen before the preceding one has finished. This enables real time data to be processed extremely quickly and efficiently, enabling the system to optimally process new information and output commands in response to new data, resulting in reactive, responsive commands being sent to the machine.

[0123] Turning back to step iii) of the method 200, at least one of the execution nodes in the graph determines 204 an output indicating an action that is to be performed by the first machine. The actions are then converted into control messages and sent 206 in step iii) to the first machine. Examples of actions, include, but are not limited to actions to control the movement of the first machine (e.g. to change direction, change speed, stop, start etc) and / or to obtain data, e.g. to take an image, or video. It will be appreciated that these are merely examples however, and that the output may be any action that can be performed by a machine.

[0124] Different types of machines, made by different manufacturers have different controls that may be written in different programming languages, with different / bespoke command structures. This has posed significant compatibility challenges in this field, as traditional, hard-coded systems have needed potentially extensive bespoke code to be written for each new machine type, or machine manufacturer added to the system, in embodiments herein, this is avoided by making the node-graph and the output of the node graph agnostic to the underlying machine that is being controiled. An asset layer may then be used to convert the agnostic actions into particular control signals that can be understood by the machine in question.

[0125] It is possible to structure the node graph in this way because even though the commands may be formatted differently between manufacturers and / or machines, fundamentally the controls are based on common principles. In embodiments here where the machines are vehicles, for example, control of such vehicles may be based on controls that boil down to ‘forwards’, ‘backwards’, ‘left’, ‘right’, ‘accelerate ‘decelerate’. For aerial vehicles, the controls may be based on roll, pitch, and yaw, for example. Thus, the action output by the node graph may be based around these generic concepts that are common to all machines (or a subset of machines) and then converted into control signals suitable for the particular asset in the asset layer.

[0126] It will be appreciated that the method 200 may comprise further steps and / or modifications to the variations described above. For example, in some embodiments, the first machine may perform two tasks concurrently.

[0127] In such embodiments, the method 200 may further comprise controlling the first machine to perform a second task, concurrently to the first task, by creating a second node graph for the second task. The second node graph may be structured from execution nodes and signal nodes in the same manner as the first node graph as described above.

[0128] The first and second node graphs may interact by sending data output from one execution node of one node graph to an execution node (e.g. such as a data-type execution node) of the other.

[0129] For example, a third execution node in the first node graph may be configured to send one or more second outputs to a fourth execution node in the second node graph. In other embodiments, a third execution node in the first node graph may be configured to receive one or more second outputs of a fourth execution node in the second node graph. In this way, data sharing between the two tasks can be performed. For example, the second outputs facilitate data sharing between the first node graph and the second node graph and / or timing information to co-ordinate processes between the first node graph and the second node graph.

[0130] As an example, the method 200 may be used to control an unmanned ground vehicle to perform a first task related to logistics support (e.g. transporting equipment from position A to position B) and a second task related to reconnaissance, e.g. taking photographs, or obtaining sensor data enroute. In such embodiments, a first node graph related to the logistics support task may send (e.g. from a third execution node in the first node graph) position data to a data-type execution node (e.g. the fourth execution node) of a second node graph related to the reconnaissance task. The second node graph may process the position data from the first node graph to determine points along the route at which the unmanned ground vehicle is to pause to take the photographs or other sensor information.

[0131] Turning to other embodiments, in other embodiments, the method 200 may be extended to coordinate tasks being performed concurrently by two (or more) remotely operable machines. For example, the method 200 may further comprise controlling a second machine that is remotely operable, to perform a third task, concurrently to the first task being performed by the first machine. In such embodiments, the method 200 may comprise creating a third node graph for the third task. One or more intermediary nodes in the first node graph may be configured to send or receive information to or from one or more other nodes in the third node graph. For example, a fifth execution node in the first node graph may be configured to send one or more third outputs to a sixth execution node in the third node graph. As another example, a fifth execution node in the first node graph is configured to receive one or more third outputs of a sixth execution node in the third node graph.

[0132] In an example where the first and third machines are drones, the method 200 may be used to control the first drone to perform a first task of surveying an area. A second drone may be controlled to follow the first drone if an execution node (e.g. the fifth execution node above) in the first node graph is configured to send the bearing of the first drone to an execution node (e.g. the fifth execution node above) in the third node graph which may e.g. be a data-type execution node in the third node graph.

[0133] In this way, the method 200 may be extended to control two or more remotely operated vehicles to perform co-ordinated tasks.

[0134] Turning now to Fig. S, which shows a block diagram of an overview of an example system architecture 500 according to some embodiments herein. The system 500 is for controlling a machine that is remotely operable (e.g. a ROM) to perform a task. The system 500 comprises a user interface 502, a server 504, a task broker 506, a first node graph (which may otherwise be referred to as a task graph) 400, a pairing algorithm 508 and a machine (asset) broker 510.

[0135] The user interface (UI) 502 or human-machine interface is in communication with the server 504 and may be configured to enable a user to provide one or more inputs to the server 504. For example, the UI 502 may be configured to enable the user to provide inputs to build or alter the node graph 400. In some embodiments, the UI 502 may include a display and one or more input devices, such as touchscreens, keypads, buttons and / or the like, which are configured to receive inputs from the user. In other embodiments, the UI 502 may also be a smartphone, tablet, or other such user device connected to the server 504.

[0136] In this embodiment, to initiate a task to be performed by a machine, a user inputs a request through the UI 502 to start a task. The task may be identified using a task identification (ID) number, and the type of task that is being requested. Each type of task is associated with a task manifest which is an uninstantiated class definition which the system attempts to instantiate and populate when the respective (type of) task is initiated for a particular machine, by a user.

[0137] The task manifests are task specific and comprise a set of rules and / or requirements that are to be satisfied by the system 500 in order for the system 500 to begin to control the machine to perform the task. For example, the task manifest may include the requirement to check that the machine exists e.g. by pinging or calling the machine. When the task manifest is instantiated for a particular task a manifest object is output. Generally, a machine may be controlled using more than one system 500 and therefore the manifest object may indicate that the machine does not “exist” e.g. is unavailable to be used, when it is being used or controlled by another system 500. As another example, the task manifest may specify conditions such as that in order for the task to be completed, image or video processing needs to be available, or in embodiments where the processing is carried out by an external system such a machine learning (ML) algorithm, that the respective ML algorithms needs to be available. In such examples when the manifest object is populated from the task manifest, the system checks that there is successful communication between the system 500 and the respective image processing, video processing and / or ML algorithm^).

[0138] Each type or class of task is also associated with a specification file. The static specification file comprises the task manifest e.g. the required inputs to start a particular task. The static specification file may also include a definition of the task, such as the task ID and the actions that have to be performed in order to complete the task. It also provides the structure of the node graph 400, including the verbs comprised in each node, and defines how the nodes are linked. The Task Manifest thus allows leveraging of the type system of the language to both a) verify that the node graph in the specification file is safely using the expected manifest items and b) that, at run-time, the machine is available and can be paired to the system.

[0139] The task broker 506 receives the inputs via the server 504 (which may be cloud-based or any other type of server) and is configured to carry out the method 200 described above.

[0140] To initiate a task, the task broker 506 receives the request to begin the task, verifies the static specification file, sends the task manifest and manifest target to a pairing algorithm 508.

[0141] The pairing algorithm 508 receives the task manifest from the task broker 506 and connects (e.g. pairs) the system 500 with the machine (asset). On receipt of the task manifest, the pairing algorithm checks that the machine fulfils each requirement. Such a requirement may include that the machine has the ability to perform a particular action, e.g. a requirement for a vehicle may include the ability to reverse.

[0142] If the machine includes all the requirements, the pairing algorithm instructs the asset broker 510 to check or assign an identification (ID) number to the machine so that the machine can be tracked in the system 500. If the asset broker 510 already has an ID number for the asset, it assigns it in the system 500. If the asset broker 510 has a different ID number of the same type, this ID number is suggested. Otherwise, the asset broker 510 may return a suggestion for an ID number to the pairing algorithm. Once an ID number has been successfully assigned to the asset (machine), the task broker 506 creates a node graph 400 from the specification file.

[0143] The task broker 506 then performs the method 200 and creates the node graph for the task (based on the specification file) and assigns the manifest object to it (thus linking the nodes in the graph to the particular machine) and then instructs the node graph to begin executing the task as described in the method 200 above. The manifest object effectively acts to link up the node graph (which is otherwise machine agnostic) with the particular machine being controlled. The node graph being linked-up to a particular machine and a particular machine’s incoming data at run-time in this way, separates the particulars of the machines from the node graphs, further contributing to the scalability, reliability and agnostic nature of the systems herein.

[0144] Once the task has ended, the task broker 506 destroys the node graph 400. [014S] The node graph 400, the static specification file, the manifest object, and any data imported to the node graph from an external source, such as the machine, can be sent through the server 504 to the UI 502 for display or manipulation by the user. Additionally, data can be stored on the server 504,

[0146] It will be appreciated that the system 500 in Fig. 5 is merely one example system architecture that may run the method 200 and that the method 200 may equally be performed by other system architectures with different configurations and modules to those described above,

[0147] Turning now to Figs. 6 and 7, which show an example node graph that may be used to control a machine according to some embodiments herein. In this example, a node graph is used to control a machine in the form of an unmanned aerial vehicle 604 to perform the task of surveying a geographical area. Fig. 6 shows that a geographical area to be surveyed can be selected by drawing a polygon 602 on a map. It will be appreciated however, that this is merely an example and that the machine may equally be another type of vehicle, such as an unmanned ground vehicle (UGV).

[0148] The polygon 602 may be determined by drawing on a map viewed on a UI, by inputting coordinates for the vertices of the polygon 602 through the UI or by a different (second) node graph providing the polygon 602 as an input to the present (first) node graph, or by any other input mechanism.

[0149] The geographical area may be surveyed for various reasons, such as locating mines in a potential minefield and then removing the located mines from the minefield. In this example, the UAV 604 is used for an intelligence, surveillance and reconnaissance (ISR) task. In particular, the UAV 604 is used to map the area within the polygon 602, and subsequently survey and collect intelligence (information) about this area and features therein. The UAV 604 may therefore comprise one or more sensors to carry out the ISR task, such as a camera, a thermal imaging sensor, or a magnetometer.

[0150] The UAV 604 and the sensors comprised therein are controlled by a node graph to execute the ISR task. An example of such a node graph is shown in Fig. 7. The node graph 700 controls the UAV 604 to map and image the area within the polygon 602. Note that in this example, the intermediary nodes are referred to as “Signal Nodes”, however these have the same functionality and structure as the intermediary nodes described above. Furthermore, the execution nodes in this example are referred to by sub-type. As such an ‘action node’ refers to an execution node that is an action-type execution node, as described above; a data node refers to an execution node that is a data-type execution node, as described above; and an intelligence node refers to an execution node that is an intelligence-type execution node as described above.

[0151] The node graph 700 is started by an entry signal node 701 which is the starting point for the node graph.

[0152] 702: signal node writes the coordinates of the polygon 602 or the area to search to a file (“area to search”).

[0153] 712: an action node sends a control message from the node graph 700 to the UAV 604 to instruct the UAV 604 to survey the area within the polygon 602 (“stage control”). The surveillance may for example include mapping or imaging the area.

[0154] 714: signal node stores the current stage of the UAV in the surveillance as instructed by the action node 712 (“current stage”).

[0155] 704: signal node transfers data corresponding to the detection area that the UAV has been instructed to survey to subsequent execution nodes (“detection location”).

[0156] 706: data node iteratively brings in real-time images collected by an image sensor comprised in the UAV 604 to the node graph 700 (“monitoring UAV image sensor”).

[0157] The data node 706 is connected to two signal nodes 708 and 710.

[0158] 708: signal node, when having a status "True”, indicates that the images from the UAV have been successfully written to a file (“monitoring UAV Image is written (“True”)”).

[0159] 710: signal node monitors UAV image data output from the data node 706 (“monitoring UAV Image output”).

[0160] The signal nodes 704 and 708 feed data into the intelligence node 716.

[0161] 716: intelligence node transforms the telemetry data (detection location) and image data collected by the UAV 604 to suitable input files for viewing on a UI. In this example, the intelligence node 716 transforms the telemetry and image data to GeoJSON files, wherein the skilled person would recognise GeoJSON as a format designed for representing simple geographical features (“monitoring UAV Telemetry and Image to GeoJson”).

[0162] The image data collected by the UAV 604 can then be matched with geographical coordinates as determined by the telemetry data, to provide a real-time image of the area within the polygon 602.

[0163] In addition, the image and telemetry data transformed by the intelligence node may be analysed using machine learning (ML) to detect and locate objects e.g. a tank or a mine.

[0164] 718: signal node monitors that the area within the polygon 602 has been geotagged using the UAV (“monitoring UAV geotagged GeoJson”).

[0165] 720: signal node monitors that the geotagged GeoJSON file is written and when having a status “True” indicates that the file is successfully written (“monitoring UAV geotagged GeoJson is written (“True”)”).

[0166] Turning now to Figs, 8 and 9 which describe an example embodiment whereby the method 200 is used to control two machines collaboratively. In this second example, the task is to breach a minefield, schematically indicated by the polygon 802 shown in Fig. 8, using an unmanned aerial vehicle (UAV) 604 and an unmanned ground vehicle (UGV) 806. The task is completed in this example by using the UAV 604 and sensors therein (as described in the First Example) to survey and image the area, and information is sent to the UGV 806 to breach the appropriate location.

[0167] it is noted that this is merely an example, and that the method could equally be applied using fewer or more vehicles (machines) and / or different numbers and / or combinations of nodes.

[0168] In this example, the UAV 604 and UGV 806 are controlled by a node graph such as the node graph 900 shown in Fig. 9 to carry out the task of surveying and breaching the minefield.

[0169] The nodes in the node graph 900 perform the following functions:

[0170] 902: data node iteratively transfers real-time telemetry data from the UAV 604 to the node graph, for example from a camera comprised in the UAV 604 (“telemetry data stream”).

[0171] 904: signal node acts as an intermediary data handler to transfer the data from the data node 716 to an intelligence node 720 (“telemetry reading memory list output signal”).

[0172] As shown in the node graph 700, a plurality of data and signal nodes are used to input realtime image data from the UAV 604 to the node graph 900 and traverse the data through the node graph 900.

[0173] 906: intelligence node scans the telemetry and image data collected by the UAV 604 and transforms the data to suitable input files for viewing on a user interface. In this example, the intelligence node 906 transforms the telemetry and image data to GeoJSON files (“scan UAV Telemetry and Image to GeoJson”).

[0174] The image data collected by the UAV 604 is matched with geographical coordinates as determined by the telemetry data, to provide a real-time image of the area within the polygon 802.

[0175] 908: signal node monitors that the UAV geotagged GeoJson data is written to a tile (“monitoring UAV geotagged GeoJson is written (“False”)”). When having a status “False” the data is not written to a file.

[0176] In parallel to the collection of image and telemetry data, a magnetometer comprised on the UAV is used to collect magnetometry data.

[0177] 910: data node transfers real-time data collected by the magnetometer into the node graph 900 (“scan UAV magnetometer ingest”).

[0178] 912: signal node acts as an intermediary data holder to store the magnetometry data (“magnetometer data holder”). For example, the magnetometry data may be stored in a tabulated format.

[0179] 914: signal node, when having a status “False” indicates that the magnetometry data is not written to a file (“magnetometer sensor ingest is written (“False”)”).

[0180] 916: intelligence node scans the magnetometer data e.g. the table containing the magnetometry data and transforms it into a suitable input file for subsequent viewing or analysis (“scan magnetometer table transform to GeoTiff’). In this example, the magnetometry data is transformed to a GeoTIFF data format, which the skilled person would understand to be a data format that enables georeferencing information to be embedded within a TIFF file.

[0181] 918: signal node acts as an intermediary data handler to transfer the data from the intelligence node 916 to an intelligence node 922 (“magnetometer GeoTiff output”).

[0182] 920: signal node, when having a status “False”, indicates that the magnetometry GeoTIFF file was not written (“magnetometer GeoTiff file written (“False”)”). The option to write this file can be changed by the user i.e. by changing the status of the signal node 916 to “True”.

[0183] 922: intelligence node transforms the magnetometry hotspots from the GeoTIFF file format to the GeoJSON file format (“scan magnetometer hotspots GeoTiff to GeoJson hotspots”). This enables the magnetometry data to be overlayed with the imaged area within the polygon 802.

[0184] 924: signal node indicates that the magnetometry hotspots were successfully written to a GeoJSON file format (“hotspot GeoJson is written”).

[0185] The signal nodes 924 ad 908 transfer the GeoJSON data formats of the telemetry, image and magnetometry data to an intelligence node 926.

[0186] 926: intelligence node calls an external model to overlay the telemetry, image and magnetometry data for viewing and optional subsequent data analysis (“scan UAV image and magnetometer is written”).

[0187] The subsequent data analysis may include analysing the telemetry, image and magnetometry data using machine learning (ML) to identify mines and their locations e.g. coordinates.

[0188] 928: signal node, when having a status “False”, indicates that the overiayed telemetry, image and magnetometry data is not saved in a file (“scan UAV and Magnetometer set value output (“False”)”).

[0189] 930: data node is provided with a selected path to breach (“decision: Select breach path”). The selected path may be determined by using the coordinates of the identified mines from an external analysis process such as the ML process described above, or via another source, e.g. such as user input..

[0190] 932: signal node acts as a data intermediary of the data indicating the breach path output from the data node 930 (“decision: Breach path output”).

[0191] 934: action node outputs a control message to the UGV 806 to move the UGV 806 along the breach path in order to clear the detected mines from the minefield i.e. the area within the polygon 802 (“breach UGV move along path”).

[0192] Thus, in this way, a node graph may be used to control a pair of machines to perform a task. It is noted that in other examples, each of the UAV 604 and UGV 806 may be controlled through more than one node graph. In such examples, each of the vehicles (or machines) may be only responsive to control signals sent from one node graph at any one time and when the task of one node graph has been completed, the vehicle is “released” and becomes available for use by another node graph.

[0193] In this way, co-ordination of different machines is possible.

[0194] Turning now to other embodiments, it will be appreciated that the method 200 may be embodied in a computer program. For example, a computer program product may comprise a computer readable medium, the computer readable medium having computer readable code embodied thereon. The computer readable code can be configured such that, on execution by a suitable computer or processor, the computer or processor is caused to perform the method or methods described herein (such as the method 200).

[0195] A computer program may take different forms, for example, source code, compiled code, executable code, or any other type of code. It will be appreciated that the source code of computer programs may be written in a wide variety of different programming languages, and may take different architectural designs. For example, the functionality described herein may be split across various different sub-routines. Furthermore, the skilled person will appreciate that many different ways of splitting the functionality between the different sub-routines will be possible. The sub-routines may be stored together in one executable file to form a self-contained program. Furthermore, computer programs may call external and / or standard libraries of computer code for performing certain sub-tasks associated with the functionality described herein.

[0196] In another embodiment, there is a computer program product comprising non-transitory computer readable media, having stored thereon a computer program as described above. Examples of computer readable media include, but are not limited to: ROM, such as a CD ROM, a semi-conductor ROM or a magnetic recording medium such as a hard disk.

[0197] In another embodiment, there is a carrier containing a computer program. Examples of carriers include but are not limited to an electronic signal, optical signal, radio signal, computer storage medium, or similar. The carrier of a computer program may be any entity or device (e.g. hardware) capable of carrying the program. As an example, a carrier may be a computer readable media as described above. In other examples a carrier may be a transmissible carrier such as an electronic or optical signal, which may be conveyed via electrical or optical cable or by radio or other means,

[0198] Variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure and the appended claims. In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. The mere fact that certain features are recited in mutually different dependent claims does not indicate that a combination of these claims cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. A system for controlling a first machine that is remotely operable to perform a first task, the system comprising one or more processors collectively configured to:i) create a first node graph for the first task, the first node graph comprising a plurality of execution nodes, each execution node defining execution logic that is to be performed on data obtained from the first machine, to process said data, and wherein the output of the first node graph is an action to be performed by the first machine to perform the first task;ii) propagate the data received from the first machine in real-time, through the first node graph to determine a first action to be performed by the first machine; andill) send a control message to the first machine to instruct the first machine to perform the first action.

2. The system of claim 1 wherein each execution node has the following structure: a verb input interface defining one or more classes for the input parameters to the respective node;a verb output interface defining one or more classes of output parameters of the respective node; anda verb core comprising the verb logic for the respective node, wherein the verb logic corresponds to the execution logic for the respective execution node.

3. The system of claim 2 wherein the data is a data stream and wherein the verb core is configured to iteratively execute the execution logic in a loop, enabling data points in the data stream to be processed in real-time.

4. The system of claim 3 wherein the verb core is configured to iteratively execute until a stop condition defined in the verb core is met, the stop condition being a condition that is met when the first task is complete.

5. The system of any one of the claims 1,2, 3, or 4 wherein the first node graph is multithreaded and / or multi-processed and the one or more processors are configured to perform simultaneous, looped, multi-threaded execution of the execution logic of each of the plurality of execution nodes in the first node graph.

6. The system of any one of the preceding claims wherein the first node graph has a directed acyclic graph, DAG, structure.

7. The system of any one of the preceding ciaims wherein each execution node is of one of the following types:a data node comprising a verb input interface acting as the entry point of the data into the first node graph, and a verb output interface for providing the data for ongoing propagation into the first node graph;an intelligence node configured to perform a transform on the data to obtain one or processed data items for use in determining the first action; andan action node configured to output the determined first action from the first node graph.

8. The system of any one of the preceding claims, wherein at least one pair of nodes in the plurality of execution nodes are separated from one another in the first node graph by an intermediary node, the intermediary node acting as a holding node for storing a first output of a first node of the pair of nodes, and for providing said first output as input to a second node of the pair of nodes.

9. The system of claim 8 wherein the intermediary node sits on an edge between the first and second nodes of the first node graph.

10. The system of claim 8 or 9 wherein the intermediary node is configured to enable the first output to be individually provided to the second node, separately from one or more other outputs of the first node.

11. The system of claim 8, 9 or 10 wherein the intermediary node is further configured to provide the first output to one or more other execution nodes in the first node graph.

12. The system of any one of the preceding claims wherein the determined action is agnostic to the hardware configuration of the first machine, and wherein, between steps ii) and ill), the system is further configured to send the action to an asset management layer of the system, for conversion of the action into the control message in a format that can be interpreted by the hardware of the first machine.

13. The system of any one of the preceding claims wherein the one or more processors are further configured to control the first machine to perform a second task, concurrently to the first task, by creating a second node graph for the second task; andwherein a third execution node in the first node graph is configured to send one or more second outputs to a fourth execution node in the second node graph; orwherein a third execution node in the first node graph is configured to receive one or more second outputs of a fourth execution node in the second node graph.

14. The system of ciaim 13 wherein the one or more second outputs facilitate data sharing between the first node graph and the second node graph and / or timing information to coordinate processes between the first node graph and the second node graph.

15. The system of any one of the preceding ciaims wherein the one or more processors are further configured to control a second machine that is remotely operabie to perform a third task, concurrently to the first task, by creating a third node graph for the third task; and wherein a fifth execution node in the first node graph is configured to send one or more third outputs to a sixth execution node in the third node graph; orwherein a fifth execution node in the first node graph is configured to receive one or more third outputs of a sixth execution node in the third node graph.

16. The system of claim 15 wherein the one or more third outputs facilitate data sharing between the first task being performed by the first machine and the third task being performed by the second machine, and / or timing information to co-ordinate processes between the first node graph and the third node graph.

17. The system of any one of the preceding claims wherein each execution node in the first node graph is configured to perform a type-safe check on compilation.

18. The system of any one of the preceding claims wherein the first machine is an unmanned aerial vehicle, unmanned marine vehicle, or unmanned ground vehicle.

19. The system of any one of the preceding claims wherein the first task to be performed by the first machine is:to map an area;to survey an area with a particular sensor;to detect an object;to detect a landmine; and / or to remove a landmine.

20. The system of any one of the preceding claims wherein the data obtained from the first machine is telemetry data from a sensor on the first machine.

21. A computer implemented method for controlling a first machine that is remotely operable to perform a first task, the method camprising:i) creating a first node graph for the first task, the first node graph comprising a plurality of execution nodes, each execution node defining execution logic that is to be performed on data obtained from the first machine, to process said data, and wherein the output of the first node graph is an action to be performed by the first machine to perform the first task;ii) propagating the data received from the first machine in real-time, through the first node graph to determine a first action to be performed by the first machine; andiii) sending a control message to the first machine to instruct the first machine to perform the first action.

22. The method of claim 21 wherein each execution node has the following structure: a verb input interface defining one or more classes for the input parameters to the respective node;a verb output interface defining one or more classes of output parameters of the respective node; anda verb core comprising the verb logic for the respective node, wherein the verb logic corresponds to the execution logic for the respective execution node.

23. The method of claim 22 wherein the data is a data stream and wherein the verb core is configured to iteratively execute the execution logic in a loop, enabling data points in the data stream to be processed in real-time.

24. The method of claim 23 wherein the verb core is configured to iteratively execute until a stop condition defined in the verb core is met, the stop condition being a condition that is met when the first task is complete.

25. The method of any one of the claims 21,22, 23, or 24 wherein the first node graph is multithreaded and / or multi-processed and the method comprises performing simultaneous, looped, multi-threaded execution of the execution logic of each of the plurality of execution nodes in the first node graph.

26. The method of any one of claims 21 -- 25 wherein the first node graph has a directed acyclic graph, DAG, structure,27. The method of any one of claims 21 - 26 wherein each execution node is of one of the following types:a data node comprising a verb input interface acting as the entry point of the data into the first node graph, and a verb output interface for providing the data for ongoing propagation into the first node graph;an intelligence node configured to perform a transform on the data to obtain one or processed data items for use in determining the first action; andan action node configured to output the determined first action from the first node graph.

28. The method of any one of claims 21 - 27, wherein at least one pair of nodes in the plurality of execution nodes are separated from one another in the first node graph by an intermediarynode, the intermediary node acting as a holding node for storing a first output of a first node of the pair of nodes, and for providing said first output as input to a second node of the pair of nodes.

29. The method of claim 28 wherein the intermediary node sits on an edge between the first and second nodes of the first node graph.

30. The method of claim 28 or 29 wherein the intermediary node is configured to enable the first output to be individually provided to the second node, separately from one or more other outputs of the first node.

31. The method of claim 28, 29 or 30 wherein the intermediary node is further configured to provide the first output to one or more other execution nodes in the first node graph.

32. The method of any one of claims 21 - 31 wherein the determined action is agnostic to the hardware configuration of the first machine, and wherein, between steps ii) and iii), the method further comprises sending the action to an asset management layer, for conversion of the action into the control message in a format that can be interpreted by the hardware of the first machine.

33. The method of any one claims 21 - 32 wherein the method further comprises controlling the first machine to perform a second task, concurrently to the first task, by creating a second node graph for the second task; andwherein a third execution node in the first node graph is configured to send one or more second outputs to a fourth execution node in the second node graph; orwherein a third execution node in the first node graph is configured to receive one or more second outputs of a fourth execution node in the second node graph.

34. The method of claim 33 wherein the one or more second outputs facilitate data sharing between the first node graph and the second node graph and / or timing information to coordinate processes between the first node graph and the second node graph.

35. The method of any one of the preceding claims wherein the method further comprises controlling a second machine that is remotely operable to perform a third task, concurrently to the first task, by creating a third node graph for the third task; andwherein a fifth execution node in the first node graph is configured to send one or more third outputs to a sixth execution node in the third node graph; orwherein a fifth execution node in the first node graph is configured to receive one or more third outputs of a sixth execution node in the third node graph.

36. The method of ciaim 35 wherein the one or more third outputs facilitate data sharing between the first task being performed by the first machine and the third task being performed by the second machine, and / or timing information to co-ordinate processes between the first node graph and the third node graph.

37. The method of any one of ciaims 21 -- 36 wherein each execution node in the first node graph is configured to perform a type-safe check on compiiation.

38. The method of any one of ciaims 21 -- 37 wherein the first machine is an unmanned aeriai vehicle, unmanned marine vehicle, or unmanned ground vehicle.

39. The method of any one of claims 21 -- 38 wherein the first task to be performed by the first machine is:to map an area;to survey an area with a particular sensor;to detect an object;to detect a landmine; and / orto remove a landmine.

40. The method of any one of claims 21 - 39 wherein the data obtained from the first machine is telemetry data from a sensor on the first machine.

41. A computer program comprising instructions which when executed on at least one processor, cause the at least one processor to carry out the method according to any one of claims 21 --40.

42. A carrier containing a computer program according to claim 41, wherein the carrier comprises one of an electronic signal, optical signal, radio signal or computer readable storage medium.

43. A computer program product comprising non-transitory computer readable media having stored thereon a computer program according to claim 41.

Images