Methods and systems for remote robot control

Abstract

A computer-implemented method for remotely controlling a robot. The method comprises, at a client device, connecting to a robot remote from the client device via a data connection over a network, the robot comprising a plurality of sensors and / or actuators, at least two of the plurality of sensors and / or actuators corresponding to a single robot modality of one or more robot modalities, receiving, by the client device, instructions to control operation of the robot, wherein the instructions comprise a first predetermined datatype corresponding to the single robot modality, the first predetermined datatype encoding data corresponding to the at least two of the plurality of sensors and / or actuators, and transmitting, by the client device, the instructions to the robot via the data connection to control operation of the robot in accordance with the instructions.

Description

[0001] Methods and Systems for Remote Robot Control

[0002] The present application relates to methods and systems for remote control of robots.

[0003] BACKGROUND

[0004] It is known that connecting and transmitting robot control instructions remotely to a device, for example a robot, may be facilitated by transmitting said instructions over a network. However, robots may have complex arrangements of sensors and actuators requiring bespoke combinations of instructions and therefore require dedicated software programs to enable a user to control the robots and to receive data from the robot. In addition, as the number of sensors and / or actuators increases, the amount of data transmitted between the robot and a client also increases. However, different robots often have different combinations of components (e.g. sensors and / or actuators), and are therefore not often interoperable. For example, different robots may require different instructions in order to provide commands to actuate the robot and / or to sense the environment in which the robot is present. Additionally, components on the robot may be replaced and / or upgraded, further changing the nature of the instructions that the robot may receive and act upon, and the data that the robot may transmit in return.

[0005] For the above reasons, robots have generally been controlled via a direct data connection between the client device and the robot, such as a wired connection or a point-to-point wireless connection between the client device and the robot. A robot may also be controlled over a local area network, where latency is minimal.

[0006] Where remote control of robots has been implemented, the need for a direct data connection has generally required use of a virtual private network (VPN) between a client device and the robot. Instructions may then be sent from the client device to the robot using the VPN as though the robot and the client device share a direct data connection. However, the above mentioned complexity and volume of data means that latencies experienced over wide area networks can result in delays between the client device sending an instruction and the robot receiving the instruction, which can prevent successful operation of the robot. Additionally, the aforementioned variability in robot design and configurations, and the possibility for configurations to change over time, means that remote control of robots is generally only implemented on a small scale and only for limited combinations of robots and types of client devices. For example, while it may be possible for one client device to be configured to remotely control one type of robot over a wide area, that client device would need re-configuring to control a different type of robot, or if the configuration of the robot changes.

[0007] SUMMARY

[0008] There is described a computer-implemented method for remotely controlling a robot. The method comprises, at a client device, connecting to a robot remote from the client device via a data connection over a network, the robot comprising a plurality of sensors and / or actuators, at least two of the plurality of sensors and / or actuators corresponding to a single robot modality of one or more robot modalities. Receiving, by the client device, instructions to control operation of the robot, wherein the instructions comprise a first predetermined datatype corresponding to the single robot modality, the first predetermined datatype encoding data corresponding to the at least two of the plurality of sensors and / or actuators. Transmitting, by the client device, the instructions to the robot via the data connection to control operation of the robot in accordance with the instructions. Robots with a plurality of sensors and / or actuators may generate a large amount of input data for the robots to process and to transmit to client devices. Alternatively, or additionally, data which the robot needs to receive may also be large. By classifying data by their respective robot modality and defining predetermined datatypes for encoding this data and corresponding to that modality, instructions may be provided to, and data may be transmitted from a robot much more efficiently. This improvement in efficiency can reduce latency both in providing instructions to the robot and receiving data from the robot, to such an extent that it enables control of the robot over a direct data connection.

[0009] In some embodiments, the instructions are encoded, by the client device, into a serialised data format for transmission over the data connection. By encoding the instructions into a serialised data format, bandwidth requirements may be reduced when compared with encoding using alternative data formats. In some embodiments, the network is a wide area network. Advantageously, the method enables a robot to be controlled over a wide area network. In some embodiments, the data connection is a peer-to-peer data connection. Advantageously, by transmitting data over a data connection that is a peer- to-peer data connection using the above method, the robot may be controlled over a direct connection without requiring a VPN, reducing latency. In some embodiments, the first predetermined datatype is one of a plurality of predetermined datatypes, the plurality of predetermined datatypes comprising at least one predetermined datatype for encoding data and corresponding to a visual modality of the one or more robot modalities. By using at least one predetermined datatype for encoding data which corresponds to a visual modality, data streams corresponding to this modality may be combined, thereby reducing redundant header data sent over the network. In some embodiments, the plurality of predetermined datatypes comprises at least one predetermined datatype for encoding data and corresponding to a motor modality of the one or more robot modalities. By using at least one predetermined datatype for encoding data which corresponds to a motor modality, data streams corresponding to this modality may be combined, thereby reducing redundant header data sent over the network. In some embodiments, the plurality of sensors and / or actuators comprise at least one camera and at least one motor. In some embodiments, the robot is a first robot of a plurality of robots and wherein the first robot has a different arrangement of sensors and / or actuators to a second robot of the plurality of robots. The above method provides a robot agnostic method for controlling a robot over a network. In other words, two robots each having a different configuration of sensors and / or actuators may be controlled using the same method for transmitting and receiving instructions. This improves the overall efficiency of development and implementation of remote robot control methods and systems.

[0010] In some embodiments, the method further comprises establishing the connection in accordance with a set of WebRTC protocols. Using the set of WebRTC protocols provides for efficient data transmittal peer-to-peer. In some embodiments, the serialised data format is a Protobuf data format. Using the Protobuf data format reduces the bandwidth required to transmit data over a network when compared with other data formats (e.g. JSON). In some embodiments, the robot is located in a different geographical location to the client device. The methods described herein provide for remote robot control. Advantageously, robots may be controlled in real-time over a data connection rather than in person via a direct connection. In some embodiments, the geographical location may be a town, city, country, continent, or region. In some embodiments, the method further comprises receiving, from the robot over the connection, data from at least two actuators and / or sensors encoded using the first predetermined datatype. By receiving data from the robot regarding its actuators and / or sensors encoded using the first predetermined datatype, the same advantages are experienced when receiving data from the robot as are experienced when transmitting data to the robot.

[0011] In some embodiments, each one of the one or more robot modalities indicates a mode in which the robot can interact with or experience an environment. By using robot modalities which indicate a mode in which the robot can interact or experience the environment, remote robot control may be implemented in a robot agnostic manner and benefit from the improvements experienced in both transmitting and receiving data over a network.

[0012] There is described a computer-implemented method for controlling a robot. The method comprises identifying a first plurality of sensors capable of receiving an input from an environment, wherein each sensor of the first plurality of sensors are different ones of the first plurality of sensors each corresponding to a first robot modality. Receiving, by the robot, a first input data stream from a first sensor of the plurality of sensors. Receiving, by the robot, a second input data stream from a second sensor of the plurality of sensors. Mapping the first input data stream and the second input data stream to a robot coordinate system of the robot. Generating a robot modality input by combining the first input data stream and the second input data stream. Transmitting data to a client device from the robot, the data comprising the robot modality input. Robots with a plurality of sensors and / or actuators may generate a large amount of input data for the robots to process and to transmit to client devices. Alternatively, or additionally, data which the robot needs to receive may also be large. By classifying data by their respective robot modality and defining predetermined datatypes for encoding this data and corresponding to that modality, instructions may be provided to, and data may be transmitted from a robot much more efficiently. This improvement in efficiency can reduce latency both in providing instructions to the robot and receiving data from the robot, to such an extent that it enables control of the robot over a direct data connection.

[0013] In some embodiments, the method further comprises receiving, by the robot from the client device, an instruction configured to actuate the robot, the instruction defined with reference to the robot coordinate system, and causing the robot to be controlled in accordance with the instruction. By transmitting the instruction with reference to the robot coordinate system, the robot may be controlled in an accurate manner. In some embodiments, each one of the plurality of sensors is a camera sensor, a LIDAR sensor, a sonar sensor, a microphone, or a radar sensor. In some embodiments, the method further comprises receiving a plurality of input data streams from the plurality of sensors, wherein each one of the plurality of input data streams corresponds to a different one of the plurality of sensors. Generating a plurality of robot modality inputs, each corresponding to a respective robot modality, comprising generating, for each of the plurality of robot modalities, a robot modality input by combining two or more of the plurality of input data streams that are associated with the robot modality. By combining data streams associated with the same robot modality in order to generate a robot modality input, redundant data may be reduced.

[0014] In some embodiments, the coordinate system is based upon a point of origin defined with reference to the robot. By basing the coordinate system on a point of origin defined with reference to the robot, transformation or actuation of the robot may be tailored specifically for different types of actuation. For example, for general positioning of the robot, using the base of the robot as the point of origin may be desirable. In another example, for actuating end effectors of the robot whilst maintaining stationary, using the end of the end effector as the point of origin may be desirable.

[0015] There is also described herein a computer readable storage medium comprising computer readable instructions as set out above. There is also described herein an apparatus comprising a processor and memory, the memory comprising computer readable instructions as set out above.

[0016] While specific embodiments of the invention have been described above, it will be appreciated that the invention may be practiced otherwise than as described. The descriptions above are intended to be illustrative, not limiting. Thus it will be apparent to one skilled in the art that modifications may be made to the invention as described without departing from the spirit of the invention.

[0017] BRIEF DESCRIPTION OF FIGURES

[0018] Figure 1 depicts an environment comprising a plurality of robots, a plurality of client devices, and a server, in which techniques described herein may be implemented.

[0019] Figure 2 depicts a robot, a set of robot hardware and software components, a client device, and a set of client device hardware and software components, in which techniques described herein may be implemented. Figure 3A depicts a front-view of a robot, in which techniques described herein may be implemented.

[0020] Figure 3B depicts a side-view of a robot, in which techniques described herein may be implemented.

[0021] Figure 4 depicts a flow diagram of a method for remotely controlling a robot.

[0022] Figure 5 depicts a flow diagram of a method for controlling a robot.

[0023] DETAILED DESCRIPTION

[0024] Figure 1 schematically depicts a system 1 for enabling remote control of one or more robots by one or more client devices.

[0025] In Figure 1 , a plurality of robots 101 are available for control by ones of a plurality of client devices 111. The plurality of robots 101 include a robot 100 and robots 101a, 101b. While three robots are shown in Figure 1 , it will be appreciated that the plurality of robots 101 may include a number of robots. The robot 100 is described in more detail below. Each of the plurality of robots 101 may be the same as or different to the robot 100. That is, the robots may have different configurations of sensors and / or actuators and may be located in different geographical locations. Generally, however, as explained in more detail below, each of the robots in the plurality of robots is configured to receive instructions from the client device 100, to execute those instructions, and / or to transmit data to the client device 100. Each the plurality of robots 101 may be at a distinct remote location. For example, any one of the plurality of robots 101 may be located in a different geographical location (e.g. town, city, postcode, coordinate, country, continent, etc.) to any other one of the plurality of robots 101.

[0026] The robot 100 may be any physical machine capable of sensing, and / or being actuated within, an environment. The robot 100 comprises one or more sensors and / or actuators to sense the environment and / or be actuated within the environment, respectively. By way of example only, the robot 100 may be provided with one or more of a camera sensor, LIDAR sensor, radar sensor, sonar sensor, pressure sensor, or capacitive sensor. As will be apparent to the skilled person, the robot 100 may be provided with any number of other sensors for detecting characteristics of the environment which the robot 100 occupies.

[0027] The robot 100 is configured to receive and transmit data over a network 130. That is, the robot 100 is equipped with a suitable network interface used to establish a data connection over the network 130. The network 130 may be a wide area network (WAN) extending over a geographical area. For example, the network 130 may be the Internet. The network 130 may be other wide area networks which may extend over smaller geographical areas such as continents, countries, cities, etc. Alternatively, the network 130 may use any appropriate connection technologies as will be apparent to the skilled person, and the robot 100 may connect to the network 130 using any appropriate connection technology. For example, the robot may connect to the network 130 using a cellular connection (e.g. 3G, LTE, 4G, 5G, etc.), WiMAX connection, etc. The robot 100 may connect to the network 130 through a local area network (LAN), for example through a Wi-Fi connection.

[0028] As described above, the robot 100 executes software that enables the robot 100 to establish the connection with the network 130, transmit data to and receive data from the network 130, as well as perform other functions. For example, the software may take the form of software executed by a central processor and / or accelerator processor that is configured to control the robot 100, and / or software (which may include ‘firmware’) installed onto one or more of the individual hardware components of the robots 100. The software executed by the robot 100 is configured to translate and / or decode instructions received from any one of the plurality of client devices 111 into instructions that are executable by any one of the plurality of robots 101 .

[0029] As described above, the robot 100 may be one of a plurality of robots 101 made available to the client device 110 (and the other client devices 111) for the client device 110 to control. The client device 110 may, for example, maintain a local list of available robots. Alternatively or additionally, the client device 110 may communicate with a server 120 to determine if any one of the plurality of robots 101 is available for connection with the client device 110, for the client device 110 to control. The client device 110 may communicate with the server 120 over a first network connection 121. The robot 100 may also communicate with the server 120 over a second network connection 122. The first network connection 121 and second network connection 122 may be any form of network connection facilitated by any form of network, and may, for example, be the same as the network 130.

[0030] The client device 110 may be one of a plurality of client devices 111. The client device 110 may send a request to the server 120 over the network connection 121 in order to determine whether or not any one of the plurality of robots 101 is available for connection with the client device 110. The server 120 may transmit data back to the client device 110 over the network connection 121 indicating that the robot 100 is or is not available for connection. In the case that the robot 100 is not available for connection, the server 120 or client device 110 may prohibit a data connection being established between the client device 110 and the robot 100. In the case that the robot 100 is available for connection, the server 120 or client device 110 may allow a connection being established between the client device 110 and the robot 100. In some examples, the server 120 may facilitate establishment of the data connection between the client device 110 and the robot 100. The facilitation may involve the server 120 initiating a “handshake” between the client device 110 and the robot 100. The handshake may involve the establishment of the data connection between the client device 110 and the robot 100, and an agreement on a set of protocols and / or policies used to manage the data connection. The initiation by the server 120 may involve the server 120 communicating with the robot 100 over the second network connection 122, and the server 120 communicating with the client device 110 over the first network connection 121.

[0031] Each of the plurality of client devices 111 is configured to receive instructions for controlling a robot. In one example, the instructions are received from a user or from another device (not shown). Instructions may be received in a first programming language. The first programming language may be any one of a number of programming languages (e.g. Python, JavaScript, Java, C, C++, C#, etc.). For example, the client device 110 may be configured to receive instructions written in the programming languages “Python” or “Java”. In one example, the instructions may be received comprising compiled or interpreted instructions.

[0032] At least some of the instructions may comprise a set of predetermined datatypes. Each of the set of predetermined datatypes is configured to encode data relating to a particular, generally single, robot modality. Data relating to a robot modality is data relating to a mode by which the robot 100 experiences or interacts with an environment. For example, a first predetermined datatype may be configured to encode data related to a “visual” mode of experiencing the environment. A visual mode may be a mode that is characterised by receiving and processing electromagnetic signals and may not be limited to electromagnetic signals in the visible spectrum. In another example, a second predetermined datatype may be configured to encode data related to a “motor” mode of interacting with the environment, e.g. relating to movement of the robot 100. In another example, a third predetermined datatype may be configured to encode data related to a “proprioception” mode relating to a robot’s understanding of its own position in space. It will be understood that one of the set of predetermined datatypes may relate to any mode by which the robot 100 can experience or interact with the environment. The set of predetermined datatypes are datatypes that may be processed by each of the plurality of robots 101. For example, the robot 100 may receive instructions conforming to the set of predetermined datatypes, and the robot 100 may be capable of processing and / or executing the instructions. In some examples, each one of the plurality of client devices 111 may comprise one or more user interface devices, including suitable visual, audio, and / or haptic feedback devices. Such interaction devices may be configured to provide visual, audio, and / or haptic feedback from a robot 101 to which the client device is connected, to a user of the client device. It will be understood that other types of feedback devices will be envisaged. For example, client devices may be provided with screens, controllers, virtual reality headsets, etc., as will be apparent to the skilled person.

[0033] Table 1 below describes an exemplary set of predetermined datatypes that may be used in examples of the techniques described herein. It will be understood that other predetermined datatypes may be envisaged and used. Advantageously, by transmitting instructions between a client device and a robot comprising a set of predetermined datatypes that relate to a particular robot modality, a significant reduction in latency between transmission of the instructions by the client device and the execution of those instructions by the robot, may be achieved. In experiments, a reduction in latency of 33% has been observed compared to the latency observed in a standard internet video call. This is in contrast to the latency observed in known methods, which often exceed the latency of a standard internet video call. As described in more detail below, the techniques described in the present application are designed to optimize, having regard to both the robots and client devices, transmission of data using internet-based communications. It will be understood that others of the client devices 111 may be similarly configured to the client device 110, and that others of the robots 101 may be similarly configured to the robot 100. That is, each of the other client devices may be configured to receive instructions in the same manner as the client device 110, and that each of the other robots 101 may be configured to receive instructions in the same manner as the robot 100.

[0034] Table 2 below describes an exemplary set of robot modalities that may be used in techniques described herein. For each one of the set of robot modalities, Table 2 further describes, for ease of understanding, a corresponding human equivalent mode of experiencing an environment, and a corresponding description of a robot equivalent experience of the environment. It will be appreciated that robot modalities relate to ways in which the robot experiences an environment and may not map directly to human senses. As such, the mapping is provided for illustration only. Advantageously, associating one or more sensors with a corresponding robot modality enabled a reduction in latency to be observed when sending data relating to those sensors between the robot and the client device(e.g. providing input from the sensors to the client device, and / or providing instructions relating to the sensors from the client device to the robot). For example, by associating ones of a robot’s sensors with a corresponding modality, this can facilitate the client device to provide feedback to a user of the client device in a manner that combines input data streams from a plurality of sensors of a corresponding robot modality to the client device and enables the client device to more easily address the plurality of sensors.

[0035] Table 1

[0036] Table 2

[0037]

[0038] Figure 2 schematically depicts a robot 200 and a client device 210 connected via a network 220 (which may be the network 130 depicted in Figure 1). The robot 200 may the robot 100 depicted in Figure 1 , or any other of the plurality of robots 101 in Figure 1. The client device may be the client device 110 depicted in Figure 1 , or any other of the plurality of client devices 111 in Figure 1. The robot 200 comprises a set of robot hardware and software components 201 . The client device 210 comprises a set of client device hardware and software components 211. While depicted together in Figure 2 for ease of explanation, it will be appreciated that the robot 200 is a separate entity to the client device 210 and need not be connected to a client device over a network.

[0039] The robot hardware and software components 201 include robot control logic 202. The robot control logic 202 comprises software that is configured to control operation of the robot 200 including others of the set of robot hardware and software components 201 for interaction with the client 210. The robot control logic 202 may be written in any one of a set of programming languages suitable for execution by the robot 200, or be expressed in any way otherwise capable of being executed by the robot 200. The robot control logic 202 may comprise one or more separate and / or interoperable software components. The robot control logic 202 may comprise software which has the dedicated function of enabling hardware components of the robot 200 to operate correctly (often known as firmware and / or drivers). The robot control logic 202 is responsible for determining how each one of the set of robot hardware and software components 201 interacts with every other one of the set of robot hardware and software components 201 . The robot control logic 202 may also determine how the robot 200 or each one of the set of robot hardware and software components 201 interacts with other entities external to the robot 200. The robot control logic 202 may be stored on the robot 200 in a computer-readable storage medium 205 associated with the robot 200. The robot control logic 202 may be processed and / or executed by one or more processors 204 associated with the robot 200.

[0040] The robot hardware and software components 201 also includes a network interface 203. The network interface 203 may include hardware and / or software configured to enable the robot 200 to communicate with other devices, including the client 210 over a network (such as the network 220). The network interface 203 may comprise a hardware component that is configured to interact directly with the network 220 over a wired and / or wireless connection. The network interface 203 may use a set of network protocols and / or policies stored in the computer-readable storage medium 205, and which determine how data is transmitted and received over the network 220. The network interface 203 may further comprise hardware and / or software components configured to communicate with other devices within the local vicinity of the robot 200, such as other robots or other connected devices (such as “Internet of Things” devices). For example, the network interface 203 may be configured to receive information from, or send data to, other devices in the local vicinity of the robot 200, such as beacons. In this way, the network interface may be considered to act as a sensor, sensing communication signals from other devices.

[0041] The set of robot hardware and software components 201 also includes a set of one or more sensors 206. Each one of the set of one or more sensors 206 is a sensor which is capable of detecting characteristics of an environment and / or enabling the robot 200 to receive an input data stream from the environment. For example, a first sensor of the set of one or more sensors 206 may be a camera, the camera being capable of detecting visual characteristics of the environment and / or enabling the robot 200 to receive a visual input data stream from the environment. Each one of the set of one or more sensors 206 is configured to have their operation controlled by the robot 200 and / or the robot control logic 202. Other examples of sensors are provided above, and it will be appreciated that the sensors 206 may include any sensors.

[0042] The robot hardware and software components 201 also include a set of one or more actuators 207. Each one of the set of one or more actuators 207 is an actuator which is capable of actuating the robot 200 (or a portion of the robot 200) in the environment. For example, a first actuator of the set of one or more actuators 207 may be a motor, the motor being capable of actuating a first portion of the robot 200 (such as an arm, wheel, effector, etc.).

[0043] The client device hardware and software components 211 includes client control logic 212. The client control logic comprises software that is configured to control operation of the others of the set of client device hardware and software components 211 for interaction with the robot 200. The client control logic 212 may be written in any one of a set of programming languages suitable for execution by the client device 210, or be expressed in any way otherwise capable of being executed by the client device 210. The client control logic 212 may comprise one or more separate and / or interoperable software components. The client control logic 212 may comprise software which has the dedicated function of enabling hardware components of the client device 210 to operate correctly (i.e. firmware and / or drivers). The client control logic 212 may be stored on the client device 210 in a computer-readable storage medium 215 associated with the client device 210. The client control logic 212 may be processed by one or more processors 214 associated with the client device 210.

[0044] The client device hardware and software components 211 also includes a network interface 213. The network interface 213 is configured to enable the client device 210 to communicate with other devices, including the robot 200 over the network 220. The network interface 213 uses a set of network protocols and / or policies, stored in the computer-readable storage medium 205, and which determine how data is transmitted and received over the network 220.

[0045] The client device hardware and software components 211 comprises a set of one or more input / output devices 216. Each one of the set of one or more input / output devices 216 is an input / output device capable of receiving an input and / or generating an output. For example, the input / output devices 216 may be configured to allow a user to input instructions for the robot 200 and to receive data / feedback from the robot 200. For example, a first input / output device of the set of one or more input / output devices 216 may be a microphone or a visual display, the microphone capable of receiving audio input and the visual display capable of generating a visual output. It will be appreciated that the client device 210 may include any input and / or output devices.

[0046] The client control logic 212 is configured to enable the client device 210 to connect to the robot 200 via a data connection over a network 220. The robot 200 may be one of a plurality of available robots each of which are available for connection. The network 220 may be the network 130 depicted in Figure 1. The client device 210 may send a request to a server (not depicted) in order to request a list of robots, of which the robot 200 is one. For example, a server may provide the client device 210 with a list of robots which are available or unavailable. The client device 210 may send a further request, upon receiving the list of available / unavailable robots, to the server in order to establish a data connection between the client device 210 and the robot 200 in response to determining that the robot 200 is available for the data connection. The client device 210 may then connect to the robot 200 via the data connection over the network 220. In one example, the client device 210 is connected to the robot 200 via a peer-to-peer data connection. It will be understood that the data connection between the client device 210 and the robot 200 may be established in any number of suitable ways. In alternative examples, the client device 210 may store an indication of one or more robots to which the client device may connect, such that the client device 210 is able to connect to the robot without requesting a list of robots from a server. For example, the client device 210 may not request a list of robots from an external source, or may use both a list of robots from an external source and / or one or more locally stored indications of robots to which the client device 210 may connect.

[0047] In one example, the data connection over the network 220 may be managed by a set of protocols and / or policies. In some examples, the protocols or policies may define a peer- to-peer connection model. It will be appreciated that any of a number of peer-to-peer data connection models may be used. In one example, the set of protocols and / or policies may be a set of WebRTC protocols. WebRTC uses a peer-to-peer connection model, whereby a connection is established between two devices directly over a network. The set of WebRTC protocols determine how data is transmitted and received between the two devices. WebRTC is implemented in a number of different programming language APIs which enable application of the set of WebRTC protocols to a large number of different devices. WebRTC has generally been used for facilitating video and / or audio calls over the Internet, and for use in remote monitoring, surveillance, and the “Internet of things” (e.g. connecting one smart device to another smart device). The set of WebRTC protocols, and peer-to-peer remote internet communication protocols more generally, may not be suitable for use with devices that transmit and / or receive relatively large amounts of data over the network 220. However, for relatively small amounts of data, the set of WebRTC protocols enable efficient transmittal and / or receiving of data between the two devices directly.

[0048] The client control logic 212 is configured to enable the client device 210 to receive instructions. The instructions may be instructions for controlling operation of the robot 200. The instructions are received by the client device 210. The instructions may be expressed in a programming language (e.g. Java, Python, C, C++, etc.), as machine code, as binary, or in any other form executable by the client device 210. For example, the client device 210 may receive first instructions expressed in Python. The client control logic 212 facilitates the instructions to be received at the client device 210 by each of the set of one or more input / output devices 216 or via the network interface 213. The instructions may be provided in accordance with a set of programming language libraries. The programming language libraries are a set of predefined functions and / or datatypes available for inclusion, by the user, in the instructions.

[0049] The instructions may comprise a set of predetermined datatypes. The predetermined datatypes may include one or more of the exemplary set of predetermined datatypes described in T able 1 above. Each one of the set of predetermined datatypes corresponds to one of a plurality of robot modalities associated with the robot 200. Each one of the set of predetermined datatypes may encode data that relates to the corresponding one of the plurality of robot modalities associated with the robot 200. Each one of the plurality of robot modalities associated with the robot is a different mode in which the robot 200 can experience or interact with an environment. For example, a first predetermined datatype may encode one or more visual data items that corresponds to a visual modality associated with the robot 200. In this example, the visual modality is a visual mode in which the robot 200 can experience the environment. For example, the “ImageSamples” datatype may be used to combine a plurality of samples of image data received from a plurality of sensors. Advantageously, transmitting data which has been encoded according to one of the set of predetermined datatypes (e.g. the predetermined datatypes in Table 1) reduces the frequency of transmissions between the robot 200 and the client device 210. This reduction in the frequency of transmission is achieved by grouping and / or combining two or more input data streams related to the same robot modality into one of the set of predetermined datatypes. The fewer transmissions achieved by grouping and / or combining data in this way reduces unnecessary header data that would otherwise be transmitted if the two or more input data streams were transmitted over multiple transmissions.

[0050] The client control logic 212 is configured to enable the client device 210 to transmit the instructions to the robot 200 over the network 220. The operation of the robot 200 is controlled in accordance with the instructions transmitted to the robot 200. The instructions may conform to a predetermined data format. The predetermined data format may facilitate transmission over the network 220. The predetermined data format may improve efficiency and / or latency of transmission over the network 220, particularly when paired with a first set of protocols used to facilitate the data connection over the network. In one example, the predetermined data format may be a serialised data format. In one example, the predetermined data format may be a Protobuf data format. Protobuf is an open source data format developed for the purpose of communicating data over a network. Protobuf is a data format which is formatted in such a way that the data is serialised and encoded in binary, the binary encoding additionally encoding a Protobuf schema used for decoding the encoded data. Advantageously, the wide range of Protobuf supported programming languages increases interoperability between devices operating on different hardware and software. Furthermore, Protobuf enables the efficient transmittal of data over a network by reducing bandwidth requirements for transmitting data when compared with other predetermined data formats. For example, Protobuf may format the data in such a way that reduces the bandwidth requirements to transmit the data over a network when compared with alternative data formats (e.g. JSON).

[0051] As discussed, using the set of predetermined datatypes leads to a reduction in the frequency of transmissions between the robot 200 and the client device 210, thereby reducing the transmission of redundant data over the network 220. This reduction in data being transmitted over the network 220 facilitates the use of a specific set of protocols and / or policies which may otherwise not be suited to the transmission of larger amounts of data. The use of the specific set of protocols and / or policies which may not be suited to the transmission of larger amounts of data may provide additional benefits to the transmission of data between the robot 200 and the client device 210. Such benefits include a more stable data connection, an improvement to privacy and / or security of the data connection, a more efficient data connection, an improvement to the latency of the data connection.

[0052] The robot control logic 202 may be configured to enable identification of sensors and / or actuators associated with the robot 200. For example, the robot software 202 may provide an interface to a user to enable the user to determine which sensors I actuators the robot provides. Alternatively or additionally, the robot control logic 202 may automatically detect sensors and / or actuators of the robot. The identification may involve storing the list in the computer readable storage medium 205 of the robot 200.

[0053] The robot control logic 202 is configured to associate (or enable a user of the robot 200 to associate) each one of the plurality of sensors and / or actuators associated with the robot 200 with a corresponding robot modality. The association may involve the computer software 202 mapping each one of the set of one or more sensors 206 and the set of one or more actuators 207 to a corresponding robot modality. The mapping may be stored in the computer readable storage medium 205 of the robot 200.

[0054] The robot control logic 202 is configured to enable the robot 200 to generate a robot modality input by combining one or more input data streams received from one or more sensors 206 associated with the robot 200 in one of the set of predetermined datatypes. As described above, the datatypes may encode data received from different sensors relating to the same robot modality. As such, two or more input data streams may be received from different ones of the set of one or more sensors 206 associated with the robot 200 and encoded into the same datatype. The encoding of two data streams in a single datatype may comprise a joining (e.g. appending) of the data of the two or more input data streams. It will be understood that the encoding of input data streams may involve any method that results in each one of the two or more input data streams being used to generate the robot modality input. For example, a first input data stream received from a first camera sensor associated with the robot 200 may be combined with a second input data stream received from a second camera sensor associated with the robot 200 by joining the first input data stream with the second input data stream. More generally, input data streams may be processed in any appropriate way to facilitate the encoding of the two input data streams in a datatype. It will be appreciated that the particular processing may depend upon the modality to which the datatype relates.

[0055] The robot modality input may further comprise a coordinate system of the robot modality input. The coordinate system may extend infinitely in all directions in 3-dimensional space. The coordinate system may comprise a point of origin to which every point in the coordinate system is relative. The point of origin may be located at any point in the coordinate system. The point of origin may comprise an x-position, a y-position, and a z- position. In one example, the point of origin is located at a base of the robot. The base of the robot 200 may include any of the lowest points on the robot 200. In one example, the base of the robot 200 is located at the centre-bottom of the robot 200. With reference to Figures 3A and 3B, the base 350 of the robot is depicted. A quaternion of the robot 200 may comprise the position of the robot 200 in the coordinate system, in addition to an orientation of the robot 200. The orientation of the robot 200 determines the direction the robot 200 is orientated in the coordinate system with respect to the position of the robot 200 in the coordinate system. The robot 200 may transmit the robot modality input to the client device 210. The robot modality input may be transmitted to the client device in response to a request being sent from the client device 210 to the robot 200. The transmission of the robot modality input may be scheduled to occur at regular intervals of time while the data connection between the robot 200 and the client device 210 is active. The client device 210 may receive the robot modality input and optionally generate an output based on the robot modality input for a user interacting with one of the set of one or more input / output devices 216 associated with the client device 210. The feedback may represent the robot modality input as an output in any number of ways. For example, a first robot modality input associated with a visual modality may be provided as feedback to the user by representing the first modality input as a visual representation by a display of the one or more input / output devices 216. For example, the feedback provided to the user may allow the user to vicariously interact and / or experience the environment in which the robot 200 is located. The client device 210 may further receive the coordinate system of the robot modality input.

[0056] The client device 210 may transmit one or more instructions to the robot 200 in response to receiving the robot modality input, or in response to a user reacting to feedback provided by any one of the set of one or more input / output devices 216. The instructions may be configured to cause the robot 200 to actuate, or may be configured in any other way cause the operation of the robot to be controlled. The instructions may be configured to cause one or more of the set of one or more actuators 207 to actuate the robot 200. For example, a first instruction may be configured to cause a first motor and / or a second motor associated with the robot 200 to operate. The instructions may be defined with reference to the coordinate system of the robot modality input. The instructions may be provided by a user to the set of one or more input / output devices 216 as first input in response to the user perceiving a first output of the set of one or more input / output devices 216. The first output may be a representation of the robot modality input provided to the user by one or more of the set of one or more input / output devices 216. The robot 200 executes the instructions upon receiving the instructions, thereby controlling the operation of the robot 200 in accordance with the instructions.

[0057] In general, the robot control logic 202 and the client control logic 212 both encode data according to the one or more predetermined datatypes depending on a corresponding robot modality. In other words, data received by the robot 200 corresponding to the same robot modality may be combined into a single predetermined datatype using the robot control logic 202 before transmitting the data to the client device 210. Additionally, data received by the client device 210 corresponding to the same robot modality may also be combined into a single predetermined datatype using the client control logic 212 before transmitting the data to the robot 200.

[0058] Figure 3A schematically depicts a front-view of a robot 300. Figure 3B schematically depicts a side-view of the robot 300. The robot may be the robot 100 depicted in Figure 1 , any other robot of the plurality of robots 101 in Figure 1 , or the robot 200 depicted in Figure 2. The robot 300 is associated with a plurality of sensors 303, 304, 305, 306, 307, 308 and / or actuators 301 , 302, 309, 310. It will be understood that the robot 300 may comprise any number of sensors and / or actuators. Each one of the plurality of sensors associated with the robot 300 may be, for example, a camera sensor, a microphone, a laser sensor, a sonar sensor, a LIDAR sensor, or any other sensor known in the art. Each one of the plurality of actuators associated with the robot may be, for example, a motor, a speaker, a light source or any other actuator known in the art. For example, the robot 300 may have a first motor 310 capable of displacing a first structure 313 of the robot 300 in respect of a second structure 314 of the robot 300 such that the angle between the first structure 313 and the second structure 314 can be controlled. In another example, the robot 300 may have a second motor 309 capable of displacing the second structure 312, thereby also displacing anything attached thereof (i.e. the first structure 313 and the first motor 310), such that the angle between the body of the robot 300 and the second structure 314 can be controlled. It will be understood that any configuration of the plurality of sensors and / or actuators is envisaged.

[0059] The robot 300 is associated with a robot coordinate system 320 which may be used by the robot 300 to orient and / or calibrate any one of the plurality of sensors and / or actuators associated with the robot 300. The coordinate system 320 may be used by the robot 300 during execution of instructions and by client devices when providing instructions. For example, if the robot receives an instruction to control a motor associated with the robot 300, the robot 300 may use the coordinate system in order to achieve a particular result (i.e. move a structure of the robot 300 into a particular position in the coordinate system, or to transport the robot 300 to the particular position). The coordinate system may be the coordinate system described with reference to Figure 2. The coordinate system 320 is determined with respect to a point of origin 321 which may be located at a base 350 of the robot 300. The base 350 of the robot 300 is the centre-bottom of the robot’s 300 body. In other words, the location of the base 350 of the robot 300 in 3D space is the midpoint of the robot 300 in the x direction (as depicted in Figure 3A), the midpoint of the robot in the z direction (as depicted in Figure 3B), and the lowest point in space in the y direction on the robot’s 300 body (as depicted in both Figure 3A and Figure 3B). It will be understood that the point of origin 321 of the coordinate system 320 may be located at any point on or relative to the robot 300, in addition to at any point in an environment 330 that the robot 300 occupies. Each one of the plurality of sensors associated with the robot 300 is capable of receiving an input data stream from the environment 330. The input data stream received by the plurality of sensors indicate characteristics of the environment 330. The characteristics of the environment 330 may be a visual, an audio, a thermal, or any other characteristic of the environment 330. The robot 300 may map received sensor inputs to the coordinate system. The robot 300 is capable of transmitting and receiving the coordinate system 320 and / or the point of origin 321 over a network. Each one of the plurality of sensors and / or actuators is associated with a corresponding robot modality. The robot modality may be the robot modality described with reference to Figure 2. The robot modality may be any one of a plurality of modes in which the environment 330 can be experienced. For example, a first sensor of the plurality of sensors may be a camera. The camera may be associated with a robot modality named “Vision”. The robot modality “Vision” is a mode in which the environment 330 can be visually experienced. In another example, a first actuator of the plurality of actuators may be a motor. The motor may be associated with a robot modality named “Motor”. The robot modality “Motor” is a mode in which the environment 330 can be physically experienced. The robot modality may be any one of the exemplary set of robot modalities described in Table 2. It will be understood that any number of robot modalities may be envisaged.

[0060] The robot 300 is capable of generating a robot modality input by combining two or more input data streams each received from a different one of the plurality of sensors each associated with the robot 300, the different ones of the plurality of sensors are each associated with the same robot modality. For example, the robot 300 may receive a first input data stream from a first sensor (for example a camera) associated with the robot modality “Vision”, and a second input data stream from a second sensor (for example a lidar sensor) also associated with the robot modality “Vision”. In this example, the first sensor and the second sensor are different ones of the plurality of sensors associated with the robot 300, and the first sensor and the second sensor are both associated with the same robot modality (i.e. the robot modality “Vision”). In this example, the robot modality input may be generated by mapping the first input data and the second input data to the coordinate system 320 and combining the first input data stream with the second input data stream. The robot modality input may further comprise the coordinate system 320 of the robot 300. The robot 300 may receive instructions configured to actuate the robot 300 with reference to the coordinate system 320. The actuation of the robot 300 causes the operation of the robot 300 to be controlled in accordance with the instruction.

[0061] Each one of the plurality of sensors may have an associated field of view. Each field of view may be with respect to the coordinate system 320. A field of view of a sensor may define an area of the environment 330 that can be detected by that sensor. For example, the robot 300 may have a first sensor 304, a second sensor 305, and a third sensor 308. The first sensor 304 may have a first field of view 304a, the second sensor 305 may have a second field of view 305a, and the third sensor 308 may have a third field of view 308a as depicted by the front-view and side-view of the robot 300 in Figures 3A and 3B respectively. Each one of the first sensor 304, the second sensor 305, and the third sensor 308 have different fields of view to every other one of the first sensor 304, the second sensor 305, and the third sensor 308. In this example, each one of the first sensor 304, the second sensor 305, and the third sensor 308 are each capable of detecting characteristics of the environment 330 in their respective field of view which are different to every other one of the first sensor 304, the second sensor 305, and the third sensor 308. It will be understood that, because of potentially differing fields of view, each one of plurality of sensors may not be capable of detecting characteristics from the environment 330 that could be detected by every other one of the plurality of sensors. It will also be understood that in some examples the respective fields of view associated with each one of the plurality of sensors may overlap. For example, the field of view associated with a fourth sensor 312 has a first field of view that overlaps 311a with a second field of view associated with a fifth sensor 311.

[0062] Advantageously, by combining two or more input data streams respectively provided by two or more of the plurality of sensors each having a different field of view, generating a robot modality input is improved by providing a robot modality input comprising a plurality of input data streams which as a whole reflect a more holistic field of view of the environment 330. Each one of the plurality of sensors contributes to the robot modality input, and in addition may compensate for shortcomings of the other ones of the plurality of sensors, at least in respect of field of view. Furthermore, by combining the two or more input data streams the generation of the robot modality input may be further be improved by reducing redundant data in the robot modality input by taking into account any overlap in the field of view respectively associated with any two or more of the plurality of sensors. Alternatively, sensor data from multiple inputs with overlapping fields of view may be combined to provide more information for the overlapping portion of the respective fields of view.

[0063] With reference to Figure 4, there is now described a computer-implemented method for remotely controlling a robot. It will be understood that the method described below may be performed by the system 1 depicted in Figure 1 or a method performed by the system 2 of Figure 2.

[0064] At step 401 , a client device connects to the robot via a data connection over a network. The robot may be the robot 100 depicted in Figure 1 , any other one of the plurality of robots 101 depicted in Figure 1. Likewise, the client device may be the client device 110 depicted in Figure 1 , or any other one of the plurality of client devices 101. The network may also be the network 130 depicted in Figure 1. The client device may be a device operated by a user, or be a fully autonomous device. In the case that the client device is a device operated by the user, the user may operate the client device by interacting with one or more input devices associated with the client device.

[0065] The data connection over the network between the client device and the robot may be established in accordance with, and / or managed by, a set of one or more network protocols and / or policies. The set of one or more protocols and / or policies determine how data is transmitted and received over the data connection. It will be understood that the set of one or more protocols and / or policies may be any set of protocols and / or policies known by a person skilled in the art to be suitable for determining how data is transmitted and received over the data connection. In one example, the set of one or more protocols and / or policies may be the set of WebRTC protocols as described above with reference to Figure 2.

[0066] At step 402, the client device receives instructions to control operation of the robot. The received instructions may be received from a user, or other device. The instructions may be received from the user or other device over a network or data connection. The received instructions may be received in the form of instructions expressed in a programming language, bytecode, machine code, binary, or any other form suitable for instructing control operation of the robot over the network or data connection. The received instructions may comprise one or more of a set of predetermined datatypes. The predetermined datatypes may be the predetermined datatypes as described above with reference to Figure 2. Each one of the set of predetermined datatypes may be any one of the exemplary set of predetermined datatypes described in Table 1. It will be understood that other predetermined datatypes may be envisaged. Each one of the predetermined datatypes may relate to one of a set of robot modalities. The set of robot modalities may be the set of robot modalities described above with reference to Figure 2. The set of robot modalities may be the exemplary set of robot modalities described in Table 2. It will be understood that other robot modalities may be envisaged.

[0067] At step 403, the client device transmits the received instructions to the robot to control operation of the robot in accordance with the instructions. The received instructions may be instructions implemented in any one of a plurality of programming languages. For example, the instructions may be implemented in a Python programming language. In another example, the instructions may be implemented in a C++ programming language. The instructions may be received as compiled or interpreted instructions depending on the specific programming language the instructions are implemented in. The client device may transmit the instructions via the data connection over the network to the robot. In one example, the client device may transmit the received instructions to the robot in order to retrieve data from the robot. In another example, the client device may transmit the received instructions to the robot in order to control the motors of the robot. As described above, the data connection may be established and / or managed by a set of protocols and / or policies. In some examples, the client device may transmit the instructions in a predetermined data format. In one example, the predetermined data format may be a serialised data format. In an example, the data format may be the Protobuf data format described above with reference to Figure 2. It will be understood that other data formats may be used.

[0068] The instructions may cause the robot to perform operations, such as physical operations (e.g. movement) and / or internal operations (such as updates to internal states). It will be understood that the control may be the control of any function of the robot. For example, an end effector or other physical structure of the robot may be operated by controlling a motor or other actuator configured to move the end effector or other physical structure. In another example, a software function of the robot may be operated by instructing the robot control logic to execute the software function. In another example, a sensor may be instructed to perform measurement of some characteristic of the environment in which the robot operates.

[0069] With reference to Figure 5, there is now described a computer-implemented method for controlling a robot. It will be understood that the method described below with reference to Figure 5 may be a method performed by the system 1 depicted in Figure 1. It will be further understood that the robot may be the robot 100 depicted in Figure 1 , or the robot 200 depicted in Figure 2.

[0070] At step 501 , a plurality of sensors and / or actuators associated with the robot are identified. Each one of the plurality of sensors is capable of detecting characteristics of an environment and correspondingly receiving an input data stream reflecting the detected characteristics of the environment. Each one of the actuators is capable of actuating the robot within the environment. Each one of the plurality of sensors may be an image sensor, LIDAR sensor, radar sensor, sonar sensor, pressure sensor, capacitive sensor, or any other sensor capable of detecting characteristics of the environment.

[0071] At step 502, each one of the plurality of sensors and / or actuators are associated with a corresponding robot modality. The robot modality represents a mode in which the robot can experience or interact with the environment. The association may further comprise any step of the association performed by the robot control logic 202 or by the robot 200 depicted in Figure 2. The robot modality may be any one of the exemplary set of robot modalities in Table 2. The robot modality may be any one of the set of robot modalities described with reference to Figure 2. It will be understood that other robot modalities will be envisaged.

[0072] At step 503, a first input data stream is received from a first sensor of the plurality of sensors. The first input data stream may be received from the first sensor in any number of ways. In particular, the first input data stream may be received in accordance with steps performed by the robot 200 depicted in Figure 2. The first input data stream may be received in any format suitable for being received for the first sensor. The first sensor may be any sensor capable of detecting characteristics from an environment and enabling an associated device to receive the first input data stream.

[0073] At step 504, a second input data stream is received from a second sensor of the plurality of sensors. The first sensor and the second sensor are different ones of the plurality of sensors, the first sensor and the second sensor each being associated with the same robot modality. The second input data stream may be received from the second sensor in any number of ways. In particular, the second input data stream may be received in accordance with the steps performed by the robot 200 depicted in Figure 2. The second input data stream may be received in any format suitable for being received from the second sensor. The second sensor may be any sensor capable of detecting characteristics from an environment and enabling an associated device to receive the second input data stream.

[0074] At step 505, a robot modality input is generated by combining the first input data stream and the second input data stream. The combining may involve the joining of the first input data stream with the second input data stream. It will be understood that the combination may involve the combination of any number of input data streams. In one example, the combination may involve encoding the first input data stream with the second input data stream in a one of a set of predetermined datatypes. The predetermined datatypes may be any one of the exemplary set of predetermined datatypes described in Table 1. Any one of the input data streams combined may relate to the same robot modality. The robot modality may be any one of the exemplary set of robot modalities described in Table 2. The robot may transmit data to a client device. In particular, the robot may transmit data including the robot modality input. For example, where the robot modality input is a vision modality input, the robot may transmit vision data to the client device. This may enable the client device to provide an output to a user that represents the environment of the robot as sensed by the first sensor and the second sensor.

[0075] At step 506, an instruction configured to actuate the robot is received. The instruction is defined with reference to a coordinate system of the robot modality input. The instruction may be configured to actuate an actuator associated with the robot. In one example, the actuator may be a motor, a microphone, or a light source. It will be understood that other types of actuators will be envisaged. The coordinate system is a coordinate system of a 3D space that the robot occupies. The coordinate system may comprise a point of origin. The point of origin is the centre point upon which every other point of the coordinate system is relative to. The point of origin may be at the base of the robot, the base being described above. It will be understood that the point of origin may be located at any point in the 3D space that the robot occupies. In For example, instructions to actuate the robot may comprise instructions to transform a3D position and / or orientation of the robot or its respective components (e.g. an end effector of the robot). In one example, the transformation may be carried out in response to the robot receiving instructions to actuate an end-effector of the robot. The transform may be defined relative to the point of origin as described above. In this example, the point of origin may be relative to a sensor of the robot, the robot’s base, or other point relative to the body of the robot. In one example, the transformation may be processed to determine an equivalent transformation relative to the robot’s base. In this example, the equivalent transformation is then sent into an inverse kinematic solver that generates a robot specific solution for actuating the robot actuators in order to achieve the received instruction.

[0076] It will be appreciated that embodiments disclosed herein can be implemented in any convenient form. For example, embodiments disclosed herein may be implemented by appropriate computer programs which may be carried on appropriate carrier media which may be tangible carrier media (e.g. disks) or intangible carrier media (e.g. communications signals). Embodiments disclosed herein may also be implemented using suitable apparatus which may take the form of programmable computers running computer programs arranged to implement the embodiments disclosed herein.

[0077] Embodiments of the subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on computer storage medium for execution by, or to control the operation of, data processing apparatus. Alternatively, or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. A computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Moreover, while a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially-generated propagated signal. The computer storage medium can also be, or be included in, one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices). The operations described in this specification can be implemented as operations performed by a processor on data stored on one or more computer-readable storage devices or received from other sources.

[0078] The term “processor” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations, of the foregoing. The apparatus can include special purpose reprogrammable logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment can realize various different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures.

[0079] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for performing actions in accordance with instructions and one or more memory devices for storing instructions and data. Devices suitable for storing computer program instructions and data include all forms of computer-readable media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD- ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry and fiber-optic platform for faster data transfer remotely.

[0080] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor including audio, for displaying information (e.g. an indication and / or alert) to the user. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback.

[0081] Although the disclosure has been described in terms of preferred embodiments as set forth above, it should be understood that these embodiments are illustrative only and that the claims are not limited to those embodiments. Those skilled in the art will be able to make modifications and alternatives in view of the disclosure which are contemplated as falling within the scope of the appended claims. Each feature disclosed or illustrated in the present specification may be incorporated in the disclosure, whether alone or in any appropriate combination with any other feature disclosed or illustrated herein.

Claims

CLAIMS:

1. A computer-implemented method for remotely controlling a robot, the method comprising, at a client device: connecting to a robot remote from the client device via a data connection over a network, the robot comprising a plurality of sensors and / or actuators, at least two of the plurality of sensors and / or actuators corresponding to a single robot modality of one or more robot modalities; receiving, by the client device, instructions to control operation of the robot, wherein the instructions comprise a first predetermined datatype corresponding to the single robot modality, the first predetermined datatype encoding data corresponding to the at least two of the plurality of sensors and / or actuators; transmitting, by the client device, the instructions to the robot via the data connection to control operation of the robot in accordance with the instructions.

2. The method of claim 1 , wherein the instructions are encoded, by the client device, into a serialised data format for transmission over the data connection.

3. The method of any claim 1 or 2, wherein the network is a wide area network.

4. The method of claim 1 , 2 or 3, wherein the data connection is a peer-to-peer data connection.

5. The method of any preceding claim, wherein the first predetermined datatype is one of a plurality of predetermined datatypes, the plurality of predetermined datatypes comprising at least one predetermined datatype for encoding data and corresponding to a visual modality of the one or more robot modalities.

6. The method of claim 5, wherein the plurality of predetermined datatypes comprises at least one predetermined datatype for encoding data and corresponding to a motor modality of the one or more robot modalities.

7. The method of any preceding claim, wherein the plurality of sensors and / or actuators comprise at least one camera and at least one motor.

8. The method of claim 7, wherein the robot is a first robot of a plurality of robots and wherein the first robot has a different arrangement of sensors and / or actuators to a second robot of the plurality of robots.

9. The method of any preceding claim, further comprising: establishing the connection in accordance with a set of WebRTC protocols.

10. The method of claim 2 or any claim dependent thereon, wherein the serialised data format is a Protobuf data format.

11. The method of any preceding claim, wherein the robot is located in a different geographical location to the client device.

12. The method of any preceding claim, further comprising: receiving, from the robot over the connection, data from at least two actuators and / or sensors encoded using the first predetermined datatype.

13. The method of any preceding claim, wherein each one of the one or more robot modalities indicates a mode in which the robot can interact with or experience an environment.

14. A computer-implemented method for controlling a robot, comprising: identifying a first plurality of sensors capable of receiving an input from an environment, wherein each sensor of the first plurality of sensors are different ones of the first plurality of sensors each corresponding to a first robot modality; receiving, by the robot, a first input data stream from a first sensor of the plurality of sensors; receiving, by the robot, a second input data stream from a second sensor of the plurality of sensors; mapping the first input data stream and the second input data stream to a robot coordinate system of the robot; generating a robot modality input by combining the first input data stream and the second input data stream; and transmitting data to a client device from the robot, the data comprising the robot modality input.

15. The method of claim 14, further comprising receiving, by the robot from the client device, an instruction configured to actuate the robot, the instruction defined with reference to the robot coordinate system; and causing the robot to be controlled in accordance with the instruction.

16. The method of claim 14 or 15, wherein each one of the plurality of sensors is a camera sensor, a LIDAR sensor, a sonar sensor, a microphone, or a radar sensor.

17. The method of any one of claims 14 to 16, further comprising:receiving a plurality of input data streams from the plurality of sensors, wherein each one of the plurality of input data streams corresponds to a different one of the plurality of sensors; generating a plurality of robot modality inputs, each corresponding to a respective robot modality, comprising: generating, for each of the plurality of robot modalities, a robot modality input by combining two or more of the plurality of input data streams that are associated with the robot modality.

18. The method of any one of claims 14 to 17, wherein the coordinate system is based upon a point of origin defined with reference to the robot.

19. A computer readable storage medium comprising computer readable instructions, the computer readable instructions, when executed by a processor, cause the processor to carry out the method of any preceding claim.

20. An apparatus comprising: a processor; a memory comprising computer readable instructions, that when executed by the processor, cause the processor to carry out the method of any of claims 1 to 18.

Images