Systems and methods for swarm intelligence in industrial robotics

Abstract

Systems and methods for industrial robotic platforms. Squads of industrial robots autonomously communicate and work together. A control center may monitor the autonomous operations. Software at the control center, squad, and robot levels forms a distributed control system that analyzes various data related to the platform for monitoring of the various systems. Artificial intelligence, such as machine learning, is implemented at the control center, squad, and / or robot levels for swarm behavior driven by intelligent decision making. Each robot may include a universal platform attached to a task-specific tooling system. A robot may be elected as a squad leader, cartographer leader, or task planning leader. A squad leader, cartographer leader, or task planning leader may perform specialized planning functions for a squad of robots. Information generated by robots performing a task may be transmitted to the control center for further development of machine learning models.

Description

OFFW.011WO PATENT SYSTEMS AND METHODS FOR SWARM INTELLIGENCE IN INDUSTRIAL ROBOTICS CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Prov. Patent App. No. 63 / 592,119, filed October 20, 2023, which is hereby incorporated by reference herein in its entirety.

[0002] This application additionally incorporates by reference U.S. Patent No. 11,738,461 and U.S. Patent No.12,005,588. BACKGROUND Field

[0003] Features for industrial robotics are described, in particular architectures, approaches and methods for operating swarms of autonomous, task specific robots, such as mining robots. Description of the Related Art

[0004] Robots are used to perform various tasks in environments that may be dangerous to a human, or difficult for a human to reach. However, existing solutions for performing such tasks may require high degrees of human interaction or control, and frequent repair, and as a result are inefficient or expensive. Some existing solutions capable of functioning in one environment may not be suitable for environments where significant communication delays are unavoidable. Improvements in this field are therefore desirable. SUMMARY

[0005] The embodiments disclosed herein each have several aspects, no single one of which is solely responsible for the disclosure’s desirable attributes. Without limiting the scope of this disclosure, its more prominent features will now be briefly discussed. After considering this discussion, and particularly after reading the section entitled “Detailed Description” one will understand how the features of the embodiments described herein provide advantages over existing approaches to industrial robotics.

[0006] Performing mining operations in locations with limited outside access presents several issues. Such locations may include remote locations on Earth (e.g., underwater, in mountainous areas, etc.) or locations in space (e.g., an asteroid, the moon, a planet, etc.). For example, communication with equipment used in the mining information may be limited by a lack of communications infrastructure in the area to be mined. Such limited communications infrastructure may result in a low bandwidth for communications, reducing or eliminating the ability to manage mining equipment at the mining location in real time, such as by not being able to handle the constant real-time transmission of video or telemetry data from the mining equipment to a user or computing system controlling the mining equipment. Additionally, in space-based mining scenarios, the time delay cause by the distance between a controlling user or computing system and the mining equipment may present additional challenges for managing the mining equipment. Another challenge presented by mining in remote locations is a limited ability to transport people or equipment to and from the mining location. This may present challenges in repairing or maintaining the mining equipment.

[0007] The systems and methods described herein for autonomously operating robotic system platforms address some or all of the issues with performing mining operations in remote locations. A colony of robots may operate autonomously as individual units and in varying degrees of collaboration with surrounding robots. As used herein the terms “robot” and “bot” may be used interchangeable. The robots may be organized in squads or groups of robots, which in turn may be organized and grouped in platoons, forming the overall colony hierarchical structure. A control center communicates with the colony of robots to enable human monitoring and operational exception management, however the control center may not nominally or actively tele-operate the colony. Computing systems (e.g., executing software) at the control center, platoon, squad, and robot levels may analyze various data related to the platform and the external environment for monitoring, communication, and control of the various systems. These computing systems may be referred to collectively as “systems” herein. Artificial intelligence, such as machine learning, may be implemented at the control center, platoon, squad, and robot levels for individual robot and swarm behavior driven by intelligent decision making. Examples of machine learning that may be implemented by the systems herein include: route optimization algorithms, graph optimization models, semantic segmentation models, neural networks (e.g., trained using reinforcement learning),and the like. The robots may communicate with each other and with the control center to work autonomously to complete an industrial task. A remote control center geographically non- collocated may communicate with multiple colony control centers.

[0008] Each robot may include of a universal platform integrated with a task- specific tooling system. The universal platform may be a standardized robotics platform that is combinable with additional tools and parts. A robot built using the universal platform may be made into a specialized robot by attaching tools or parts useful for a particular task. The use of a universal platform may improve redundancy when mining in remote locations, as the specialized tools and parts can be moved between universal platforms, and a particular robot is easily converted between different uses by the replacement of the tools and parts attached to the universal platform. The robots may be mining robots, with a mining-specific tooling system attached to the universal platform, also referred to herein as a framework etc., and configured for mining tasks. The platform is modular and may be used for other industrial applications and / or robot types, such as construction, manufacturing, demolition, satellite swarms, fuel production, disaster recovery, communications, remote power, and others, deployed terrestrially on-land and underwater, in free space, on the Moon, Mars and other celestial bodies. When a new species (e.g., a type, model, and so on of a robot) is identified to be added to the suite of species in the swarm robotic architecture catalog of species, payload modules dedicated to performing functions associated with the new species may be developed to be integrated with the robotic universal platform both at the hardware and software level. Since the robotic architecture is modularized at the hardware and software subsystem level, the techniques described herein may accelerate the ability to easily build and integrate new robotic species into existing and new colonies to fulfill new functions and industrial tasks. In the hardware layer, key subsystems may be modularized in the universal platform (e.g., power, thermal management, mobility, data processing, structural support). In the software layer, the system may include a distributed layered architecture where firmware and software modules dedicated to universal and payload functions interface with the data processing layer through an application programming interface (“API”) capable of handling different layers of operating protocols, such as CAN, RS232, ROS, UDP, TCP / IP, etc. Additional details may be found in U.S. Patent No. 11,738,461 and U.S. Patent No. 12,005,588, which is incorporated herein by reference.

[0009] In order to fulfill a certain industrial task, robots may be grouped in squads, so that each squad will include an optimized number of bots of a certain species in order to fulfill the task. Squad robots may be configured to share bot data between each other in order to collaborate autonomously as a group and achieve the successful execution of the industrial task. Squads performing the same or different industrial tasks in a certain region of the deployment site, or to fulfill a specific function across the entire deployment site, may be organized in platoons. Bots belonging to the same platoon may be tagged so that data may be shared among them to autonomously perform collaborative tasks as a group.

[0010] Operations and telemetry transacted in the system may be handled uniformly in some embodiments. Modules to manage processes, oversee data processing, perform housekeeping and logging of data may be part of the software architecture in different layers of complexity, and completely modularized for scalability, flexibility and ease of integration of new payload modules. An example colony of robots may be architected so that the robots may operate in complete autonomy as an individual unit, a squad, a platoon or the colony itself. Each robot may include a hardware and software stack configured to autonomously execute an industrial task, optionally with no required human intervention. Algorithms and controls (e.g., hard coded, computer vision, linear, non-linear, machine learning, etc.) and an entire data processing infrastructure may handle the autonomous operation of the system, for example by broadcasting robot status information and / or sensor data throughout the communications network.

[0011] Data packets that may include relevant information and bot, squad, platoon and colony identifiers may be shared among the colony in a distributed data architecture. For example, the computing systems may create a mesh network between network elements (e.g., robots, communication controllers, etc.). For example, the packet may include a header based in part on the identifier, which may be used to route the packet to the correct bot or other computing system in the distributed data architecture. In this example, the payload of the packet may include the relevant information (e.g., current bot status, communication status, network status, etc.). The algorithms and controls at each robot unit will filter and analyze the relevant data packets shared by the bots in the same squad, platoon and colony. Bots collaborating in the same squad may autonomously transmit and receive data packets dedicated to the squad, and may be shared at a higher frequency and volume based on proximity ofoperations than the data packets dedicated to the platoon or colony, to sustain operations at the squad level. Bots collaborating in the same platoon may autonomously transmit and receive data packets dedicated to the platoon. Bots collaborating in the same colony may autonomously transmit and receive data packets dedicated to the colony. The above-described control center may, as an example, monitor the autonomous operations. For example, the control center may monitor data packets in the network (e.g., the mesh network) at low or high frequency rates depending on the priority level of the information included in the data packet. The control center may, in some embodiments, have the ability to take over control of any bot unit in the colony at any given time to perform manual intervention, exception management, testing or training operations.

[0012] As noted previously herein, performing mining operations in remote locations may make access to such locations difficult, in addition to the challenges in communicating with computing systems in remote mining locations. This may make it difficult to position a human operator within range of a mining robot, or in a location with sufficient communications bandwidth with the mining robot, to allow the human to control or direct the robot.

[0013] To address the problem of controlling robots in such locations, the present disclosure describes the training, deployment, evaluation, and use of machine learning models to control a robot to perform navigation and task-related actions. The machine learning models may be trained to perform specific tasks, or to be generalized to perform various different tasks by generating control signals for a robot on which the machine learning model executes. In some embodiments, multiple machine learning models may be deployed for execution on a robot to perform different functions based on the function for which each machine learning model was trained. For example, a first machine learning model may be trained to manage task planning for a squad of robots and deployed on a squad leader robot, and a second machine learning model may be trained to cause a robot to extract ore from a location using a particular tool. In this example, the robot may execute the task planning machine learning model if it is elected as a squad leader, and may execute the ore extraction machine learning model if the robot is assigned an ore extraction task.

[0014] Training of machine learning models to control a robot are additionally described in the present disclosure. Machine learning model training may be performed invarious arrangements. For example, centralized training with decentralized execution, decentralized training with decentralized execution, or centralized training with centralized execution, may be used as known by those skilled in the art. Additionally, reinforcement learning may be used to train machine learning models. The policy for which machine learning models are trained may be defined to optimize certain functions, such as the collection of ore by an excavator bot, or positioning of an ore heap by a hauler bot. Additionally, some existing systems may be integrated for use in the training of a machine learning model training pipeline. For example, OpenAI®Gym environments may be used to train and compare models trained by reinforcement learning. A reinforcement learning layer of the model training pipeline may include the classes which are specific to reinforcement learning / imitation learning classes. OpenAI®Gym environment class objects communicate with the environment controller class objects through a platform. Advantageously, by updating models based on data received following a bot’s action improved models may be generated that may be applied to unknown environments with improved success.

[0015] Additionally, in a distributed multifunction and multiple species operation performed by a swarm of robots doing industrial work, a layered / level-based architecture may assist in optimizing the effectiveness and productivity of the swarm with heterogeneous functionalities operating towards a common high-level mission, as described below herein. Such an architecture allows the swarm to self-organize at different levels in its topology. The systems and methods described herein provide an innovative approach to implementing different levels or layers of intelligence, planning, communication and control, in environments where access to information and command latencies and throughputs are different depending on how close a user, or computing system, is to the task. The more detached the user or computing system is from a task, the more the job must deal with large- scale, asynchronous, long-horizon planning and optimization. The lower the layer (e.g., the closer the actions being performed are to directly interaction with a physical environment), the more the task is localized in its application and the less it is concerned with what other robots are doing elsewhere in the swarm. This approach is based in part on managing functions provided by different levels or layers of the system, managing bandwidth in communications by efficiently communicating necessary information between users or computing systems,adjusting the topology of the robotic swarm, and updating systems in response to the dynamic nature of an operational environment where tasks are being performed.

[0016] Software systems may be divided into logical layers as part of a software system architecture. Such logical layers may aid in an understanding of the functions each element of the present system is expected or able to perform. For example, the logical layers may include an optimization layer, a planning layer, a coordination layer, a communications layer, an autonomous navigation layer, and a bot autonomous operations layer. Elements of the system may function within one layer, or a plurality of layers. The optimization layer may include software responsible for improving one or more machine learning models (e.g., neural networks) over time. The planning layer may include software responsible for offline planning required by one or more industrial robots to execute a mission or solve other planning problems. Offline planning may include generating plans, determining or analyzing goals, etc. The communication layer may include software responsible for enabling communication between individual industrial robots. The autonomous navigation layer may include software responsible for allowing one or more industrial robots, individually or as part of a squad, to navigate an environment. The bot autonomous operations layer may include software to enable industrial robots to perform robot-specific and / or domain-specific operations autonomously.

[0017] In one aspect, a system for the autonomous operation of industrial bots is described. The system includes one or more colonies. Each colony includes one or more squads. The squads may be grouped or not grouped in platoons. Each squad includes a plurality of bots. Each bot is configured to operate autonomously and includes a universal platform coupled with a payload stack, the payload stack being one of a plurality of payload stacks with which the universal platform may be coupled, and where the bot is configured to perform a payload-specific industrial task using the payload stack. The system may further include a colony control center configured to remotely communicate with the one or more squads.

[0018] Various embodiments of the various aspects may be implemented. In some embodiments, the plurality of bots may be configured to communicate with each other and the colony control center via a colony communications network. Two or more of the squads may communicate with each other via a colony communications network. The universal platform further may include a mobility system configured to move each bot. According to another embodiment, the mobility system may include a tracked system, a wheeled system, or a leggedsystem. The universal platform may include a control system configured to be operated by a robotic control algorithm. The robotic control algorithm may include an artificial intelligence or machine learning package. The universal platform may include a data processing system where each data packet includes a data packet header including identification information related to each bot. The identification information may include one or more of the following: a colony identifier, a platoon identifier, a squad identifier, a bot identifier, a bot location identifier, a bot position identifier, health data, performance data, operational data, housekeeping data and / or sensor data. The universal platform may include a hardware platform stack and a software platform stack, and wherein the universal platform is configured to use the hardware platform stack and the software platform stack to autonomously operate the payload stack to perform the payload-specific industrial task and to communicate with other bots and / or the colony control center. A communication system may receive operational data from the one or more colonies and transmit update data to the one or more colonies via a colony communications network. The colony control center may monitor and support the plurality of bots, initialize systems, perform exception management, analyze the operational data and to generate the update data based on analysis of the operational data. A user interface may enable a user to monitor and control the one or more colonies.

[0019] According to another aspect of the disclosed technology, the colony control center may support operations required to train, update, and evaluate machine learning models used to enable the autonomous performance of various tasks by one or more bots of a colony. Computing systems of the colony control center may be optimized for the training of machine learning models. Alternatively, general purpose computing systems may perform the operation of training and evaluating machine learning models. The machine learning models evaluated at the colony control center may be distributed to one or more bots, for example by transmission to a communication coordinator, squad leader, or directly to a bot. Lossless compression of a machine learning model may be used to minimize the transmission bandwidth required to transmit the machine learning model.

[0020] Some embodiments described herein relate to a system. The system includes a colony control center comprising non-transitory computer-readable memory storing computer-executable instructions; and one or more processors in communication with the computer-readable memory and configured to execute the computer-executable instructionsto: receive a request to initiate a mission, the mission comprising a task, wherein the task is associated with mining in a real-world environment, and wherein the mining is performed via a plurality of bots organized into a squad within the real-world environment, wherein the bots include at least one of: an excavator bot, a hauler bot, or a battery bot; determine, based on pre- mission check information, that the plurality of bots is ready to execute the task; transmit an instruction to the plurality of bots comprising task information; receive bot information associated with execution of the task, wherein the bot information is used to adjust one or more machine learning models associated with the colony center and / or the bots; the plurality of bots organized into the squad, wherein individual bots include individual processors and at least one tool associated with mining, wherein the plurality of bots communicate with each other, wherein the plurality of bots are configured to elect a cartographer leader and a squad leader in response to receipt of the task from the colony center, wherein the cartographer leader is configured to: maintain a shared map in real time, wherein to maintain the shared map the cartographer leader transmits map updates to other bots in the squad, wherein the squad leader is configured to: determine a plurality of actions to be executed by at least one bot of the plurality of bots based at least in part on the shared map, and wherein based on completion of the actions, one or more of the bots of the plurality of bots are configured to transmit bot information to the colony control center to trigger adjustments to the one or more machine learning models.

[0021] Some embodiments described herein relate to a method. The method orchestrates actions of a plurality of bots organized in a squad by a colony control center, the colony control center including one or more computers, and the method includes: receiving a request to initiate a mission, the mission comprising a task, wherein the task is associated with mining in a real-world environment, and wherein the mining is performed via the plurality of bots organized into a squad within the real-world environment, wherein the bots include at least one of: an excavator bot, a hauler bot, or a battery bot; determining, based on pre-mission check information, the plurality of bots is ready to execute the task; transmitting an instruction to the plurality of bots comprising task information, wherein the task information causes the plurality of bots to perform the task based in part on a map shared between the squad of bots and a plurality of actions generate by a squad leader of the squad; receiving bot information from the plurality of bots; and updating a machine learning model based on the bot information.

[0022] Some embodiments described herein relate to a method. The method performed by a plurality of bots organized in a squad comprises: transmitting pre-mission check information indicating at least a portion of the plurality of bots is ready to execute a task; in response to transmitting the pre-mission check information, receiving an instruction comprising task information; selecting, from the plurality of bots, a cartographer leader to generate map information for an area where a task associated with the task information will be performed; scanning, by the cartographer leader, the area to generate a shared map, wherein the shared map is shared between bots in the squad; maintaining, by the cartographer leader, the shared map in real time based on scan information received from the plurality of bots; selecting, from the plurality of bots, a squad leader to determine a plurality of actions to be executed by at least one bot of the plurality of bots based at least in part on the shared map and the instruction; and executing, by at least one bot of the plurality of bots, the plurality of actions; in response to executing the plurality of actions; transmitting bot information to a colony control center to cause the colony control center to train a machine learning model based on the bot information. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The foregoing and other features of the present development will become more fully apparent from the following description and appended claims, taken in conjunction with the accompanying drawings. Understanding that these drawings depict only several embodiments in accordance with the development and are not to be considered limiting of its scope, the development will be described with additional specificity and detail through use of the accompanying drawings. In the following detailed description, reference is made to the accompanying drawings, which form a part hereof. In the drawings, similar symbols typically identify similar components, unless context dictates otherwise. The illustrative embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented here. It will be readily understood that the aspects of the present development, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, and designed in a wide variety of different configurations, all of which are explicitly contemplated and make part of this disclosure.

[0024] FIG. 1 is a block diagram of illustrative data flows in an embodiment of a colony control network structure, including a colony control network in communication with a plurality of communication coordinators, each communication coordinator in communication with a plurality of robotic squads, and with each squad having a plurality of industrial robots configured for performing various tasks.

[0025] FIG.2 is a block diagram of illustrative data flows between an embodiment of a colony control system and a communication coordinator.

[0026] FIG.3 is a block diagram of an embodiment of a robotic squad including a plurality of industrial robots.

[0027] FIG.4 is a flow diagram of an example routine for performing a task using a robotic squad in communication with a colony control center.

[0028] FIG. 5 is a flow diagram an example routine for a colony control network providing mission instructions to a robotic squad.

[0029] FIG.6A is a flow diagram of an example routine for the election of a squad leader by a robotic squad.

[0030] FIG. 6B is a diagram of an example deployment of a robotic squad in communication with a communication coordinator.

[0031] FIG.7 is a flow diagram of an example routine for generating planning tasks from a mission plan received from a colony control network by a robotic squad.

[0032] FIG. 8 is a flow diagram of an example routine for an industrial robot to perform an assigned task.

[0033] FIG. 9 is a flow diagram of an example routine for training a machine learning model to be deployed to an industrial robot.

[0034] FIG. 10 is a block diagram of illustrative data flows within and between robots for performing collaborative simultaneous localization and mapping.

[0035] FIG. 11 is an illustrative example of assigning network identifiers to a robotics squad.

[0036] FIG.12 is an illustrative example of a plurality of robotic squads deployed in an industrial environment.DETAILED DESCRIPTION

[0037] The following detailed description is directed to certain specific examples of the development. Reference in this specification to “one example,” “an example,” or “In some implementations” means that a particular feature, structure, or characteristic described in connection with the example is included in at least one example of the disclosed technology. The appearances of the phrases “one example,” “an example,” or “In some implementations” in various places in the specification are not necessarily all referring to the same example, nor are separate or alternative examples necessarily mutually exclusive of other examples. Moreover, various features are described which may be exhibited by some examples and not by others. Similarly, various requirements are described which may be requirements for some examples but may not be requirements for other examples.

[0038] Various examples will now be described with reference to the accompanying figures, wherein like numerals refer to like elements throughout. The terminology used in the description presented herein is not intended to be interpreted in any limited or restrictive manner, simply because it is being utilized in conjunction with a detailed description of certain specific examples of the development. Furthermore, examples of the development may include several novel features, no single one of which is solely responsible for its desirable attributes or which is essential to practicing the disclosed technology described herein. A. Colony Control Architecture

[0039] FIG.1 illustrates an example of an industrial robot colony 100. The colony 100 includes a colony control center 122, a cartographer leader 114, a plurality of communication coordinators 112A-112M, and a plurality of robotic squads 108A-108N each including a plurality of industrial robots (e.g., 102A-102N). Additionally, the colony 100 may be divided into logical layers based on the level of communication between components of the system 100. For example, below the lowest logical divider 110 may be each robotic squad 108A-108N, where the industrial robots within each robotic squad communicate directly with each other. The robotic squads 108A-108N, in some implementations, may communicate with each other via the communication coordinators 112A-112M above the first logical division 110. The communication coordinators 112A-112M may be located below the second logicaldivider 120, and communicate with each other via the colony control center 122 located above the second logical division 120.

[0040] The colony 100 may be a distributed, autonomous system that is heavily intelligent at one or more levels, with the industrial robots (or “bots”) operating in a distributed architecture as a swarm, individually, or in collaboration, organized in subgroups as pertaining to specific tasks, and where the control center may act as a mission manager, e.g. monitoring and supporting the independent operation of the swarm, for example by training and providing updated machine learning models for the tasks an industrial robot is to perform. The control center may thus be an observational entity, not directors of actions. In some implementations, only in rare cases where operations deviate from the norm due to exceptions or anomalies (not status quo operations) does the control center’s role change into supervisory control for brief periods. As may be appreciated, in some configurations the control center may be located at such a distance that sustained direct control of an individual industrial robot, swarm, platoon, etc. is challenging due to communication lag. Also, there may be rare instances where bots (or any permutation of a combination of bots) require confirmation from a control center for the execution of a task or a series of tasks. Thus, the systems described herein may include the autonomous, intelligent actions of the bots, and correspondingly, the autonomous, intelligent actions and collaboration between and amongst various permutations of bot combinations.

[0041] Each robotic squad (e.g., robotic squad 108A) or “squad” includes a respective plurality of industrial robots (e.g., industrial robots 102A-102N) configured for performing various industrial-specific tasks. Examples of industrial tasks can include mining- related tasks such as excavation, surveying, or providing assistance to other industrial robots. The bots, in some embodiments, may each be built on a universal platform with common structural, mechanical, electrical and computing systems, coupled with an interchangeable payload component. Each payload component is integrated with the universal platform and configured for a specific industrial task to be performed by the bot. The bots include one or more processors in communication with one or more memories storing instructions thereon that when executed by the one or more processors perform the industrial task. The task may be performed autonomously by the bot and / or in collaboration with other autonomous bots to achieve an overall collaborative industrial objective. The bots may communicate via a temporary network for the squad to which the bot is assigned. For example, bots maycommunicate via a mesh network between bots, or via a ring network. The system 100 allows for management and control of the bots.

[0042] Any number of colonies, platoons, squads, and bots may be implemented, depending on the industrial objective. In some embodiments, a colony control center 122 may manage a plurality of communication coordinators 112A-112M facilitating communication between the colony control center 122 and any squad, platoon, and / or any bot of the colony. Further, a communication coordinator (e.g., communication coordinator 112A) may be in communication with any number of squads and any number of colony control centers. As shown in the present example, communication coordinator 112A is in communication with squad 108A and squad 108B, while communication coordinator 112M is in communication with squad 108B and 108N. In some embodiments, a first communication coordinator (e.g., communication coordinator 112A) and a second communication coordinator (e.g., communication coordinator 112M) may both be in communication with one or more of the same squads, enabling redundancy for communications between a squad and the colony control center 122. In some embodiments, a communication coordinator 112A may be in communication with a platoon comprising a plurality of squads, and may communicate with a platoon leader. In further embodiments, a squad leader may be elected by one or more of the squads in communication with a communication coordinator. In such embodiments, the squad leader may be the only bot in its squad to communicate with the communication coordinator, and may act as a relay between individual bots in the squad and the communication coordinator. Further, the squad leader may take in a multi-layer 2-dimensional semantic world representation (e.g., based on information received from a cartographer leader, or another bot or computing system storing mapping information) and generate a list of action commands for completion of a task. For example, the squad leader may receive information via a mesh network connecting the bots in the squad leader’s squad from a first bot indicating the first bot has a low battery level. In this example, the squad leader may then generate a task plan including a series of actions for a battery bot to approach the first bot and replace the battery of the first bot. The list of commands may be transmitted to one or more bots, where the action commands are queued for execution.

[0043] Each communication coordinator 112A-112M may be located in a particular geographic site. Each communication coordinator 112A-112M may be in a differentlocation from one or more of the other communication coordinators. The communication coordinators may be co-located with a respective platoon or squad, or in a different location. The colony control center 122 may be located in a different geographic site from one or more of the communication coordinators 112A-112M. The colony control center 122 may be co- located with one or more of the squads 108A-108N.

[0044] As discussed herein, any number of bots may form a squad based on the objective the squad is designed to achieve. Additionally, each squad may include a different plurality of bots, wherein the type of each bot is selected based on a mission objective. For example, squad 108A may be tasked with a surveying mission requiring bots 102A, 102B, and 102N. Squad 108B may then be tasked with a follow up mission based on the results of the surveying mission performed by squad 108A, the follow up mission requiring bots 104A, and 104B through 104N. Some or all of bots 104A to 104N may be the same bots as in squad 108A. A mining mission may then be determined to be appropriate, for example by the colony control center 122, and squads 108C (not shown)-108N comprising bots 106A, and 106B through 106N may be deployed.

[0045] In some embodiments, each of the above-mentioned squads may be deployed simultaneously in various combinations, such that different mission objectives are being achieved. For example, a surveying squad may survey new locations while mining squads mine a previously-surveyed location. The total number of squads deployed at a specific geographic site may be defined as a site colony. The total number of squads of each type deployed in the colony may be optimized based on performance and economic metrics. The minimum set of bots required to perform an end-to-end industrial objective may be referred to as the minimum viable platoon (MVP).

[0046] The colony control center 122 may provide command and control, planning, guidance, and overall supervisory functions for the colony 100. In some implementations, the colony control center 122 may include a remote communication system, a command and control system, and / or a user interface. The colony control center 122 may include one or more computers, servers, switches, databases, etc. configured to monitor, control, process, store and update the colony data. The user interface (e.g. displays, HMI, AR, VR, etc.) may be configured to enable a user to remotely monitor and control the colony and / or colony control center 122. Additionally, the user interface may be configured to allow a user to provide orupdate parameters related to a mission, for example bot types, squad size, mission area, etc. Further, the user interface may allow the user to monitor the progress of a mission during mission performance by the bots, and allow the user to pause the mission if the system identifies an error. Additionally, the user interface may be used during a mission analysis phase to allow the user to perform an analysis of mission results. Mission results may be analyzed, for example, by referring to analyzing the maps at different time intervals, analyzing a Gantt chart generated prior to the mission, or analyzing the successful and failed tasks associated with the mission.

[0047] The colony control center 122 may further be used to store, update, and provide machine learning models for other elements of the colony 100, for example as described in relation to FIG. 9 below herein. The colony control center 122 may use classical techniques (e.g., rules-based approaches) or machine learning models (e.g., a neural network, a large language model, a clustering algorithm, etc.) for planning purposes (e.g., capacity planning, mission task planning based on mission instructions received from a user, etc.). The classical techniques or machine learning model may, for example, associate squads of bots with specific real-world environments, analyze a mission area as represented in map information (e.g., a point cloud, 2-dimensional grid, semantically labelled image information, etc.), and the like.

[0048] The colony control center 122 may include a colony communications network / communications system (e.g., communication coordinator 112A). The colony communications network / communications system may receive data of various types from any number of bots, squads, platoons, and / or colonies. The colony control center 122 may be supervisory and analyze the incoming data (among other tasks) and generate update data based thereon to achieve a general objective. The update data may include revisions to existing commands, priorities, behaviors, missions, plans, tasks, operational thresholds, virtual fences, environmental data (e.g., rainfall, temperature, etc.) and / or general high-level operating instructions, as will be discussed in further detail in relation to FIG.2 below.

[0049] As an example of update data, a stop order, such as “cease all operations until go order is given,” may be sent to the bots and / or other nodes of the system, due to an anomaly, like a mine cave-in or mine shutdown or emergency on construction site. As anotherexample, the bot mission area maybe updated from one designated mine quadrant or panel to another, such a with the command “find another suitable area instead of the current one.”

[0050] As another example, the bot mission area may be updated from mining gold to silver, such as “go find silver instead of gold.” As another example, the system may be instructed to speed up or slow down task / behavior rate due to an interface with a human process step, such as “slow down excavation by 10% so the human process step of supervisory inspections can keep up.” As another example, tele-operational data may be instructed that allows a user to guide / drive a bot around a construction site, for example, using an interface for a tablet, mobile phone, laptop, etc.

[0051] These and other kinds of update data may be communicated to any number of bots, squads, platoons, and / or colonies via the colony communications network / communications system. Communication of the update data may be continuous (e.g., not serial). Communication of the update data may occur in real-time, with an infinite number of parallel streams being communicated, multi-directionally.

[0052] The colony control center 122 may be distributed across any number of bots, squads, platoons, colonies and / or other nodes. Some embodiments may use “edge” computing where operations are completed at each node in a system. Edge computing may include a distributed computing paradigm that brings computation and data storage closer to the location where it is needed, for example to improve response times and save bandwidth. The colony control center 122 may not be a direct “command and control” type of system, but instead a “supervisory guidance” system, where general directions are provided and the system determines the best way to complete those general directions. The “industrial objective(s)” may be an example of an overall supervisory guidance provided. More specific guidance may be provided to help achieve any specific industrial objectives and may include revisions to existing tasks, behaviors, missions, plans. Examples of industrial objectives or tasks thereof may include supervision to locate and chisel a wall with soft rock hardness, to move away from rock above a threshold rock hardness, to find softer rock and / or a particular type of rock, to determine a size and shape for a virtual geo-fence within which to operate, other suitable supervisions, or combinations thereof.

[0053] In some embodiments, the cartographer leader 114 may be in communication with one or more squads and / or one or more communication coordinators. Thecartographer leader 114 may be configured to receive surveying, location, or other data used to generate maps. The cartographer leader 114 may further be configured to generate, based on received data, one or more maps of an area, for example an area in which it is desirable to deploy a squad for a mining operation. The cartographer leader 114 may store one or more machine learning models configured to generate such maps. Further, the cartographer leader 114 may allow for increased collaboration between squads and platoons by allowing for the combination of localization and mapping information generated by each bot, squad, and / or platoon into a unified map of an area. The unified map may allow for increased map quality compared to maps generated by individual squads or platoons. Advantageously, having the cartographer leader 114 located at a point closer to the deployment area of the bots allows for reduced communication latency as compared to transmitting and receiving map data from the colony control center 122. In some embodiments, a communication coordinator may include a cartographer leader 114. In some embodiments, the cartographer leader 114 may be collocated with a communication coordinator. The cartographer leader 114 may be in communication with every squad and / or bot, alternatively the cartographer leader 114 may be in communication with a subset of squads (e.g., the subset of squads in communication with a communication coordinator associated with the cartographer leader 114).

[0054] In further embodiments, the cartographer leader 114 may be a bot in a squad of bots that is selected (e.g., by election of the squad) to perform scanning and / or mapping tasks for the squad by generating scan information with a sensor (e.g., a LIDAR sensor, image sensor, depth sensor, computer vision information, and so on.). The cartographer leader 114 may update or generate a shared map for the squad, for example based on sensor data collected by the cartographer leader 114 or received by the cartographer leader 114 (e.g., via the mesh network enabling communication between the bots). Updates to the shared map may be made in real-time based on additional sensor data and / or information received from another bot in the squad. This may be performed as part of the collaborative simultaneous localization and mapping (“CSLAM”) described in further detail with respect to FIG. 10 below herein. The cartographer leader 114 may additionally segment the map into a 2D grid, which may assist in efficient route planning for bots in the squad. Further, the cartographer leader 114 may semantically segment map information, for example to identify obstacles, other bots, ore deposits, piles, objects, and the like, to provide additional context to the shared mapinformation. The semantic segmentation may be performed by a machine learning model trained to semantically segment point cloud, image, or other information used to generate, or included in, the map information. The 2D grid, or any representation of map information, may include segmentation information such as layers or labels for a portion for the map that identify obstacles, other bots, ore deposits, piles, objects, and the like.

[0055] The divide and conquer approach to operations described herein may include a system where a single bot of the colony 100 does not have to do everything nor be only one size. The specialization allows for each bot to do one or a select number of tasks proficiently. It may thus be easy to train on one function where there is a flexibility in collaboration of functions. The system may allow for rapid reconfiguration of the weighting of each function to respond quickly to changes or progress in the execution of a particular industrial objective. These may manifest in various ways, e.g., not only changing the functions of each bot as needed but adapting and improving and especially right sizing the workforce to the size of the job permanently, so that there is never an overcapacity.

[0056] FIG.2 is a block diagram of illustrative data flows between an embodiment of a colony control center 122 and a communication coordinator 112. The colony control center 122 of this example includes a planning engine 202, a communications engine 204, a mine optimization engine 208, a model optimization engine 206, a system goal store 212, and a swarm data store 214.

[0057] The communication coordinator 112 of this example receives information from one or more squads and / or bots (not shown). The communication coordinator 112 may then transmit the received information as bot activity data 215. As will be described, the bot activity data 215 may combine information from a multitude of sources. Further, the bot activity data 215 may include compressed information such that transmission to the colony control center 122 may be accomplished in limited bandwidth environments. In some embodiments, the communication coordinator 112 may perform the compression. The type of compression used may depend on an available transmission bandwidth, an available received data throughput of the colony control center 122, or on other factors affecting the amount of data which may be transmitted or received.

[0058] In some embodiments, the bot activity data 215 may be transmitted in packets, and the communication coordinator 112 may structure the bot activity data 215 byadding, for example, a header and other identifiers to the bot activity data 215 such that the original data may be reconstructed by the colony control center 122. Further, the communication coordinator 112 may be configured to provide redundancy, for example by transmitting bot activity data 215 in segments and awaiting a confirmation or acknowledgement of the data received by the colony control center 122 before transmitting further segments. Additionally, the communication coordinator 112 may be configured to retransmit bot activity data 215, for example when the bot activity data 215 is determined to not have been received by the colony control center 122, or when the bot activity data 215 is determined to have been corrupted when received by the colony control center 122.

[0059] Bot activity data 215 may include information from data sources such as position information, mapping information, task information (e.g., an indication of progress towards completing an assigned task), and / or model data associated with a machine learning model implemented by a bot (e.g., a machine learning model configured to calculate the motion, path, and so on, which may be required to complete a task or reach a location). In some implementations, the communication coordinator 112 may act only as a relay, and all packetization and compression of bot activity data 215 may occur at a bot, for example a squad leader. In such implementations, a squad leader may combine information from some or all of the bots in an associated squad, and transmit the data to the communication coordinator 112, which then transmits the information to the colony control center 122.

[0060] The bot activity data 215, in this example, is received by a communications engine 204. The communications engine 204 is a component of the colony control center 122 configured to manage communications between the colony control center 122 and the communication coordinator 112. For example, the communications engine 204 may receive packetized data from the communication coordinator 112 and confirm that each expected packet is received in an uncorrupted state (e.g., by comparison of the received data to a checksum). When the communications engine 204 determines that less than all of the packets have been received, or that one or more packets are corrupted, the communications engine 204 may request the communication coordinator 112 resend the affected data. The communications engine 204 may additionally be configured to request only unreceived or corrupted packets, thereby minimizing the bandwidth required to address faults in communications. The communications engine 204 shown here is further configured to determine the correct sub-module of the 122 to which received data should be provided. For example, when bot status information associated with a bot is received from the communication coordinator 112, the communications engine 204 may transmit the bot status information to the swarm data store 214. In another example, when the bot activity data 215 includes information associated with a bot’s completion of a task, such information may be provided to the system goal store 212 for storage and later processing.

[0061] The model optimization engine 206 shown here is a component of the colony control center 122 configured to optimize machine learning models used by the colony 100. Machine learning models are used by various components of the colony 100, for example the colony control center 122 and the bots, to allow for task planning, scheduling, multi-agent path finding, generating a control signal to enable performance of tasks or actions, and multi- robot task allocation. The machine learning models may be trained using reinforcement learning. Different targets may be used to train the machine learning models. For example, the machine learning models may be trained to optimize at least one of: power efficiency for a task, an amount of ore collected, time to complete a task, and the like.

[0062] The model optimization engine 206 may update models by training such models using data received from one or more bots when completing assigned tasks. For example, the model optimization engine 206 may determine that a current model has not provided an optimal solution based on received bot activity data 215, or may receive an indication a model requires retraining from another source (e.g., a supervising user of the system). As an example, the model optimization engine 206 may determine that a current version of a machine learning model is performing with less accuracy or efficiency than a previously-stored version of a machine learning model configured for the same task. In such an example, the model optimization engine 206 may retrain the machine learning model on previous data, such that the trained machine learning model functions similarly to the previous version. Alternatively, the model optimization engine 206 may store previous versions of the machine learning model in the swarm data store 214 and may replace the machine learning model with the previous version. In another example, the model optimization engine 206 may receive updated bot activity data 215 generated, at least in part, based on the output of a machine learning model operating on a bot.

[0063] The model optimization engine 206 may then train the machine learning model using the updated bot activity data 215 to generate an updated machine learning model. The model optimization engine 206 then transmits the updated machine learning model to a bot via the communication coordinator 112 replacing the machine learning model with the updated machine learning model, thereby improving the receiving bot’s ability to complete a task.

[0064] Training of machine learning models by the model optimization engine 206 may be performed offline (e.g., while a bot using a machine learning model is powered down to recharge). Alternatively, some machine learning models may be trained by the model optimization engine 206 online (e.g., a mine optimization machine learning model used by the mine optimization engine 208), such that training occurs while the model is in use based on current data. In some embodiments, the model optimization engine 206 may generate a virtual environment in which various machine learning models are trained in a simulated environment, allowing for evaluation of the performance of machine learning models before they are deployed for real-world use.

[0065] To generate the simulated environment, the model optimization engine 206 may start by processing the shared map and / or bot sensor information into a 2D grid representation, which includes high level semantic world state information represented in distinct layers such as: obstacle location, robot locations, goals, other object locations required for planning. This 2D grid representation data is saved in a database and is accessible for reinforcement learning coordination, planning, navigation policies. The outputs of the policies are robot actions that are forwarded to the model optimization engine 206 for further use and model analysis. Thus, the simulation may be based on a real-world environment and output associated with accomplishing a mission. For example, the output may reflect metrics as described below with respect to Figure 4. The simulation may also obtain movements of the bots, such that it can replay the movements or actions of the bots in a real-world mission.

[0066] The planning engine 202 is a component of the colony control center 122 configured to determine a plan. A plan may be a series of tasks required to accomplish a goal of the colony 100 stored in the system goal store 212. A goal may be received by the colony control center 122, for example from a supervising user, and indicate an objective the colony 100 is intended to accomplish (e.g., mining a specific area to find water ice). Alternatively, agoal may be determined by the colony control center 122 based on a received objective. For example, where the objective is to mine a specific area for water ice, the planning engine 202 may determine a goal is to survey the specific area. The planning engine 202 may then determine a plan to complete the goal, for example the plan may include forming a squad with a surveyor robot, a battery robot, and a charging robot, and sending the squad to the specific area. In some embodiments, the planning engine 202 may use one or more machine learning models to generate plans, for example machine learning models trained by the model optimization engine 206.

[0067] The mine optimization engine 208 is a component of the colony control center 122 which optimizes the mining of an area by performing offline planning of tasks for the excavation of a mine. Offline task planning may be performed for platoons, squads, and bots of the colony 100. The mine optimization engine 208 may use one or more machine learning models to, given a mine geometry, decide on an optimal layout for mining the ore of the mine, for example determining an optimal number of layers for a mine, and the optimal number of stopes per layer. Mine optimization engine 208 may further provide determined information to the planning engine 202 and the planning engine 202 may then develop task plans based on the information generated by the mine optimization engine 208. The communications engine 204 may then communicate the plan data 217 comprising assignment of a mining area and the optimal mining configuration to the appropriate platoon, squad, or bot. A squad leader may then use the plan data 217 to generate tasks for each member of the squad. Swarm Robotics Coordination – Example Hardware and Communication

[0068] FIG. 3 is a block diagram of an example squad 300 of industrial robots coordinating the completion of a task or series of tasks. The squad 300 in this example includes three bots, bot A 330, bot B 340, and bot N 350. While three bots are shown here to simplify discussion, it should be recognized that any number of bots may in a squad. The bots of this example include a navigation module 302, a squad coordination module 304, a communication module 306, a task planning engine 308, and a multi-agent path planner 310. The bots share information allowing for the coordinated completion of assigned tasks or plans as coordination data 320. In some examples, as shown here, the bots may communicate directly with eachother. Alternatively, some or all of the communication of coordination data 320 may be facilitated through a squad leader as discussed in further detail with respect to FIG. 6A. Alternatively, some or all of the communication of coordination data 320 may be facilitated by a communication coordinator (e.g., communication coordinator 112 not shown here).

[0069] The navigation module 302 is a module of the bot configured to generate control instructions for motors of the bot, allowing the bot to navigate its environment and reach an intended location. For example, the navigation module 302 may include systems implementing obstacle avoidance techniques such as receiving image input data from a camera, LIDAR, or other sensor of the bot, or from another bot as coordination data 320, and identifying obstacles based on the received input data. The navigation module 302 may then calculate a path avoiding known obstacles, and adjust the control instructions sent to a motor of the bot to enable the bot to follow the path that is not obstructed by known obstacles (e.g., as described below in relation to FIG. 10). Additionally, the navigation module 302 may perform path planning based on mapping data provided to the bot by other bots, other squads, other platoons, or the colony control center 122.

[0070] The navigation module 302, in some embodiments, may receive data from the bot with which it is associated indicating the functioning of mechanical or electrical components of the bot (e.g., a performance characteristic of a motor), and adjust instructions sent to the components of the bot based on such received data. For example, the navigation module 302 may receive feedback from a battery of the bot indicating insufficient power to operate an electric motor at full torque.

[0071] The navigation module 302 may then adjust instructions sent to the motor to accommodate the reduced torque while allowing the bot to continue following a planned path, such as by running the motor for a longer time than would be needed if the motor could be run at its full torque. Some or all functions of the navigation module 302 may be performed by one or more machine learning models trained to perform such functions, for example by the model optimization engine 206.

[0072] The squad coordination module 304 is a module of the bot configured to manage coordination of the bot with the other bots of the same squad. Additionally, when a bot is elected as squad leader (e.g., based on votes received from bots in the squad, based on round-robin), the squad coordination module 304 may perform various squad leader tasks. Forexample, the squad coordination module 304 may work with the task planning engine 308 to develop tasks allowing for coordinated actions between multiple bots, such as coordinating excavation of an excavation bot with a hauler bot to ensure that materials excavated by the excavation bot are moved as needed to follow the overall plan. The squad coordination module 304 may generate coordination data 320, which may include timing information.

[0073] The communication module 306 is a module of the bot configured to manage communication between each bot, and between the bots and the communication coordinator 112. The communication module 306 may operate various antennae of the bot, and manage communication protocols associated with inter-bot communication, including the generation of outbound coordination data 320, the receiving and processing of incoming coordination data 320. Additionally, the communication module 306 manages software communication functions. For example, when a bot uses a plurality of communication protocols, for example a first protocol associated with inter-bot communication and a second protocol associated with communication between a bot and the communication coordinator 112, the communication module 306 formats the data according to each protocol such that data is interpretable by a receiver. Further, the communication module 306 manages incoming information received from other elements of the colony 100, for example by parsing incoming data packets and transmitting the information of the packets to the appropriate module or subsystem of the bot.

[0074] The task planning engine 308 is a module of the bot configured to perform online task planning for an individual bot. Further, when a bot is the squad leader, the task planning engine 308 may perform task planning functions associated with being the squad leader. Such tasks are discussed in further detail below in reference to FIG.6A.

[0075] The multi-agent path planner 310 is a module of the bot configured to perform multi-agent path planning functions. For example, coordinated simultaneous localization and mapping (CSLAM), collision avoidance using sensors of the bot, allowing one or more bots to follow a guide bot (e.g., when a navigation issue is detected), and the like. Planning may additionally be performed using machine learning models, such as neural networks which reduce a search space associated with actions a route. Further, a machine learning model may be used to assist in navigation. For example, computer vision models (e.g., convolutional neural networks, vision-transformer models, etc.) may be used to analyze imageinformation gathered by one or more bots of the squad, segmentation models may be used to semantically segment map or image information to identify objects (e.g., points of interest, obstacles, other bots, etc.) or apply labels (e.g., objects, navigable routes, ore, other bots, and so on), or models to coordinate movement between bots (e.g., using V2X techniques). Additionally, machine learning models may be used to control or direct movement, for example to generate a control signal for a motor of a bot to cause the bot to navigate towards a particular destination (e.g., based on map information, task information, etc.).

[0076] The multi-agent path planner 310 may, with the communication module 306, communicate with other bots of a squad to share pathfinding information, for example an intended path for the bot to reach a destination point, so that the squad may ensure each bot avoids collision with each other bot. Further, by sharing pathfinding information between the bots, the multi-agent path planner 310 may allow the squad to receive additional details or information about the local area not identified during scanning and mapping of the area. For example, the operation of a first bot may alter the terrain on which the bots are operating (e.g., by creating tracks in lunar regolith). A second bot may then find the tracks and determine that a portion of the track now creates a navigation obstacle. The multi-agent path planner 310 of the second bot may then inform other bots of the squad for future pathfinding. Swarm Robotics System Coordination

[0077] FIG. 4 illustrates an example routine 400 for performing a task using a robotic squad in communication with a colony control center 122. The routine 400 begins at block 402 in response to the colony control center 122 receiving a request to being a mission including one or more tasks to be performed by a squad of bots (e.g., squad 108A). The request may, for example, be received by a user providing direct input to a computing system performing the functions of the colony control center 122, or from a second computing system in communication with the colony control center 122 via a network (e.g., a network coordinated by a communication coordinator).

[0078] At block 404, the colony control center 122 performs pre-mission checks. The pre-mission checks may include a pre-mission check of a communication coordinator 112, a bot 102A, a squad 108A, or any other element of the colony 100 associated with a task of the mission. Pre-mission checks may include the colony control center 122 transmitting a requestfor a current status to an element of the colony 100, or receiving a current status from an element of the colony 100. For example, the colony control center 122 may request a current status from the squad 108A that will be used to perform the mission, and in response the squad 108A may transmit current status information for each bot in the squad 108A. The status information may include information including results of functioning tests (e.g., a function test on a motor of a bot), a current software version, current machine learning model stored in memory, current battery capacity, or any other information useful for determining whether the element of the colony 100 is capable of, or prepared to, perform a task to complete the mission.

[0079] At block 406, the colony control center 122 transmits information to the bots (e.g., the squad 108A) that will be used to perform the tasks of the mission.

[0080] At block 408, the squad (e.g., squad 108A) elects leaders to manage certain functions of the squad. For example, a squad leader may be elected to coordinate the performance and completion of various tasks useful to the squad. In another example, a cartographer leader may be elected to perform various scanning and mapping tasks in order to provide map information useful for planning navigation of various bots of the squad to locations where tasks will be performed.

[0081] At block 410, the squad leader or cartographer leader plans a scanning task. The scanning task may cause the cartographer leader to perform surveying and mapping functions to generate a map of the area where the mission will be performed. The area may be defined by the initial mission request received by the colony control center 122. Alternatively, the area may be determined by the squad leader or the colony control center 122 based on information included in the mission request.

[0082] At block 412, the cartographer leader performs the scanning task to update a shared map. The information generated by the scanning task may be generation of a point cloud or topological map. The point cloud or topological map may include semantic labels. Semantic labels may include identification in the point cloud or topological map of regions of interest, or object. For example, a semantic label may indicate an obstacle (e.g., a hole, a rock, a bot, and the like), an ore location, a pile location, a charging station, and the like. The shared map may be shared between all bots in the squad.

[0083] At block 414, the squad leader generates bot instructions for a bot to complete a task for the mission. In some embodiments, the bot instructions may be generatedby a machine learning model executed by the squad leader. The squad leader may provide at least one of the mission, the task, the shared map, or bot information as input to the machine learning model to cause the machine learning model to generate a bot instruction. In alternative embodiments, one or more tasks may be associated with a bot instruction, for example in a rules-based approach, and when a certain task is determined by the squad leader as part of the mission, the associated bot instructions may be retrieved from memory and transmitted to the bot that will perform the task. Further, while the squad leader is discussed as transmitting bot instructions in the present examples, each bot may include the bot instruction or machine learning model to generate a bot instruction, and generate the bot instruction itself based on the task.

[0084] At block 416, the bot performs the task. The bot may perform the task based on the generated bot instruction. Examples of tasks include scanning and mapping tasks, excavation, hauling, battery replacement, and other functions related to completing the mission.

[0085] At block 418, the colony control center 122 receives bot information from the bots that have performed a task associated with the mission. The bot information may be received from each bot individually, for example via the communication coordinator 112. Alternatively, each bot in a squad may transmit their information to the squad leader, and the squad leader may transmit the bot information to the communication coordinator 112, or the colony control center 122 directly. The bot information may include, for example, navigation information indicating a path taken by a bot, task performance information associated with the actions taken by the bot to complete a task, or a current status of the bot after the task has been completed. Bot information may include, for example, metrics associated with mining. For example, metrics may relate to ore extracted, time till completion, damage incurred by bots, adjustments required to a shared map generated by a cartographer leader (e.g., an adjustment may include a bot navigating to an area, and making adjustments to the shared map indicating the map was incorrect), and so on. As another example, metrics may relate to computer vision, such as how accurately the bots are able to detect objects or segmentation information (e.g., detect ore), and so on. Navigation information may additionally include any obstacles or issues encountered by the bot during traversal of a path. For example, if the bot collides with an object, or a wheel of the bot fails to function during traversal.

[0086] At block 420, the colony control center 122 compares the bot information to expected performance of the bot. The expected performance may be based on a prediction generated by a machine learning model for the bot’s performance. Comparison of bot information to expected performance is discussed in further detail below with respect to FIG. 5.

[0087] The colony control center 122 may update machine learning models associated with the center 122 and / or the bots. For example, the updates may cause updates to an ML model used by the center 122 that is used for planning of the bots. The updates may cause updates to the ML model used by the center 122 associated with instructing the bots to perform specific actions (e.g., mining actions, based on analyses of a real-world environment in which the bots are located). The updates may also cause updates to the bots. For example, a computer vision stack may be updated to more accurately detect objects or other segmentation information. As another example, an ML model associated with navigation, generating a map, techniques to form a mesh network and deal with packet loss, and so on, may be updated. As described herein, the updates may be based on simulations performed by the center 122.

[0088] At block 422, the colony control center 122 updates a machine learning model based on the received bot information as described below with respect to FIG.5. Swarm Robotics Coordination – Example Control Center Routine

[0089] FIG. 5 illustrates an example routine 500 for control center operation performed by, for example, the colony control center 122. The routine 500 begins at block 502 when a request to begin a mission is received by the colony control center 122.

[0090] At block 504, an indication is received by the colony control center 122 of the pre-mission check status of at least one communication coordinator 112. In some embodiments, pre-mission check information may be presented to a user of the system, and the user may indicate the pre-mission check information indicates the communication coordinator 112 is in a condition to operate as expected and perform the mission.

[0091] At block 506, an indication is received of pre-mission check status from at least one bot. Bot information may be received from each bot, a squad leader of each squad, and / or a communication coordinator 112 managing communications with one or more squads and / or platoons. In some embodiments, bot information may be presented to a user of thesystem, and the user may indicate the bot information indicates the associated bot (e.g., bot A 330) is in a condition to operate as expected and perform the mission.

[0092] At block 508, mission instructions are transmitted to at least one bot by the colony control center 122. For example, the colony control center 122 may transmit mission instructions to a bot operating as squad leader of a squad via an associated communication coordinator 112. Mission instructions may include any information needed to perform the mission requested at block 502. For example, excavation quadrant definitions, mine information, a location and / or number of layers to be excavated, a location and / or number of stopes to be excavated, information associated with a layer and / or stope (e.g., available sunlight for charging, distance from a base, material drop off location, other bots and / or squads nearby, etc.), map information, etc.

[0093] At block 510, the colony control center 122 receives bot data. The bot data may be received from each bot, or from a squad leader, via the communication coordinator 112 associated with the squad. Bot data may include status information (e.g., operational status of hardware or software components of the bot, mission status, etc.), mission performance information (e.g., operational time, tasks completed, etc.), model information for one or more machine learning models operating on the bot, and any other information the colony control center 122 may request, or the bot may determine is necessary to provide.

[0094] At block 512, model evaluation is performed in a simulated environment based on the received bot data. Model evaluation may be performed by the model optimization engine 206. Model evaluation may begin with generating or retrieving a virtual environment representing the physical environment the model data was generated in (e.g., the area of operation of a bot that transmitted the model information as part of its bot data). The model may be retrieved from a storage location of the colony control center 122. Alternatively, the model may be transmitted by the bot as part of the model information. The model may then be provided with the received bot data and evaluated for performance in the simulated environment to ensure the model is performing as a user or the system expects.

[0095] Further, in some embodiments, multiple machine learning models may be evaluated against each other to identify a preferred machine learning model. For example, OpenAI®Gym may be used to develop, train, and evaluate multiple reinforcement learning models to generate the preferred machine learning model to be deployed. Models may betrained by reinforcement learning in, for example, a centralized training or decentralized training arrangement. Swarm Robotics Coordination – Example Squad Leader Operation

[0096] FIG. s illustrates an example routine 600 for squad leader management of a squad of bots. To better understand the operation of routine 600, FIG.6B illustrates an example squad operating under the control of squad leader 665, and FIG. 6A will be described in reference to FIG. 6B.

[0097] In FIG. 6B, a squad leader 665 is in communication with communication coordinator 675 (e.g., a communication coordinator 112 associated with the squad), bots 670A to 570N, and a cartographer leader 660.

[0098] Routine 600 begins at block 602 in response to the squad leader 665 receiving an indication that it has been elected squad leader. In some embodiments, the squad leader 665 may be elected by the bots 670A-570N, for example using a ring network structure where each bot provides information to the token as it is passed through the squad. Alternatively, the squad leader 665 may be determined based on an availability of processing resources. In another alternative, the squad leader 665 may be determined based on a predicted risk of damage or loss of function, where a bot determined to be least likely to suffer damage or loss of function is selected as the squad leader.

[0099] At block 604, the squad leader 665 determines an operational area 655 for the squad. The operational area 655 may be defined by a mission plan or task received from the colony control center 122, indicating an area to be excavated by the squad associated with the squad leader 665. In some embodiments, the squad leader 665 may define the operational area 655 to be an area less than a mission area identified in the mission plan. For example, where the mission plan defines a mission area comprising two or more stopes, the squad leader 665 may determine a first stope of the two or more stopes at which the squad should begin. The squad leader 665 may then define the first stope as the operational area 655. The operational area 655 may be redefined as a second stope of the two or more stopes when the first stope has been excavated, or the squad has otherwise completed its tasks in the first stope.

[0100] At block 606, the squad leader 665 transmits at least one scanning and mapping task to a cartographer leader 660. For example, the squad leader 665 may instruct thecartographer leader 660 to perform scanning and mapping of operational area 655 to generate a map of the area for use when determining tasks for the bots based on a mission plan. Operational area 655 may, for example, be a stope, a layer, or a mine. The scanning and mapping task may, in some embodiments, identify the bot to act as the cartographer leader 660. The scanning and mapping task may identify a resolution of mapping information to be generated, and / or a type of mapping information to be generated (e.g., identifying locations likely to include a material of interest to be mined). The scanning and mapping task may be generated by applying information available to the squad leader (e.g., mission information, task information, mission area information, etc.) as input to a machine learning model to generate a set of actions to complete the scanning and mapping task.

[0101] At block 608, the squad leader 665 receives an indication from the cartographer leader 660 that the scanning and mapping task is complete. The indication may include the scanning and mapping information for the operational area 655. The squad leader 665 may transmit the scanning and mapping information to some or all of the bots of the squad. Alternatively, the cartographer leader may transmit the scanning and mapping information to some or all of the bots of the squad, thereby reducing the risk of data corruption based on repeated transmissions.

[0102] At block 610, an operational task and an associated operational task plan are transmitted by the squad leader 665 to each bot. For example, the squad leader 665 may transmit to a first bot configured to perform excavation (e.g., an excavation bot) a task of excavating a location identified as having a material of interest by the scanning and mapping information received from the cartographer leader. The squad leader 665 may then transmit a task plan to the excavation bot indicating a series of actions to take to accomplish the excavation task. As an example, the leader 565 may navigate to the area, perform a test excavation to ensure the material is what was expected from the scanning and mapping information, confirm the results of the test excavation indicate excavation should continue, if the results indicate excavation should continue then excavate. For this example, the squad leader 665 may transmit a second operational task and operational task plan to a hauler bot configured to collect and haul excavated material. The operational task may be to collect material excavated by the excavator bot, and the operational task plan includes a series of steps required to accomplish this operational task. This may continue until each bot needed toaccomplish the current portion of the mission plan has been assigned an operational task and operational plan. The operational task may include one or more actions generated by a machine learning model executed by the squad leader based on information available to the squad leader (e.g., shared map information, mission information received from the colony control center 122, task information, etc.).

[0103] At decision block 612, a determination is made as to whether the excavation of the operational area 655 is complete. For example, each excavator bot of the squad may indicate it has completed its assigned excavation operational task, and the squad leader 665 may then determine that the operational area 655 has been excavated. If the operational area 655 has been excavated, the routine 600 moves to block 614. If the area excavation is not completed, the routine 600 returns to block 606.

[0104] At block 614, the squad leader 665 transmits bot data to the colony control center 122. As stated previously herein, the order of blocks in this example routine 600 are intended to be illustrative only, and do not limit the order the steps described with reference to any routine may be performed. It should be noted, therefore, that data transfer between the squad leader 665 and the colony control center 122 may be performed at any point in time, and may be performed continuously throughout a mission. The bot data may be transmitted via the communication coordinator 675 when direct communication between the squad leader 665 and the colony control center 122 is not possible or not desirable. Scanning and Mapping Operations – Cartographer Leader

[0105] FIG. 7 illustrates example routine 700 for generation of scanning and mapping data for an area by a cartographer leader. The routine 700 begins at block 702 when the cartographer leader receives a scanning and mapping task, and navigates to a destination site indicated by the task, for example as described above with respect to FIG. 7.

[0106] At block 704, the cartographer leader completes scanning and mapping of the area of the destination site (e.g., operational area 655). The cartographer leader may complete scanning and mapping by dividing the destination site into a series of areas in a grid formation and effectuate scanning and mapping of each grid in a predetermined sequence. Alternatively, the scanning and mapping of the destination site may be completed by generating a graph of the area comprising nodes and edges, the nodes representing locationswhere the cartographer leader is to stop and perform scanning and mapping tasks, and the edges representing paths the cartographer leader is to take between nodes. The scanning and mapping task then may be optimized using algorithms designed to provide solutions to the traveling salesman problem (e.g., a branch and bound algorithm). It should be recognized that other methods are known for performing initial mapping of an area, and any such algorithm may be used to complete the scanning and mapping task of the cartographer leader.

[0107] At block 706, the cartographer leader plans navigation for the other bots in the cartographer leader’s squad. In embodiments where the cartographer leader used a graph- based mapping, the cartographer leader may use the information derived from the scanning and mapping task to efficiently route each bot in the squad to a node associated with each bot’s task. For example, by determining edges leading between nodes that avoid obstacles a bot may not be able to traverse (e.g., a large hole). In embodiments where a grid-based mapping was performed, the locations in the grid may have been assigned values (e.g., the vertical lines may be alphabetical values, and the horizontal lines may be assigned numerical values). Each bot may then be associated with a grid location in which the bot is to perform an operational task, and the cartographer may indicate the series of grid locations to be traversed for the bot to reach its destination grid location. The bot may then use its multi-agent path planner 310 to perform pathfinding operations to navigate the series of grid locations to reach the destination grid location.

[0108] In the context of multi-robot pathfinding, bots navigate from one location to another to perform their assigned task or tasks. Therefore, a graph representation may be useful for planning the path for each bot (an example of the generation of the graph representation is discussed further detail with respect to FIG. 10 below), where the graph represents the operational environment. In the graph, vertices may be locations of bots, task locations, or obstacles. Edges may represent paths between locations, and may be associated with a cost, such as a traversal cost representing the predicted amount of energy, or risk of damage, associated with navigating the edge. A collision-free or conflict-free path may then be determined for each bot in the graph to reach a task location. Types of conflict include a vertex conflict where two obstacles or bots are simultaneously in the same position, an edge conflict where two bots attempt to traverse the same edge at the same time, or a swappingconflict where two bots attempt to exchange positions at the same time in two different directions.

[0109] Further, in some embodiments, the planning of navigation tasks may be performed by the squad leader or a squad planning leader based on information generated by the cartographer leader. The squad planning leader may be elected by the squad, or may be determined based on determined available resources of one or more bots (e.g., a bot with enough electrical power, processing power, etc., to perform planning tasks for the squad).

[0110] At block 708, the cartographer leader receives data from other bots of the cartographer leader’s squad indicating the bots’ paths travelled so that the cartographer leader’s navigation planning may be evaluated. The cartographer leader may update mapping and scanning data based on the data received from the other bots of the squad, for example based on the excavation of a location, the discovery of an obstacle (e.g., an unstable area not suited for traversal), the discovery of additional excavation locations, comparison of an expected time for traversal to an actual time for traversal for a bot, etc.

[0111] At block 710, the cartographer leader transmits the squad data it has received to the colony control center 122. In some embodiments, the cartographer leader transmits the information to the squad leader, or the communication coordinator 112 associated with the squad, to transmit the squad data to the colony control center 122. Bot Operation

[0112] FIG.8 illustrates example routine 800 for bot operation within a squad. The routine 800 begins at block 802 when a bot receives navigation and task information from the squad task planner. In some embodiments, the squad task planner is the same bot as the squad leader. Additionally, navigation information may be received directly from a cartographer leader, as described in routine 700.

[0113] At block 804, the bot navigates to the location of the next task it has been assigned using the received navigation information. For example, the multi-agent path planner 310 of the bot may process navigation information received from the cartographer leader indicating a destination location and a path. The multi-agent path planner 310 may then use a collaborative pathfinding algorithm to analyze the path information and determine an optimal set of movements for traversal between the bot’s current location and the destination location.Additionally, the multi-agent path planner 310 may operate continually as the bot traverses the path, updating the path information during traversal to account for unexpected obstacles, a need to perform additional data gathering (e.g., when an unstable area is reached, the bot may pause to determine the extent of the unstable area before continuing traversal to the destination location), and performing other functions to optimize the path taken. Alternatively, a squad planning leader in charge of task planning for the squad may provide the navigation plan to some or all bots of the squad, and the bot may navigate to the location of the next task using the planning information provided by the planning leader.

[0114] At block 806, the bot identifies task instructions associated with the destination location it has navigated to. In some embodiments, the bot may have received operational task instructions from the squad leader, the operational task instructions associated with the destination location. For example, a battery charger bot may have received an operational task to charge its battery array by moving to a location with access to sunlight for solar recharging. When the battery charger bot reaches the destination location for solar recharging, the bot may identify an operational task instruction to orient solar panels for optimal energy generation. The bot may then determine a series of tasks using its task planning engine 308 to accomplish the operational task, and begin to adjust the orientation of the solar panels, taking measurements of energy production at each location. When the orientation energy generation information has been processed, the bot may then identify the optimal location for initial orientation of the solar panels. The bot’s task instructions may further include gradual reorienting of the solar panels from the initial position to track the location of the sun as it moves through the sky. The bot’s task planning engine 308 may then generate a series of instructions indicating time to move, and orientation to move to, for the bot, so that it may maintain optimal charging as long as sunlight, or another energy source, is available.

[0115] At block 808, the bot performs the task associated with its current location, following the task instructions. The bot may perform online task planning and determine based on a task to be accomplished the movement of parts of the bot required to complete the task. Continuing the task of the battery charger bot in block 806, the bot may perform the series of tasks generated by the task planning engine 308 to allow for optimal charging, and continue to reorient the solar panels to allow for optimal charging until the sun is no longer visible, or received energy is below a minimum threshold.

[0116] At block 810, the bot performs a hardware and software health check. The bot may request that some or all of its components report current status information as part of the hardware and software check. For example, the bot may request each motor provide its current status information. Further, the bot may request mechanical parts perform a mechanical function test. For example, when testing a motor, the bot may rotate the motor by 30, 60, 90, and 180 degrees to ensure fine motor control is functioning as expected, rotate the motor at a speed for a predetermined amount of time to ensure the motor is able to operate continuously, operate the motor against a load to test the motor’s torque, etc. For components related to software, the bot may use checksums or other sanity checks to ensure that data has not been corrupted or lost, and that processing circuitry has not been damaged (e.g., by high energy radiation).

[0117] At block 812, the bot generates updated bot information and transmits the updated bot information to the squad leader, for example via the communication module 306 of the bot. Machine Learning Model Training

[0118] FIG. 9 illustrates example routine 900 for training and / or updating a machine learning model. The routine 900 begins at block 902 when the model optimization engine 206 generates predictions for a current status using a new machine learning model being trained and / or evaluated. The evaluation of new machine learning models may occur at the colony control center 122, or at another computing device in communication with the colony control center 122. The predictions may be based off historical data previously received from one or more bots, and / or simulated data generated be a computing system. In some embodiments, a plurality of new machine learning models may generate predictions from the same set of data, and all such new machine learning models may be compared substantially simultaneously.

[0119] At block 904, the model optimization engine 206 receives current status information for a bot or function of a bot, for example from a squad leader. The status information may include pathfinding data, mapping and scanning data, communication data, operational task information, tool movement information, information associated with the completion of a task (e.g., the time to complete excavation of a site), energy use information,or any other information generated by a bot and received by the colony control center 122. For example, the current status information may be the information transmitted by the squad leader 665 of FIG.5 to the colony control center 122.

[0120] At block 906, the model optimization engine 206 compares a currently deployed model’s predictions to the current status information. For example, the currently deployed model may be the machine learning model operating on the bot from which current status information was received. The predictions of the currently deployed model may have been stored previously at the colony control center 122 after being received from a bot (e.g., a squad leader of a squad). Alternatively, the predictions may have been transmitted with the current status information, or following the current status information, for the purpose of comparison and evaluation. The current status information may be associated with a function performed and / or planned by the currently deployed machine learning model. Comparing the currently deployed model’s predictions to the current status information may include determining a difference between various metrics associated with the predictions. For example, where a currently deployed multi-agent pathfinding model was used to generate a path for a bot, a comparison may be performed between the generated path and the path actually taken by the bot (e.g., after adjustments to the path are made to account for obstacles or to avoid collisions with other bots).

[0121] At decision block 908, the model optimization engine 206 determines whether the new model has generated more accurate predictions than the deployed model previously generated, based on the current status information. If the new model has provided more accurate predictions, then the routine 900 moves to block 910 and the new model is deployed to the system. If the new model has not provided more accurate predictions than the currently deployed model, the routine 900 returns to block 902 and a new model is tested.

[0122] Alternatively, the model optimization engine 206 may determine the new model is more efficient (e.g., required reduced computing resources, executes in a shorter amount of time, requires less input information to generate an accurate result, etc.), and may determine to deploy the new model based on improved efficiency. In some cases, a balance may be sought between efficiency and accuracy, and threshold values may be set for efficiency and accuracy in order to optimize the model towards a desired target. For example, a high accuracy may be preferred for manipulation tasks such as replacing a battery in a bot, and so athreshold that prefers accuracy over efficiency may be set for selecting a model to be deployed to a battery bot. In another example, accuracy requirements for performing an excavation task by an excavation bot may be lower, and so efficiency may be preferred and a threshold set to prefer efficiency when comparing models. Example Collaborative Localization and Mapping System

[0123] FIG. 10 illustrates example data flows within and between robots for performing collaborative simultaneous localization and mapping. The collaborative simultaneous localization and mapping may be performed by a collaborative SLAM system 1010A - 1010N within one or more bots in a squad. Each collaborative SLAM system 1010A may be in communication with a sensor module 1005.

[0124] The sensor module 1005 may include, for example, at least one of a stereo camera, an RGB camera, LIDAR, an inertial measurement unit, a global positioning system, or additional sensor types useful for capturing information about an environment in which a bot operates. The sensor module 1005 may communicate with the collaborative SLAM system 1010A via the sensor controller 1015.

[0125] The collaborative SLAM system 1010A includes a sensor controller 1015, a sensor synchronization module 1020, an odometry module 1030, a loop closure detection module 1025, a neighbor management module 1035, a multi-robot pose graph optimization module 1040, and a SLAM communication module 1045A.

[0126] The sensor controller 1015 manages communication between the sensor module 1005 and the components of the collaborative SLAM system 1010A. Additionally, the sensor controller 1015 may perform control functions related to the operation of the sensors of the sensor module 1005. For example, the sensor controller 1015 may manage power sent to the sensor module 1005 or individual sensors of the sensor module 1005. In another example, the sensor controller 1015 may set a detection frequency (e.g., a number of samples taken by a sensor within a timeframe) for sensors, or may control when a sensor is on or off. The sensor controller 1015 transmits received sensor information to the odometry module 1030 and the loop closure detection module 1025 for further processing.

[0127] The sensor synchronization module 1020 receives sensor information from the sensor controller 1015. The sensor synchronization module 1020 may use time informationincluded in the sensor information received from different sensors of the sensor module 1005 to synchronize time samples from each sensor. The synchronized time samples may be packetized, and provided in groups (e.g., a packet including a one second interval of sensor information from each sensor for the same second) to the odometry module 1030 or another element of the collaborative SLAM system 1010A for further processing and use. The sensor synchronization module 1020 may perform additional sensor information processing functions, such as detecting false or inconsistent sensor information.

[0128] The odometry module 1030 uses motion data for the bot in which the collaborative SLAM system 1010A is located to estimate a change in the bot’s position over time. The change in position may be estimated based on one or more of a number of motor revolutions, wheel revolutions, sensor information (e.g., information generated by an inertial measurement unit), or a length of time for which a motor was activated. Information from the odometry module 1030 is provided to the graph optimization module 1040 for further processing and use. The odometry module 1030 may additionally transmit sensor information to the graph optimization module 1040.

[0129] The loop closure detection module 1025 performs loop closure determination functions as known in the art for localization and mapping using SLAM.

[0130] The neighbor management module 1035 receives information from the SLAM communication module 1045A to process location, pose, and other information related to the localization and mapping function performed by the collaborative SLAM system 1010N of another bot. The neighbor management module 1035may determine a relative position between the bot where the collaborative SLAM system 1010A is operating, and another bot (e.g., a second bot in the same squad). The neighbor management module 1035 may additionally tag information received from a bot, so that this information may be associated with the bot from which it was received later during processing.

[0131] The multi-robot pose graph optimization module 1040 represents poses and relationships between poses of two or more robots as a graph. In the graph, nodes represent pose information, and edges represent spatial constraints or measurements between the poses. The constraints represented by the edges may be determined based on odometry information from the odometry module 1030, and loop closure information received from the loop closure detection module 1025. By performing pose graph optimization, an optimal set of posessatisfying the spatial constraints may be determined, as is known in the art. Pose graph optimization advantageously improves accuracy, consistency, and efficiency of localization.

[0132] The SLAM communication module 1045A may be in communication with any number of SLAM communication modules (e.g., SLAM communication module 1045N) of another collaborative SLAM system 1010N. The SLAM communication module 1045A may communicate wirelessly with the SLAM communication module 1045N, or any number of other SLAM communication modules, in order to send and receive information related to localization and mapping for each bot. Example Network Address Assignment

[0133] FIG. 11 illustrates assigning network identifiers to a robotics squad using an IP addressing scheme according to some embodiments. The IP addressing scheme is used to assign unique IP addresses to devices on a network. It involves dividing the IP address space into subnets and assigning blocks of addresses to each subnet. One common scheme is to use Class C networks, which uses a subnet mask of 255.255.255.0. This means that the first three octets of the IP address are used to identify the network, and the last octet is used to identify the host. Other IP addressing schemes may be used, for example a location-based scheme, a role-based scheme, or an application-based scheme where IP addresses are determined by an application executed by the device.

[0134] The present example IP address table 1100 uses a combination location and role-based IP addressing scheme. In this example, the first octet represents a location where the bot is located. The second octet represents a squad to which the bot is assigned. The third octet represents a type of the bot (e.g., surveyor, hauler, battery bot, etc.). The fourth, and last, octet is a number assigned between 1 and 254. The fourth octet value may be assigned randomly with collision detection to ensure the final IP address is unique, or may be assigned sequentially (e.g., the first bot to connect is assigned a value of 1, the second bot is assigned a value of 2, and so on). Example Operating Environment

[0135] FIG. 12 illustrates an example plurality of robotic squads deployed in an industrial operating environment. For example, the operating environment may be an area inwhich the robotic squads are instructed to mine. Robotic squads, as described previously herein, may be assigned to excavate a stope of the mine, and each stope may represent an operating environment for a respective robotic squad of the plurality of robotic squads. Further, the operating environment may comprise locations for storage of robots and / or mined materials, recharging areas, organization or deployment areas, and / or any other area or zone type useful for the purpose of organizing the mining of the operating environment by the plurality of robotic squads. Terminology

[0136] While the above detailed description has shown, described, and pointed out novel features of the development as applied to various examples, it will be understood that various omissions, substitutions, and changes in the form and details of the systems or processes illustrated may be made by those skilled in the art without departing from the spirit of the development. As will be recognized, the present development may be embodied within a form that does not provide all of the features and benefits set forth herein, as some features may be used or practiced separately from others. The scope of the invention is indicated by the appended claims rather than by the foregoing description. All changes which come within the meaning and range of equivalency of the claims are to be embraced within their scope. The systems, devices, and methods may thus be practiced in many ways.

[0137] It will also be appreciated by those of skill in the art that parts included in one example are interchangeable with other examples; one or more parts from a depicted example may be included with other depicted examples in any combination. For example, any of the various components described herein and / or depicted in the Figures may be combined, interchanged or excluded from other examples. The use of headings is for ease of reading only, and is not meant to limit the scope of the disclosure in any way. Any features or examples from one heading section may be applied to any other features or examples of other heading sections.

[0138] The flow chart sequences discussed herein are illustrative only. A person of skill in the art will understand that the steps, decisions, and processes embodied in the flowcharts described herein may be performed in an order other than that described herein.Thus, the particular flowcharts and descriptions are not intended to limit the associated processes to being performed in the specific order described.

[0139] With respect to the use of substantially any plural and / or singular terms herein, those having skill in the art may translate from the plural to the singular and / or from the singular to the plural as is appropriate to the context and / or application. The various singular / plural permutations may be expressly set forth herein for sake of clarity.

[0140] It will be understood by those within the art that, in general, terms used herein are generally intended as “open” terms (e.g., the term “including” should be interpreted as “including but not limited to,” the term “having” should be interpreted as “having at least,” the term “includes” should be interpreted as “includes but is not limited to,” etc.). The term “comprising” as used herein is synonymous with “including,” “containing,” or “characterized by,” and is inclusive or open-ended and does not exclude additional, unrecited elements or method steps.

[0141] It will be further understood by those within the art that if a specific number of an introduced claim recitation is intended, such an intent will be explicitly recited in the claim, and in the absence of such recitation no such intent is present. For example, as an aid to understanding, the following appended claims may contain usage of the introductory phrases “at least one” and “one or more” to introduce claim recitations. However, the use of such phrases should not be construed to imply that the introduction of a claim recitation by the indefinite articles “a” or “an” limits any particular claim containing such introduced claim recitation to examples containing only one such recitation, even when the same claim includes the introductory phrases “one or more” or “at least one” and indefinite articles such as “a” or “an” (e.g., “a” and / or “an” should typically be interpreted to mean “at least one” or “one or more”); the same holds true for the use of definite articles used to introduce claim recitations. In addition, even if a specific number of an introduced claim recitation is explicitly recited, those skilled in the art will recognize that such recitation should typically be interpreted to mean at least the recited number (e.g., the bare recitation of “two recitations,” without other modifiers, typically means at least two recitations, or two or more recitations).

[0142] Furthermore, in those instances where a convention analogous to “at least one of A, B, and C, etc.” is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., “a system having at least one ofA, B, and C” would include but not be limited to systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.). In those instances where a convention analogous to “at least one of A, B, or C, etc.” is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., “a system having at least one of A, B, or C” would include but not be limited to systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.). It will be further understood by those within the art that virtually any disjunctive word and / or phrase presenting two or more alternative terms, whether in the description, claims, or drawings, should be understood to contemplate the possibilities of including one of the terms, either of the terms, or both terms. For example, the phrase “A or B” will be understood to include the possibilities of “A” or “B” or “A and B.”

[0143] All numbers expressing quantities used in the specification and claims are to be understood as being modified in all instances by the term “about,” unless otherwise indicated. Accordingly, unless indicated to the contrary, the numerical parameters set forth in the specification and attached claims are approximations that may vary depending upon the desired properties sought to be obtained by the present invention. At the very least, and not as an attempt to limit the application of the doctrine of equivalents to the scope of the claims, each numerical parameter should be construed in light of the number of significant digits and ordinary rounding approaches. For example, terms such as about, approximately, substantially, and the like may represent a percentage relative deviation, in various examples, of ±1%, ±5%,, ±10%, or ±20%.

[0144] The term “model,” as used in the present disclosure, can include any computer-based models of any type and of any level of complexity, such as any type of sequential, functional, or concurrent model. Models can further include various types of computational models, such as, for example, artificial neural networks (“NN”), language models (e.g., large language models (“LLMs”)), artificial intelligence (“AI”) models, machine learning (“ML”) models, multimodal models (e.g., models or combinations of models that can accept inputs of multiple modalities, such as images and text), and / or the like.

[0145] All references cited herein are incorporated herein by reference in their entirety. To the extent publications and patents or patent applications incorporated byreference contradict the disclosure contained in the specification, the specification is intended to supersede and / or take precedence over any such contradictory material.

Claims

CLAIMS WHAT IS CLAIMED IS:

1. A system comprising: a colony control center comprising: non-transitory computer-readable memory storing computer-executable instructions; and one or more processors in communication with the computer-readable memory and configured to execute the computer-executable instructions to: receive a request to initiate a mission, the mission comprising a task, wherein the task is associated with mining in a real-world environment, and wherein the mining is performed via a plurality of bots organized into a squad within the real-world environment, wherein the bots include at least one of: an excavator bot, a hauler bot, or a battery bot; determine, based on pre-mission check information, that the plurality of bots is ready to execute the task; transmit an instruction to the plurality of bots comprising task information; receive bot information associated with execution of the task, wherein the bot information is used to adjust one or more machine learning models associated with the colony center and / or the bots; the plurality of bots organized into the squad, wherein individual bots include individual processors and at least one tool associated with mining, wherein the plurality of bots communicate with each other, wherein the plurality of bots are configured to elect a cartographer leader and a squad leader in response to receipt of the task from the colony center, wherein the cartographer leader is configured to: maintain a shared map in real time, wherein to maintain the shared map the cartographer leader transmits map updates to other bots in the squad, wherein the squad leader is configured to:determine a plurality of actions to be executed by at least one bot of the plurality of bots based at least in part on the shared map, and wherein based on completion of the actions, one or more of the bots of the plurality of bots are configured to transmit bot information to the colony control center to trigger adjustments to the one or more machine learning models.

2. The system of claim 1, wherein the shared map is a 2-dimensional grid associated with the real-world environment.

3. The system of claim 1, wherein the cartographer leader is further configured to semantically segment the shared map to identify objects, wherein semantically segmenting the shared map associates segmentation information with the objects within the shared map.

4. The system of claim 1, wherein the plurality of bots in the squad form a mesh network to communicate information between bots of the squad.

5. The system of claim 1, wherein the cartographer leader is further configured to generate the shared map based on scan information.

6. The system of claim 1, wherein the cartographer leader is further configured to update the shared map based on information received from a second bot of the plurality of bots in the squad.

7. The system of claim 1, wherein the cartographer leader is further configured to: transmit scan information to the colony control center to cause the colony control center to update the shared map; and receive the updated shared map from the colony control center.

8. The system of claim 1, wherein the squad leader is further configured to generate an action to cause the cartographer leader to scan an area to generate scan information and update the shared map.

9. The system of claim 8, wherein the squad leader is further configured to generate an action to cause a bot of the squad of bots to mine the area, wherein the action generated is based in part on the updated shared map.

10. The system of claim 1, wherein the system further comprises a communication coordinator configured to relay information between at least one bot of the squad of bots and the colony control center.

11. The system of claim 1, wherein adjusting the one or more machine learning models is based on simulation information, the simulation information being based on the real-world environment and the bot information.

12. A method for orchestrating actions of a plurality of bots organized in a squad by a colony control center, the colony control center including one or more computers, and the method comprising: receiving a request to initiate a mission, the mission comprising a task, wherein the task is associated with mining in a real-world environment, and wherein the mining is performed via the plurality of bots organized into a squad within the real-world environment, wherein the bots include at least one of: an excavator bot, a hauler bot, or a battery bot; determining, based on pre-mission check information, the plurality of bots is ready to execute the task; transmitting an instruction to the plurality of bots comprising task information, wherein the task information causes the plurality of bots to perform the task based in part on a map shared between the squad of bots and a plurality of actions generate by a squad leader of the squad; receiving bot information from the plurality of bots; and updating a machine learning model based on the bot information.

13. The method of claim 12, wherein updating the machine learning model comprises training the machine learning model in a simulated environment based on reinforcement learning.

14. The method of claim 12, further comprising: updating the map based on scan information received from a cartographer leader of the plurality of bots; and transmitting the updated map to the plurality of bots.

15. The method of claim 12, further comprising: displaying a user interface to a user comprising a display portion to receive the request to initiate the mission.

16. The method of claim 12, further comprising: displaying a user interface comprising a 2-dimensional grid representation of the map comprising semantic segmentation information identifying objects represented within the map.

17. A system comprising one or more computers and non-transitory computer storage media storing instructions that when executed by the one or more computers, cause the one or more computers to perform the method of claims 12-16.

18. A method performed by a plurality of bots organized in a squad comprising: transmitting pre-mission check information indicating at least a portion of the plurality of bots is ready to execute a task; in response to transmitting the pre-mission check information, receiving an instruction comprising task information; selecting, from the plurality of bots, a cartographer leader to generate map information for an area where a task associated with the task information will be performed; scanning, by the cartographer leader, the area to generate a shared map, wherein the shared map is shared between bots in the squad; maintaining, by the cartographer leader, the shared map in real time based on scan information received from the plurality of bots;selecting, from the plurality of bots, a squad leader to determine a plurality of actions to be executed by at least one bot of the plurality of bots based at least in part on the shared map and the instruction; and executing, by at least one bot of the plurality of bots, the plurality of actions; in response to executing the plurality of actions; transmitting bot information to a colony control center to cause the colony control center to train a machine learning model based on the bot information.

19. The method of claim 18, wherein the plurality of bots comprises at least one of: a battery bot, an excavator bot, or a hauler bot.

20. The method of claim 18, wherein selecting the squad leader comprises receiving at a first bot of the plurality of bots a vote from each bot of the plurality of bots via a mesh network used for communicating between individual bots of the plurality of bots.

21. The method of claim 18, wherein selecting the cartographer leader comprises the squad leader identifying a bot of the plurality of bots having sufficient available computing resources to generate and update the shared map, and wherein the bot is the cartographer leader, and wherein the bot has at least one of: a LIDAR, a camera, or a depth sensor.

Images