Systems and methods for physics-informed neural networks for robot mapping

Abstract

Methods and systems for autonomous robot motion planning are disclosed. A system for autonomously controlling a robot in an unknown environment comprises a robotic device having at least one drive mechanism for moving at least a part of the robot. A processor is in communication with the robotic device and a memory is in communication with the processor. The memory has instructions store thereon that, when executed, cause the processor to obtain a plurality of local sensor data of the unknown environment from a first viewpoint. The processor processes the plurality of local sensor data to create a plurality of sample points and a plurality of speed values. The processor trains a neural network based on a pairing of a first sample point from the plurality of sample points and a first speed value from the plurality of speed values.

Description

Docket No.172842.00032 (70611) 1 SYSTEMS AND METHODS FOR PHYSICS-INFORMED NEURAL NETWORKS FOR ROBOT MAPPING CROSS-REFERENCE TO RELATED APPLICATION(S)

[0001] This application claims priority to and incorporates by reference U.S. provisional patent application no. 63 / 644,362, filed May 8, 2024, the content of which is hereby incorporated by reference in its entirety. STATEMENT OF GOVERNMENT SUPPORT

[0003] N / A TECHNICAL FIELD

[0004] The technology discussed below relates to motion planning and environment mapping, such as for applications in robotic and / or autonomous movement. BACKGROUND

[0005] Motion Planning (MP) is one of the components of an autonomous robot system that aims to interact physically with its surrounding environments. MP algorithms find path solutions from the robot’s start state to the goal state while respecting all constraints, such as collision avoidance. The quest for fast, scalable MP methods has led from traditional approaches that exhibit promising performance in high-dimensional spaces. However, a significant bottleneck in state-of-the-art MP methods is the need for expert trajectories from traditional MP methods, limiting their application to high-dimensional scenarios where large-scale data generation is time-consuming.

[0006] Additionally, many existing approaches require prior knowledge of an existing environment (e.g., a mapping) in order to develop a trajectory through a traditional MP method. Thus, traditional MP approaches do not function well (or at all) in scenarios in which an autonomous or robotic system encounters a new environment, or in which the obstacles within an environment have meaningfully changed since a mapping was obtained.

[0007] As the demand for autonomous robot systems continues to increase, a need has arisen for more advanced robot motion planning technologies to meet the growing demand for improved processing path solutions of autonomous robots in motion planning, QB\182506.00043\96318469.1Docket No.172842.00032 (70611) 2 along with a rapid and reliable way to map novel environments to support such motion planning. SUMMARY

[0008] The following presents a simplified summary of one or more aspects of the present disclosure, in order to provide a basic understanding of such aspects. This summary is not an extensive overview of all contemplated features of the disclosure, and is intended neither to identify key or critical elements of all aspects of the disclosure nor to delineate the scope of any or all aspects of the disclosure. Its sole purpose is to present some concepts of one or more aspects of the disclosure in a simplified form as a prelude to the more detailed description that is presented later.

[0009] In some aspects, the present disclosure can provide a system for autonomously controlling a robot in an unknown environment. The system can include a robotic device having at least one drive mechanism for moving at least a part of the robot. A processor can be in communication with the robotic device. A memory can be in communication with the processor and can have instructions stored thereon that, when executed, can cause the processor to obtain a plurality of local sensor data of the unknown environment from a first viewpoint. The processor can process the plurality of local sensor data to create a plurality of sample points and a plurality of speed values. The processor can train a neural network based on a pairing of a first sample point from the plurality of sample points and a first speed value from the plurality of speed values.

[0010] These and other aspects of the disclosure will become more fully understood upon a review of the drawings and the detailed description, which follows. Other aspects, features, and embodiments of the present disclosure will become apparent to those skilled in the art, upon reviewing the following description of specific, example embodiments of the present disclosure in conjunction with the accompanying figures. While features of the present disclosure may be discussed relative to certain embodiments and figures below, all embodiments of the present disclosure can include one or more of the advantageous features discussed herein. In other words, while one or more embodiments may be discussed as having certain advantageous features, one or more of such features may also be used in accordance with the various embodiments of the disclosure discussed herein. Similarly, while example embodiments may be discussed below as devices, systems, or methods embodiments it should be understood that such example embodiments can be implemented in various devices, systems, and methods. QB\182506.00043\96318469.1Docket No.172842.00032 (70611) 3 BRIEF DESCRIPTION OF THE DRAWINGS

[0011] FIG.1 is a flow diagram illustrating an example process for unknown environment mapping according to some embodiments.

[0012] FIG. 2 is a flow diagram illustrating an example process for autonomous robot motion planning training according to some embodiments.

[0013] FIG. 3 is a flow diagram illustrating an example process for autonomous robot motion planning according to some embodiments.

[0014] FIG.4 illustrates an example neural network model for autonomous robot motion planning according to some embodiments.

[0015] FIG. 5 is an illustration showing an example mapping and motion planning process according to some embodiments.

[0016] FIG.6 is a diagram illustrating an example Active NTFields pipeline according to some embodiments.

[0017] FIGS. 7A and 7B illustrate a comparison between four maze environments according to some embodiments.

[0018] FIGS. 8A and 8B illustrate a comparison for mapping in Gibson environments according to some embodiments.

[0019] FIG. 9 illustrates another comparison for mapping in Gibson environments according to some embodiments.

[0020] FIG. 10 illustrates a real-world example environment according to some embodiments.

[0021] Fig.11 illustrates a real-world cabinet environment. DETAILED DESCRIPTION

[0022] The detailed description set forth below in connection with the appended drawings is intended as a description of various configurations and is not intended to represent the only configurations in which the subject matter described herein may be practiced. The detailed description includes specific details to provide a thorough understanding of various embodiments of the present disclosure. However, it will be apparent to those skilled in the art that the various features, concepts, and embodiments described herein may be implemented and practiced without these specific details. In some instances, well- known structures and components are shown in block diagram form to avoid obscuring such concepts. QB\182506.00043\96318469.1Docket No.172842.00032 (70611) 4

[0023] The disclosure provided below will first provide general processes for mapping an unknown environment according to the novel mapping / planning approach herein, for motion planning development, and for runtime processes of using such approaches for robotic navigation and motion. After these general descriptions, a discussion of the inventors’ validation work will be provided. While the validation work regards certain specific attributes or embodiments, it should be recognized that these efforts are also intended to provide a quantification of how the techniques herein improve upon the field of mapping and motion planning. Thus, the Examples and Experiments described herein are not meant to be limiting of the scope of the disclosure, but rather as illustrative examples of real world advantages and benefits of this disclosure. Example Unknown Environment Mapping Process

[0024] FIG. 1 is a flow diagram illustrating an example process 100 for mapping an unknown or partially-unknown environment using an autonomous robot (or other device capable of self-planned movement), in accordance with some aspects of the present disclosure. As described below, a particular implementation can omit some or all illustrated features / steps, may be implemented in some embodiments in a different order, and may not require some illustrated features to implement all embodiments. In some examples, an apparatus (e.g., computing device 410, processor 412 with memory 414, etc.) in connection with FIG.4 (described below) can be used to perform example process 100. However, it should be appreciated that any suitable apparatus or means for carrying out the operations or features described below may perform process 100. Thus, while the general term “robot” is used herein to refer to a device that may be performing some or all of the processes disclosed herein, it should be understood that this term is meant to be encompassing of a variety of devices that sense and / or map an environment, and plan a course of motion within that environment (e.g., robotic arms; aquatic, terrestrial or aerial drones; autonomous vehicles; autonomous devices; general purpose or purpose-specific robots; autonomous mapping and exploration devices; etc.).

[0025] At step 112, the process 100 can obtain local sensor data of an unknown or partially unknown environment. In some examples, the local sensor data can include raw input from internal sensors, such as a robot odometer, accelerometer, or other sensor, or a database of recorded relative heading and distance measurements obtained from sensors associated with the robot’s mode of movement (e.g., wheels, tracks, etc.). In other examples, the local sensor data can be from an outward “perception” or projected scanning sensor, such as LiDAR-based scanning data, infrared scanning data, optical QB\182506.00043\96318469.1Docket No.172842.00032 (70611) 5 sensor data, time of flight data, and / or depth images. In some embodiments, LiDAR or similar reflective time of flight data can provide data in the form of rays, while depth images can provide sampled pixels, which may be converted into rays using an intrinsic matrix. In further embodiments, a portion of an environment may be represented by movement and obstacle data from prior movement of the robot, a portion of an environment may be represented by perception sensors, and / or a portion of an environment may be interpolated from any of such data.

[0026] At step 114, the process 100 can process the local sensor data to create various sample points and speed values. In some examples, rays or pixels associated with the sensor data can be transformed into stratified spatial points. In some examples, the sample points and speed values can indicate a distance to a surface of one or more obstacles, objects, walls, etc., or to free space regions, in the environment. The associated speed values may reflect traversability characteristics. In some examples, the sample points may be points in a configuration space or c-space (or other lower dimensional projection of the environment) that may be processed for a c-space-specific encode. In further examples, the sample points may correspond to sample pixels that may be converted to rays during processing. For example, the processing of the data may use approximation methods that use a minimum or maximum threshold for one or more parameters.

[0027] At step 116, the process 100 can optionally obtain a pre-trained neural network, and apply the neural network to generate, in whole or in part, an arrival time field based on at least some pairings of sample points and speed values. For example, the neural network may be trained to approximate a solution to a partial differential equation for various sample points, such as the Eikonal equation, using a physics-informed loss function. In other words, the neural network may be trained to estimate the time, resource usage, or other “cost” for the device to move to given points in the mapped space relative to a defined target or goal. In some examples, the output of the neural network (after providing as inputs the sample points and / or other sensor values of the environment), may be a continuously differentiable time field, for use in subsequent motion planning tasks for that device in that environment.

[0028] Optionally, step 116 may entail training or fine tuning the neural network: in some examples, the training may be performed in accordance with all or some of the steps of process 200, as described below with respect to FIG. 2. For example, the neural network may be trained using a physics-informed loss function (e.g., Eikonal equation loss function), or other type of loss function capable of applying time field estimations. QB\182506.00043\96318469.1Docket No.172842.00032 (70611) 6

[0029] At step 118, the process 100 can determine a next viewpoint for further exploration, based on a trained exploration policy algorithm, which may be separate from the trained neural network. Thus, the policy algorithm may be a processing network configured to optimize environmental perception coverage and information gain, while balancing navigation utility and resource constraints and achieving user-defined or programmatic goals. In such examples, the policy algorithm may be a function that determines outcomes in information gain and resource usage if the device were to move to various points in the known space, and can choose the next viewpoint (or a sequence of viewpoints) that provide optimal outcomes. In some examples, the policy network may utilize distance indications and / or travel paths, in combination with ground truth speed or travel duration indications, to determine the next viewpoint. In some examples, the next viewpoint can provide for additional sections or portions of the unknown environment to be explored and / or mapped. In some examples, an exploration strategy associated with the movement of a device (i.e., robot) attached to the sensor may dictate the selection of the next viewpoint. For example, the strategy may chose a next viewpoint using a neural network model trained in capturing maximized scene coverage. In other examples, a heuristic or rule-based planner may be used instead of a learned policy. In such examples, a device may test various next viewpoints and determine outcomes, and advance to next viewpoints based on determined advantages. The exploration strategy may consider factors such as uncertainty in the arrival time field, variance in traversability estimates, or coverage gaps in the observed environment. In some embodiments, the exploration strategy operates independently of the arrival time estimation network, though in other embodiments they may be jointly trained or share components.

[0030] In further examples, a next viewpoint may be determined not merely by movement of the device itself, but also by movement of one or more components of the device such as external arms that may extend to lift a sensor (e.g., a LiDAR sensor) over or around obstacles to gain a more advantageous view of the environment.

[0031] At each viewpoint, the device may obtain further sensor information, and process the arrival time neural network to update a full arrival time field map of the environment.

[0032] At step 120, the process 100 determines if there is a next viewpoint that would achieve a threshold of exploration of the unknown environment, or if sufficient information (e.g., a threshold coverage) for the environment has been obtained. In some examples, the threshold of exploration can be determined by setting required a minimum distance (e.g., a clearance distance) from one or more detected obstacles. For example, a QB\182506.00043\96318469.1Docket No.172842.00032 (70611) 7 user may define thresholds dminand dmax, wherein d provides the minimum distance from a robot or sensor (in points of C-space) to the obstacle. If there is, the process 100 moves back to step 112, and repeats steps 112, 114, 116, and 118 until the threshold has been met.

[0033] Once the threshold described above with respect to step 120 has been met, the process 100 may optionally perform step 122. At step 122, the process 100 can store an arrival field map for the environment or move a robot in accordance with the generated map of the unknown environment. In some examples, the arrival field map may display expected time of arrivals at various points within a given environment. The stored arrival field map may be exported for use in other device performing exploration and motion planning in the same or similar environments. Example Autonomous Robot Motion Planning Training Process

[0034] FIG.2 is a flow diagram illustrating an example process 200 for autonomous robot motion planning training (e.g., training one or more networks used in motion planning, as described herein), in accordance with some aspects of the present disclosure. In some examples, process 200 may be performed prior to deployment of a device, or prior to a device commencing mapping of a new environment, to generate a trained model for use during execution of process 100 or process 300. In other examples, process 200 (or portions thereof) may be used to update or fine-tune a deployed model during runtime (e.g., during exploration, or upon detection of new or changed obstacles or other environmental features, or to account for changes in resource constraints of the device (e.g., degradation of battery life, changes in speed caused by wear of the device’s components, etc.). As described below, a particular implementation can omit some or all illustrated features / steps, may be implemented in some embodiments in a different order, and may not require some illustrated features to implement all embodiments. In some examples, an apparatus (e.g., computing device 410, processor 412 with memory 414, etc.) in connection with FIG.4 (described below) can be used to perform example process 200. However, it should be appreciated that any suitable apparatus or means for carrying out the operations or features described below may perform process 200.

[0035] At step 212, the process 200 can obtain device (e.g., a robot) configuration data to be utilized as training data. In some examples, the robot configuration training data can include one or more start configurations and one or more goal configurations, such as locations, poses, or states within a robotic device’s configuration space (c-space). In some examples, the start configuration and the goal configuration data can be a location QB\182506.00043\96318469.1Docket No.172842.00032 (70611) 8 (e.g., an absolute location, a relative location, or any other suitable location) or a coordinate (e.g., an absolute coordinate, a relative coordinate, or any other suitable coordinate) in a configuration space. In some examples, the configuration can be used by a robot manipulator (e.g., joint angles of a robotic arm, grasping mechanism, etc.), a vehicle (e.g., position and steering angle), etc. In further examples, the coordinate can include a global positioning system-based coordinate, a radar-based position, etc. In some examples, a configuration space can be a data representation of all possible configurations that a robot can attain in its workspace. In some examples, the configuration space can be a transformed / simulated space from the physical space in which the robot is located, of finite size. In some examples, the configuration space can be obtained by shrinking the robot to a point while growing the obstacles by the size of the robot. In further examples, the configuration space can have a suitable dimension. For example, if a robot has a 6- DOF (degree-of-freedom) arm, the configuration space would be a six-dimensional space that includes all the possible joint angles that the arm can take.

[0036] In some examples, the training data can contain sampled start and goal pairs in robot c-space randomly. In some examples, robot c-space to ^െ0.5,0.5^ can benormalized on each dimension. Then, the c-space sample range can be defined as a hypercube, i.e., all dimensions have the same range scale. Furthermore, the speed can be computed for those configurations using the speed model. In some examples, the workspace point cloud can include 20000 points on the workspace obstacle surface, which is converted into 128 ൈ 128 ൈ 128 occupied voxel grid. In some examples, theconfiguration space may represent spatial positions, joint angles, actuation states, or combinations thereof, each of which maybe represented in the configuration space as continuous, bounded, or discretized values, and / or may be normalized to a standardized range (e.g., [-0.5, 0.5] in each dimension).

[0037] In some examples, the process 200 can obtain an obstacle-free configuration space (c-space). In some examples, the obstacle-free configuration space can indicate a configuration space excluding one or more obstacle configurations. In some examples, although the c-space is the obstacle-free configuration space, the configuration space might include one or more obstacles, which can be ignored or overcome by the robot. In further examples, the start configuration and the goal configuration are on the obstacle- free configuration space. In some examples, the start configuration and the goal configuration are randomly sampled configurations on the obstacle-free configuration space. In other examples, the configuration space may be derived from real-world scans QB\182506.00043\96318469.1Docket No.172842.00032 (70611) 9 or simulated environments (e.g., voxelized obstacle point clouds), which may allow training of networks that are tailored to a particular use case of the associated robotic device. In some implementations, actual workspace obstacles or objects may be represented in a voxel matrix or mesh format from a scan of a real-world example (or actual) work environment, and subsequently converted into configuration space constraints applicable to the given device.

[0038] In further examples, there can be multiple sets (training epochs) of the start configuration and the goal configuration for training the neural network model or the machine learning model, which is described in connection with step 220.

[0039] In some examples, the configuration space and the environment space can be denoted as ^^ ⊂ ℝௗ and ^^ ⊂ ℝ^, where ^^^,^^^ ∈ ℕ represents their dimensionality. Theobstacles inenvironment, denoted as ^^^^^ ⊂ ^^ , form a formidable robotconfiguration space (c-space) defined as ^^^^^ ⊂ ^^ . Finally, the feasible space in theenvironment and c-space (i.e., the obstacle-free c-space) is represented as ^^^^^^ൌ ^^\^^^^^ and ^^^^^^ ൌ ^^\^^^^^ , respectively. The objective of robot motion planningalgorithms is to find a trajectory ^^ ⊂ ^^^^^^ that connects the given robot start ^^^ ∈ ^^^^^^and goal ^^^ ∈ ^^^^^^ configurations.

[0040] At step 214, the process 200 can configure input and output channels or layers of the neural network so that applying a start configuration and a goal configuration to the neural network model will result in outputting a time field. In some examples, the motion planning problems can be viewed as the solution to a partial differential equation (PDE), specifically focusing on solving the Eikonal equation. In some examples, the physics governed by an Eikonal equation can be modeled using a deep neural network.

[0041] The Eikonal equation, a first-order non-linear PDE, allows finding the shortest trajectory between the start configuration (^^^) and the goal configuration (^^^) under speed constraints by relating a predefined speed model ^^^^^^ at configuration ^^^to the arrival time ^^^^^^, ^^^^ from ^^^ to ^^^ as follows:^ൌ∥ ∇^^^^^^^^,^^^^ ∥ Equation 1

[0042] Theof the arrival time ^^^^^^, ^^^^ functionwith respect to ^^^. Therefore, finding a trajectory connecting the given start and goal requires solving the PDE under a predefined speed model and arrival time function. The arrival time function can be factorized as follows: QB\182506.00043\96318469.1Docket No.172842.00032 (70611) 10 ^^^^^∥^ ି ^, ^^^^ ൌ ೞ^^∥ ఛ^^ೞ,^^^ Equation 2

[0043] At step 216, determine a predicted speed indication and / or arrival time, based onthe goal configuration, and the time field. In some examples, the neural network model may be designed to produce ^^^^^^,^^^^, the factorized time field for the given ^^^(the start configuration) and ^^^(the goal configuration). Alternatively the neural network may return an intermediate representation (e.g., a vector field, embedding, or latent feature) from which τ can be derived. The predicted speed may represent physical traversal time, energy cost, or other motion-related metrics depending on the training objective.

[0044] At step 218, the process 200 may obtain a ground truth speed indication. In some examples, the ground truth speed indications may be obtained from existing observations. Moreover, in further examples, the ground truth speed indications may include a batch of points received from a memory of a local or remote device.

[0045] At step 220, the process 200 may train the neural network model based on a loss function, between the predicted speed indication (or time field) and the ground truth speed, travel duration or arrival time. In some examples, the loss function may be a physics-informed residual loss derived from the Eikonal equation. For example, the loss may correspond to a fixed number of iterations and may be calculated using the following calculation: ^^^^^∗, ^^^ ൌ ^ௌ∗ଶ ^^ೞ^ ^^ ^െ 1^ ^ ^^ௌ^∗ଶ ൫^^൯ െ^

[0046] Inand S the ground truth speed. In other examples, gradient-based or adversarial training approaches may be used, so as to refine accuracy, better enforce consistency, or ensure broader coverage over the configuration space. In some embodiments, process 200 may be executed as a preliminary, pre-deployment training phase to produce a fixed model (thereby ensuring stability and similar desirable benefits). In other embodiments, process 200 may be used in a periodic refinement or updating phase, post-deployment, for example in which the robot collects new data during exploration (e.g., start and goal configurations, and ground truth speed / travel durations) and periodically retrains or fine-tunes the model to improve performance over time. Example Autonomous Robot Motion Planning Process

[0047] FIG.3 is a flow diagram illustrating an example process 300 for autonomous robot motion planning in accordance with some aspects of the present disclosure. As described QB\182506.00043\96318469.1Docket No.172842.00032 (70611) 11 below, a particular implementation can omit some or all illustrated features / steps, may be implemented in some embodiments in a different order, and may not require some illustrated features to implement all embodiments. In some examples, an apparatus (e.g., computing device 410, processor 412 with memory 414, etc.) in connection with FIG. 4 can be used to perform example process 300. However, it should be appreciated that any suitable apparatus or means for carrying out the operations or features described below may perform process 300.

[0048] At step 312, the process 300 obtains a start configuration and a goal configuration. In some examples, the start configuration and the goal configuration at step 312 are substantially similar to the start configuration and the goal configuration at step 212 of FIG.2. For example, the start configuration and the goal configuration can be a location (e.g., an absolute location, a relative location, or any other suitable location) or a coordinate (e.g., an absolute coordinate, a relative coordinate, or any other suitable coordinate) on a configuration space. In some examples, the goal configuration may refer to a specific arrangement or position of one or more components on a robot (e.g., an arrangement of joints or links). For example, the goal configuration may vary based on a number of degrees of freedom or positions that robot is able to achieve.

[0049] At step 314, the process 300 applies the start configuration and the goal configuration to a trained neural network model to obtain a time field. In some examples, the trained neural network model was trained via steps 212–220 of FIG. 2. In some examples, any type of neural network capable of determining a time field may be used. For example, the neural network may be a recurrent neural network (RNN), a long short- term memory network (LSTM), a gated recurrent unit (GRU), a multi-dimensional neural network, or the like.

[0050] At step 316, the process 300 determines an arrival field map based on a plurality of gradients of the arrival time map. In some examples, the determined arrival field map determined at step 316 is substantially similar to the stored arrival field map at step 122 of FIG. 1. For example, the arrival field map correspond to expected time of arrivals at various points within a given environment, for a specific robot or device (given the possible configurations for that robot or device).

[0051] At step 318, the process 300 determines a map for an unknown environment based on the arrival field map, the start configuration, and the goal configuration. In some examples, the map can be reconstructed based on further sensing. In some examples, the map may be displayed as a depth image stream. In some examples, the format or display QB\182506.00043\96318469.1Docket No.172842.00032 (70611) 12 of the map may vary based on a type of environment. For example, the map may represent a Gibson environment.

[0052] At step 320, the process 300 may optionally move a robot according to the generated map of the unknown environment. In some examples, the generated map may include predictions of various obstacles within the environment. Therefore, when moving the robot, the movements may avoid these obstacles, while still achieving a goal configuration.

[0053] In some examples, process 100 described above with respect to FIG. 1 may be performed simultaneously with process 300, also described above with respect to FIG.3. In particular, a robot or device can begin with using local observations to create partial map of the unknown environment. Then, based on the goal configuration, can perform motion planning to reach the next viewpoint location to obtain new observations. Example Autonomous Robot Planning System

[0054] FIG.4 shows a block diagram illustrating a system for autonomous robot motion planning according to some embodiments. In some examples, a computing device 410 can obtain or receive robot configuration training / runtime data 402 including a start configuration and a goal configuration, a configuration space, a work space, obstacles on the configuration space, obstacles on the workspace, an obstacle-free configuration space, an obstacle-free workspace, and / or a ground truth speed indication from a user and / or a system via the communication network 430, produce a path solution from the start configuration to the goal configuration, and move a robot according to the path solution.

[0055] In further examples, the computing device 410 can include a processor 412. In some embodiments, the processor 412 can be any suitable hardware processor or combination of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), a microcontroller (MCU), etc.

[0056] In further examples, the computing device 410 can further include a memory 414. The memory 414 can include any suitable storage device or devices that can be used to store suitable data (e.g., robot configuration training / runtime data 402, a configuration space, a work space, obstacles on the configuration space, obstacles on the workspace, an obstacle-free configuration space, an obstacle-free workspace, and / or a ground truth speed indication, neural network model, etc.) and instructions that can be used, for example, by the processor 412 to obtain robot configuration training data, the robot configuration training data comprising a start configuration and a goal configuration; QB\182506.00043\96318469.1Docket No.172842.00032 (70611) 13 apply the start configuration and the goal configuration to a neural network model to obtain a time field; determine a predicted speed indication based on the start configuration, the goal configuration, and the time field; obtain a ground truth speed indication; train the neural network model based on a loss between the predicted speed indication and the ground truth speed indication; obtain an obstacle-free configuration space; combine a first output of the first encoder and a second output of the second encoder using a non-linear symmetric operator to produce a combined output; apply the start configuration and the goal configuration to a trained neural network model to obtain a time field; determine a path solution based on the predicted speed indication, the start configuration, and the goal configuration; and move a robot according to the path solution.. The memory 414 can include any suitable volatile memory, non-volatile memory, storage, or any suitable combination thereof. For example, memory 414 can include random access memory (RAM), read-only memory (ROM), electronically- erasable programmable read-only memory (EEPROM), one or more flash drives, one or more hard disks, one or more solid state drives, one or more optical drives, etc. In some embodiments, the processor 412 can execute at least a portion of process 100, 200, or 300 described above in connection with FIG.1, 2, or 3.

[0057] In further examples, computing device 410 can further include communications system 418. Communications system 418 can include any suitable hardware, firmware, and / or software for communicating information over communication network 430 and / or any other suitable communication networks. For example, communications system 418 can include one or more transceivers, one or more communication chips and / or chip sets, etc. In a more particular example, communications system 418 can include hardware, firmware and / or software that can be used to establish a Wi-Fi connection, a Bluetooth connection, a cellular connection, an Ethernet connection, etc.

[0058] In further examples, computing device 410 can receive or transmit information (e.g., robot configuration training / runtime data 402, a configuration space, a work space, obstacles on the configuration space, obstacles on the workspace, an obstacle-free configuration space, an obstacle-free workspace, and / or a ground truth speed indication, neural network model, a path solution 404, etc.) and / or any other suitable system over a communication network 430. In some examples, the communication network 430 can be any suitable communication network or combination of communication networks. For example, the communication network 430 can include a Wi-Fi network (which can include one or more wireless routers, one or more switches, etc.), a peer-to-peer network QB\182506.00043\96318469.1Docket No.172842.00032 (70611) 14 (e.g., a Bluetooth network), a cellular network (e.g., a 3G network, a 4G network, a 5G network, etc., complying with any suitable standard, such as CDMA, GSM, LTE, LTE Advanced, NR, etc.), a wired network, etc. In some embodiments, communication network 430 can be a local area network, a wide area network, a public network (e.g., the Internet), a private or semi-private network (e.g., a corporate or university intranet), any other suitable type of network, or any suitable combination of networks. Communications links shown in FIG. 4 can each be any suitable communications link or combination of communications links, such as wired links, fiber optic links, Wi-Fi links, Bluetooth links, cellular links, etc.

[0059] In further examples, computing device 410 can further include a display 416 and / or one or more inputs 420. In some embodiments, the display 416 can include any suitable display devices, such as a computer monitor, a touchscreen, a television, an infotainment screen, etc. to display the report, the human activity indication 440, or any suitable result of the path solution. In further embodiments, and / or the input(s) 420 can include any suitable input devices (e.g., a keyboard, a mouse, a touchscreen, a microphone, etc.). Example Method and Embodiments

[0060] Let the environment be denoted as ^^ ∈ ℝଷ with its obstacle and obstacle-freespace as ^^^^^ and ^^^^^^, respectively. The robot configuration space is indicated as ^^ ∈ℝௗ with dimension ^^ , where ^^^^^ and ^^^^^^ indicate the obstacle and obstacle-freeThe objective of environment mapping is for a robot to explore environment ^^ to recover its features ^^. The traditional mapping methods describe the features as maps, indicating obstacles and obstacle-free space. The modern approaches also recover the environment’s geometry as the SDF. The SDF provides the signed distance of any point in the environment to its obstacle geometry’s surface. These features allow motion planning methods to find a collision-free path connecting a robot’s given start and goal configurations to navigate the mapped environment. The collision avoidance constraints for the robot path are satisfied by leveraging the environment features ^^ recovered during mapping. However, traditional and modern mapping features require an extra computational mechanism to find the robot’s motion path. In this paper, a new mapping feature is introduced called the arrival time field, which is the solution to the Eikonal equation. QB\182506.00043\96318469.1Docket No.172842.00032 (70611) 15

[0061] The Eikonal equation is a first-order non-linear equation that represents a robot moving from a start ^^^to a goal ^^^within a speed constraint ^^^^^^ defined over robot configurations. The speed function outputs a scalar value and is designed so that the robot’s speed is high in the free space and low near the obstacle region. The Eikonal equation relates the speed constraint to the robot arrival time ^^ between the start ^^^and goal ^^^, i.e., ^ ௌ^^^^ൌ∥ ∇^^^^^^^^,^^^^ ∥ (1).

[0062] is the shortest arrival time between the given start andthe above equation using the grid-based Fast Marching Method (FMM) to recover the shortest path based on arrival time between the given start and goal. However, these methods lacked continuity and were computationally intractable in higher dimensional spaces. Recent work introduced a Laplacian-based viscosity term in the Eikonal equation and a new factorized formulation of arrival time, which allowed for solving Equation 1 using neural networks. The factorized arrival time is described as follows: ^^^^^∥^ೞି^^∥ ^, ^^^^ ൌఛ^^ೞ,^^^ (2)

[0063] The neural given start and goal to parameterize theabove equation, which leads to arrival time ^^ and its gradient. When the target point is in the obstacle space, the ^^ is predicted to be 0, making ^^ very large, and when the target is the same as the start, ∥ ^^^ െ ^^^ ∥ is 0, making arrival time ^^ equal to zero. The estimatedarrival time ^^ and its gradient are then used to predict the speed using the following viscosity Eikonal equation with a Laplacian. ^ ௌ^^^^ൌ∥ ∇^^^^^^^^,^^^^ ∥^^^Δ^^^^^^^^,^^^^,െ0.05^^^^ (3) where ^^ is a hyperparameter. Theiselliptic and has a unique solution around obstacles. Finally, the predicted speed is compared against the following ground truth speed function ^^∗via isotropic loss function to train the neural network. ^^∗^^^^ ൌ ^^^^ೞ^ௗ^ೌ^ ൈ clip^^^^^^,^^^^^^,^^^^^,^^^^௫^ (4)

[0064] The function ^^ provides the minimum distance from robot points of C-Space point ^^ to the obstacle ^^^^^, clipped between user-defined thresholds ^^^^^and ^^^^௫, and scaled by factor ^^^^^^௧. Note that the output of ^^∗is a scalar value and is based on the minimum distance of robot configuration to the obstacle. Once the neural network is trained, it provides the arrival time field between any given start and goal, leading to a path solution. QB\182506.00043\96318469.1Docket No.172842.00032 (70611) 16

[0065] New Time Field Factorization: Although the factorization in Equation 2 has desirable properties for motion planning, the arrival time ^^ increases steeply as ^^decreases to zero. Hence, when computing the gradients of Equation 1 to train the neural network, the sharp features around obstacles often lead to incorrect local minima. Therefore, in P-NTFields, a Laplacian term in Equation 3 and a speed scheduler was introduced. The former leads to a unique Eikonal equation solution, whereas the latter progressively reduces the speed around obstacles as the training epoch increases to prevent convergence to an incorrect solution. The Laplacian is computationally expensive to calculate as it requires the second derivation of the neural network, and the progressive speed scheduling requires a large number of training epochs. Both strategies make P- NTFields unsuitable when aiming for a fast mapping approach that learns to infer the arrival time field of the environment on the fly. Hence, a new factorization of the arrival time is introduced as described in the following. ^^^^^ ଶ^, ^^^^ ൌ log^^^^^^^, ^^^^^ ∥ ^^^ െ ^^^ ∥(5).

[0066] It can be seen that 1 / ^^ is replaced in Equation 2 with log^^^^ଶ. The latter is more flattened and changes less steeply as ^^ decreases, preventing sharp features nearobstacles. Shown in the experiments is that the new definition recovers accurate arrival time fields without needing Laplacian or speed scheduling. Furthermore, by solving the Eikonal equation (Equation 1) via chain rule with the new arrival time definition in Equation 5, the speed ^^ becomes as follows: ^^^^^ఛ^^ೞ,^^^ / ୪୭^^ఛ^^ೞ,^^^^ ^^ ൌ(6)

[0067] Neural Architecture: The neural architecture is inspired by spectral distance (SD) formulation . The SD ^^௪^^^^,^^^^ between two points ^^^and ^^^is an approximate alternative to the geodesic distance with the following form: ^^௪^^^^,^^^^ଶൌ ∑^^ୀ^ ^^^^^^^^^^^^^^^^ െ ^^^^^^^^^ଶ, (7) where ^^^^⋅^ and ^^^are the eigenfunctions andcollision-free space. The ^^^⋅^ is a weight function. In practice, ^^ smallest eigenvalues with correspondingeigenfunctions are used to estimate the spectral distance. Furthermore, the ^^ eigenfunctions can be seen as a feature vector for a given point ^^. Since the shortest arrival time solution of the Eikonal equation is also the geodesic distance, the neural time field network design was based on on the abovementioned SD formulation. QB\182506.00043\96318469.1Docket No.172842.00032 (70611) 17

[0068] The eigenfunctions of the Laplace operator are equivalent to the Fourier features of the given environment. To better capture the Fourier features, random Fourier positional encoding was combined with the SIREN neural network. Given the robot start ^^^ ∈ ^^^^^^ and goal ^^^ ∈ ^^^^^^ , their Fourier positional encoding are obtained as^^^^^ ^ ൌ ^cos^2^^^^்^^ ^, sin^2^^^ ்follows:^ ^ ^ ^^^^^^^^^^ ^ ൌ ^cos^2^^^^்^^ ^, sin^2^^ ்(8) where ^^ is a fixed latent ^^ ^^ ^^^^^,random code. These encodings are then passed through the SIREN neural network, denoted as ^^, which is a multilayer perceptron with sine activation functions. The sine activation function captures the high-frequency features and allows smooth differentiation for gradient computation in the method. Let the combination of positional and SIREN be defined as Φ^^^^ ൌ ^^^^^^^^^^ . Next, as inspired by Equation 7, thesefeatures are subtracted and a square is taken, i.e., Φ^^^^^ ⊗^Φ^^^^^ ൌ ^^^^^^^^^^^^ െ^^^^^^^^^^^^ଶ(9)

[0069] The above squared subtraction is also relevant to the symmetric operator introduced in NTFields. This operator enforces the arrival time field’s symmetric property, i.e., the arrival time from start to goal and from goal to start should be the same. Therefore, a symmetric operator between latent encodings of robot start and goal configurations was used. Specifically, let latent encodings be ^^ and ^^ for start and goal. The symmetric operator was defined as the concatenation of min and max, i.e., ^max^^^,^^^, min^^^, ^^^^. Hence, the squared subtraction operator in Equation 9, inspiredby spectral distance, also acts as a symmetric operator, ensuring the symmetric property of the arrival time field. Furthermore, it is observed that the squared subtraction operator instead of concatenation of min and max reduces the feature size, leading to neural network training efficiency. Finally, the resulting features from Equation 9 are given to another multi-layer perceptron, denoted as ^^, that predicts the ^^. In summary, the neural time field network is summarized as follows: ^^^^^^, ^^^^ ൌ ^^^Φ^^^^^ ⊗Φ^^^ ^ ^^(10).

[0070] Speed Inference: The neural framework mentioned above predicts ^^^^^^,^^^^ for the given start and goal pairs. The predicted ^^ and its gradients with respect to start∇^ೞ^^^^^^, ^^^^ and goal ∇^^^^^^^^, ^^^^ were used to estimate the speeds ^^^^^^^ and ^^^^^^^according to Equation 6. These predicted speeds at the start and goal are then utilized for training the network by comparing them against the expert speed model, as described herein. QB\182506.00043\96318469.1Docket No.172842.00032 (70611) 18

[0071] Path Inference: Once the neural time field network is trained, its predictions and gradients were utilized to recover the path solution between any start and goal as follows. First, the network predicts ^^^^^^, ^^^^ , and it was used to predict speeds and also toparameterize Equation 5 to determine the arrival time field ^^^^^^,^^^^. Next, the path solution in was recovered the same way as in NTFields, i.e., gradients were followed bidirectionally from start to goal and from goal to start as described below: ^^^ ← ^^^ െ ^^^^ଶ^^^^^∇^ೞ^^^^^^, ^^^^^^ ← ^^ െ ^^^^ଶ^^^ ^∇ ^^(11) ^^ ^ ^^ ^^^^,^^^^scaling factor. The above iterative procedure isand the goal are within the threshold. It should be emphasized that gradient descent does not ensure convergence, and the algorithm may terminate unsuccessfully after a predefined number of iterations.

[0073] Active Neural Time Fields: This section describes the approach to gathering data online in unknown environments for training the neural network on the fly. The prior work assumes the environment to be known, and therefore, the dataset for training the neural time field models is gathered offline. The dataset comprises randomly sampled start and goal points and their ground truth speed values. As described herein (Equation 4), the ground truth speed ^^∗is computed for each sampled point based on its minimum distance from the obstacles. The NTFields and P-NTFields can directly acquire this distance from a known environment by calculating the distance between the point and the nearest obstacle mesh. However, in the setting, the environment is unknown, and the robot needs to explore, obtain data, and determine its ground truth speed values to train the neural networks. Therefore, this section provides procedures to actively create such data and define their ground truth speed for online training of the neural networks.

[0074] Local Perception Processing: The local perception processing is inspired by the iSDF framework. The raw input of data comprises robot odometry and the sensor readings. It was assumed that sensor data to be either LiDAR-based scanning or depth images. The LiDAR provides scanning in the form of rays. If the perception is from depth images, pixels were sampled and converted to rays using the camera intrinsic matrix. Furthermore, these rays were transformed to world coordinates and randomly select ^^ number of rays from the given scan. For each ray, ^^ ∈ ℕ stratified points along weresampled its range. This means that each ray of length ^^ ∈ ℝ will be split into ^^ / ^^ bins,QB\182506.00043\96318469.1Docket No.172842.00032 (70611) 19 and in each bin, one sample is selected. Hence, ^^ ∈ ℕ sample points were gathered,denoted as ^^^^^ ⊂ ^^, from ^^ rays.

[0075] Next, the ground truth speed values for all sampled points was determined. In the setting, the actual minimum distance to the nearest obstacle is unknown since the environment will only be partially observed during exploration. This contrasts with prior methods where the environment mesh is provided, and the nearest distance to obstacles is readily available. Therefore, in the framework, approximating the minimum distance of sampled points to the obstacles was performed. The distance between the point and the nearest surface point in the given scan were calculated. Although using all LiDAR points or depth points can be more accurate for distance estimation, it is computationally expensive. Therefore, only ^^ sampled rays were used to select the nearest obstaclesurface. For 3D point robot, workspace sampled points ^^^^^are the C-Space points ^^^^^. However, in the case of a higher-dimension manipulator, a further operation is used to get C-Space samples and their expert speed values. Specifically, ^^^^^was used as the manipulator end effect positions and determine their configuration samples ^^^^^via inverse kinematic solution. Next, for these C-Space samples, the robot surface weredetermined to compute the minimum distance to the obstacle. Once the approximate minimum distance of all sampled points from the obstacle is available, the expert speed model was defined based on Equation 4. In summary, this module processes local perception, creating sample points ^^^^^and their expert speed values ^^^∗^^.

[0076] Online Training: Note that the data arrives in streams.a memory buffer, ℳ, was maintained that stores all sample points and their ground truth speed gathered over time. However, training the timefield neural network on the complete memory is computationally expensive. Therefore, a batch of points and their ground truth speed were randomly sampled from the memory. Let the batch be denoted as ℬ ⊂ ℳ,comprising sample points and their ground truth speed values. Next, the memory buffer was augmented, ℳ ൌ ℳ ∪ ^^^ ∗^^^, ^^^^^ ^, and the batch buffer, ℬ ൌ ℬ ∪ ^^^^^^, ^^^∗^^^ , with new data. The batchshuffed and pairs ofwere formed. These pairs act as start and goal points for training the neural model. The neural network trains on these pairs for a fixed number of iterations. Note that buffer ℬ contains samples from new and old perception data, and random shuffling allows start and goal pairs to spread across different parts of the observed environment. Hence, it allows neural QB\182506.00043\96318469.1Docket No.172842.00032 (70611) 20 networks to learn to generate the arrival time field of a fully observed environment over time.

[0077] The sampled start and goal pairs were passed through the neural time field network to obtain the ^^. The ^^ and its gradients parametrize Equation 6 to infer the speed ^^. Next, the loss of predicted speed against ground truth speed was computed as follows and was used to train the neural networks for a fixed number of iterations. ^^^^^∗, ^^^ ൌ ^^^^∗^^^^^ / ^^^^^ ଶ^^ െ 1^ ^(12) ^^^^∗^^^ ଶ^ ^^ 1^time was proposed, the loss was alsoloss with squared root and square makes the loss smoother and more flat than MSE and isotropic loss functions. It was observed that the new loss function aids in training the neural network faster than prior loss functions introduced by NTFields and P-NTFields.

[0079] Next Viewpoint Selection: Once the neural network is trained on the given local perception, the next viewpoint was selected to obtain the subsequent observation for learning. The active exploration selects the next best viewpoint to gather the data via the robot’s onboard sensor. Since the focus of this work is not on devising novel exploration strategies, a standard exploration approach was employed for maximizing the scene coverage. Two maps were considered: the visitation map and the occupancy map. The latter assigns a binary number 0 or 1 to each cell in the map. A cell gets a value of 0 if it belongs to a collision-free space; otherwise, it gets a value of 1. The former, the visitation map, tracks the cells that have been observed by the robot sensor. A cell can either be fully observed, partially observed, or unobserved. Initially, all cells in the visitation map are considered unobserved, and in the occupancy map as collision-free. As the robot explores, any cell viewed by the robot sensor within the pre-defined range ℎ ^ ^^^^௫ isconsidered observed. The cells beyond range ℎ until the sensor depth limit ^^^^௫ areconsidered partially observed. Furthermore, if the surface obstructs the ray, it is marked that cell as an obstacle point in the occupancy map. The range value h was defined lower than the sensor’s max limit to prune out far-distant points from local observation.

[0080] To select the next location for exploration, the occupancy and visitation map were employed as follows. First, the maps were split into regions comprising 4 ൈ 4 grid cells.In the visitation map, a region is considered partially observed if more than 25% of its cells are partially observed. Second, the arrival time to the center of all partially observed QB\182506.00043\96318469.1Docket No.172842.00032 (70611) 21 regions was computed using the neural model from the robot’s current location. A region with minimal arrival time is selected. Third, from the region chosen, the obstacle-free grid was selected by utilizing the occupancy map as the next location to visit for exploration. Finally, the path from the robot’s current location to the next location is calculated using the arrival time field by following the procedure described herein. Note that the neural time field generator can be tightly integrated into any other exploration strategy. Since the next best viewpoints are usually at the boundaries of explored and unexplored regions, the locally learned time fields can provide a path without needing any external path planner.

[0081] Performance Comparison: This section demonstrates the method performance over NTFields and P-NTFields. Fig. 5 shows the speed and time fields of the method, NTFields, and P-NTFields in comparison to FMM as ground truth. The color shows the speed fields and the contours show the arrival time fields from a start point. From Fig.5 contours, the approach and P-NTFields get a similar result as FMM, while NTFields generate incorrect time fields shown in the red box. To quantitatively compare the results of all PINN methods, points were sampled from their predicted arrival time and compute absolute error to the ground truth time field value given by the FMM. The time fields have an error of 0.044 േ 0.050 , P-NTFields have an error of 0.048 േ 0.047 , andNTFields have an error of 0.44 േ 0.42. Note that the NTField error is almost ten timeshigher. Regarding training times, the method takes 15 seconds, while P-NTFields and NTFields take 18 minutes and 10 minutes, respectively. While the method and P- NTFields have similar results, the latter is about 72 times faster than the former in terms of training speed. Hence, this performance comparison validates the method’s effectiveness and its suitability for online continual learning in mapping tasks.

[0082] Mapping Comparison: This section demonstrates the method performance on mapping tasks over KinectFusion, iSDF, and nvblox. Eight Gibson environments were used to demonstrate the 3D mapping of indoor scenes. Due to the three baselines requiring a given trajectory, the active exploration strategy was first run to obtain a series of depth images and camera poses. With these raw data, all methods reconstruct the maps of all indoor scenes. For LazyPRM, the roadmap was incrementally grown from local observations, and the average reconstruction time is 20 seconds. All other methods were compared over SDF quality and mapping efficiency, as the speed fields can be seen as truncated SDF. In Figs. 8A and 8B, two environments of SDF and their zero-level-set mesh are presented. The method, iSDF, and nvblox capture a similar result as the ground QB\182506.00043\96318469.1Docket No.172842.00032 (70611) 22 truth, while KinectFusion can only reconstruct the SDF very close to the obstacles as it cannot support large truncated regions. However, from the zero-level-set mesh, KinectFusion and nvblox can reconstruct high-quality obstacles, whereas the method and iSDF cannot capture many details. Note that the focus is on the downstream motion planning applications, and rough obstacle details are fine for collision avoidance, as also validated by the motion planning experiments in the next section. Table 1 shows the SDF error and reconstruction time. To compute the SDF error, points were sampled within the truncated region and calculate the absolute error of SDF value against ground truth results. In the table, it can be seen that the SDF error is similar to iSDF. Regarding mapping efficiency, although the method is slower than the baseline methods as it gathers informative features, the mapping time is comparable with baseline methods and suitable for online tasks.

[0083] Table 1: Quantitative comparison of the method against iSDF, Kinect Fusion (KFusion), and nvblox in mapping the Gibson environments. SDF Performance Metrics SDF Error Frame Time (s) Mapping Time (s) This method 0.09 ± 0.10 2.12 ± 0.06 149.74 ± 18.24 iSDF 0.09 ± 0.11 0.94 ± 0.02 69.90 ± 7.57 KFusion 0.47 ± 0.27 1.71 ± 0.01 126.08 ± 16.06 nvblox 0.07 ± 0.07 0.12 ± 0.00 9.16 ± 3.70

[0084] Motion Planning Comparison: This section demonstrates the method performance on motion planning tasks over MPOT, RRTConnect with smoothing, and LazyPRM with smoothing. Note that MPOT is the most recent and best available method that outperformed various state-of-the-art motion planning methods. Furthermore, the reconstructed maps of eight Gibson environments from the mapping results were used. Hence, two planners, MPOT and RRTConnect, use the reconstructed map from iSDF, KinectFusion, and nvblox. MPOT is a GPU-based method, and pm;y the grid map was loaded to GPU. RRTConnect is a CPU-based method, and the grid map was loaded on both GPU and CPU. LazyPRM uses its own reconstructed roadmap and uses graph search with smoothing to find the path. In contrast, the method can directly generate motion planning paths by following Equation 11, demonstrating the effectiveness of arrival time field mapping. QB\182506.00043\96318469.1Docket No.172842.00032 (70611) 23

[0085] 100 start and goal points in each environment were randomly chosen and the above-mentioned motion planning methods were run to get the final paths. The path length, computational time, and success rate were compared. Fig.9 shows the paths where the method and KinectFusion+MPOT generate short and smooth paths, but iSDF+MPOT and nvblox+MPOT generate damping results. RRTConnect only considers the SDF zero- level-set; thus, all RRTConnect generate similar results. Table 2 presents the statistical results. The method’s computational time is about 30 times faster than MPOT and achieves 98% success rate. In addition, the method generates smooth paths with a safe margin to obstacles, while RRTConnect generates paths close to the obstacle, resulting in relatively shorter lengths. Additionally, iSDF is slower due to its slower collision query through the neural network compared to a grid-based collision query. In summary, the results validate that the online reconstructed active NTFields achieve the best performance for motion planning.

[0086] Table 2: Comparison for Motion Planning in Eight Gibson Environment Methods Performance Metrics Time (s) Length SR (%) This method (G) 0.01±0.05 0.48±0.43 98.28 LazyPRM (C) 0.34±1.54 0.41±0.22 99.28 iSDF+MPOT (G) 1.42±0.52 0.62±0.27 91.43 KFusion+MPOT (G) 0.28±0.08 0.55±0.16 93.71 nvblox+MPOT (G) 0.27±0.05 0.58±0.14 96.57 iSDF+RRTConnect (G) 1.44±2.13 035±0.20 83.85 KFusion+RRTConnect (G) 0.89±1.36 0.38±0.21 91.57 nvblox+RRTConnect (G) 0.51±0.99 0.39±0.21 88.71 KFusion+RRTConnect (C) 0.20±0.48 0.40±0.22 94.71 nvblox+RRTConnect (C) 0.27±0.46 0.40±0.22 96.29

[0087] Real-Word Experiments - TurtleBot4 in indoor environment: In this experiment, a TurtleBot4 robot equipped with a RealSense2 camera is used to explore an indoor environment with furniture. The camera poses are from the TurtleBot4 odometry, while depth images are from the RealSense2 camera. Fig. 10 shows the method of incrementally constructing the map with incoming frames. Furthermore, this method uses the reconstructed arrival time field in the partially observed environment to reach the next QB\182506.00043\96318469.1Docket No.172842.00032 (70611) 24 viewpoint for sensing. Thus, it does not require any external motion planner during mapping. In Fig. 10, the color represents the speed fields, and the contour lines indicate the time fields. It takes 65 seconds for the robot to actively explore and reconstruct the arrival time field of the whole environment. These results highlight the capability of this approach in mapping a real-world indoor environment.

[0088] Once the map reconstruction is done, any start and goal pair in the map can be fed into the neural network, and a path is generated in a bidirectional way using Eq. 11. To evaluate this method of motion planning performance, 100 starts and goals were randomly sampled in the real environment. On average, this method computation times remained around 0.02 seconds with a 98% success rate. Finally, the bottom row of Fig. 10 depicts an example path. The start location was chosen near the sofa and the goal location was chosen behind a chair. It takes only 0.02 seconds to generate a valid smooth trajectory, avoiding collision with all the furniture.

[0089] Fig. 11 illustrates a real-world cabinet environment. This method reconstructed the arrival time fields using the in-hand camera. The top and bottom rows show two cases of tje motion planning problems in this confined environment. The first case shows the manipulator going into the cabinet, and the second case shows the manipulator crossing the cabinet’s middle level to reach the given target at the top level. The end-effector was highlighted to show the narrow passage condition of Case 2. In this scenario, this method takes only 0.02 s to find the path, whereas LazyPRM (CPU) takes 3.72 s, KinectFusion+RRTConnect (CPU) takes 4.15 s, and nvblox+MPOT (GPU) takes 4.63 s.

[0090] UR5e Manipulator in cabinet environment: The UR5e robotic arm was used with a hand-held RealSense2 camera to navigate a realistic cabinet setting. This approach generates an arrival time field map in 6 DOF C-Space and completes this process in 115 seconds. By comparison, alternative workspace maps reconstruction methods like iSDF, KinectFusion, and nvblox require significantly less time – 6 seconds, 2.47 seconds, and 0.55 seconds, respectively. Note that the time difference is primarily because this method maps the higher-dimensional (6D) C-Space, whereas baseline methods map the 3D workspace. The baseline SDF methods cannot scale to C-Space mapping. Furthermore, the LazyPRM constructed from incremental local observations takes 162 seconds to build the roadmap in C-Space and is slower than this method. For motion planning, the reconstructed map was used and tests were conducted using 100 randomly selected start and goal pairings near obstacles. This method demonstrated a QB\182506.00043\96318469.1Docket No.172842.00032 (70611) 25 quick average planning time of 0.03 seconds with a 91% success rate, as detailed in Table 3, achieving a computational time approximately 40 times faster than that of LazyPRM. Fig. 11 showcases two instances of motion planning within this environment. The first scenario illustrates the manipulator initiating movement from an open area and entering the cabinet, while the second shows the manipulator navigating across the cabinetΓÇÖs levels to reach a specified target. Notably, the manipulator begins from a confined position deep within the cabinet in Case 2. Furthermore, in this particular scenario (case 2), this method was able to find a solution in just 0.02 seconds, whereas LazyPRM (CPU) tool 3.72 seconds, KinectFusion+RRTConnect (CPU) takes 4.15 seconds, and nvblox+MPOT (GPU) takes 4.63 seconds. These experiments demonstrate the scalability of this approach to high-dimensional C-Space and real-world confined environments.

[0091] Table 3: Comparison for motion planning in the UR5E manipulator in cabinet environment with 100 randomly sampled start and goals near obstacles. Methods Performance Metrics Time (s) Length SR (%) This method (G) 0.03±0.01 2.25±0.72 91.00 LazyPRM (C) 1.36±0.81 3.05±0.89 87.00 iSDF+MPOT (G) 3.63±1.29 4.04±0.77 89.00 KFusion+MPOT (G) 2.18±0.06 3.99±0.48 79.00 nvblox+MPOT (G) 2.11±0.39 4.15±0.56 88.00 iSDF+RRTConnect (G) 3.92±8.20 2.33±1.04 89.00 KFusion+RRTConnect (G) 2.71±4.56 2.18±0.83 85.00 nvblox+RRTConnect (G) 3.16±3.53 2.30±1.09 84.00 KFusion+RRTConnect (C) 1.64±1.35 2.48±1.21 82.00 nvblox+RRTConnect (C) 1.90±5.63 2.32±0.96 82.00

[0092] In the foregoing specification, implementations of the disclosure have been described with reference to specific example implementations thereof. It will be evident that various modifications may be made thereto without departing from the broader spirit and scope of implementations of the disclosure as set forth in the following claims. The specification and drawings are, accordingly, to be regarded in an illustrative sense rather than a restrictive sense. Practical Applications and Example Embodiments QB\182506.00043\96318469.1Docket No.172842.00032 (70611) 26

[0093] In some examples, moveable devices (such as a robotic arm, an automated vehicle, etc.) can be configured to perform the processes described by FIGS. 1-3. A moveable device may be coupled to and / or in electrical communication with one or more cameras or other sensing devices. In one example, a robotic arm may use the above-described systems and methods of mapping unknown environments and / or motion planning. In particular, the robotic arm may be encountering new scenarios of obstacles in its operating environment. In another example, an autonomously-moving robot or drone may use the above-described systems and methods to navigate various environments. For example, a robot or drone may perform emergency operations using these systems and methods to navigate a building that is on fire. In this example, obstacles may include physical objects, as well as areas reaching or exceeding specified temperatures. In another example, an autonomous delivery vehicle may utilize the above-described systems and methods of mapping and motion planning while performing delivery operations in an outdoor environment, where the scene may be actively changing. Other practical applications can include robotic arms attached to moving robots and / or drones and other devices operating in environments related to manufacturing, construction, service, emergency, etc.

[0094] The presently disclosed subject matter is further described in the paper appended hereto as Appendix, titled “Physics-informed Neural Mapping and Motion Planning in Unknown Environments”. The paper attached is intended to provide examples of various embodiments, but not to limit the scope of the inventions described herein or therein. QB\182506.00043\96318469.1

Claims

Docket No.172842.00032 (70611) 27 CLAIMS WHAT IS CLAIMED IS:

1. A system for autonomously controlling a robot in an unknown environment, the system comprising: a robotic device having at least one drive mechanism for moving at least a part of the robot; a processor in communication with the robotic device; and a memory in communication with the processor and having instructions stored thereon that, when executed, cause the processor to: obtain a plurality of local sensor data of the unknown environment from a first viewpoint; process the plurality of local sensor data to create a plurality of sample points and a plurality of speed values; train a neural network based on a pairing of a first sample point from the plurality of sample points and a first speed value from the plurality of speed values.

2. The system of claim 1, wherein the instructions stored on the memory further cause the processor to: determine a second viewpoint, different from the first viewpoint, based on the neural network; and determine if the second viewpoint achieves an exploration threshold for the unknown environment.

3. The system of claim 2, wherein the instructions stored on the memory further cause the processor to: receive a plurality of exploration threshold comprising a minimum distance from one or more detected obstacles, wherein the second viewpoint is further determined based on plurality of exploration thresholds.

4. The system of claim 1, wherein the instructions stored on the memory further cause the processor to: QB\182506.00043\96318469.1Docket No.172842.00032 (70611) 28 generate a map and an arrival field of the unknown environment based on the plurality of local sensor data; and store the arrival field for the unknown environment based on the plurality of local sensor data.

5. The system of claim 4, wherein the instructions stored on the memory further cause the processor to engage the robotic device to move the robot according to the map and the arrival field of the unknown environment.

6. The system of claim 4, wherein the map for the unknown environment is further generated based on an Eikonal equation.

7. The system of claim 4, wherein the map for the unknown environment is generated without a plurality of trajectories. is further obtained using an Eikonal equation. wherein the arrival field map is further generated using an Eikonal equation 19. The method of claim 15, wherein the map is generated without a plurality of trajectories. QB\182506.00043\96318469.1

Images