System, apparatus, and method for automated elevator navigation using robotic devices

Abstract

Systems, apparatus, and methods for robot learning, planning, and skill execution, including elevator navigation, are described. In one or more embodiments, the robotic device may need to autonomously utilize the elevator, for example, by using its autonomy features and / or by relating to an application programming interface (API) to communicate with the elevator.

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims the priority and benefit of U.S. Application No. 63 / 465,779, filed on May 11, 2023, and U.S. Application No. 63 / 465,791, filed on May 11, 2023, the entire contents of which are incorporated herein by reference.

[0002] The present disclosure generally relates to the fields of robot learning, planning, and skill execution. In particular, the present disclosure relates to methods and apparatuses for automated elevator navigation of robotic devices.

Background Art

[0003] Robots can be used to perform and automate various tasks. Robots can perform tasks by moving through environments such as buildings or hospitals. Robots can be equipped with wheels, tracks, or other moving components that enable the robot to move autonomously within the environment. Some robots are trained to move through a single plane within the environment. Additionally, some robots do not have the ability to move through stairs and access different floors of multi - story buildings. Some robots have limited ability to operate an elevator system, interpret changing altitudes, interact with the elevator system, or maneuver around obstacles within the elevator system to reach a destination located on a different floor than the robot's starting position.

[0004] Furthermore, most commercial robots are designed to operate in structured environments, such as factories and warehouses. Unstructured environments, such as hospitals and homes, which involve humans, can present additional challenges for programming robots. In unstructured environments, robots cannot rely on complete knowledge of their surroundings, but they must be able to perceive changes in their surroundings and adapt based on those changes. Therefore, in unstructured environments, robots need to continuously or repeatedly acquire information about their environment in order to make autonomous decisions and perform tasks. Often, the movement of a robot, for example, the movement of an arm or end effector in an environment, is also constrained by objects and other obstacles in the environment, which further adds to the challenges of robot perception and manipulation. Given the uncertain and dynamic nature of unstructured environments, robots typically cannot be pre-programmed to perform tasks.

[0005] Therefore, there is a need for robotic systems that can perceive and adapt to dynamic and unstructured environments, and perform advanced tasks within those environments without relying on pre-programmed operational skills. [Overview of the Initiative]

[0006] Systems, devices, and methods for robot learning, planning, and skill execution, including elevator navigation, are described. In one or more embodiments, a robotic device may need to autonomously utilize an elevator, for example, by using its autonomy features and / or by relating to an application programming interface (API) to communicate with the elevator. In some embodiments, a robotic device may navigate using an elevator by implementing one or more of the following: elevator lobby management, elevator call button operation, elevator entry, passenger management, floor button operation, floor detection, and elevator exit. In some embodiments, a robotic device may implement intention or current state through social cues. In some embodiments, a robotic device may implement data recording, for example, for additional training and / or planning, and / or to report information to one or more users.

[0007] In some embodiments, the device comprises a memory and a processor, the processor being operably coupled to the memory and executing instructions stored in the memory to retrieve information relating to an elevator car, including the floor plan of the elevator car and the configuration of the elevator button interface within the elevator car; determining, for each value of a first parameter from a plurality of parameters, the probability of successfully pressing a target button over a range of values ​​of one or more second parameters from a plurality of parameters other than the first parameter, by simulating pressing the target button over a plurality of trials at that value of the first parameter and over a range of values ​​of one or more second parameters, wherein the plurality of parameters include the location of the target button, the position of a robotic device, and the pose of the operating elements of the robotic device; generating a reachability map indicating the probability of successfully pressing the target button over a range of values ​​of one or more second parameters based on the results of simulating pressing over a plurality of trials for each value of the first parameter; and storing the reachability map for each value of the first parameter in memory.

[0008] In some embodiments, the device comprises an operating element including a plurality of joints and segments, configured to be positioned in a plurality of poses to initiate a trajectory for pressing one or more target buttons; a transport element configured to move along a surface; one or more sensors; and a processor operably coupled to the operating element, the transport element, and the one or more sensors, wherein the processor is configured to determine the location of the target buttons in the elevator car; determine which pose out of the plurality of poses the operating element is positioned in; use a reachability map associated with the poses to select a position of the transport element that is likely to successfully press the target buttons, given the location of the target buttons; and navigate to the position of the transport element along the path.

[0009] In some embodiments, the device comprises an operating element including a plurality of joints and segments, a transport element configured to move along a surface, one or more sensors, and a processor operably coupled to the operating element, the transport element, and one or more sensors, wherein the processor is configured to: determine the location of a target button in an elevator car; compare the location of the target button with a plurality of predefined locations of the target button; and, in response to determining, based on the comparison, that the location of the target button does not match any of the plurality of predefined locations, identify a set of predefined locations that are closest to the location of the target button from the plurality of predefined locations; and determine a reachability map indicating the possibility of pressing the target button at the location of the target button and over a range of predefined locations of the transport element, based on a set of reachability maps indicating the possibility of pressing the target button within the set of predefined locations and over a range of predefined locations of the transport element.

[0010] In some embodiments, the method includes: calling at least one elevator car to arrive at the elevator lobby in a robotic device; navigating to a first location in the elevator lobby to await the arrival of at least one elevator car using the transport elements of the robotic device; determining in the robotic device that an elevator car is approaching the elevator lobby; navigating to a second location to board the approaching elevator car using the transport elements; monitoring the state of the elevator car using one or more sensors of the robotic device to determine when to board the elevator car; boarding the elevator car by navigating from the second location to a third location in the elevator car in response to the determination to board the elevator car; and selecting a destination floor by pressing a target button in the elevator car. [Brief explanation of the drawing]

[0011] [Figure 1]This is a block diagram illustrating a system including a robotic device in several embodiments. [Figure 2] This is a block diagram illustrating the configuration of a robotic device according to several embodiments. [Figure 3] This is a block diagram illustrating the configuration of a control unit associated with a robotic device according to several embodiments. [Figure 4] This is a schematic diagram illustrating several embodiments of a robotic device. [Figure 5] This is a schematic diagram illustrating several embodiments of a robotic device. [Figure 6] This is a schematic example diagram of layers of environment maps generated by a robotic device, according to several embodiments. [Figure 7] This flowchart illustrates, in several embodiments, a method involving navigation using an elevator by a robotic device. [Figure 8] This flowchart illustrates, in several embodiments, a method for managing behavior and / or actions within an elevator lobby using a robotic device. [Figure 9] This flowchart illustrates, in several embodiments, how a robotic device can be used to board an elevator. [Figure 10] This flowchart illustrates, in several embodiments, a method for managing the behavior and / or actions of a person riding an elevator using a robotic device. [Figure 11] This flowchart illustrates a method for monitoring the relative location of a robotic device to one or more floors, according to several embodiments. [Figure 12] This flowchart illustrates a method for exiting an elevator using a robotic device, according to several embodiments. [Figure 13] This flowchart illustrates a method of preemption or replanning using a robotic device, according to several embodiments. [Figure 14]An exemplary representation of an elevator map interpreted by a robot device according to some embodiments. [Figure 15] Depicts a state machine flow for elevator navigation by a robot device according to some embodiments. [Figure 16] Depicts a behavior tree for lobby management by a robot device according to some embodiments. [Figure 17] Schematically illustrates beam classification by a robot device according to some embodiments. [Figure 18] Depicts a decision tree for determining the elevator door state by a robot device according to some embodiments. [Figure 19] Illustrates an exemplary pipeline for acoustic signal classification using a convolutional neural network (CNN) model according to some embodiments. [Figure 20A] Provides a comparison of accuracy, validation loss, and error for multiple different algorithms used to process acoustics from an elevator according to some embodiments. [Figure 20B] Provides a comparison of accuracy, validation loss, and error for multiple different algorithms used to process acoustics from an elevator according to some embodiments. [Figure 20C] Provides a comparison of accuracy, validation loss, and error for multiple different algorithms used to process acoustics from an elevator according to some embodiments. [Figure 21A] Schematically depicts obstacles, fiducials, and other elements within an elevator scene as seen by a robot device according to some embodiments. [Figure 21B] Schematically depicts obstacles, fiducials, and other elements within an elevator scene as seen by a robot device according to some embodiments. [Figure 22]An example of a reachability map depicting the probability of a robot device successfully pressing an elevator button in a grid at the base position inside an elevator car according to an embodiment. [Figure 23] A set of reachability maps corresponding to each elevator button and elevator tack pose of an elevator car having different layouts according to an embodiment is depicted. [Figure 24] A subset of the reachability map that can be used by a robot device to select an elevator tack pose that can enable the robot device to successfully press a specific button inside an elevator car according to an embodiment is shown. [Figure 25A] A robot device in three exemplary elevator tack poses according to an embodiment is depicted. [Figure 25B] A robot device in three exemplary elevator tack poses according to an embodiment is depicted. [Figure 25C] A robot device in three exemplary elevator tack poses according to an embodiment is depicted. [Figure 26A] A top view of a robot device pressing an elevator button with the base of the robot device positioned at orientation theta according to an embodiment is depicted. [Figure 26B] Reachability maps generated when the base of the robot device is positioned at one orientation and multiple orientations respectively according to an embodiment are shown. [Figure 27A] A reachability map depicting coordinates where the reachability map can be cropped to reduce the computational load of the robot device according to an embodiment is shown. [Figure 27B] A reachability map depicting coordinates where the reachability map can be cropped to reduce the computational load of the robot device according to an embodiment is shown. [Figure 28] An example of the determination of a reachability map when a button location different from the sampled button location is given according to an embodiment is depicted. [Figure 29A] This flowchart illustrates a decision tree for determining the tuck pose and base position for elevator operation, according to an embodiment. [Figure 29B] This flowchart illustrates a decision tree for determining the tuck pose and base position for elevator operation, according to an embodiment. [Figure 30] This flowchart illustrates a method for determining elevator tack pose and base position according to one embodiment. [Figure 31] This flowchart illustrates a method for determining tuck pose and base position using a neural network model, according to one embodiment. [Figure 32A] This flowchart illustrates a method for navigating using an elevator with a robotic device, according to one embodiment. [Figure 32B] This flowchart illustrates a method for navigating using an elevator with a robotic device, according to one embodiment. [Figure 33] This is a schematic block diagram of a system for operating an elevator car using an intermediate server, according to one embodiment. [Figure 34] This is a flowchart illustrating an example of a floor detection training method for a robotic device, according to several embodiments. [Modes for carrying out the invention]

[0012] This specification describes systems, apparatus, and methods for robot learning, planning, and skill execution. In some embodiments, the systems, apparatus, and methods described herein relate to robotic devices capable of learning skills through human demonstration, exploration, planning, and interaction, and performing the learned skills in an unstructured environment. In some embodiments, the systems, apparatus, and methods described herein relate to robotic devices capable of learning, planning, and performing skills, including autonomous navigation using an elevator. overview

[0013] In some embodiments, the systems, apparatus, and methods described herein relate to a robot (also referred to herein as a “robot device”) capable of learning and performing skills (e.g., operational skills) for navigating using an elevator. For example, the robot can acquire and perform operational skills using machine learning techniques. After learning the skills, the robot can plan and / or perform the skills in different environments. The robot can learn and / or perform skills based on sensor data (e.g., perceived visual information, torque or force data, etc.) and / or data from external sources (e.g., hospital databases, elevator management software, etc.).

[0014] In some embodiments, robots may be designed to interact with and collaborate with humans to perform tasks. In some embodiments, robots may use common social behaviors to behave in a socially predictable and acceptable manner around humans. Robots may also use common social behaviors to execute predefined sound files to request assistance (e.g., to help the robot, to warn people to stay away from the robot). Mobile robots may also be designed to navigate within an environment while interacting with humans within that environment. For example, a robot may be programmed to utter specific phrases to navigate around humans, move aside to allow humans to pass, and communicate intentions using gaze while navigating. In some embodiments, robots may have sensors that allow the robot to perceive and track humans in its surrounding environment and use that information to trigger gaze and other social behaviors. Robots may also be programmed to utter phrases to instruct people to open doors, press elevator buttons, etc.

[0015] In some embodiments, a robotic device may be capable of learning and / or performing skills in unstructured environments, such as dynamic environments and / or human environments, where the robotic device does not have complete information about the environment in advance. Unstructured environments may include, for example, indoor and outdoor settings and may include one or more people or other objects that can move within the environment. Since most natural or real-world environments are unstructured, robotic devices that can adapt and operate in unstructured environments, such as the robotic devices and / or systems described herein, can offer a significant improvement over existing robotic devices that are unable to adapt to unstructured environments. Unstructured environments may include indoor settings (e.g., buildings, offices, houses, elevators, rooms, etc.) and / or other types of enclosed spaces (e.g., airplanes, trains, and / or other types of mobile compartments), as well as outdoor settings (e.g., parks, beaches, outdoor gardens, fields). In one embodiment, the robotic device described herein may operate in an unstructured hospital environment.

[0016] In some embodiments, robot learning can be performed in a factory before robot deployment or in the field after robot deployment (e.g., in a hospital). In some embodiments, the robot may be taught and / or adapted to operate in a given environment by a user who has not received training in robotics and / or programming. For example, the robot may have a learning algorithm that leverages natural human behavior and may include tools that can guide the user through a demonstration process.

[0017] Examples of learning, planning, and execution by robotic devices are described in U.S. Patent Application Publication No. 2021 / 0379758, filed on March 1, 2021, entitled "Systems, apparatuses, and methods for robotic learning and execution of skills," and in International Patent Application Publication PCT / US2022 / 015710, filed on February 8, 2022, entitled "Systems, apparatuses, and methods for robotic learning and execution of skills including navigation and manipulation functions," the disclosures of which are incorporated herein by reference.

[0018] In some embodiments, a robotic device may be configured to navigate a multi-floor building using an elevator. The robot may be trained to navigate in and out of the elevator to move to destinations on different floors. The robot may be trained to identify and determine the floor on which it is currently located and to continue navigation using a floor plan for that floor. For example, the robot may be configured to determine the current floor or a nearby floor while it is in the elevator. This allows the robot to determine when to exit the elevator. The robot may be trained to navigate around or through different types of elevators, different button panels, different elevator cars (also referred to herein as "elevators"), different elevator lobbies (also referred to herein as "elevator bays"). The robot may be trained to navigate around or through elevators based on the number of occupants (e.g., doctors, patients, nurses, etc.) and obstacles (e.g., hospital beds, wheelchairs, etc.). The robot may be trained to enter and exit elevators quickly without becoming immobile. For example, a robot can manipulate the orientation of its arm or body when navigating around obstacles or positioning itself within an elevator. The robot can also utter phrases requesting assistance when obstacles obstruct its path.

[0019] Figure 1 is a high-level block diagram illustrating system 100 in several embodiments. System 100 may include one or more robotic devices 110. The robotic devices 110 may be communicably coupled to one or more databases 106, servers 104, and / or other devices 108 via a network 102.

[0020] Network 102 can be any type of network (e.g., local area network (LAN), wide area network (WAN), virtual network, telecommunications network) implemented as a wired / or wireless network and used to operably connect computing devices including robot device 110, database 106, server 104, and other devices 108. As will be described in more detail herein, in some embodiments, for example, computing devices are connected to each other via an Internet Service Provider (ISP) and the Internet (e.g., network 102). In some embodiments, a connection can be defined between any two computing devices via network 102. As shown in Figure 1, for example, a connection can be defined between robot device 110 and any one of database 106, server 104, and / or other devices 108. In some embodiments, computing devices can communicate with each other (e.g., send data to each other and / or receive data from each other) and with network 102 via an intermediate network and / or alternative network (not shown in Figure 1). Such intermediate and / or alternative networks may be of the same and / or different types as network 102. Each computing device may be any type of device configured to transmit data on network 102 in order to send data to and / or receive data from one or more other computing devices.

[0021] The robotic device 110 may be configured to perceive information about the environment, learn skills, plan and / or execute trajectories, etc. The robotic device 110 may be configured to interact with the environment and / or learn environmental constraints through human demonstration and input, exploration, and / or execute those skills within the environment. In some embodiments, the robotic device 110 may use an elevator to learn, plan, and / or execute skills associated with autonomous navigation. More detailed illustrations of exemplary robotic devices are depicted and described with reference to Figures 2 and 3.

[0022] The robotic device 110 can transmit data to and / or receive data from other robotic devices via the network 102. For example, the robotic device 110 can transmit information it perceives about the environment (e.g., the location of objects) to other robotic devices and receive information about the environment from other robotic devices. The robotic devices 110 can also transmit and / or receive information from each other in order to learn and / or perform skills. For example, the robotic device 110 can learn a skill in an environment and transmit a model representing that learned skill to other robotic devices, which, upon receiving the model, can use to perform the skill in the same or a different environment. The robotic device 110 can be in the same or a different location from other robotic devices. For example, the robotic device 110 and other robotic devices can be located in the same room of a building (e.g., a hospital building) so that they can learn and / or perform skills together (e.g., moving heavy or large objects). Alternatively, robotic device 110 may be located on a first floor of a building (e.g., a hospital building), and other robotic devices may be located on a second floor of the building, and they can communicate with each other to relay information about the different floors (e.g., where objects are located on those floors, where resources may exist, etc.). In some embodiments, robotic device 110 located at a first location or site (e.g., a first hospital) may learn to navigate (e.g., using an elevator) and transmit models, plans, and / or other information associated with navigation to other computing devices (e.g., a server 104) and / or other robotic devices located at that location or other locations.

[0023] In some embodiments, the server 104 may be a dedicated server for managing the robotic device 110 and / or other devices 108. The server 104 may be located in the same or different location as the robotic device 110 and / or other devices 108. For example, the server 104 may be located in the same building as one or more robotic devices (e.g., a hospital building) and managed by a local administrator (e.g., a hospital administrator). Alternatively, the server 104 may be located in a remote location (e.g., a location associated with the manufacturer or provider of the robotic devices). In some embodiments, the robotic device 110 may be configured to capture data and / or learn or perform skills, and can transmit this data, skills, etc., to the server 104, allowing the server 104 to transfer it from the robotic device 110 to other devices 108, such as other robotic devices located remotely. In other words, the robotic device 110 may be trained with training data in one or more elevators in a particular building / hospital, and this data and / or training can be passed to other robotic devices in the same building or site or in different buildings or sites.

[0024] The database 106 can store information that can be made accessible to the robotic device 110. In some embodiments, the database 106 can be a hard drive, a database, cloud storage, a network-attached storage device, or other data storage device. In some embodiments, the database 106 can store sensor data, trained models, marker location information, etc., which include information about one or more components of the robotic device 110. In some embodiments, the database 106 can store information about patients, hospital staff, hospital supplies and / or equipment, etc.

[0025] In some embodiments, the system 100 may include other devices 108 that can be configured to operate and / or perform specific functions. In a hospital setting, for example, the other devices 108 may be diagnostic and / or therapeutic devices that connect to a network 102 and can communicate with other computing devices, including robotic devices 110 and / or a database 106. In some embodiments, the other devices 108 may include computing devices operated by an administrator to control and / or provide guidance to one or more robotic devices 110.

[0026] Any computing device as described herein (e.g., robot device 110, database 106, server 104, and other devices 108) may include a user interface that allows a user (e.g., a nearby user or robot supervisor) to control the operation of robot device 110. For example, a user may interrupt and / or modify the execution of one or more actions performed by robot device 110. These actions may include, for example, navigation behavior, maneuvering behavior, head behavior, sound / light, and / or other components of robot device 110. In some embodiments, a robot supervisor may remotely monitor and control robot devices 110 for safety reasons. For example, a robot supervisor may command robot device 110 to stop or modify the execution of an action to avoid endangering a human or causing damage to robot device 110 or another object in the environment. In some embodiments, a nearby user may guide or demonstrate an action to robot device 110. In some embodiments, robot device 110 may be configured to solicit user intervention at specific points in time during the execution of an action. For example, the robot device 110 may request user intervention at times such as when the robot device 110 cannot determine certain information about itself and / or the environment around it, when the robot device cannot determine a trajectory to complete an action or navigate to a particular location, or when the robot device 110 is pre-programmed to request user input (e.g., during learning using an interactive learning template, as further described below). In some embodiments, the robot device may request feedback from the user at specific points in time during learning and / or execution. For example, the robot device 110 may prompt the user to specify the information that the robot device should collect (e.g., information associated with an operating or transporting element, information associated with the surrounding environment) and / or when to collect the information (e.g., the timing of keyframes during demonstration).Alternatively or additionally, the robot device 110 may require the user to tag past or present behavior of the robot device in a particular context as positive or negative examples of the behavior. The robot device 110 may be configured to use this information to improve future performance of the behavior in a particular context.

[0027] Figure 2 schematically illustrates a robotic device 210 according to several embodiments. The robotic device 210 includes a controller 212, a user interface 240, at least one operating element 250, and at least one sensor 270. Additionally, in some embodiments, the robotic device 210 optionally includes at least one transport element 260. The controller 212 includes a memory 220, a processor 214, a system bus 216, and at least one input / output interface ("I / O interface") 208. The memory 220 can be, for example, random access memory (RAM), a memory buffer, a hard drive, a database, erasable programmable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), read-only memory (ROM), etc. In some embodiments, memory 220 stores instructions that cause the processor 214 to execute modules, processes, and / or functions associated with sensing or scanning the environment, learning skills, and / or performing skills.

[0028] The processor 214 of the controller 212 can be any suitable processing device configured to operate and / or perform functions associated with observing the environment, learning skills, and / or performing skills. For example, the processor 214 may be configured to perform skills by generating a model of skills based on sensor information, or by using the model to generate a trajectory for performing the skills, as further described herein. More specifically, the processor 214 may be configured to perform modules, functions, and / or processes. In some embodiments, the processor 214 can be a general-purpose processor, a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a graphics processing unit (GPU), and the like.

[0029] The system bus 216 can be any suitable component that enables the processor 214, memory 220, storage 230, and / or other components of the controller 212 to communicate with each other. The I / O interface 208 connected to the system bus 216 can be any suitable component that enables communication between the internal components of the controller 212 (e.g., the processor 214, memory 220, storage 230) and external input / output devices such as the user interface 240, operating elements 250, transport elements 260, and sensors 270.

[0030] The user interface 240 may include one or more components configured to receive inputs and send outputs to other devices and / or users who operate those devices, for example, a user who operates the robotic device 210. For example, the user interface 240 may include a display device 242 (e.g., a display, touchscreen, etc.), an acoustic device 244 (e.g., a microphone or speaker), and optionally, one or more additional input / output devices ("I / O devices") 246 configured to receive inputs and / or generate outputs to the user.

[0031] The operating element 250 can be any suitable component capable of manipulating and / or interacting with stationary and / or moving objects, including, for example, a human being. In some embodiments, the operating element 250 may include a plurality of segments connected to one another via joints that can provide translation along one or more axes and / or rotation about one or more axes. The operating element 250 may optionally include an end effector that can engage with and / or otherwise interact with objects in the environment. For example, the operating element may include a gripping mechanism that can releasably engage (e.g., grip) with an object in the environment for lifting and / or transporting the object. Other examples of end effectors include, for example, a vacuum engagement mechanism, a magnetic engagement mechanism, a suction mechanism, and / or a combination thereof. In some embodiments, one or more operating elements 250 may be retractable into the housing of the robotic device 210 when not in use in order to reduce one or more dimensions of the robotic device. In some embodiments, the operating element 250 may include a head or other humanoid component configured to interact with the environment and / or one or more objects (including a human being) in the environment. In some embodiments, the operating element 250 may include a transport element or base (e.g., transport element 260). A detailed diagram of an exemplary operating element is shown in Figure 4.

[0032] The transport element 260 can be any suitable component configured for movement, such as wheels or tracks. One or more transport elements 260 may be provided on the base portion of the robotic device 210 to enable the robotic device 210 to move around an environment. For example, the robotic device 210 may include multiple wheels that enable the robotic device to navigate around a building, such as a hospital. The transport elements 260 may be designed and / or dimensionally determined to facilitate movement through narrow and / or constrained spaces (e.g., small corridors and hallways, small rooms such as storage rooms). In some embodiments, the transport elements 260 may be rotatable around an axis and / or movable relative to each other (e.g., along tracks). In some embodiments, one or more transport elements 260 may be retractable into the base of the robotic device 210 when not in use to reduce one or more dimensions of the robotic device. In some embodiments, the transport elements 260 may be an operating element (e.g., an operating element 250) or form part thereof.

[0033] The sensor 270 can be any preferred component that enables the robot device 210 to capture information about the environment around the robot device 210 and / or objects in the environment. The sensor 270 can include, for example, an image capture device (e.g., a camera such as a red-green-blue-depth (RGB-D) camera or a webcam), an acoustic device (e.g., a microphone), an optical sensor (e.g., a light-detecting and ranging or lidar sensor, a color-detecting sensor), a proprioceptive sensor, a position sensor, a tactile sensor, a force or torque sensor, a temperature sensor, a pressure sensor, a motion sensor, a sound detector, and the like. For example, the sensor 270 can include at least one image capture device, such as a camera, for capturing visual information about objects and the environment around the robot device 210. In some embodiments, the sensor 270 can include a tactile sensor, such as a sensor that can transmit force, vibration, touch, and other non-visual information to the robot device 210.

[0034] In some embodiments, the robotic device 210 may have humanoid features, such as a head, body, arms, legs, and / or a base. For example, the robotic device 210 may include a face having eyes, a nose, a mouth, and other humanoid features. These humanoid features may form and / or be part of one or more operating elements. Although not schematically described, the robotic device 210 may also include actuators, motors, couplers, connectors, power supplies (e.g., onboard batteries), and / or other components that link, actuate, and / or drive different parts of the robotic device 210.

[0035] Figure 3 is a block diagram illustrating a controller 312 in a schematic manner according to several embodiments. The controller 312 may include components similar to those of the controller 212 and may be structurally and / or functionally similar to the controller 212. For example, the controller 312 includes a processor 314, memory 320, I / O interface 308, and system bus 316, which may be structurally and / or functionally similar to the processor 214, memory 220, I / O interface 208, and system bus 216, respectively. The controller 312 may be located on a robotic device and / or on a remote server connected to one or more robotic devices.

[0036] Memory 320 stores instructions that can cause the processor 314 to execute modules, processes, and / or functions, which are optionally exemplified as active sensing 322, planning 325, execution / decision-making 326, and perception / reasoning 328, as well as learning 324. Active sensing 322, learning 324, planning 325, execution / decision-making 326, and perception / reasoning 328 may be implemented as one or more programs and / or applications tied to hardware components (e.g., sensors, operating elements, I / O devices, processors, etc.). Active sensing 322, learning 324, planning 325, execution / decision-making 326, and perception / reasoning 328 may be implemented by one or more robotic devices. For example, a robotic device may be configured to implement active sensing 322, learning 324, planning 325, execution / decision-making 326, and perception / reasoning 328. As another example, the robotic device may be configured to implement active sensing 322, learning 324, planning 325, and execution / decision-making 326. As yet another example, the robotic device may be configured to implement active sensing 322, learning 324, planning 325, execution / decision-making 326, and perception / reasoning 328. Although not described, memory 320 may also store programs and / or applications associated with the operating system, as well as general robotic actions (e.g., power management, memory allocation, etc.).

[0037] In some embodiments, the active sensing 322 may include active sensing or scanning of the environment, as described herein. In some embodiments, the active sensing 322 may include active scanning of the environment and / or sensing or perception of information associated with the environment, objects in the environment (e.g., including people in the environment), and / or one or more conditions associated with a robotic device or system.

[0038] Learning 324 may include modules, processes, and / or functions that, when implemented, cause the robotic device to learn behaviors, skills, environmental constraints, and / or other information associated with autonomous activities performed by the robotic device. In some embodiments, a skills model 334 may be generated and / or updated based on Learning 324. In some embodiments, Learning 324 may include configuring the robotic device with specific information such as elevator maps, corridor maps, and floor plans of a particular building in which the robotic device is operating. Learning 324 may include setting predefined configurations for the robotic device, such as the dimensions of safety bubbles and thresholds for reactions (e.g., warnings, requests for assistance). Planning 325 may include teaching boarding and / or exit procedures for the robotic device for different elevators, doors, etc., within a hospital where the robotic device is deployed.

[0039] Plan 325 may include modules, processes, and / or functions that enable a robotic device to generate a plan or trajectory for performing one or more actions or skills, or, if implemented, cause a robotic device to generate them. As further described herein, a plan or trajectory is a path between points that are represented as the future position of the robotic device between a first time (t) and a future time (t+n, where n can represent the number of time steps into the future). For example, the trajectory is {(t1,x1,y1,theta1),(t2,x2,y2,theta2),...(t n ,x goal ,y goal Theta goalIt can be expressed as )}, where the goal represents the final waypoint or desired position of the robotic device. In some implementations, plan 325 may include pre-planning for generating a cache of plans for performing an action or skill, and each plan may be associated with various conditions and / or information, such as sensor data, elevator data, context information, and environmental constraints. In some embodiments, plan 325 may include immediate planning or real-time determination of a plan for performing an action or skill.

[0040] The execution / decision-making 326, when implemented, may include modules, processes, and / or functions that cause a robotic device to execute a plan for actions and / or skills from, for example, a plan from the plan 325. In some embodiments, the execution / decision-making 326 may include determining which actions and / or skills to execute in a given environment. In some embodiments, the execution / decision-making 326 may include preempting the execution of a particular action or skill (or preempting the continued execution of a particular action or skill) based on changes in the dynamic environment. For example, the execution / decision-making 326 may be configured to determine the direction and / or action that a robotic device should take (e.g., taking an alternative route, exiting a desired floor in a building, pressing one or more buttons on an elevator to arrive at and exit a desired floor, predicting the arrival of an elevator car, etc.). In a hospital setting, the execution / decision-making 326 can enable the robotic device to navigate through the environment while avoiding people (e.g., doctors, nurses, staff, patients, etc.) and objects (e.g., wheelchairs, hospital beds, etc.), to determine whether to wait for or call an elevator, and to identify the safe space the robotic device will occupy within the elevator car.

[0041] The Perception / Inference 328, when implemented, includes a module, process, and / or function that causes a robotic device to perceive and / or infer information about objects in its environment. The Perception / Inference 328 can operate based on information collected by one or more robotic devices (e.g., sensor data) and / or associated information such as state information, data tracking and / or analysis. In some embodiments, the Perception / Inference 328 may be configured to access and / or acquire information from third-party systems (e.g., security system data, hospital floor plan data, elevator system data, etc.) and use and / or analyze that data with or without using data collected by one or more robotic devices to perform data tracking, analysis, etc. In some embodiments, the Perception / Inference 328 may be configured to analyze information associated with humans, such as hospital staff and / or patients in a hospital.

[0042] The controller 312 may include a storage device 330 or be operably coupled to this storage device. The storage device 330 stores information related to the environment and / or objects in the environment, skill learning and / or execution (e.g., tasks and / or social behaviors), and / or state information of the robotic device. The storage 330 stores, for example, state information 344, skill models 334, inference models 336, decision models 338, object / constraint information 340, state information 334, maps 346, machine learning libraries, etc.

[0043] The object / constraint information 340 may include information relating to physical objects in the environment. An object may include any type of physical object located in the environment, including objects that define a space or opening (e.g., surfaces or walls defining doorways, hospital beds, wheelchairs, etc.). An object may be stationary or movable. Examples of objects in an environment, such as a hospital, include equipment, fixtures, instruments, tools, furniture, and / or people (e.g., nurses, doctors, patients, etc.). The object / constraint information 340 may include information that identifies or quantifies different characteristics of an object, such as location, color, shape, and surface features. The object / constraint information 340 may also identify codes, symbols, and other markers associated with physical objects, such as quick response or "QR" codes, barcodes, tags, etc. In some embodiments, the object / constraint information 340 may include information characterizing objects in the environment, such as a crowded elevator car, a button panel in an elevator car which is a specific type of button panel, an elevator lobby, a corridor, and / or indicators in an elevator. The object / constraint information 340 can enable the controller 312 to identify physical objects in the environment. The object / constraint information 340 may include information associated with objects and / or conditions in the environment that may restrict the operation of robotic devices in the environment. For example, the object / constraint information 340 may include information associated with the size, configuration, and / or location of objects in the environment (e.g., supply containers, rooms, doorways, etc.) and / or information indicating that access to a particular area (e.g., rooms, corridors, etc.) is restricted. The object / constraint information 340 may influence the learning and / or execution of one or more actions in the environment. Thus, environmental constraints can be part of each model for skills performed within a context that includes environmental constraints.

[0044] State information 344 may include information about the state of a robotic device (e.g., robotic device 210) and / or the environment in which the robotic device is operating (e.g., a building such as a hospital). Map 346 may include information associated with a representation of the environment (e.g., map and / or representation 600) and / or information obtained from a third-party system that is tracked and / or analyzed by a data tracking and analysis element such as perceptual / inference 328 or a processor 302 that performs perceptual / inference 328 (e.g., electronic medical records of a hospital, security system data, insurance data, vendor data, etc.). Examples of map 346 include floor plan data, elevator system data, room data, corridor data, emergency exit data, etc.

[0045] In some embodiments, state information 344 can indicate the location of a robotic device within an environment, such as a room, floor, enclosed space, or elevator. For example, state information 344 can indicate the location of a robotic device within a map. State information 344 can also include information that identifies a designated space (e.g., a safe zone, a hazard zone, etc.), a hazard queue, etc. In some cases, state information 344 can include the dimensions of a designated space in 3D. State information 344 can also include the locations of one or more objects (or markers representing and / or associated with objects) within the environment, such as within a map. Thus, state information 344 can identify the position of a robotic device relative to one or more objects.

[0046] Map 346 may include a stored representation of the environment, along with static and / or dynamic information about objects in the environment (e.g., fixtures, equipment, etc.) and social context information associated with human and / or social scenes in the environment, such as those shown in representation 600 of Figure 6. For example, representation 600 may include, for example, a navigation layer 610, a static semantic layer 620, a social layer 630, and a dynamic layer 640. The navigation layer 610 provides a general layout or map of a building that can identify several floors 612 having walls 614, stairs 616, and other built-in elements in the building (e.g., corridors, openings, boundaries). The static semantic layer 620 identifies objects and / or spaces within the building, such as elevators 622, objects 624, and doors 626. The static semantic layer 620 can identify which elevators 622 or other spaces are accessible or inaccessible to the robotic device. In some embodiments, the static semantic layer 620 can provide a three-dimensional map of objects located within the building. The social layer 630 provides social context information 632, which includes information associated with people within the building. The dynamic layer 640 provides information about objects and other elements within the building that may move and / or change over time. For example, the dynamic layer 640 can track movement 644 and / or change 646 associated with an object 642. In one embodiment, the dynamic layer 640 can monitor the expiration date of an object 642 and identify when that object 642 has expired.

[0047] The information within representation 600 may be accessible and / or managed by, for example, a control unit 602 of a robot device (e.g., structurally and / or functionally similar to controller 312). Control unit 602 may include components similar to those of other control units described herein (e.g., control units 202 and / or 302). Control unit 602 may include a storage device (e.g., storage device 330, similar to other storage elements described herein) that stores state information 604 (e.g., similar to state information 344), object / constraint information 606 (e.g., similar to object / constraint information 340), and / or map 608 (e.g., similar to map 346). Control unit 602 may be located on a robot device and / or on a remote server connected to one or more robot devices. Robot devices may be configured to update and maintain state information 604, object / constraint information 606, and / or map 608 when the robot devices collect information about their surrounding environment.

[0048] For example, information learned by the robotic device from active sensing 322, learning 324, planning 325, execution / decision-making 326, and / or perception / reasoning 328 may be fed into different layers of representation 600 and organized for future reference by the robotic device (and / or other robotic devices). For example, the robotic device may, as further described herein, rely on information learned about different objects in the environment (e.g., elevators or doors) to decide how to mediate between different behaviors (e.g., waiting for the elevator doors to open before entering the elevator car, waiting for a person to move before entering / exiting the elevator car, seeking assistance to press a button for an elevator direction request, seeking assistance to press a button for a desired floor inside the elevator car, seeking assistance to clear a path for the robotic device, etc.).

[0049] Referring back to Figure 3, the skill model 334 includes models generated to perform different actions and represents skills learned by the robotic device (e.g., by implementing learning 324). In some embodiments, the skill model 334 is a model that can be used to learn and / or perform a variety of actions or skills, including tasks and / or behaviors. For example, the skill model 334 may include information associated with objects involved in the performance of the skill (e.g., objects manipulated by the robotic device, objects that the robotic device interacts with during the performance of the skill, objects that the robotic device considers while performing the skill). In some embodiments, each model 334 is associated with a set of markers tied to different physical objects in the environment. The marker information can indicate which markers are associated with a particular model 334. Each model 334 may also be associated with sensor information collected, for example, through one or more sensors of the robotic device during other demonstrations of kinesthetic teaching and / or skills. The sensor information may optionally include manipulator information associated with manipulator elements of the robotic device when the robotic device performs an action during a demonstration. Operational element information may include, for example, joint position and configuration, end effector position and configuration, base or transport element position and configuration, and / or forces and torques acting on joints, end effectors, transport elements, etc. Operational element information may be recorded at specific points in time during the demonstration and / or execution of the skill (e.g., keyframes), or alternatively, throughout the entire demonstration and / or execution of the skill. Sensor information may also include information associated with the environment in which the skill is demonstrated and / or performed, such as the location of markers in the environment. In some embodiments, each Model 334 may be associated with a success criterion. The success criterion may be used to monitor the execution of the skill. In some embodiments, the success criterion may include information associated with visual and tactile data perceived using one or more sensors, such as a camera, force / torque sensor, etc.Success criteria can be linked to, for example, visually detecting the movement of an object, sensing the forces acting on the components of a robotic device (e.g., weight from an object), or sensing the engagement between the components of the robotic device and an object (e.g., changes in pressure or force acting on a surface).

[0050] The inference model 336 includes a model that can infer or predict different information about objects in the environment, the environment, robotic devices, etc. In some embodiments, the inference model 336 may include an algorithm for determining the arrival of an elevator and / or which floor the elevator is on. For example, the inference model 336 may include a machine learning algorithm, such as a CNN model, which can be used to process acoustic data (and / or other types of sensor data) to determine whether an elevator is about to arrive. In some embodiments, the inference model 336 may include an algorithm for predicting changes in the dynamic environment (e.g., human movement, human behavior, etc.).

[0051] The decision-making model 338 includes a model for guiding the robotic device when implementing one or more actions (e.g., executing one or more trajectories or plans). In some embodiments, the decision-making model 338 may be associated with preempting or halting the execution of certain actions or skills, for example, to avoid safety issues or impossible movements. In some embodiments, the decision-making model 338, when implemented by the processor 314, allows the robotic device to mediate or decide on multiple plans for execution.

[0052] In some embodiments, the storage device 330 may include machine learning libraries, which may include modules, processes, and / or functions related to different algorithms for machine learning and / or model generation of different skills. In some embodiments, the machine learning libraries may include methods such as convolutional neural networks, Bayesian belief networks, and Hidden Markov Models (HMMs). An example of an existing machine learning library in Python is scikit-learn. The storage device 330 may also include additional software libraries related to, for example, robot simulation, motion planning and control, kinematic teaching and perception, etc. Although specific machine learning models are described herein, it can be understood that the machine learning libraries may include additional forms of learning, for example, to detect objects (e.g., people, doors, buttons), to open and / or close elevator doors, to select a base position in an elevator to press an elevator button, etc.

[0053] In some embodiments, an initial set of state information 344, object / constraint information 340, and / or algorithms (e.g., models 334, 336, 338) may be provided to the robotic device, for example, via a remote administrator or supervisor. The robotic device may adapt and / or add to its knowledge of state information 344, object / constraint information 340, and / or algorithms based on its own interactions with the environment or people in the environment (e.g., patients, nurses, doctors, etc.), demonstrations, etc., and / or via additional user input. Alternatively or additionally, the robotic supervisor may update the robotic device's knowledge of state information 344, object / constraint information 340, and / or algorithms based on new information collected by the robotic device or other robotic devices (e.g., similar or the same environment, e.g., other robotic devices in a hospital) and / or provided to the robotic supervisor by an external party (e.g., a supplier, administrator, manufacturer, etc.). As new information regarding the environment or skills is provided to the robotic device and / or robotic supervisor, such updates may be provided repeatedly and / or sequentially.

[0054] The I / O interface 318 can be any suitable component that enables communication between the internal components of the controller 312 and external devices such as a user interface, operating elements, transport elements, and / or a computing device. The I / O interface 318 may include a network interface 319 that can connect the controller 312 to a network (e.g., network 102 as depicted in Figure 1). The network interface 319 enables communication between the controller 312 (which may be located on a robotic device or another network device communicating with one or more robotic devices) and a remote device such as a computing device that may be used by a robotic supervisor to monitor and / or control one or more robotic devices. The network interface 319 may be configured to provide wireless and / or wired connectivity to a network (e.g., network 102).

[0055] Figure 4 schematically illustrates an operating element 450 in several embodiments. The operating element 450 can form part of a robotic device, such as robotic devices 102 and / or 200. The operating element 450 can be implemented as an arm including two or more segments 452 connected to each other via joints 454. The joints 454 can enable one or more degrees of freedom. For example, the joints 454 can provide translation along one or more axes and / or rotation about one or more axes. In one embodiment, the operating element 450 can have seven degrees of freedom provided by the joints 454. Although four segments 452 and four joints 454 are depicted in Figure 4, those skilled in the art will understand that the operating element may include a different number of segments and / or joints.

[0056] The operating element 450 includes an end effector 456 which can be used to interact with objects in the environment. For example, the end effector 456 may be used to engage with and / or manipulate different objects. Alternatively or additionally, the end effector 456 may be used to interact with movable or dynamic objects, including, for example, a human being. In some embodiments, the end effector 456 may be a gripper that can releasably engage with or grasp one or more objects. For example, an end effector 456 implemented as a gripper may be able to lift and move an object from a first location (e.g., a storage room) to a second location (e.g., an office, a room, etc.).

[0057] Multiple sensors 453, 455, 457, and 458 may be arranged on different components of the operating element 450, such as the segment 452, the joint 454, and / or the end effector 456. Sensors 453, 455, 457, and 458 may be configured to measure sensor information, including environmental information and / or operating element information. Examples of sensors include position encoders, torque and / or force sensors, touch and / or tactile sensors, image capture devices such as cameras, temperature sensors, pressure sensors, optical sensors such as lidar sensors, and microphones. In some embodiments, sensor 453 arranged on segment 452 may be a camera configured to capture visual information about the environment. In some embodiments, sensor 453 arranged on segment 452 may be an accelerometer configured to measure the acceleration of segment 452 and / or calculate its moving speed and / or position. In some embodiments, sensor 455 arranged on joint 454 may be a position encoder configured to measure the position and / or configuration of joint 454. In some embodiments, the sensor 455 disposed on the joint 454 may be a force or torque sensor configured to measure the force or torque applied to the joint 454. In some embodiments, the sensor 458 disposed on the end effector 456 may be a position encoder and / or a force or torque sensor. In some embodiments, the sensor 457 disposed on the end effector 456 may be a touch sensor or tactile sensor configured to measure the engagement between the end effector 456 and an object in the environment. Alternatively or additionally, one or more of the sensors 453, 455, 457, and 458 may be configured to record information about one or more objects and / or markers in the environment. For example, the sensor 458 disposed on the end effector 456 may be configured to track the location of an object in the environment and / or the position of an object relative to the end effector 456. In some embodiments, one or more of the sensors 453, 455, 457, and 458 may also track whether an object, such as a person, has moved in the environment.Sensors 453, 455, 457, and 458 can transmit the sensor information they record to a computing device located on the robotic device (e.g., onboard control units such as control units 202 and / or 302), or sensors 453, 455, 457, and 458 can transmit sensor information to a remote computing device (e.g., a server such as server 120).

[0058] The operating element 450 may optionally include a coupling element 459 that allows the operating element 450 to be releasably coupled to a robotic device, such as one of the robotic devices described herein. In some embodiments, the operating element 450 may be coupled to a fixed location on the robotic device and / or can be coupled to multiple locations on the robotic device (e.g., to the right or left side of the body of the robotic device, as shown in Figure 5). The coupling element 459 may include any type of mechanism that can couple the operating element 350 to the robotic device, such as a mechanical mechanism (e.g., fasteners, latches, mounts), a magnetic mechanism, or a friction fit.

[0059] Figure 5 schematically illustrates a robotic device 500 in several embodiments. The robotic device 500 includes a head 580, a body 588, and a base 586. The head 580 may be connected to the body 588 via a segment 582 and one or more joints (not shown). The segment 582 may be movable and / or flexible to allow the head 580 to move relative to the body 588. The head 580, segment 582, etc., may be examples of operating elements and may include similar functions and / or structures to other operating elements described herein.

[0060] The head 580 includes one or more image capture devices 572 and / or other sensors 570. The image capture devices 572 and / or other sensors 570 (e.g., LiDAR sensors, motion sensors, microphones, etc.) can enable the robotic device 500 to scan the environment and acquire a representation of the environment (e.g., a visual representation or other semantic representation). In some embodiments, the image capture device 572 can be a camera. In some embodiments, the image capture device 572 can be made movable so that the image capture device can be used to focus on different areas of the environment around the robotic device 500. The image capture devices 572 and / or other sensors 570 can collect sensor information and transmit the sensor information to a computing device or processor mounted on the robotic device 500, such as a control unit 202 or 302. In some embodiments, the head 580 of the robotic device 500 can have a humanoid shape and may include one or more human features, such as eyes, nose, mouth, ears, etc. In such embodiments, the image capture device 572 and / or other sensors 570 may be implemented as one or more human features. For example, the image capture device 572 may be implemented as an eye on the head 580. In some cases, the sensor 570, including a microphone, may receive acoustic data, for example, from a user speaking to the robotic device or from an object (e.g., an arriving elevator), to perform actions, perform inferences, and so on.

[0061] In some embodiments, the robotic device 500 can use an image capture device 572 and / or other sensors 570 to scan the environment for information about objects in the environment, such as physical structures, devices, articles, people, etc. The robotic device 500 can perform active sensing, or it can initiate sensing or scanning in response to a trigger (e.g., user input, detected event, or environmental change). In some embodiments, the robotic device 500 can perform adaptive sensing, where sensing may be performed based on stored knowledge and / or user input. For example, the robotic device 500 may identify areas in the environment for scanning for objects based on prior information the robotic device has about objects.

[0062] In some embodiments, the robotic device 500 may also know to sense or scan different areas of a scene more precisely based on human input. For example, a human may indicate to the robotic device 500 that a particular area of ​​a scene contains one or more objects of interest, and the robotic device 500 may scan those areas more precisely to identify those objects. In such embodiments, the robotic device 500 may include an input / output device 540 such as a display with a keyboard or other input device, and / or a touchscreen, as schematically depicted in Figure 5. In some embodiments, the robotic device 500 may scan the environment and identify, for example, that an object, such as a human, is moving within the environment. In some embodiments, the robotic device 500 may perform active sensing so that the robotic device can adjust its actions in near real-time.

[0063] As schematically depicted in Figure 5, the base 586 may optionally include one or more transport elements implemented as wheels 560. The wheels 560 can enable the robotic device 500 to move around an environment, for example, a hospital. The robotic device 500 also includes at least one operating element implemented as an arm 550. The arm 550 may be structurally and / or functionally similar to other operating elements described herein, for example, operating element 450. The arm 550 may be fixedly attached to the body 588 of the robotic device 500, or optionally, the operating element 550 may be releasably coupled to the body 588 via a coupling element (e.g., coupling element 459) that can be attached to the coupling portion 584 of the robotic device 500. The coupling portion 584 may engage with the coupling element 459 and be configured to provide an electrical connection between the arm 550 and an onboard computing device (e.g., control unit 202 or 302), which can power and / or control components of the arm 550 and receive information collected by sensors (e.g., sensors 453, 455, 457, and 458) disposed on the operating element 550. The head component 580 may be connected to the control unit 202 or 302 and may include lights that provide information about the robot's status to objects around the robot. In some embodiments, the head component 580 may include one or more lights (e.g., on a headband) that are connected to the control unit 202 or 302 and provide information about the robot's status to objects around the robot. Elevator Navigation

[0064] In some embodiments, the robotic devices described herein may be configured to navigate using an elevator. However, autonomous elevator navigation faces various challenges, including, for example, elevator opening and closing speeds, strict physical constraints, and shared space. A robotic device such as those described herein may recognize and / or interact with one or more of the following: an elevator interface including buttons and / or API interfaces; various ways in which the elevator may be occupied; recovery behavior for quickly entering and exiting the elevator without becoming immobile; an indicator of which floor the elevator or robotic device is on (e.g., when the elevator is moving); and an indicator of elevator arrival.

[0065] In some embodiments, robotic devices as described herein may be trained using information about elevator infrastructure and the building structures of various buildings. The robot can navigate efficiently and automatically using models or maps of elevator and floor layouts. The robot may be configured to recognize, pass through, and / or interact with humans and obstacles. For example, the robot can interact with humans, such as requesting assistance, warning humans to be vigilant about the robot, and requesting user input from humans. The robot may use indicators to determine which floor the robot is on, which direction the elevator is moving, etc. The indicators may include signs that show floor levels, floor rooms, elevator lobbies, room numbers, the direction the elevator is heading, elevator buttons indicating a specific floor, etc.

[0066] In some embodiments, robotic devices as described herein may be configured to perform navigation while avoiding collisions with humans and equipment, causing inconvenience to hospital staff (e.g., blocking and / or coming into contact with humans), blocking elevator doors during opening and closing, etc. In a hospital setting, the robot may also be trained to navigate without damaging the hospital infrastructure and / or equipment, without damaging the robot itself, without becoming immobile, and without causing inconvenience to hospital staff. The robot may also be trained to navigate efficiently while adhering to navigation policies in order to arrive at the destination more quickly.

[0067] In some embodiments, the problem space of an elevator navigation system may be represented by a hierarchy of data objects, each containing information acted upon by various perception, navigation, and / or operation subsystems. The data can be interpreted by a hierarchy of state machines and behavior trees to perform actions. In some embodiments, the data may be captured in a representation that includes the environment of the elevator lobby and elevator car, divided into parts that map to robotic capabilities. For example, the lobby area may be represented as a 2D polygon for mapping to localization and navigation. The up and down buttons for calling the elevator may be represented as groups of obstacles, walls, and targets relative to a reference frame located at points on the wall. The elevator car may be represented as a 2D polygon for mapping to localization and navigation. The floor buttons within the elevator car may be represented as groups of obstacles, walls, and targets relative to a frame located at points on the wall. The elevator car door may be represented by a bounding box and a center of gravity mapped onto the perception subsystem. The components of the data representation may be linked hierarchically so that a robotic device can efficiently scan the data to reach the next actionable component. For example, the lobby area can be linked to the up and down button panels, the lobby area can be linked to the elevator car, the elevator car can be mapped to the floor button panel, and the elevator car can be mapped to the elevator door.

[0068] In some embodiments, data may be interpreted and acted upon by a robotic device employing a decision-making system. The decision-making system may include behavioral trees for performing complex robotic tasks (e.g., pressing a call button, determining that the correct elevator car has arrived, and boarding the correct elevator car). The behavioral trees may be embedded in a state machine using a hybrid approach that enables decision-making at two levels: a state machine (e.g., explicitly defined paths based on high-level discrete decisions) and a behavioral tree (e.g., executing overlapping and most efficient paths to achieve a goal based on available knowledge and outcomes). In some embodiments, the decision-making system may also include generating a behavioral tree for pressing a call button, e.g., a behavioral tree supporting the execution of an arm operation plan or trajectory. In some embodiments, the decision-making system may also include determining that the correct elevator car has arrived and boarding the elevator car. In exemplary implementations, this may include multiple subtrees that can be executed in parallel or sequentially. For example, a robotic device may execute subtrees related to perception, the execution of specific tasks, etc. Perceptual information can be updated continuously or periodically, for example, as the robotic device continues to capture information about its state and / or environment. This perceptual information can be used to determine which plans to execute and when to execute them, for example, whether to board an elevator. In particular, the tree can ensure that the robotic device implements the most immediate action necessary for boarding, or performs actions that contribute to boarding until boarding becomes possible.In some embodiments, the robotic device may use a hierarchical task network (HTN), a planning domain description language (PDDL), or other planning approaches for navigating using an elevator, such as pressing a call button, determining when the target elevator has arrived, or determining when to board the target elevator. In some embodiments, information from one or more perception systems can be used together to determine whether the elevator has arrived, its direction, and when its doors will open. In some embodiments, the decision-making system may also include considering various obstacles. For example, both state machines and behavioral trees may have route and recovery behaviors, which take into account various obstacles associated with attempting complex robotic tasks in a dynamic human environment.

[0069] In some embodiments, one or more perception systems (e.g., multimodal and / or multisensor (acoustic and visual, etc.) systems that may employ one or more machine learning algorithms) can be used to capture and interpret data about the environment, robotic devices, etc. These perception systems can be combined with APIs for specific systems to add a level of confidence to the perceived systems and / or be used for self-supervised learning. For example, one perception system may be for elevator door detection. Representations of elevator doors can enable an efficient method for determining their open or closed state so that a robot can navigate through the doors. In some embodiments, laser data can be used to determine whether there is a sufficiently wide path through a bounding box representing an elevator door. Another perception system may be for elevator arrival detection. Elevator arrival detection can analyze acoustic data (e.g., including sounds produced by the elevator) to indicate the imminent arrival and direction of the elevator car. In some embodiments, a 1D convolutional network may be trained on the collected acoustic data and used to determine the elevator arrival. Further perception systems may be used to receive and / or interpret data from integrated systems, such as elevator monitoring systems (e.g., elevator APIs). These systems, which integrate directly with elevator hardware, may be superimposed on top of perception systems to ensure that the most accurate information about the elevator's status is supplied to the decision-making system.

[0070] Further details of these systems, data representations, and / or decisions are explained with reference to the following diagram.

[0071] Figures 7–11 are flowcharts illustrating a method 700 for elevator navigation, which may be performed by one or more robotic devices or a system including robots (e.g., system 100) according to several embodiments. The method 700 may be performed by a single robotic device, such as one of the robotic devices described herein. Alternatively or additionally, the method 700 may be performed sequentially or simultaneously by multiple robotic devices and / or other computing devices, each performing a part of the method 700. While specific steps, processes, or modules are depicted in Figures 7–11, it can be understood that specific steps, processes, or modules may be combined and / or bypassed using end-to-end learning, for example, implemented using a neural network.

[0072] Figure 7 provides a high-level flow of Method 700 for elevator navigation. When a robotic device arrives at an elevator lobby, it may be configured to perform a series of decisions between arriving at the elevator lobby and entering the elevator. In some implementations, the robotic device may be trained to perform decisions and / or actions based on contextual information. For example, the robotic device may determine the state of the elevator, such as whether the elevator is en route, whether the elevator has not yet been called, or whether the elevator has departed. The robotic device may also determine the state of the elevator doors, such as whether they are opening, open, closing, or closed. The robotic device may also determine the likelihood of successfully navigating through an open door. For example, the elevator lobby and / or elevator car may contain obstacles, and the robotic device may, based on the likelihood of success, select a specific path, a verbal cue for assistance or warning, etc., to navigate around the obstacles in substantially real time. The robotic device may also determine the state of the elevator car in order to navigate around obstacles within the elevator car and to navigate towards open space for the robotic device to position itself.

[0073] In 710, the robotic device can determine whether the elevator has been called and / or call the elevator. In some embodiments, the robotic device may be configured to communicate with an elevator API and use the elevator API to determine the elevator status and / or whether the elevator has been called (by another human or robotic device). An elevator that can support an API that can be used to call, track, and / or send the elevator car may have an elevator API that the robotic device can call to replace button-pressing actions and / or elevator sensor processes. For example, the elevator API may be used to call the elevator without the robotic device having to physically move toward a call button panel to press a button for the desired direction and / or for the elevator.

[0074] In some embodiments, a robotic device can determine whether a button on a call button panel has already been pressed, for example, by a human or another robotic device. For example, a robotic device can determine, based on image data captured by a camera, whether an elevator call button is lit or otherwise indicates that the elevator call button has already been pressed.

[0075] In some embodiments, a robotic device may be configured to move toward a call button panel to call an elevator. The robot may execute a path that includes one or more waypoints to move toward the elevator call button. In some implementations, the robotic device may be trained to recognize, pass through, and / or interact with people and obstacles. For example, the robot may utter a phrase based on obstacles that obstruct the robot's path toward the call button panel and press a button. The robotic device may arrive at a predefined call button interaction waypoint for the elevator lobby, perceive the location or position of the elevator call button, and press the correct button within the call button panel to call the elevator to move in the desired direction (e.g., up or down). When the robotic device reaches a predefined call button interaction waypoint and operates the elevator call button, it may remain safe from other people in the elevator lobby by, for example, using a safety bubble feature to temporarily pause its arm movement when someone enters the safety bubble. For example, when the robot interacts with the button, the robot may temporarily pause its arm movement when a person enters the safety bubble.

[0076] In some embodiments, a robotic device can use visual information, such as reference markers, to determine its environment when calling an elevator. In particular, a robotic device can perceive information about the environment using visual feedback from the robotic device's camera or sensors. In some cases, a robotic device can identify reference markers and, based on the known locations of those reference markers relative to the elevator call buttons, execute one or more trajectories to successfully press the elevator call buttons. Reference markers may be attached to the wall near and / or at the location of each elevator call button. In some cases, a robotic device may include reference markers, for example, on the arm of the robotic device, which facilitate the positioning of a part of the robotic device relative to the elevator call buttons. For example, a robotic device can use its reference detector to determine the locations of wall reference markers and arm reference markers and, by following trajectories, position the arm reference markers relative to the wall reference markers to press the desired elevator call buttons, based, for example, the stored relative positions between the elevator call buttons and the wall reference markers, and the location of the arm reference markers relative to the end effector of the robotic arm. In some cases, robotic devices can also use camera sensors to perceive the pose or location of elevator call buttons and robotic arms without reference markers.

[0077] In 720, the robotic device can monitor the state of the elevator lobby and the arrival of a target elevator (e.g., a desired elevator for boarding). The robotic device can recognize the floor in which it is located and monitor and determine when the elevator will arrive on that floor. In some embodiments, the robotic device can use indicators to determine which floor the robotic device is on, which direction the elevator is going, and / or when the elevator will arrive. The indicators may include signs indicating the floor level, floor room, elevator lobby, room number, the direction the elevator is heading before arrival, etc. In some embodiments, the robotic device can use acoustic data and / or other sensor data to determine when the elevator will arrive. For example, the robotic device can capture acoustic data associated with sounds in the elevator lobby and process that acoustic data using an algorithm (e.g., a trained CNN) to predict whether the elevator is about to arrive and / or whether the arriving elevator is going up or down. In some embodiments, the robotic device can communicate with an elevator management system (e.g., an elevator API) to identify when the elevator will arrive. For example, a robotic device can use the elevator API to determine whether the elevator is going up or down, which floor it is on, and the estimated arrival time of the elevator. The robotic device can also use the elevator API to check whether the elevator is about to arrive and whether the arriving elevator is heading in the direction desired by the robotic device.

[0078] While the robotic device is in the elevator lobby, it can navigate to one or more locations or waypoints in 730. For example, the robotic device may navigate to a waypoint or area within the elevator lobby to monitor an elevator and / or await its arrival. When the elevator lobby includes multiple elevator doors, the robotic device can navigate to a waypoint or area that allows the robotic device to observe each elevator door within the elevator lobby (e.g., using its sensors). In some embodiments, the robotic device may move from a waypoint or area for observation to a second waypoint or area (e.g., a location near an elevator door) to await the arrival of a particular elevator. In response to determining (e.g., based on sensor data and / or an elevator API) that an elevator is about to arrive, the robotic device may be configured to move toward a location or area to prepare to board and / or enter the elevator once the elevator doors open. In an elevator lobby with multiple elevator doors, the robotic device may move toward a location near the elevator door of a particular elevator that the robotic device intends to board before its arrival.

[0079] In some implementations, a robotic device can move towards a waypoint with a tolerance range or navigate towards a spatial domain. In some cases, a robotic device may be configured to implement some degree of slack or flexibility in its navigation system to identify a more reliable or feasible plan / trajectory. In some embodiments, when implementing such navigation, a robotic device may consider the shape of the environment around the robotic device in the form of cost maps, potential fields, inverse velocity planners, or geometric shapes derived through machine learning inference. In some embodiments, a robotic device can obtain information through perceptual input (e.g., sensor data) that recognizes objects in the environment and specific policies (e.g., do not stand next to a door). The robotic device can then select an optimal (or more optimal) position under the constraints of these various conditions and execute a plan to navigate to that point.

[0080] As described above, the robotic device may operate with a built-in safety zone or safety bubble, and the robotic device may take a safe position or stop its operation when an object (e.g., a person) enters or overlaps with the safety zone. In some embodiments, the robotic device may be configured to mitigate complexities such as interference from obstacles in the robotic device's path or destination, information collisions (e.g., an elevator arriving but the elevator doors not opening, inconsistencies in map data, etc.), and interruptions / failures of action (e.g., malfunctions in the robotic device's arm or body).

[0081] In some embodiments, a robotic device can navigate to one or more waypoints based on the state of the elevator lobby. For example, the robotic device can determine the occupancy level of the elevator lobby and identify an available (e.g., unoccupied) waypoint or area for the robotic device to occupy while waiting for the arrival of the target elevator. In some cases, if a lobby waypoint is occupied, the robotic device can select a different lobby waypoint, taking constraints (e.g., obstacles) into consideration. For example, if the robotic device determines that its original goal pose or waypoint {x,y,theta} is already occupied, the robotic device can select an alternative goal {x2,y2,theta2} to navigate to. If the goal pose {x2,y2,theta2} is blocked or occupied while the robotic device is navigating to the target pose {x2,y2,theta2}, the robotic device can select another goal {x3,y3,theta3}. Various waypoints or poses can be predetermined based on semantic data. In some cases, if the path to a particular waypoint is blocked (e.g., due to an obstacle), the robotic device can select a different path to reach the lobby waypoint.

[0082] When the target elevator arrives, at 740, the robotic device can take steps to board the target elevator. The robotic device can determine the state of the elevator upon arrival. In particular, the robotic device can determine whether the elevator doors are opening, open, closing, closed, and / or not operating. The robotic device can also determine whether an object (e.g., a person) is blocking the path to the elevator. The robotic device can monitor for when there are no longer any moving objects in the open doorway in order to navigate into the elevator. The robotic device can also estimate whether a particular elevator has enough space for the robot, for example, due to the current occupancy of the elevator car and / or the number of individuals entering the elevator. Using perceptual input, the robotic device can autonomously enter the target elevator, determining when to enter the elevator, when to pause, when to yield, when not to enter the elevator (e.g., due to spatial constraints), and / or where to position itself for boarding the elevator after entering.

[0083] In some embodiments, a robotic device can classify obstacles and / or objects it perceives (e.g., using one or more sensors) to determine which plan or trajectory to execute. For example, a robotic device can classify objects into people, beds, wheelchairs, etc., and determine whether to board a particular elevator. A robotic device can also track dynamic objects or obstacles to determine an appropriate plan to execute. The robotic device may base its decision on which plan to execute on one or more predefined criteria or policies. Such policies may be applied on a priority basis, for example, with a first policy taking precedence over a second policy.

[0084] While a robotic device is boarding an elevator (or determining whether to board an elevator), the elevator's state can be continuously monitored by the robotic device. The robotic device can receive continuous input or information about the elevator's state (e.g., from sensors, elevator APIs, users, etc.) as the robot navigates the elevator. The elevator state may include information that the robotic device can use to determine whether to board the elevator and / or when to board it. For example, the robotic device may choose to pause, resume, or calculate a different trajectory or waypoint depending on a measure of how likely the elevator is to be boarded (e.g., the probability of successful boarding based on contextual information).

[0085] After the robotic device boards the elevator, it can navigate to an elevator boarding waypoint or area at 750. The elevator boarding waypoint can be a safe area or zone that the robotic device can occupy while riding the elevator. In an embodiment, the robotic device can enter the elevator and move to a safe zone within the elevator at the earliest possible opportunity. The robotic device can reach an interaction waypoint and press a floor button within the elevator. If a floor button is blocked by another person or object, the robotic device can, for example, make an audible signal to the person requesting them to press the desired floor button. In some cases, the robotic device can remain at the elevator spot from which it is easiest to exit.

[0086] Elevator boarding waypoints may be selected to correspond to flow or occupancy conditions (e.g., boarding and / or exiting flows, the position of people and / or other objects in the elevator). While moving to a boarding waypoint, the robotic device may perceive information about the elevator car (e.g., using its sensors) and, if the original elevator boarding waypoint is occupied, may select a new elevator boarding waypoint. In some embodiments, the robotic device may also move to a different boarding waypoint, for example, if the first boarding waypoint is obstructing or blocking a person, another robotic device, or other object. In some embodiments, the robotic device may navigate to a first boarding waypoint to select a floor (e.g., by pressing a floor button) and then navigate to a second boarding waypoint for the rest of the elevator ride. The second boarding waypoint may be within an area that does not obstruct others from the floor selection panel. In some embodiments, the robotic device may also navigate to a third boarding waypoint suitable for exiting the elevator when the robotic device determines that the elevator is approaching its desired floor. Various waypoints or poses can be defined using semantic data and / or determined immediately by optimizing one or more constraints (e.g., maximum distance from elevator walls and / or people or other objects inside the elevator). In the case of optimization, the robotic device can continuously or repeatedly perceive information about its environment as it enters the elevator, navigates within the elevator, and has greater visibility to the elevator's occupancy grid, and can apply this information to update its waypoint goals. If a waypoint goal is occupied, the robotic device can attempt to find the next optimal location, or if it cannot find the next optimal location, it can fall back using social cues and / or implement actions to recover from the obstacle.

[0087] In some embodiments, a human can interact with the robotic device while riding it. For example, a user can provide input to the robotic device (e.g., by speaking to it and / or by operating a keyboard, joystick, or other input device). The user can request the robotic device to move to a new location or waypoint, exit the elevator on a floor, and / or take other actions. In some embodiments, the robotic device can use a behavior tree-based decision management system to select the correct action, taking into account the robot's state, its observations, and / or other information provided to the robot (e.g., user input). The robotic device may be configured to determine and monitor its own state, such as where the robotic device is located within the elevator car, which floor the elevator car is on, and the state of the elevator doors. The robotic device may have a set of recovery behaviors in case the robot fails to perform its intended action in a scenario. The recovery behaviors may be designed to ensure that the robotic device complies with navigation boarding and / or exit policies, for example, by not causing inconvenience to other individuals in the elevator (e.g., hospital staff or visitors).

[0088] The robotic device may be configured in 760 to indicate and / or perform floor selection. In some embodiments, the robotic device may navigate to a floor selection waypoint or region, which may be located around a floor button panel inside the target elevator. The robotic device may store or access the elevator layout (e.g., in onboard memory) to know where the floor button panel is located. Alternatively or additionally, the robotic device may use active scanning (e.g., using its sensors) to identify markers (e.g., references, buttons, etc.) to determine the location and layout of the floor button panel and perform floor selection. To press a floor button, the robotic device may perform movement of its operating element (e.g., arm, end effector). In some cases, the target elevator may be congested, preventing the robotic device from accessing the floor selection waypoint and / or performing floor selection. In such cases, the robotic device can perceive the crowded environment and execute pre-programmed acoustic files to request assistance in selecting a desired floor, or to request a human to clear a path for the robotic device to approach a floor selection waypoint. In some cases, the robotic device can use an elevator API to transmit signals representing the selection of a desired button instead of physically pressing a floor button.

[0089] In 770, the robotic device can determine its position relative to the floors of a building. For example, the robotic device can use various sensors to determine the pressure, drift, and / or velocity of the robotic device as the target elevator moves up or down. The robotic device may be configured to have floor detection hardware, firmware, and / or software that can analyze one or more sensor inputs to estimate relative height and apply that information to determine the relative position of the robotic device relative to various floors of a building. In some embodiments, the robotic device can use sensor data when entering the elevator as a reference value for its current floor, and then determine the relative height and changes in floor during the elevator ride. The robotic device may be configured to determine the floor despite pressure changes and drift over time. In some cases, the robotic device can use an elevator API to determine the direction the target elevator is going and / or which floor the target elevator is arriving at or passing through during the elevator ride.

[0090] In 780, the robotic device can exit a target elevator, for example, upon arrival at a desired floor. For example, the robotic device can exit the elevator automatically and / or autonomously using sensory input. The robotic device can detect that the elevator is about to arrive at a target floor and when the elevator doors will open. Similar to entering an elevator, the robotic device can perform object classification into people, beds, wheelchairs, etc., and track dynamic obstacles. The robotic device can then select an appropriate plan to execute to exit the elevator, while adhering to one or more policies specific to exiting an elevator (e.g., not blocking exits, not colliding with other objects, etc.). As with other policies described herein, elevator exit policies may be ranked according to priority. In some embodiments, the robot can use an elevator API to check and determine whether its elevator is approaching a desired floor. In response to determining that the elevator is approaching a desired floor, the robotic device can move to a waypoint or area for exiting the elevator.

[0091] Figure 8 provides a more detailed flow of lobby management and elevator status determination by a robotic device according to an embodiment. In particular, the steps depicted in Figure 8 provide further details to steps 720-730 depicted in Figure 7.

[0092] In 721, the robotic device may optionally navigate to one or more elevator lobby waypoints or areas. In 721, the robotic device may monitor the elevator lobby for the arrival of a target elevator. In 701, the robotic device may acquire data from an elevator monitoring system, and / or in 702, the robotic device may acquire sensor data. The data from the elevator monitoring system and / or sensor data may be used by the robotic device to determine when the target elevator will arrive. For example, the robotic device may use its sensors to capture information about the elevator lobby, including sounds, objects, the location of elevator doors, and the layout of the elevator lobby. In some embodiments, the robotic device may infer the arrival of an elevator by scanning elevator floor indicators that show which floor the elevator is on.

[0093] In 722, the robotic device can determine whether a target elevator (e.g., an elevator traveling in the direction requested by the robotic device) is about to arrive at the elevator lobby. When the robotic device determines that the target elevator is about to arrive (722: yes), the robotic device can approach the target elevator in 732. Approaching the target elevator may include navigating to a suitable waypoint or area for boarding the target elevator. The waypoint or area may be close to a door associated with the target elevator. The waypoint or area may also meet certain social requirements, for example, allowing potential people and / or other objects to exit the arriving elevator. In 724, the robotic device may detect that the target elevator has arrived, for example, based on data from an elevator monitoring system or API, sensor data, etc.

[0094] In 725, the robotic device can determine the state of the target elevator. Determining the state of the target elevator may include determining whether the elevator doors are closed or open, whether an object is moving through the doorway, and determining the elevator's occupancy status (or available space within the elevator). The elevator state may be updated continuously or repeatedly while the robotic device is monitoring whether it is boarding the elevator and while it is boarding the elevator. In other words, the robotic device can constantly monitor information related to the elevator state while determining whether it is boarding the elevator and / or is currently boarding it. Such information can enable the robotic device to determine whether it should board the elevator and / or continue boarding it. For example, depending on the elevator boarding conditions or other elevator state information, the robotic device may determine whether it should pause, resume, or recalculate robot navigation (e.g., whether to execute a new plan or select a new waypoint).

[0095] Optionally, while the robotic device is monitoring the elevator lobby and / or the target elevator, it may determine in 726 that the target elevator has been changed, i.e., that a better or more suitable elevator is about to arrive. For example, after an elevator is called, two or more elevators may arrive in the elevator lobby. If another elevator is about to arrive in the elevator lobby or has arrived, the robotic device may determine whether that other elevator is suitable for the robotic device to board. For example, another elevator that arrives earlier or has fewer occupants (or more space) may be more suitable for the robotic device to board. If the target elevator should be changed (726: yes), the robotic device may, in 732, execute a new navigation path to approach the new target elevator.

[0096] The robotic device can continue monitoring the elevator lobby and the status of the target elevator until it determines that the target elevator is ready for boarding. In 727, when the robotic device determines that the target elevator is ready for boarding, in 740, the robotic device can board the target elevator. Further details of elevator lobby management and / or elevator status monitoring are described with reference to Figures 16-18 below.

[0097] Figure 9 provides a more detailed flow of boarding an elevator by a robotic device according to an embodiment. In particular, the steps depicted in Figure 9 provide further details of step 740 depicted in Figure 7. As depicted in Figure 9, the robotic device can perform navigation and board the elevator.

[0098] A robotic device can be configured to autonomously enter an elevator using sensory input when performing the task of entering an elevator. Based on the sensory input, the robotic device can determine which elevator to enter, when to pause, when to yield, and where to position itself for boarding after entering the elevator. While performing the task of entering the elevator, the robotic device can classify potential objects as people, beds, wheelchairs, etc., and track dynamic obstacles. Based on its sensory input, the robotic device can select and execute its navigation boarding policy.

[0099] In some embodiments, the robotic device can perform elevator boarding while adhering to one or more predefined policies, such as avoiding collisions with people or equipment, avoiding causing inconvenience to people, avoiding blocking potential paths for people, and avoiding damage to itself or other equipment, furniture, walls, etc.

[0100] In 741, the robotic device may monitor the elevator state of the target elevator, for example, as described in 725 above. The robotic device may monitor the occupancy status of the elevator car or the space inside the elevator car, objects moving through the elevator doorways (e.g., people or other objects exiting, people or other objects entering, etc.). In some embodiments, the robotic device may monitor the elevator state in 702 based on sensor data and / or in 703 based on a history of attempts to reach pauses and / or waypoints. For example, the robotic device may use its sensors to perceive elevator states such as occupancy status, door open / closed, and elevator car layout. The robotic device may also recall a history of its attempts to reach a particular waypoint or area, for example, to board the elevator or to stand inside the elevator. If the robotic device recalls previous failures to reach a particular waypoint, it may adjust its action plan.

[0101] In 742, the robotic device can determine whether there is space and / or no obstacles at the entrance to the target elevator. If there is no space and / or obstacles, the robotic device may optionally implement one or more social cues. For example, if, after a predetermined period, the robotic device still determines that the entrance is blocked, the robotic device may use social cues to communicate with other people around it. The robotic device may utilize semantic dialogue to request help from others (e.g., "Could you call an elevator going up?", "Could you please make space for me to board the elevator?"), acoustic and / or visual cues to communicate intentions (e.g., arm movements, displays on a touchscreen), hazard cues to warn potential pedestrians of the robotic device's movement, obstacle status cues, and to continue providing information to people nearby. In some embodiments, the robotic device may use illuminated headbands and / or eye states to discreetly indicate its status, changes in its status, and / or intentions. In some embodiments, the robotic device may use acoustic cues to indicate the type of obstacle. In some embodiments, social scenarios may include calling for help from people within a building, signaling to people who may not have visibility of the robotic device to avoid unsafe navigation, and indicating awareness of the robotic device to people around it. Calling for help may include directly requesting help (e.g., pressing a button) or moving out of the robotic device's path. Signaling to people who may not have visibility of the robotic device may include transmitting preventative communications when the robotic device is not visible to potential pedestrians. Indicating awareness of the robotic device to people around it may include interacting with people and performing voice commands, including socially appropriate language, to make people feel more comfortable, happier, and / or more tolerant of the robotic device's presence or behavior.

[0102] The robotic device can then continue waiting to board the target elevator at 745, monitor the elevator status at 741, and determine at 742 when the robotic device can board the elevator.

[0103] If there is space at the elevator entrance and no obstacles (742: yes), the robotic device can determine a path or plan through the elevator door. The path may include performing navigation to an elevator boarding waypoint inside the target elevator. In some embodiments, the robotic device may select a navigation plan from a cache (e.g., several predefined plans for navigating through the elevator entrance). The robotic device may select a plan based on its current pose or position relative to the elevator entrance (or other physical feature). In some embodiments, the robotic device may plan a trajectory onto the elevator in real time (e.g., substantially in real time, e.g., within a few seconds). In 746, the robotic device may begin executing the plan through the elevator door.

[0104] While executing the plan, the robotic device can check whether any conditions or changes require preemption, such as pausing and / or replanning the execution. If the robotic device determines that preemption is necessary (747: yes), it can stop executing the plan and re-determine the path through the elevator door at 748. If preemption is not necessary (747: no), the robotic device can terminate execution of the path through the elevator door at 749. In some embodiments, the robotic device can navigate to a waypoint or pose, or any of several waypoints or poses, while receiving as perceptual inputs (a) the state of the robotic device in space and velocity, and (b) a recent history of previous attempts to reach its waypoint or pose.

[0105] In some embodiments, the robotic device may implement sanity checks to determine, for example, whether preemption may be required in advance. The robotic device may ensure that it identifies points of obstacles in an incorrect plan, such as points that the robotic device cannot execute. In some embodiments, for a given time step, the robotic device may evaluate the local trajectory (e.g., the plan for the next few meters) and apply sanity checks to ensure that the path is kinematically followable or feasible. If the check fails, the robotic device declares that the position is unexecutable. Points that are unexecutable due to either collisions or kinematic problems with the path are marked as points where the robotic device must stop. The robotic device can then determine a different plan or path to execute.

[0106] As described above, a robotic device can be configured to navigate to waypoints with a certain tolerance, that is, to navigate to a spatial region relative to a specific waypoint. Therefore, when boarding an elevator, for example, to board the elevator, a robotic device can be configured to navigate to a spatial region within the elevator. In some embodiments, a user may be able to specify waypoints or poses to which the robotic device can be driven within a predetermined tolerance.

[0107] In some embodiments, a robotic device may move its arms and / or other components to a predetermined pose before boarding an elevator. The predetermined pose may be a tuck pose designed for safe navigation. The predetermined pose may also be suitable for small spaces inside an elevator, such as a crowded elevator. The predetermined pose may also allow the robotic device to easily operate its arms and / or end effectors to press floor buttons or perform other actions within the elevator.

[0108] Figure 10 provides a more detailed flow of elevator occupancy management according to an embodiment. The steps depicted in Figure 10 provide further details to steps 750-760 depicted in Figure 7.

[0109] The robotic device can initiate passenger management once it has completed its navigation to the elevator. While riding the elevator, a user (e.g., a hospital employee) can request that the robotic device exit the elevator on any floor. The robotic device can choose an action to perform by considering its state and one or more observations, using a behavioral tree-based decision management system. The robotic device can infer its state from different components, such as knowledge of the current floor from its floor detection system, its location from its navigation system, the feasibility of reaching a waypoint for pressing a floor button from its cost map and perception system, and the state of the elevator door from its door detection perception system. In some embodiments, the robotic device is designed to have a set of recovery behaviors in case the robotic device fails to perform its selected action in each scenario. These recovery behaviors are designed to ensure that the robotic device adheres to the policies described above and does not cause inconvenience to the user (e.g., a hospital employee or visitor).

[0110] In some embodiments, passenger management may include determining the elevator floor status. A robotic device may monitor the elevator floor status based on inputs from one or more sensors and / or analysis thereof. The determined floor status may be used to notify other actions, such as when to prepare to exit the elevator, when to recover from an obstacle during elevator ride and exit at an intermediate floor, etc. Further details on determining the elevator floor status are illustrated with reference to Figure 11.

[0111] In some embodiments, passenger management may also include handling disembarkation requests. For example, a robotic device may be configured to receive input from a user and act based on that input. In one embodiment, a user may command the robotic device to disembark the elevator early, and the robotic device may exit the elevator on an intermediate floor.

[0112] In some embodiments, passenger management may also include recovery from failures. For example, a robotic device may be configured to implement one or more recovery behaviors based on, for example, the current state of the robotic device and / or the environment. The robotic device may select a recovery path when there is a failure caused by a failure in the robotic process, or by environmental factors such as a crowd that does not allow the robotic device to exit its intended floor.

[0113] In an exemplary implementation, a robotic device can monitor the occupancy status within an elevator using data from sensors, including one or more 2D LiDAR sensors and / or RGBD cameras. The robotic device can merge the occupancy data from the sensors to resolve conflicting data. For example, if a first sensor captures data indicating that there is no obstacle at a first location, but a second sensor captures data indicating that there is an obstacle at the first location, the robotic device can determine, based on the ranking applied to the sensor data, whether an obstacle is present, whether certain sensor data may be faulty, and so on.

[0114] In 751, the robotic device can navigate to a waypoint or area for boarding the elevator. In some embodiments, the robotic device may remain at this waypoint or area until, for example, the robotic device arrives at its destination floor and exits the elevator. Alternatively, in some embodiments, the robotic device may navigate to one or more additional waypoints or areas.

[0115] Optionally, in 761, the robotic device may check whether it can reach the floor button panel in the elevator. If the robotic device cannot reach the floor button panel, in 762, the robotic device may determine an alternative waypoint to access the floor button panel or employ a social signal. A social signal may include requesting assistance from a human or another robotic device nearby. For example, the robotic device may request assistance from a human or another robotic device to select a floor. If the robotic device can reach the floor button panel, in 763, the robotic device may perform a floor button selection.

[0116] In some embodiments, the robotic device may implement an elevator tack selection framework. During the training phase, the robotic device may generate workspace and reachability maps for each of several candidate elevator tack poses and operational strategies. Then, during execution, the robotic device can select an elevator tack pose that maximizes the success of the task (e.g., maximizing the reachable area within the elevator), taking the goal button pose into consideration. The robotic device can then use the selected pose and operational strategy to select the optimal position or location for the robotic device's base to be for the elevator tack pose. The robotic device can then navigate to this position (e.g., a waypoint) within the elevator. Once positioned, the robotic device can perform an arm operation to press the elevator floor button. While performing the arm operation, the robotic device can capture information about itself, its surrounding environment, objects within that environment, and / or the elevator in order to determine whether to change or abort the arm operation.

[0117] In some embodiments, the robotic device may have multiple end effectors, for example, to account for the variability of different elevators. For example, the robotic device may have an arm capable of supporting multiple end effectors, thereby enabling different button-pressing capabilities for different button and panel configurations inside the elevator car. In exemplary embodiments, the robotic device may have end effectors projecting radially from its final arm link and end effectors projecting axially, resulting in a larger robot base reachability footprint inside the elevator car. When the robotic device supports multiple end effectors, the robotic device can be trained with different grippers and / or end effectors. In some embodiments, the robotic device may be configured to transfer operations trained on an old gripper and / or end effector to a new gripper and / or end effector without the robotic device needing to be completely retrained for the new gripper and / or end effector.

[0118] In some embodiments, the robotic device may not need to select a floor via a floor button panel, and therefore step 761 may be omitted. For example, the robotic device may communicate with an elevator API to direct the elevator to its floor. In some embodiments, the robotic device may need to select a floor before boarding the elevator, for example, if the floor selection is made in the elevator lobby. In such embodiments, steps 761-763 may be performed by the robotic device in the elevator lobby.

[0119] Optionally, in 752, the robotic device may determine whether to move to a different boarding waypoint or area. For example, the robotic device may determine to navigate to a different boarding waypoint after selecting a floor to avoid blocking the floor selection panel. The robotic device may also determine to navigate to a different boarding waypoint when it determines that it is approaching or nearing its desired floor. The robotic device may also determine to move to a different boarding waypoint, for example, if there is less or more available space in the elevator, or if people need to exit the elevator. If the robotic device determines to navigate to a different boarding waypoint or area (752: yes), the robotic device may navigate to that different boarding waypoint or area in 754.

[0120] Figure 11 provides a more detailed flow of floor detection performed by a robotic device while riding an elevator, according to an embodiment. In particular, the steps depicted in Figure 11 provide further details of step 770 depicted in Figure 7.

[0121] In 771, the robotic device can detect or monitor its relative position to different floors of a building (e.g., a hospital). The robotic device can estimate its relative height, which can be used to determine its relative position to various floors of the building. In 701, the robotic device can acquire data from an elevator monitoring system, and / or in 702, it can acquire sensor data. In some embodiments, the robotic device can use sensor data when entering an elevator as a reference for its current floor, and then use it to determine the relative height of the robotic device and the change in floor during its elevator ride. This technique allows the robotic device to accurately determine its floor despite pressure changes and drift of sensor readings over time. In some embodiments, the robotic device can use its sensors to perceive the elevator environment, such as an elevator floor indicator, and use that information to detect its floor. In some embodiments, the robotic device can receive information from an elevator API and use that information to determine its floor.

[0122] In exemplary implementations, a robotic device can detect its floor using one or more of the following: a camera, laser data, a barometer, an inertial measurement unit (IMU), an accelerometer, etc. The robotic device may, for example, receive acceleration data from an IMU and / or accelerometer, which can be used to provide vertical odometry data. The robotic device may receive pressure readings from a barometer, which can be used to determine height, for example, by interpolating over calibration data associated with a known height. The robotic device may combine (e.g., fuse) these outputs to provide an estimate of the relative height between the robotic device and its floor. In some embodiments, specific constraints can be applied to the determined height or floor, for example, to avoid unrealistic jumps or changes in height.

[0123] Optionally, at 772, the robotic device may check whether the elevator is approaching its target floor or desired floor. If the robotic device is approaching the target floor (772: yes), the robotic device may optionally navigate at 753 to a waypoint that is more optimal or suitable for exiting the elevator. At 773, the robotic device may detect that the elevator doors have opened at the target floor, and at 780, the robotic device may proceed to exit the elevator.

[0124] Figure 12 provides a more detailed flow of exiting the elevator by a robotic device according to an embodiment. In particular, the steps depicted in Figure 9 provide further details of step 780 depicted in Figure 7.

[0125] The robotic device may be configured to autonomously exit the elevator using perceptual inputs to determine, for example, the elevator status, the occupancy of the robotic device and the elevator doors when performing the exit procedure. Similar to elevator entry, as described above with respect to Figure 9, the robotic device during elevator exit can actively scan and / or perceive information about its environment, classify objects in the environment as people, beds, wheelchairs, etc., and track the movement of dynamic obstacles. The robotic device can use this information to select and execute its navigation exit policy, for example, as described above with respect to elevator entry.

[0126] In 781, the robotic device can monitor the state of the elevator in which it is located. This can be similar to 741, as described above with respect to Figure 9. In some embodiments, the robotic device can monitor the elevator state in 702 based on sensor data and / or in 703 based on a history of attempts to reach pauses and / or waypoints. The robotic device can monitor the occupancy of the elevator car or the space inside the elevator car, objects moving through the elevator doorways (e.g., people or other objects exiting, people or other objects entering, etc.), etc. The robotic device can also recall a history of its attempts to reach a particular waypoint or area in order to exit the elevator, for example. If the robotic device recalls previous obstacles to reaching a particular waypoint, it can adjust its action plan.

[0127] In 782, the robotic device can determine whether there is space and / or no obstacles at the elevator entrance. This can be the same as in 742, as described above with respect to Figure 9. If there is no space and / or a moving obstacle, the robotic device can optionally use a social signal in 784 to request a human to allow the robotic device to exit. This can be the same as in 744, as described above with respect to Figure 9. In 785, after implementing the social signal, the robotic device can wait to exit the elevator. This can be the same as in 745, as described above with respect to Figure 9. In 781, the robotic device can continue monitoring the elevator status to determine when the exit path is clear.

[0128] Optionally, in 791, the robotic device may obtain a map of the new floor. For example, the robotic device may access and / or obtain a map of the floor to which it has arrived. In some embodiments, the robotic device may obtain the map before going to the new floor using the elevator, for example, while the robotic device is still in the elevator lobby waiting to board the elevator. In some embodiments, the robotic device may already have a stored copy of the map of the new floor. For example, the robotic device may store maps of all floors in the building (or, for example, a subset of the building floors that the robotic device is most likely to navigate). Thus, in some embodiments, the robotic device does not need to receive a map of the new floor.

[0129] In 783, the robotic device can determine a path through the elevator doors based on, for example, its current pose or waypoint, a map of the new floor, and waypoints on the new floor that the robotic device plans to navigate. This can be similar to 743, as described above with respect to Figure 9. The robotic device can perform navigation to reach a goal waypoint on the new floor, thereby exiting the elevator on the new floor. In some embodiments, the robotic device can select navigation from a cache of pre-planned trajectories. In some embodiments, the robotic device can perform improvisational planning to generate a trajectory for exiting the elevator.

[0130] At 786, the robotic device can begin executing the route through the elevator door. This can be the same as 746, as described above with respect to Figure 9. While executing the plan, the robotic device can check whether any conditions or changes require preemption, such as pausing and / or replanning the execution. If the robotic device determines that preemption is necessary (787: yes), it can stop executing the plan at 788 and re-evaluate the route through the elevator door. If preemption is not necessary (787: no), the robotic device can terminate the execution of the route through the elevator door at 789. This can be the same as 747-749, as described above with respect to Figure 9.

[0131] Figure 13 is a flowchart illustrating a method by which a robotic device replans or determines a new goal, according to several embodiments. The method depicted in Figure 13 can be performed continuously and / or periodically by the robotic device, for example, when the robotic device is navigating using an elevator. Thus, the method described in Figure 13 can cause the robotic device to terminate, pause, delay, or otherwise adjust any of the steps described with reference to Figures 7 to 12.

[0132] In 704, the robotic device may monitor for faults, safety concerns, or other conditions and / or events. For example, the robotic device may monitor one or more conditions associated with predefined policies (e.g., elevator boarding policy, elevator exit policy, elevator entry policy, lobby management policy, etc.). In some embodiments, the condition or event may be whether an object has entered the safety bubble of the robotic device. The robotic device may perform active scanning and / or perception of its environment, for example, in 702 by acquiring sensor data from one or more of its sensors, and / or in 792 by acquiring data from other sources (e.g., elevator API, elevator monitoring system, hospital system, etc.).

[0133] In 705a, the robot device may determine whether it needs to correct its position or pose. If the robot device determines that it needs to correct its position or pose (for example, due to detecting conditions and / or events as described above), the robot device may correct its position or pose in 705b. In 706a, the robot device may determine whether it needs to modify or interrupt the execution of its plan or trajectory. If the robot device determines that it needs to modify or interrupt its execution, the robot device may modify or interrupt its execution in 706b. In 707a, the robot device may determine whether it needs user input. For example, the robot device may encounter an obstacle or problem that it cannot deal with autonomously (for example, a failure that it cannot recover from autonomously). If user input is needed, the robot device may obtain user input in 707b. If there are no conditions or events that require replanning and / or user input (i.e., 705a, 706a, 707a: No), the robotic device may continue its operation as planned.

[0134] In some embodiments, a robotic device can be trained to perform floor detection (e.g., which floor the robotic device is near or on). For example, a robotic device can be trained by capturing sensor data (e.g., data while the robotic device is riding an elevator) against reference data (e.g., data on the robotic device's starting floor). Such data can then be used to determine the relative height of the robotic device, which can then be correlated with changes in the floor. As another example, a robotic device can implement a trained machine learning algorithm (e.g., trained on pressure data, temperature data, IMU data, camera images, etc.) to detect the nearest floor to the robotic device.

[0135] Figure 14 is an illustrative representation of an elevator map 1500 as interpreted by a robotic device, according to several embodiments. The elevator map 1500 can depict a top-down view of an elevator lobby 1502 that can include doors to a plurality of elevators 1504a, 1504b, and 1504c. In particular, the elevator lobby 1502 can include three elevator doors to three elevators 1504a, 1504b, and 1504c. A robotic device having this map can understand that there are three doors from which the robotic device can board one of the three elevators 1504a, 1504b, and 1504c. In the map, the elevator lobby 1502 may be marked using a first bounding box, and the elevator locations may be marked using second, third, and fourth bounding boxes, respectively. During lobby management, as described above, the robotic device can use a map to determine a pose or waypoint to navigate to board one of the three elevators 1504a, 1504b, and 1504c, for example, based on detecting elevator arrival.

[0136] Figure 15 depicts the architecture of a state-machine flow 1600 for elevator navigation by a robotic device, according to several embodiments. The state-machine flow 1600 includes elements associated with observations, states, and / or actions. State elements define the states of the robotic device. Observation elements are or may include processes and / or modules that the robotic device can execute to observe and / or perceive data. Action elements are or may include processes and / or modules that the robotic device executes actions.

[0137] Similar to the lobby management described above with reference to Figures 8 and 9, a robotic device in lobby management state 1611 can receive data from the elevator state detection sensing system 1601 and the elevator arrival detection sensing system 1603. A robotic device in lobby management state 1611 can also perform actions including, for example, requesting an elevator (1621) and / or entering an elevator (1623).

[0138] Once the robot device enters the elevator, it can operate in passenger management state 1613. As described above with reference to Figures 10-12, in passenger management state 1613, the robot device can, for example, receive data from the floor detection perception system 1605 to determine, for example, which floor the robot device is on while the elevator is moving. In passenger management state 1613, the robot device can perform actions such as requesting a floor to bring the elevator to a desired floor (1625) and exiting the elevator (1627).

[0139] Figure 16 depicts the architecture of a behavior tree 1700 for lobby management by a robotic device, according to several embodiments. The behavior tree 1700 may include one or more processes, modules, and / or functions that the robotic device can execute during lobby management. The behavior tree 1700 may include a root node 1701. Two child nodes can be executed in parallel or sequentially from the root node 1700, or different parts of a node can be executed simultaneously and / or sequentially. For example, a perception node 1711 and a task node 1721 can be executed in parallel, or the task node 1721 can be executed after the perception node 1711 has been executed. In some embodiments, the perception node 1711 and the task node 1721 can be executed continuously or periodically, for example, when the robotic device is in the lobby.

[0140] The perception node 1711 can refer to a task that perceives, observes, and / or scans contextual information. In particular, the perception node 1711 may include performing lobby perception 1713. Lobby perception 1713 can refer to perceiving or sensing information about the elevator lobby (e.g., whether the elevator has been called, whether the elevator is about to arrive, the occupancy status of the elevator lobby, etc.). The information obtained by the perception node 1711 can update the robot device's representation of the elevator lobby environment and notify the robot device of one or more decisions or actions it may take, for example, under task node 1721.

[0141] Task node 1721 may contain multiple child nodes that can be organized from left to right in the order of scanning. These child nodes may include, for example, elevator change node 1722, elevator ready node 1724, elevator arrival expected node 1726, and / or one or more additional nodes. Child nodes under task node 1721 may be executed according to a predefined ranking or order. Each node 1722, 1724, 1726, and 1724 may be associated with a condition, state, or event that can be checked based on information obtained from lobby perception 1713. When elevator change node 1722 is running, it determines whether the best elevator to board has changed, and if it has changed, the elevator change node proceeds to elevator change ready node 1731 to perform one or more actions associated with the elevator change. The elevator ready node 1724, while running, determines whether the target elevator is ready (for example, whether the elevator has arrived and its doors are open for boarding). If the target elevator is ready, the elevator ready node transitions to the elevator boarding node 1733 and boards the target elevator. The elevator arrival prediction node 1725, while running, determines whether the target elevator is about to arrive. If the elevator is about to arrive, the elevator arrival prediction node transitions to the approaching elevator node 1735 and approaches the arriving elevator. In some embodiments, the robotic device may perform one or more additional tasks, such as observing the elevator lobby, moving to avoid obstacles, or moving to avoid blocking other objects.

[0142] In some embodiments, a behavior tree can be embedded in a state machine that enables decision-making at two levels. The state machine can explicitly define paths based on high-level discrete decisions. These high-level discrete decisions can describe the overall task objective of the state machine. For example, the decision to press a button or wait for an elevator is a high-level decision and may involve more complex instructions to be executed. Deciding when to stop the movement of a robotic device's arm can be a low-level discrete decision, and such a decision may not need to be further extended.

[0143] In some embodiments, a behavior tree may have multiple overlapping paths in providing the most efficient path to achieve a goal. Paths may overlap, for example, when multiple different traverses of the tree can reach the same goal in different ways using the same nodes. In some cases, the most efficient path can be determined by prioritizing actions that lead the robot device directly to a goal (e.g., a waypoint) and switching to more complex actions when an action toward the goal fails. In some cases, a behavior tree can serve as a way to organize backup plans. In some cases, the robot device can start on the simplest path toward the goal, and at each point where something goes wrong (e.g., an action fails or the situation does not allow for an action), the robot device can branch off and perform alternative actions that were created and / or trained to be performed during the setup phase.

[0144] In some embodiments, the behavior tree 1700 can be interpreted as two trees, for example, one for perception (e.g., perception node 1711) and one for tasks (e.g., task node 1721). Each of the perception node and the task node can be considered as its own behavior tree. Thus, the perception tree (with perception node 1711 as the root node) and the task tree (with task node 1721 as the root node) can be executed in parallel or sequentially and / or executed (or parts thereof can be executed in parallel or sequentially). The perception node 1721 can determine the state of the elevator lobby, for example, using audio cues and laser scans of the elevator lobby. In some cases, a robotic device can perceive the elevator lobby by integrating visual cues. The task node 1721 can be responsible for controlling the actions of the robotic device. In some embodiments, information from the perception node 1711 can be supplied to the task node 1721 to ensure that the latter is acting on the latest information obtained by the former.

[0145] In some embodiments, the behavior tree 1700 may aim to get the robotic device onto the elevator as quickly as possible. Getting it onto the elevator as quickly as possible may include the robotic device observing both elevators from a central spot. When the correct elevator is about to arrive, the robotic device positions itself for boarding without blocking the doors or people exiting the arriving elevator. When the doors are open, the navigation system attempts to determine whether it is possible to board the elevator (e.g., there is space, there are no obstacles to movement at the doors, etc.) and where the robot should navigate within the elevator (e.g., the goalway point). In other words, the behavior tree 1700 can ensure that the most immediate action is taken to board, or that the robotic device performs actions that contribute to boarding until it is possible. It should be understood that training the robotic device to navigate and / or perform actions may be performed before the robotic device is deployed, for example, so that the elevator lobby is represented in data. The location of the elevator, the location of the doors, the location of the call button, the location of a good spot to observe the elevator, and the location of a smart spot to approach the elevator can all be trained before deploying the robotic device for autonomous elevator navigation.

[0146] In some embodiments, detecting whether an elevator door is open or closed can be based on the percentage of different beams. Figure 17 illustrates different possible light beams that can be associated with door detection analysis according to some embodiments. As shown in Figure 17, a sensor (e.g., a lidar sensor) of a robotic device located at the sensor origin can transmit a signal (e.g., a light or laser beam) toward the elevator door, which is represented as a bounding box. The door bounding box can represent the space or area that the elevator door is expected to occupy, or the space or area separating the elevator lobby from the elevator. The signal can be configured to target the door bounding box, return to the sensor origin, and determine the classification of the signal and the classification of the elevator door (e.g., door state).

[0147] In some embodiments, the door bounding box can be translated into an angular range and a distance from the sensor origin. Based on this information, the robotic device can determine whether one or more beams emanating from the sensor origin are hit beams, near-range beams, or miss beams. A near-range beam is a beam that stops before a minimum threshold distance, a hit beam is a beam that stops at the expected door distance (i.e., the door bounding box), and a miss beam is a beam that travels further than the door distance.

[0148] Figure 18 illustrates a decision tree 1900 for determining the state of an elevator door (e.g., open or closed) by a robotic device, according to several embodiments. The decision tree 1900 can be implemented by either the robotic device described herein or other computing devices. The decision tree 1900 may include determining the elevator door state using sensor data, such as beam information, as described with respect to Figure 17. The decision tree 1900 may include determining the number or percentage of near beams, far beams, and hit beams. The determination or classification of whether the elevator door is open, closed, or undetermined / unknown may be based, for example, on the total number of returned beams and the proportion or percentage of beams that are near beams, far beams, and hit beams.

[0149] In 1903, if the total number of returned beams is insufficient (e.g., less than a predetermined number or less than a predetermined percentage of the total number of beams), the door state is undetermined / unknown. If the total number of returned beams is sufficient, in 1905, if the percentage of beams that are far-reaching (i.e., extending beyond the door bounding box) is greater than a predetermined percentage X%, the door state is open. If the percentage of beams that are far-reaching is less than X%, in 1907, if the percentage of beams that are hit (i.e., extending to the distance of the door bounding box) is greater than a predetermined percentage Y%, the door state is closed. Otherwise, the door state is undetermined / unknown. It can be understood that other door classification schemes may also be used to determine the door state.

[0150] In some embodiments, elevator arrivals can be determined using a trained classifier, such as a CNN. Figure 19 illustrates an exemplary pipeline 2200 for classifying elevator arrivals using a CNN 2308 according to some embodiments. Classification can be performed by any of the robotic devices and / or other computing devices described herein. Pipeline 2200 may include receiving input acoustics 2302 (e.g., raw sound data or a file). Pipeline 2200 may include preprocessing the input acoustics 2302 2304, which includes, for example, converting the input acoustics 2302 into a Mel-scaled spectrogram 2306 or other format.

[0151] The Mel-scaled spectrogram 2306 can be input to a CNN (e.g., a linear classifier model) which can output predictions of elevator arrival. For example, the output prediction may be a binary output indicating, for example, whether the elevator is about to arrive or not. Alternatively, the output prediction may be a multi-classified output indicating, for example, whether an ascending elevator is about to arrive, a descending elevator is about to arrive, or the elevator is not about to arrive. CNN2308 can be a pre-trained CNN which may be trained and / or validated using labeled audio data, for example, audio data indicating elevator arrival and / or elevator ascending or descending, and labeled accordingly. In some embodiments, the CNN can be trained with data captured by a deployed robotic device, for example, based on comparing the captured acoustic data with elevator API information. In some examples, the weights of CNN2308 can be initialized using Kaiming's usual initialization strategy. In some cases, batch normalization may follow each convolutional layer in CNN2308. In some cases, CNN2308 can be trained using cross-entropy loss.

[0152] Various CNNs were trained using a training dataset of acoustic data labeled and associated with elevator sounds. The CNN results were compared with those of an existing speech recognition model (Wav2Vec2). Figures 20A–20C illustrate the model accuracy and validation loss comparisons from acoustic signal classification using CNNs and the pre-acoustic model (Wav2Vec2). Various augmentations were implemented to generate larger training datasets when training the CNNs. Augmentations included enhancements via time shifting and / or Mel spectrogram masking. Time shift augmentation can include shifting the acoustic input from left to right by a random amount. Mel spectrogram masking can include using a sampled, masked spectrogram input. Mel spectrogram masking can include frequency masking and / or time masking. Frequency masking can include randomly masking and excluding a range of consecutive frequencies by adding horizontal bars to the spectrogram. Time masking involves randomly blocking and excluding a range of time from the spectrogram by using vertical bars.

[0153] Figure 20A depicts a comparison plot of model accuracy. Model accuracy was calculated against a separate validation dataset not used in the training process. Accuracy was calculated by dividing the number of correct predictions by the total number of predictions. Figure 20B depicts a comparison plot of validation loss. Validation loss was also calculated against a pending validation dataset. Validation loss was based on cross-entropy. As shown in Figures 20A-20B, the CNN trained with all data augmentations had the highest accuracy and the lowest validation loss. Figure 20C shows the confusion matrix of the CNN trained with all data augmentations, showing a low percentage of inaccurate predictions. The matrix compares the actual target value with that predicted by the machine learning model. In Figure 20C, none indicates that the elevator is not about to arrive, a bell indicates that the ascending elevator is about to arrive, and two bells indicate that the descending elevator is about to arrive.

[0154] Figures 21A and 21B illustrate exemplary screenshots for floor button management by a robotic device according to several embodiments. As shown in Figure 21A, screenshot 2400A depicts the field of view of a robotic device facing an elevator door in an elevator lobby. From the robotic device's sensors, the robotic device can perceive objects such as collision object 2401, collision object 2404, reference 2402, call button panel 2403, and 2D door bounding box 2405. Collision objects 2401 / 2404 can be perceived as obstacles by the robotic device. Reference 2402 can be a reference point for the robotic device during elevator navigation. The call button panel 2403 can represent an object associated with a waypoint for the robotic device to press a button to call the elevator. The 2D door bounding box 2405 can indicate an area in the elevator lobby / shaft where the elevator will arrive.

[0155] As shown in Figure 21B, screenshot 2400B can depict the field of view of a robotic device from inside an elevator. The field of view may include a 2D bounding box 2415, a reference 2412, a floor button panel 2413, etc. The floor button panel 2413 may represent the space within the field of view that the robotic device can interact with to press a desired button. In some cases, the floor button panel 2413 may be the reference 2412. This field of view may also show a 2D door bounding box 2415, which may be the space that the robotic device can interpret as an elevator doorway. Elevator Pose Selection

[0156] A robotic device may utilize an elevator tack pose (e.g., the pose or configuration of the robotic device's operating element) when operating an elevator car, as considered with respect to Figure 11. When operating an elevator car, the robotic device may face the following constraints: (1) the robotic device has a limited time to press the elevator button, and (2) the space available for the robotic device to move its arm and base to the desired position is limited by the size, layout, and / or occupancy of the elevator. Both of these factors can influence the selection of the elevator tack pose. The elevator tack pose is the starting position of the trajectory of the robotic device's arm for pressing the elevator button. The elevator tack pose is designed to position the robotic device's arm closer to the body to reduce interference with nearby people. Given that each elevator car may have a different configuration, different tack poses should be adjusted to suit different elevator car configurations in order to improve the success of the robotic device when pressing the elevator button. Implementing different elevator tack poses can also increase the number of possible base positions, as a particular elevator tack pose may allow for safe and skillful movement of the robot device to otherwise restricted base positions. Different elevator tack poses may have different success probabilities for pressing elevator buttons, depending on the base location of the robot device, the size and layout of the elevator, and the location of the button panel within the elevator. A mechanism for selecting an elevator tack pose from various elevator tack poses to maximize or increase the success rate of pressing elevator buttons is described below with reference to Figures 22 to 33.

[0157] Figure 22 is an example of a reachability map that shows the probability or likelihood of a robotic device successfully pressing an elevator button as a grid of base positions within an elevator car. The elevator floors are represented by an XY plane, and points on this plane represent the positions of the robot base within the elevator. In one embodiment, the x-axis of the reachability map (i.e., the axis labeled "X") corresponds to coordinates in meters along the width of the elevator car, with the center of the elevator car's width located at coordinate 0.0. The y-axis of the reachability map (i.e., the axis labeled "Y") corresponds to coordinates in meters along the depth of the elevator car, with the center of the elevator car's depth located at coordinate 0.0. The reachability map can be generated by determining the number of times the button was successfully pressed over multiple attempts at a given base position within the elevator car. The reachability map indicates from which base positions within the elevator car the robotic device will not succeed in pressing the button, or is likely not to succeed (i.e., “no successes”), will succeed in pressing the button at least once, or is likely to succeed (i.e., “at least one success”), or will succeed in pressing the button on all attempts, or is likely to succeed (i.e., “all successes”). Base positions from which the button will not be pressed are marked with an “X”, base positions from which the button will be pressed at least once on all attempts are marked with a square, and base positions from which the robotic device will succeed in pressing the button on all attempts are marked with a circle. While the “X”, squares, and circles are used to indicate the probability or level of success for a given position, it can be understood that the robotic device (such as any of the robotic devices described herein) may use other information to indicate or classify levels of success associated with multiple given positions within the elevator. In some embodiments, the reachability map for each elevator car is generated offline so that a robotic device can quickly load an existing reachability map and navigate to a base position that gives a high probability of success to a given elevator car and / or elevator bay.

[0158] Figure 23 illustrates a set of reachability maps corresponding to each elevator button and elevator tack pose for elevator cars with different layouts. The results of the reachability maps may depend on whether the button panel is located on the left side of the elevator (i.e., "left-side elevator") or on the right side of the elevator (i.e., "right-side elevator"). The results of the reachability maps may also depend on which button on the button panel the robotic device intends to press and the elevator tack pose the robotic device starts with. As shown in the figure, reachability maps are generated for each elevator button (button 1, button 2, button 3, button 4, button 5), taking into account each elevator tack pose (tack pose 0, tack pose 1, tack pose 2) for left-side and right-side elevators. In some embodiments, the set of reachability maps may be used in an algorithm employed by the robotic device to inform the robotic device which elevator tack pose and / or base position is likely to lead to successful pressing of a given elevator button. As shown in the figure, the reachability map corresponding to the left elevator results in a lower success rate than the reachability map corresponding to the right elevator, suggesting that the robotic device may need to select from a smaller subset of base positions when entering the left elevator in order to successfully press one of the elevator buttons.

[0159] Figure 24 shows a subset of reachability maps that a robotic device can use to select the elevator tack pose that is most likely to allow the robotic device to successfully press a specific button inside the elevator car. As shown in Figure 24, the button to be pressed is "button 3". In some embodiments, the robotic device may compare the reachability maps of three elevator tack poses corresponding to "button 3" and select the elevator tack pose to use (e.g., the reachability map with the largest solid-line shaded area) based on the total number of "success" coordinates. For example, in some embodiments, the robotic device may select "elevator tack pose 0" because the reachability map corresponding to this elevator tack pose has the largest number of "success" coordinates. In some embodiments, the robotic device may select the elevator tack pose to use based on the largest number of connected "success" coordinates (e.g., the reachability map with the largest number of consecutive "success" or solid-line shaded areas). In some embodiments, "success" coordinates may be considered connected if they have at least one adjacent "success" coordinate. In some embodiments, a "success" coordinate can be considered connected if it has adjacent "success" coordinates in all basic directions (i.e., up, down, left, and right). In some embodiments, the robotic device may select "elevator tack pose 0" because the reachability map corresponding to this elevator tack pose has the maximum number of connected "success" coordinates.

[0160] Figures 25A–25C depict a robotic device in three exemplary elevator tuck poses. As shown in Figure 25A, the robotic device is positioned in "tuck pose 0". As shown in Figure 25B, the robotic device is positioned in "tuck pose 1". As shown in Figure 25C, the robotic device is positioned in "tuck pose 2". Each elevator tuck pose may be configured to position the arm of the robotic device to occupy the smallest amount of space. In some embodiments, the elevator tuck pose may be configured to allow the robotic device to easily and skillfully move the arm to press an elevator button.

[0161] Figure 26A depicts a top view of a robotic device pressing an elevator button, with the robotic device's base positioned in two exemplary orientations. Each base position can have multiple orientations, and therefore the robot's location within the elevator can be represented in three dimensions. In some embodiments, a reachability map can be generated by determining the number of times the button is successfully pressed over multiple attempts at a given base position (e.g., base position and orientation) within the elevator car. As shown in the left panel of Figure 26A, the base of the robotic device is positioned such that a vector or axis (shown as a white arrow) extending from the front point of the base creates a 0-degree angle with the x-axis (i.e., the depth of the elevator car). As shown in the right panel of Figure 26A, the base of the robotic device is positioned such that a vector or axis (shown as a white arrow) extending from the front point of the base creates an angle theta with the x-axis. In some embodiments, the angle theta from which the base of the robotic device can be oriented is in the range of approximately -90 degrees to approximately 90 degrees with respect to the x-axis. In some embodiments, the angular thetas that the base of the robotic device can be oriented are in the range of approximately -80 to approximately 80 degrees, approximately -75 to approximately 75 degrees, approximately -60 to approximately 60 degrees, and approximately -45 to approximately 45 degrees with respect to the x-axis. In some embodiments, the range of possible angular thetas of the base that the robotic device can use is constrained by whether the orientation of the robotic device is socially acceptable so as not to disturb people in its vicinity. For example, an angular theta of the base that results in the front of the robotic device facing away from the elevator button panel may not be considered because such an orientation may appear socially abnormal to other people in the elevator.

[0162] In some embodiments, the probability of successfully pressing each elevator button can be calculated for each angle theta of the base at each base location. Figure 26B shows the transformation of an exemplary two-dimensional reachability map to an exemplary three-dimensional reachability map. As shown, the three-dimensional reachability map has a first dimension (labeled "X") corresponding to the depth of the elevator car, a second dimension (labeled "Y") corresponding to the width of the elevator car, and a third dimension corresponding to the angle theta. This reachability map shows the probability that a robotic device will successfully press an elevator button at a given base position and angle theta. Certain embodiments of the reachability map in Figure 26B may be substantially similar to the reachability maps in Figures 22–24, and therefore, certain embodiments of the reachability map in Figure 26B are not described herein with respect to Figure 26. In some embodiments, the robotic device may compare three-dimensional reachability maps for each elevator button and each elevator tack pose and select the elevator tack pose to use (e.g., the reachability map with the largest solid line shaded volume) based on the total number of "success" coordinates. In some embodiments, the robotic device may select the elevator tack pose to use based on the maximum number of connected "success" coordinates (e.g., the reachability map with the largest continuous solid line shaded volume).

[0163] Figures 27A and 27B show reachability maps that depict coordinates that can be cropped to reduce the computational load on a robotic device. When operating an elevator car, the robotic device must be able to quickly estimate the optimal elevator tack pose and base positioning so as not to miss the elevator car or cause delays to people around it. Therefore, the algorithm used by the robotic device may prioritize reducing computation time. As shown in Figure 27A, the reachability maps for the left and right elevators are cropped according to the successful base position. To reduce the computational load on the robotic device and therefore increase the efficiency with which the robotic device operates the elevator car, the reachability map may be cropped so that the robotic device considers only a subset of coordinates within a region of interest (ROI) when estimating the optimal base position and elevator tack pose. In some embodiments, the reachability map may be cropped so that only the coordinates closest to the elevator panel are within the ROI. In some embodiments, the reachability map may be cropped so that only coordinates that are “successful” (resulting in a 100% success rate) and / or “at least one success” (resulting in at least one successful attempt) are within the ROI. In some embodiments, the reachability map may be cropped so that base positions that are “no success” (do not result in success) are not considered by the robotic device. Although shown as a square, the ROI can be any shape that outlines a subset of coordinates being analyzed. By implementing an ROI that considers only successful base positions, unnecessary calculations performed by the robotic device can be reduced, thus improving the overall efficiency of the algorithms used by the robotic device to select elevator tack poses and base positions.

[0164] As shown in Figure 27B, general reachability maps for the left and right elevator cars are cropped according to the footprint of the target elevator car. Elevator cars can vary considerably in terms of dimensions, footprint, and button panel location. Therefore, generating a reachability map for each specific elevator can be computationally exhausting and time-consuming. The computational load on the robotic device can be reduced, and therefore the efficiency with which the robotic device operates the elevator car can be increased by generating a general reachability map and cropping the general reachability map according to the specific footprint of the target elevator car. In some embodiments, the general reachability map can be generated by generating a reachability map that is expected to cover an area larger than all or most elevators. The general reachability map can be cropped so that the robotic device considers only a subset of coordinates that fall within the footprint of the target elevator car when estimating the optimal base position and elevator tack pose. In some embodiments, certain base positions that may lead to successful pressing of an elevator button in an uncropped reachability map may no longer be considered in a cropped reachability map because these base positions may be too close to the edge of the elevator car footprint. Figures 27A–27B show reachability maps generated for a generic elevator floor plan, but reachability maps may be generated additionally and / or alternatively for generic elevator interface buttons, as will be further described with respect to Figure 28. Reachability maps may be generated based on a set of base positions (e.g., base location and orientation), button locations (e.g., button location and orientation), and tack poses. In some embodiments, reachability maps for multiple button locations may be calculated based on a set of base positions and tack poses.For example, a set of button positions and a set of tuck poses (e.g., tuck pose 0, tuck pose 1, tuck pose 2 shown in Figures 25A to 25C) may be predefined, and reachability maps across multiple base positions may be generated for each button position and tuck pose.

[0165] A reachability map for a general-purpose elevator button interface can be generated to reduce the amount of time a robotic device spends operating a given elevator. The general-purpose elevator button interface can be a commonly used elevator interface and / or a theoretical elevator button interface. When a robotic device interacts with a specific real-world elevator button interface, a reachability map for a target button position on that specific elevator button interface can be interpolated from a reachability map generated for the general-purpose elevator button interface. Figure 28 illustrates an example of determining a reachability map for a target button position Bx that differs from the set of button positions B1, B2, ... Bn of a general-purpose left-side elevator button interface. As illustrated, the general-purpose left-side elevator button interface has 20 button positions illustrated by a grid of hexagons B1, B2, ... Bn. Although illustrated using 20 button positions, the general-purpose elevator button interface may include any number of preferred button positions or a number of predefined positions. Reachability maps can be generated for each button position B1, B2, ... Bn, and for each tuck pose (e.g., tuck pose 0, tuck pose 1, and tuck pose 2). Each reachability map includes reachability information for multiple base positions within the elevator car (e.g., base location and orientation).

[0166] A reachability map for a target button position Bx can be calculated based on reachability maps of adjacent button positions within a generic left-side elevator button interface (e.g., button positions within a predefined distance or threshold from the target button, or button positions overlapping the target button by a specific threshold). For example, in some embodiments, the reachability map corresponding to the target button position Bx can be determined based on a filling algorithm. The filling algorithm may include any preferred interpolation method, such as mean, K nearest neighbor, natural neighbor, linear interpolation, polynomial interpolation, or any other preferred method. Once an updated reachability map is generated, the tack pose and base position that yield the best performance (e.g., maximum number of successes) can be selected for the target button position Bx.

[0167] Figures 29A and 29B illustrate a flowchart illustrating a decision tree for determining the tack pose and base position for elevator operation. As shown in Figure 29A, for a given elevator car ("car"), there may be two target buttons ("button 1" and "button 2"). For each button, the robot device generates and evaluates a reachability map and determines that elevator tack pose 1 ("tack 1") is optimal for both target buttons. After determining the elevator tack pose, the robot device, considering the reachability map corresponding to button 1 and elevator tack pose 1, determines that base position 1 ("base 1") and / or base position 2 ("base 2") are optimal locations for the robot device to use. Additionally, considering the reachability map corresponding to button 2 and elevator tack pose 1, base position 1 and / or base position 2 are optimal locations for the robot device to use.

[0168] As shown in Figure 29B, for a given elevator car ("car"), there may be two target buttons ("button 1" and "button 2"). For each button, the robot device generates and evaluates a reachability map and determines that for button 1, elevator tack pose 1 ("tack 1") and elevator tack pose 2 ("tack 2") are the optimal elevator tack poses. For button 2, elevator tack pose 3 ("tack 3") is the optimal elevator tack pose. Considering the reachability maps corresponding to button 1 and elevator tack pose 1, the robot device determines that base position 1 ("base 1") and base position 2 ("base 2") are the optimal locations for the robot device to use. Considering the reachability maps corresponding to button 1 and elevator tack pose 2, base position 1 is the optimal location for the robot device to use. Given a reachability map for button 2 and elevator tack pose 3, base position 3 ("base 3") and base position 4 ("base 4") are optimal locations for the robotic device to use.

[0169] Figure 30 is a flowchart illustrating a method 2500 for determining elevator tack pose and base position according to one embodiment. As shown, certain steps of the method for determining elevator tack pose and base position may be performed offline by a robotic device, while certain steps of the method may be performed online by a robotic device (shown in bold). In some embodiments, the initial input 2501 for the method includes the rotation range of the robotic device (i.e., the range of base orientation), the specifications of the target elevator car (e.g., dimensions, location of button panel, etc.), and the desired resolution of the reachability map. The initial input 2501 may be used in 2502 to generate a reachability map. After generating one or more reachability maps, the robotic device determines the reachable base position in 2504, and this information is supplied in 2506 to the button tack pose mapping step. In some embodiments, the reachability map may be cropped so that only reachable base positions are considered. In 2506, button tack pose mapping generation is performed, and the robotic device maps elevator tack pose options that are likely to result in a high success rate to each of the elevator buttons. For example, in some embodiments, for each elevator button, the robotic device may rank the elevator tack poses based on the maximum number of solid-shaded coordinates in the reachability map (or cropped reachability map) generated in 2502. In some embodiments, the robotic device may select a subset of preferred elevator tack poses based on the ranking. The map of preferred tack poses for each button 2508 is output from 2506 and used as input for the tack pose selection step in 2510. In some embodiments, 2506 and 2510 may occur simultaneously. For example, tack poses may be ranked, and the selection of tack poses may be at least partially based on the ranking. For example, both the ranking and selection of tack poses may be performed online in a single step. In some embodiments, the tack pose selection step 2510 may be optional.In some embodiments, the tack pose can be selectively adjusted based on the surrounding environment. In some embodiments, the tack pose selection step 2510 can be performed online by the robot device, as the robot device may consider obstacles that are monitored in real time when selecting the elevator tack pose. For example, elevator tack pose 0 may be the optimal tack pose according to the button-tack pose mapping. However, the robot device may need to select elevator tack pose 1 in order to fit within the available space in the elevator car. After the elevator tack pose has been determined, the robot device may detect the arrival of the elevator car 2512 and, in 2514, use this information to perform the navigation area selection step. The navigation area selection step 2514 can be performed online by the robot device, as the robot device may need to consider obstacles that are monitored in real time when selecting the base position. For example, a person may be standing in the optimal base position, and therefore the robot device must adapt the navigation area selection 2514 to take this obstacle into account. After the robotic device selects a navigation area, it may navigate to the selected location while in an elevator tack pose, and then proceed to press a target button.

[0170] Figure 31 is a flowchart illustrating a method for determining tuck pose and base position using a neural network model according to one embodiment. This method can be performed by any of the robotic devices described herein. Figure 31 can represent an end-to-end learning approach, thereby providing specific inputs (e.g., sensor data, elevator information) to achieve control of a robotic device without requiring intermediate steps such as reachability maps or obstacle detection. Even if, as depicted in Figure 31, a neural network model may be used for elevator tuck pose and base position selection. In an exemplary method, initial inputs to the model include the elevator footprint (e.g., coordinates of two corners) and button IDs and / or button positions. The neural network model may then output a selected elevator tuck pose and a region of base position that the robotic device can navigate, taking the initial inputs into consideration. The neural network may be configured to have a specific network architecture that learns a number of steps, processes, or modules required to select the elevator tuck pose and base pose, and may be trained using previously recorded data from a robotic device in operation. While neural networks are illustrated with reference to Figure 31, it can be understood that other types of machine learning algorithms, such as end-to-end learning approaches, can be used to determine tuck poses and base positions.

[0171] Figure 32A is a flowchart illustrating a method 2600 for using an elevator car by a robotic device according to one embodiment. In 2601, the robotic device retrieves information about the elevator bank, and in 2602, the robotic device defines parameters for generating reachability maps (e.g., resolution of the reachability map, elevator footprint, location of button panel, number of base orientations to consider, etc.). In 2604, the robotic device generates reachability maps for each elevator car, button, and elevator tack pose. In 2606, the robotic device generates a button-tack pose map for each elevator car in the bay. Specific aspects of the reachability maps and button-tack pose maps may be substantially similar to the reachability maps described with reference to Figures 22-28 and the button-tack pose map described with reference to Figure 30, and therefore specific aspects of the reachability maps and button-tack pose maps are not described herein with reference to Figure 31. In some embodiments, the robotic device may, at 2608, select an optimal elevator tack pose for each elevator car in the elevator bay. In some embodiments, the optimal elevator tack pose for a given elevator car is selected based on the elevator tack pose that yields the highest success rate for that elevator car (i.e., the highest area of ​​solid-shaded coordinates on the reachability map). At 2610, the robotic device detects the arrival (or upcoming arrival) of an elevator car. At 2612, the robotic device skillfully moves to enter the selected elevator tack pose for the elevator car about to arrive. The robotic device may then, at 2614, detect obstacles within the elevator car. At 2616, the robotic device navigates the base to an optimal position for reachability, taking into account any obstacles detected by the robotic device. In some embodiments, the robotic device may optionally update the base position at any time to account for the presence of new obstacles.

[0172] Figure 32B is a flowchart illustrating a method 2700 for using an elevator car by a robotic device according to one embodiment. In 2701, the robotic device retrieves information about the elevator bank, and in 2702, the robotic device defines parameters for generating reachability maps (e.g., resolution of the reachability map, elevator footprint, location of button panel, number of base orientations to consider, etc.). In 2704, the robotic device generates reachability maps for each elevator car, button, and elevator tack pose. In 2706, the robotic device generates a button-tack pose map for each elevator car in the bay. Specific aspects of the reachability maps and button-tack pose maps may be substantially similar to the reachability maps described with reference to Figures 22-28 and the button-tack pose map described with reference to Figure 30, and therefore specific aspects of the reachability maps and button-tack pose maps are not described herein with reference to Figure 33. In some embodiments, the robotic device may, at 2708, select the elevator tack pose with the highest probability of success for all elevator cars in a given bay. By selecting one elevator tack pose for all elevator cars in the bay, the computational load on the robotic device may be reduced, and therefore the efficiency of the robotic device may be improved. At 2710, the robotic device may skillfully move to enter the selected elevator tack pose. At 2710, the robotic device detects the arrival (or upcoming arrival) of an elevator car. Optionally, at 2714, the robotic device may readjust the selected elevator tack pose based on the arriving elevator car. At 2716, the robotic device may then detect obstacles in the elevator car. In some embodiments, the robotic device may update the elevator tack pose selection based on obstacles detected in the elevator car. At 2718, the robotic device navigates the base to the optimal position for reachability, taking into account any obstacles detected by the robotic device.In some embodiments, the robotic device may optionally update its base position at any time to account for the presence of new obstacles. Elevator Request System

[0173] To enable robotic devices to navigate elevators across multiple floors within a hospital setting, it is crucial to have a robust and efficient elevator calling system. Several considerations arise when calling an elevator in a hospital environment due to limitations of the hospital server and / or hospital network. For example, the hospital server may be configured to receive only one external connection at a given time due to bandwidth constraints and / or security issues. Furthermore, unpredictable internet connectivity can exist, causing connections to the hospital server to fail unexpectedly. In some cases, connection failures can only be resolved by restarting the system. Another possible constraint is that the status of a temporarily unavailable elevator may not be visible to the caller (e.g., the robotic device).

[0174] Figure 33 is a schematic block diagram of a system 2800 for operating an elevator using an intermediate server 2802 according to one embodiment. As shown, the system 2800 includes a hospital server 2804, one or more elevator cars 2806, and one or more robotic devices 2810, as well as an intermediate server 2802 configured to communicate with an elevator interface 2809, which is optional. The system 2800 may enable a robotic device 2810 to accomplish the following tasks: (1) calling at least one available elevator car 2806 in the elevator bay (i.e., via an elevator application programming interface (API)); (2) calling an elevator car 2806 to a desired floor inside the elevator car 2806 with inadequate or limited Wi-Fi connectivity; and / or (3) receiving immediate status updates about the elevator car 2806 at a predefined frequency (e.g., 1 Hz).

[0175] As shown in the figure, the intermediate server 2802 may be configured to request data from and / or receive data from the hospital server 2804. In some embodiments, the intermediate server 2802 may be any suitable server. In some embodiments, the intermediate server 2802 may be a cloud-hosted server. In some embodiments, the hospital server 2804 and the intermediate server 2802 may have a single half-duplex connection, meaning that the intermediate server 2802 cannot simultaneously transmit and receive data from the hospital server 2804. Thus, the intermediate server 2802 may request a status update from the hospital server 2804, and the hospital server may then send the status update. In some embodiments, the hospital server 2804 and the intermediate server 2802 may use the Modbus protocol (e.g., Modbus batch polling) so that the hospital server 2804 can initiate a status update when requested by the intermediate server 2802.

[0176] In some embodiments, the intermediate server 2802 may include a status update loop 2801 configured to periodically request status updates from the hospital server 2804. In some embodiments, the intermediate server 2802 may be configured to accumulate requests from a plurality of robotic devices 2810 and transmit the requests sequentially to the hospital server. Upon receiving a request, the hospital server 2804 may send a status update to the intermediate server 2802. In some embodiments, the status update may include the elevator car status and / or server connectivity status for any or all elevator cars. In some embodiments, the status update loop 2801 may detect disconnections of the intermediate server 2802 and / or robotic devices 2810 from the hospital server 2804. In some embodiments, status update requests may be used as probes to actively detect communication failures within the system 2800. In some embodiments, if the server connectivity status indicates that a communication failure has persisted for a predefined period, the intermediate server 2802 may restart (e.g., automatically) to resolve the communication failure and / or reduce the likelihood of a communication failure while the robot device 2810 requests the elevator car 2806.

[0177] In some embodiments, upon receiving a status update from the hospital server 2804, the intermediate server 2802 may store the elevator car state 2803. In some embodiments, the intermediate server 2802 may include a state reader loop 2805 configured to periodically read the state 2803 received from the hospital server 2804. The state reader loop 2805 may begin reading the state 2803 at a frequency of approximately 0.1 Hz to approximately 20 Hz (including all and partial ranges within that range). In some embodiments, the state reader loop 2805 may begin reading the state 2803 at a frequency of approximately 0.5 Hz to approximately 2 Hz (including all and partial ranges within that range). In some embodiments, the state reader loop 2805 may begin reading the state 2803 at a frequency of approximately 1 Hz (i.e., the loop may read the state 2803 every second). In some embodiments, the state 2803 may be automatically transmitted to the robot device 210 without a request from the robot device 2810. For example, state 2803 may be transmitted to the robot device 2810 at a frequency between 0.1 Hz and approximately 20 Hz (including the entire and partial ranges within that range). Thus, the robot device 2810 may have immediate connection status and elevator car status updates before and during the call of the elevator car 2806. In some embodiments, state 2803 may be transmitted to the robot device 2810 upon request from the robot device 2810.

[0178] The elevator interface 2809 may include a user interface tool that stores the real-time position of the elevator car 2806 based on a state 2803 stored in the intermediate server. In some embodiments, the elevator interface 2809 may visualize the position of the elevator car 2806 (e.g., real-time position) based on a state 2803 stored in the intermediate server 2802. In some embodiments, the real-time position of the elevator car 2806 may be visualized for the user and may not be stored. The elevator interface 2809 may be a program configured to run on a computing device outside of the robot device 2810, the intermediate server 2802, and the hospital server 2804.

[0179] As shown in the figure, the robot device 2810 and / or the elevator interface 2809 can communicate bidirectionally with the intermediate server 2802 using full-duplex interaction (e.g., via the WebSocket protocol). In some embodiments, as the robot device 2810 approaches the elevator bay, the robot device 2810 may send requests to each elevator car 2806 in the elevator bay where the robot device 2810 is located. In some embodiments, the robot device 2810 may sequentially send requests to each elevator car 2806, and subsequently the intermediate server may sequentially transmit the requests to each elevator car 2806 to the hospital server 2804. The hospital server 2804 may then send a signal to move one or more available elevator cars 2806 to the floor where the robot device 2810 is located. In some embodiments, the robot device 2810 may continue sending requests to the intermediate server 2802, and therefore to the elevator car 2806, until the robot device 2810 boards and / or until the elevator car 2806 times out.

[0180] In some embodiments, the elevator calling process (i.e., the algorithm for requesting an elevator) may be located outside the main state machine of the robot device 2810 (e.g., on a ROS node). For example, the robot device 2810 may send a request to the elevator calling process, which may evaluate the connection to the intermediate server 2802 and call one desired elevator as an immediate call test or connectivity test. In some embodiments, when the elevator calling process calls a desired elevator, it may receive a state 2803 from the intermediate server 2802. If the state and / or connectivity of the elevator car 2806 is confirmed, the main state machine of the robot device 2810 may perform elevator monitoring and boarding, while the elevator calling process continues calling the elevator car 2806 until the robot device 2810 successfully boards the elevator car 2806 and / or the elevator car times out. The system 2800 may enable the robotic device 2810 to detect temporary and / or immediately unrecoverable connection failures to the hospital server 2804. The system 2800 may enable the robotic device 2810 to quickly detect when the elevator is about to open when it calls the elevator. The robotic device may begin boarding when at least one elevator opens.

[0181] In some embodiments, the intermediate server 2802 is optional and may be a hospital server 2804. For example, the robotic device 2810 may receive car status information and / or connectivity status information from the hospital server 2804 without the intermediate server 2802 and request an elevator. In some embodiments, the hospital server 2804 may directly transmit and / or receive information from the robotic device 2810 and / or a computing device associated with the robotic device 2810. Floor detection

[0182] Figure 34 is a flowchart of Method 900 for training a robotic device to perform floor detection, according to an embodiment. Method 900 can be performed before having the robotic device autonomously navigate using an elevator within a building.

[0183] As described above, the robotic device needs to be able to determine when it should exit the elevator. In some embodiments, the robotic device may rely on an elevator API to determine when it has arrived at its target floor and to exit the elevator on that floor. In some embodiments, the robotic device may rely on input (e.g., from a remote user) to exit the elevator. However, in such cases, the remote user needs to have a way of determining and / or verifying the robot's current location relative to the floor. In some embodiments, the robotic device may rely on floor indicator data, such as captured by the robotic device's imaging capture device. In some embodiments, the robotic device may need to rely on sensor data, such as barometer data, IMU data, and acceleration data, to determine and / or verify its current location relative to the floor. Method 900 provides an example of how the robotic device's sensor data may be calibrated to determine and / or verify the floor.

[0184] The average floor-to-ceiling height of the floor is approximately 9 feet or 2.74 meters. Therefore, the robotic device can be calibrated to have a relative height accuracy of less than half the floor height. In 902, method 900 includes acquiring robotic sensor data (e.g., barometer data, IMU data, altimeter data, etc.) on the first floor of the building or elevator.

[0185] The robotic device can ride an elevator within the building to another floor, and in 904, the robotic device can detect, for example, when the elevator has stopped on the new floor, based on speed and / or accelerometer data. In 908, the robotic device can determine whether a pressure spike has occurred. A pressure spike may indicate that the sensor data is noisy and / or unreliable. If a pressure spike occurs (908: yes), the robotic device may optionally issue a warning in 910. In 912, the robotic device can acquire sensor data on the new floor.

[0186] In 914, the robotic device may continue to cycle through the steps of detecting when the robotic device arrives at a new floor, determining whether there is a pressure spike, and / or acquiring sensor data on the new floor, until the robotic device is on the top floor (or the final floor for training). During training, a user may be in the elevator with the robotic device and can instruct the robotic device when it reaches the top floor. Alternatively or additionally, a remote user may indicate to the robotic device that it is on the top floor. In some embodiments, the robotic device may recognize the total number of floors the building may have and can count to determine when the robotic device has reached the top floor and has collected enough sensor data to set up its floor detection algorithm.

[0187] In 916, the robotic device can match ordered sensor data with ordered floors of a building or elevator. In other words, the robotic device can associate sensor data captured at each detected floor with the corresponding floor of a building or elevator. In 918, the robotic device (or another computing device or user) can determine if there is a mismatch between the ordered sensor data and the ordered floor data. If a mismatch exists (918: yes), in 920, the data can be manually adjusted to match the sensor data with the correct floor. If no mismatch exists, in 922, the ordered sensor data and floor data can be stored. This data can later be used by the robotic device to determine which floor is near the robotic device or which floor the robotic device is located on (or the elevator the robotic device is in). An exemplary method 900 for training a robotic device to perform floor detection is depicted in Figure 34, but it can be understood that any number of floor detection training methods can be used.

[0188] It should be noted that any one or more of the embodiments and designs described herein can be conveniently implemented using one or more machines programmed according to the teachings herein (e.g., one or more computing devices used as user computing devices for electronic documents, one or more server devices such as document servers, etc.). Appropriate software coding can be readily prepared by a skilled programmer based on the teachings of this disclosure. The embodiments and implementations considered above that employ software and / or software modules may also include appropriate hardware to support the implementation of machine-executable instructions of the software and / or software modules.

[0189] Such software may be a computer program product that uses a machine-readable storage medium. A machine-readable storage medium is any medium capable of storing and / or encoding a sequence of instructions for execution by a machine (e.g., a computing device) and can cause a machine to execute any one of the methodologies and / or embodiments described herein. Examples of machine-readable storage media include, but are not limited to, magnetic disks, optical disks (e.g., CDs, CD-Rs, DVDs, DVD-Rs, etc.), magneto-optical disks, read-only memory "ROM" devices, random access memory "RAM" devices, magnetic cards, optical cards, solid-state memory devices, EPROMs, EEPROMs, and any combination thereof. As used herein, machine-readable media are intended to include a single medium, as well as a collection of physically separate media, such as a collection of compact disks or one or more hard disk drives combined with computer memory. As used herein, machine-readable storage media does not include a transient form of signal transmission.

[0190] Such software may also include information (e.g., data) carried as a data signal on a data carrier, such as a carrier wave. For example, machine-executable information may be included as a data-carried signal embodied on a data carrier, where the signal encodes a sequence or portion thereof of instructions for execution by a machine (e.g., a computing device), and any relevant information (e.g., data structures and data) that causes the machine to execute any one of the methodologies and / or embodiments described herein.

[0191] Examples of computing devices include, but are not limited to, e-book readers, computer workstations, terminal computers, server computers, handheld devices (e.g., tablet computers, smartphones, etc.), web appliances, network routers, network switches, network bridges, any machine capable of executing a sequence of instructions specifying the actions to be taken by that machine, and any combination thereof. For example, a computing device includes and / or may include a kiosk.

[0192] All combinations of the aforementioned and additional concepts considered herein (where such concepts are not contradictory) are intended to be part of the subject matter disclosed herein. Technical terms explicitly adopted herein, which may also appear in any disclosure incorporated by reference, should be given meanings that best correspond to the specific concepts disclosed herein.

[0193] The drawings are primarily illustrative and are not intended to limit the scope of the subject matter described herein. The drawings are not necessarily to scale, and in some cases, various aspects of the subject matter disclosed herein may be exaggerated or enlarged in the drawings to facilitate understanding of different features. In the drawings, similar reference numerals generally refer to similar features (e.g., functionally similar and / or structurally similar elements).

[0194] This application as a whole (including the cover page, title, headings, background art, summary, brief description of drawings, detailed description, embodiments, abstract, figures, appendices, and others) illustrates various embodiments that can be used to carry out the embodiments. The advantages and features of this application are merely representative samples of embodiments and are not exhaustive and / or exclusive. Rather, they are presented to aid in the understanding and teaching of embodiments and do not represent all embodiments. Accordingly, certain aspects of this disclosure are not considered herein. The fact that alternative embodiments may not be presented for certain parts of the innovation, or that alternative embodiments not further described may be available for parts, should not be considered to exclude such alternative embodiments from the scope of this disclosure. Many of these undescribed embodiments incorporate the same principles of the innovation, and it will be understood that other undescribed embodiments are equivalent. Accordingly, it should be understood that other embodiments may be utilized and functional, logical, operational, organizational, structural, and / or topological modifications may be made without departing from the scope and / or spirit of this disclosure. Therefore, all examples and / or embodiments are considered non-limiting throughout this disclosure.

[0195] Furthermore, aside from the purpose of reducing space and repetition, no inferences should be drawn in relation to embodiments discussed herein to embodiments not discussed herein. For example, the logical and / or topological structures of any program components (component collections), other components, and / or any combination of any set of features described throughout the drawings and / or are not limited to a fixed sequence of operations and / or arrangement. Rather, any disclosed order is illustrative, and it should be understood that all equivalents are contemplated by this disclosure, regardless of the order.

[0196] The term “automatically” is used herein to describe actions that occur without direct input or prompting from an external source, such as a user. Automatically occurring actions may occur periodically, sporadically, in response to detected events (e.g., user login), or according to a predetermined schedule.

[0197] The term "determining" encompasses a wide variety of actions, and therefore "determining" can include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, database, or other data structure), and verifying. It can also include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), and resolving, selecting, choosing, and establishing.

[0198] The phrase "based on" does not mean "based solely on" unless otherwise specified. In other words, the phrase "based on" explains both "based solely on" and "at least on."

[0199] The term “processor” should be interpreted broadly to include general-purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, etc. In some contexts, “processor” can refer to application-specific integrated circuits (ASICs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), etc. The term “processor” can refer to combinations of processing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration.

[0200] The term "memory" should be interpreted broadly to encompass any electronic component capable of storing electronic information. The term "memory" can refer to various types of processor-readable media, including random-access memory (RAM), read-only memory (ROM), non-volatile random-access memory (NVRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, and registers. Memory is said to be electronically communicating with the processor if the processor can read information from and / or write information to it. Memory integrated into a processor electronically communicates with the processor.

[0201] The terms “instruction” and “code” should be interpreted broadly to include any type of computer-readable statement. For example, the terms “instruction” and “code” can refer to one or more programs, routines, subroutines, functions, procedures, etc. “Instruction” and “code” can include a single computer-readable statement or many computer-readable statements.

[0202] The term "module" can refer to a separate but interrelated unit from which a program can be built or a complex activity can be analyzed. A module may be an extension of a main program dedicated to a specific function. A module may be code that is added as a whole or designed to be easily reusable.

[0203] Some embodiments described herein relate to computer storage products that include a non-temporary computer-readable medium (which may also be referred to as a non-temporary processor-readable medium) having instructions or computer code for performing various computer implementation operations. The computer-readable medium (or processor-readable medium) is non-temporary in the sense that it does not itself contain a temporary propagating signal (e.g., a propagating electromagnetic wave that carries information on a transmission medium such as space or a cable). The medium and the computer code (which may also be referred to as code) may be designed and constructed for a particular purpose. Examples of non-temporary computer-readable media include, but are not limited to, magnetic storage media such as hard disks, floppy disks, and magnetic tapes; optical storage media such as Compact Disc / Digital Video Disc (CD / DVD), Compact Disc-Read Only Memory (CD-ROM), and holographic devices; magneto-optical storage media such as optical discs; carrier signal processing modules; and hardware devices specifically configured to store and execute program code, such as application-specific integrated circuits (ASICs), programmable logic devices (PLDs), read-only memory (ROM), and random access memory (RAM) devices. Other embodiments described herein relate, for example, to computer program products that may include the instructions and / or computer code considered herein.

[0204] Some embodiments and / or methods described herein can be implemented by software (running on hardware), hardware, or a combination thereof. Hardware modules may include, for example, general-purpose processors, field-programmable gate arrays (FPGAs), and / or application-specific integrated circuits (ASICs). Software modules (running on hardware) can be represented in a variety of software languages ​​(e.g., computer code), including C, C++, Java®, Ruby, Visual Basic®, and / or other object-oriented, procedural, or other programming languages ​​and development tools. Embodiments of computer code include, but are not limited to, files containing microcode or microinstructions, machine instructions such as those generated by a compiler, code used to generate web services, and high-level instructions executed by a computer using an interpreter. For example, embodiments may be implemented using imperative programming languages ​​(e.g., C, Fortran, etc.), functional programming languages ​​(e.g., Haskell, Erlang, etc.), logic programming languages ​​(e.g., Prolog), object-oriented programming languages ​​(e.g., Java, C++, etc.), or other suitable programming languages ​​and / or development tools. Further examples of computer code include, but are not limited to, control signals, encryption codes, and compression codes.

[0205] Various concepts can be embodied in one or more ways, and at least one example of such is provided. The actions performed as part of the method can be ordered in any preferred manner. Thus, while shown as sequential actions in the illustrative embodiment, embodiments can be constructed in which actions are performed in a different order than illustrated, including the simultaneous execution of several actions. In other words, such features should be understood as not necessarily limited to a specific execution order, but rather limited to any number of threads, processes, services, servers, etc., that can be executed sequentially, asynchronously, concurrently, in parallel, simultaneously, synchronously, etc., in a manner consistent with the disclosure. Thus, some of these features may be mutually contradictory in that they cannot coexist simultaneously in a single embodiment. Similarly, some features are applicable to one aspect of the innovation and not to others.

[0206] In addition, this disclosure may include other innovations not described herein. The applicant reserves all rights in such innovations, including the right to embody such innovations and to file additional, continuation, partial, divisional, and so on, applications therefor. Accordingly, it should be understood that the merits, embodiments, examples, functional, characteristic, logical, operational, organizational, structural, topological, and / or other aspects of this disclosure should not be considered limitations to the disclosure or to equivalents of embodiments as defined by the embodiments. Depending on the specific needs and / or characteristics of individual and / or corporate users, database configuration and / or relational models, data types, data transmission and / or network frameworks, syntax structures, etc., various embodiments of the technologies disclosed herein can be implemented in a manner that allows for the great flexibility and customization described herein.

[0207] All definitions defined and used herein should be understood to take precedence over dictionary definitions, definitions in documents incorporated by reference, and / or the ordinary meanings of the terms defined.

[0208] As used herein and in its embodiments, the indefinite articles "a" and "an" should be understood to mean "at least one" unless explicitly stated otherwise.

[0209] As used herein and in its embodiments, the phrase “and / or” should be understood to mean “either or both” of the elements thus combined, that is, elements that exist associatively in some cases and disjunctively in others. Multiple elements enumerated by “and / or” should be similarly interpreted as “one or more” of the elements thus combined. Other elements other than those specifically identified by the “and / or” clause may exist at will, whether related to or unrelated to those specifically identified elements. Thus, as a non-restrictive example, a reference to “A and / or B,” when used in conjunction with open-ended language such as “comprising,” may in one embodiment refer to A only (optionally including elements other than B), in another embodiment to B only (optionally including elements other than A), in yet another embodiment to both A and B (optionally including other elements), and so on.

[0210] Where used herein and in embodiments, “or” should be understood to have the same meaning as “and / or” as defined above. For example, if items in a list are separated, “or” or “and / or” shall be interpreted as inclusive, that is, including at least one of several elements or lists of elements, but also including two or more, and optionally including additional unlisted items. Only terms that are explicitly indicated in the opposite way, such as “one of” or “exactly one of” or, when used in embodiments, “consisting of,” refer to including exactly one element of several elements or lists of elements. In general, where used herein, the term “or” shall be interpreted to indicate exclusive substitutes (i.e., “one or the other, but not both”) only when preceded by terms of exclusivity such as “either,” “one of,” “only one of,” or “exactly one of.” Where used in embodiments, “essentially consisting of” shall have its usual meaning as used in the field of patent law.

[0211] As used herein and in embodiments, the phrase “at least one” with respect to a list of one or more elements should be understood to mean at least one element selected from any one or more elements in the list of elements, but not necessarily including at least one of each and all elements specifically enumerated in the list of elements, and not excluding any combination of elements in the list of elements. This definition also allows for the optional presence of elements other than those specifically identified in the list of elements referred to by the phrase “at least one,” whether related to or unrelated to those specifically identified elements. Therefore, as a non-restrictive example, "at least one of A and B" (or equivalently "at least one of A or B", or equivalently "at least one of A and / or B") may refer, in one embodiment, to at least one which optionally includes two or more elements, i.e., A without B (and optionally including elements other than B); in another embodiment, to at least one which optionally includes two or more elements, i.e., B without A (and optionally including elements other than A); and in yet another embodiment, to at least one which optionally includes two or more elements, i.e., A, and at least one which optionally includes two or more elements, i.e., B (and optionally including other elements), and so on.

[0212] In embodiments and in the above specification, all transitional phrases such as “equipped with,” “include,” “carry,” “have,” “contain,” “accompany,” “hold,” and “composed of” should be understood to be open-ended, meaning “includes but is not limited to.” Only the transitional phrases “consist of” and “essentially consist of” are closed or semi-closed transitional phrases, respectively, as described in the United States Patent Office Manual of Patent Examining Procedures, Section 2111.03.

Claims

1. It is a device, Memory and The system comprises a processor operably coupled to the memory, wherein the processor executes instructions stored in the memory. Information relating to an elevator car, including the floor plan of the elevator car and the configuration of the elevator button interface inside the elevator car, Determining the probability of successfully pressing the target button over a range of values ​​of one or more second parameters from the plurality of parameters other than the first parameter, for each value of the first parameter from a plurality of parameters, by simulating pressing the target button over a plurality of trials at that value of the first parameter and over the range of values ​​of the one or more second parameters, wherein the plurality of parameters include the location of the target button, the position of the robot device, and the pose of the operating element of the robot device. For each value of the first parameter, based on the results of simulating the pressing over the multiple trials, a reachability map is generated showing the probability of successfully pressing the target button over the range of values ​​of one or more second parameters, A device configured to store the reachability map for each value of the first parameter in the memory.

2. The apparatus according to claim 1, wherein the position of the robotic device includes the location and orientation of the transport element of the robotic device.

3. The first parameter is the position of the robot device, and the one or more second parameters include the location of the target button and the pose of the operating element. The processor determines, for each value of the first parameter, the probability of successfully pressing the target button over the range of values ​​of one or more second parameters, For each of the multiple predefined positions of the robot device, the action of pressing the target button is simulated across a range of predefined locations of the target button and multiple predefined poses of the operating element. For each of the aforementioned trials, the simulation is repeated for each of the aforementioned predefined positions of the robot device, The apparatus according to claim 1, configured to determine, for each of the plurality of predefined positions of the robot device, the percentage of the plurality of attempts representing success in pressing the target button.

4. The first parameter is the location of the target button, and the one or more second parameters include the position of the robot device and the pose of the operating element. The processor determines, for each value of the first parameter, the probability of successfully pressing the target button over the range of values ​​of one or more second parameters, For each of the multiple predefined locations of the target button, the action of pressing the target button is simulated across a range of predefined positions of the robot device and multiple predefined poses of the operating element. For each of the aforementioned trials, the simulation is repeated for each of the aforementioned predefined locations of the target button, The apparatus according to claim 1, configured to determine, for each of the plurality of predefined locations of the target button, the percentage of the plurality of attempts representing success in pressing the target button.

5. The first parameter is the pose of the operating element, and the one or more second parameters include the location of the target button and the position of the robot device. The processor determines, for each value of the first parameter, the probability of successfully pressing the target button over the range of values ​​of one or more second parameters, For each of the multiple predefined poses of the control element, the action of pressing the target button is simulated within a range of the predefined location of the target button and the predefined position of the robot device. Repeat the simulation for each of the multiple predefined poses of the operation element for multiple trajectories, The apparatus according to claim 1, configured to determine, for each of the plurality of predefined poses of the operating element, the percentage of the plurality of attempts that represent success in pressing the target button.

6. The processor is configured to generate the reachability map by generating a plurality of reachability maps for a plurality of predefined poses of the operation element for each value of the first parameter. The aforementioned processor, Determining which of the plurality of predefined poses of the operating element resulted in the maximum number of positions of the robot device that resulted in successfully pressing the target button, The apparatus according to claim 1, further configured to select the pose as the optimal pose for the elevator car.

7. The apparatus according to claim 1, wherein the processor is configured to generate the reachability map for each value of the first parameter in an offline mode in which the robot device is not performing one or more plans associated with navigation or operation.

8. It is a device, An operating element comprising multiple joints and segments, configured to be positioned in multiple poses to initiate a trajectory for pressing one or more target buttons, A transport element configured to move along a surface, One or more sensors, The operating element, the transport element, and a processor operably coupled to one or more sensors, wherein the processor To determine the location of the target button inside the elevator car, To determine which of the aforementioned multiple poses the operation element is positioned in, Using the reachability map associated with the pose, and assuming the location of the target button, select the position of the transport element that is most likely to successfully press the target button, A device configured to navigate to the location of the transport element according to a route.

9. The aforementioned processor, The system is further configured to determine a set of positions of the transport element that are likely to successfully press the target button, The apparatus according to claim 8, wherein the processor is configured to select the position of the transport element from the set of positions and based on one or more environmental constraints.

10. The apparatus according to claim 9, wherein one or more of the environmental constraints include the location of an object or obstacle.

11. The aforementioned processor, The apparatus according to claim 9, wherein one or more sensors are further configured to scan for one or more obstacles along the path or inside the elevator car.

12. The aforementioned processor, The apparatus according to claim 11, further configured to select from the set of positions of the transport element a position that avoids collision with the one or more obstacles detected in the path of the robot device in response to the detection of one or more obstacles within the path of the robot device.

13. The aforementioned processor, In response to the detection of one or more obstacles within the path of the robot device, the pose of the operating element is modified to avoid collision with the one or more obstacles detected within the path of the robot device. The apparatus according to claim 11, further configured to use the reachability map associated with the modified pose to select the location of the transport element that is likely to successfully press the target button, given the location of the target button.

14. It is a device, An operating element including multiple joints and segments, A transport element configured to move along a surface, One or more sensors, The operating element, the transport element, and a processor operably coupled to one or more sensors, wherein the processor To determine the position of the target button inside the elevator car, The process involves comparing the position of the target button with a plurality of predefined positions of the target button, Based on the above comparison, in response to determining that the position of the target button does not match any of the plurality of predefined positions, the set of predefined positions closest to the position of the target button is identified from the plurality of predefined positions, A device configured to determine, at the location of the target button and over the range of the predefined locations of the transport element, based on a set of reachability maps that indicate the possibility of pressing the target button within the set of predefined locations and over the range of the predefined locations of the transport element.

15. The aforementioned processor, The apparatus according to claim 14, further configured to generate a reachability map indicating the possibility of pressing the target button at each of the plurality of predefined positions of the target button and over the range of predefined positions of the transport element.

16. The operating element is configured to be positioned in a plurality of poses for initiating a trajectory of the operating element to press one or more target buttons. The aforementioned processor, The apparatus according to claim 14, wherein for each unique combination of a pose from the plurality of poses and a predefined position from the plurality of predefined positions, the apparatus is configured to generate a reachability map indicating the possibility of pressing the target button for that unique combination of pose and predefined position, and over the range of predefined positions of the transport element.

17. The aforementioned processor, The apparatus according to claim 16, further configured to select a pose in which the operating element is positioned from the plurality of poses based on at least one of the reachability maps generated for the target button.

18. The aforementioned processor, Using one or more of the aforementioned sensors, the environment surrounding the operating element and the transport element is scanned to determine whether there is one or more obstacles in the environment. The apparatus according to claim 14, further configured to determine a position for moving the transport element within the elevator car based on the reachability map determined for the position of the target button and the location of the one or more obstacles, in response to the determination that there is one or more obstacles.

19. The processor uses the predefined set of locations, The apparatus according to claim 14, configured to identify by identifying one or more predefined positions from the plurality of predefined positions that overlap with the position of the target button by a threshold amount.

20. The processor uses the predefined set of locations, The apparatus according to claim 14, configured to identify by identifying one or more predefined positions that are within a predefined distance from the position of the target button from the plurality of predefined positions.

21. It is a method, In a robotic device, the system calls at least one elevator car to arrive at the elevator lobby, Using the transport element of the robotic device, to navigate to a first location in the elevator lobby in order to await the arrival of at least one elevator car, The robot device determines that the elevator car is approaching the elevator lobby, Using the aforementioned transport element, navigate to a second location for boarding the approaching elevator car, Using one or more sensors of the robot device, monitor the state of the elevator car and determine when to board the elevator car. In response to determining to board the elevator car, the elevator car is navigated from the second location to the third location, thereby determining to board the elevator car. A method comprising pressing a target button inside the elevator car to select a destination floor.

22. The method according to claim 21, wherein calling the at least one elevator car includes sending a request to call the at least one elevator car to an intermediate device that communicates with a server configured to control the operation of the at least one elevator car, and the intermediate device, in response to receiving the request, sends a message to the server causing the server to control the at least one elevator car to arrive at the elevator lobby.

23. Calling at least one elevator car includes sending a request to call a plurality of elevator cars to an intermediate device that communicates with a server configured to control the operation of the plurality of elevator cars, wherein the intermediate device, in response to receiving the request, sends a message to the server causing the server to control the plurality of elevator cars to arrive at the elevator lobby. The aforementioned method, The method according to claim 21, further comprising, in response to detecting that an elevator car has arrived in the elevator lobby based on the monitoring, sending a message to the intermediate device notifying the intermediate device that an elevator car has arrived, and the intermediate device sending a message to the server instructing the server to stop controlling additional elevator cars from arriving in the elevator lobby.

24. Monitoring the state of the elevator car is The method according to claim 21, comprising requesting information from an intermediate device regarding the arrival status of at least one elevator car.

25. Boarding the elevator car is Using one or more of the aforementioned sensors, monitor the area near the entrance / exit of the elevator car. Based on the aforementioned monitoring, the path through which the elevator car passes through the aforementioned entrance and exit is determined, The method according to claim 21, further comprising moving the transport element of the robotic device into the elevator car through the entrance and exit of the elevator car and along the path.

26. The further includes removing the elevator car, and the removal of the elevator car is Using one or more of the aforementioned sensors, the state of the elevator car is monitored while the elevator car is inside. Based on the aforementioned monitoring, the path through which the elevator car passes through the aforementioned entrance and exit is determined, The method according to claim 25, comprising moving the transport element of the robot device outside the elevator car through the entrance and exit of the elevator car and along the path.

27. Monitoring the state of the elevator car while inside the elevator car, Using one or more sensors of the robot device, monitor when the elevator is approaching the target floor. The method according to claim 26, further comprising using one or more sensors of the robot device to monitor when the elevator door is opened on the target floor.

28. To determine that more than one elevator car has arrived at the elevator lobby, In response to determining that more than one elevator car has arrived at the elevator lobby, the system selects one elevator car to board based on scanning the environment of the elevator lobby or the more than one elevator car, The method according to claim 21, further comprising:

29. Monitoring the status of the elevator car and determining when to board the elevator car is, Transmitting multiple light beams toward a bounding box that indicates the location of the elevator car door, (i) the number of light beams extending over a distance shorter than the bounding box, (ii) the number of light beams extending over a distance corresponding to the bounding box, and (iii) The number of light beams extending over a distance longer than the bounding box is determined, The method according to claim 21, comprising determining that the door or the elevator car is open in response to determining that the number of light beams extending beyond the distance corresponding to the bounding box is greater than a predetermined threshold.

Images