System and method for autonomous motion planning and navigation of robots

Abstract

The present technique relates to navigation (e.g., path planning and movement) of a robot in an indoor environment shared with humans. The technique includes detecting human behavior, including but not limited to human motion, over time, modeling the human behavior using the human behavior history, and planning the robot's motion or movement using the modeled human behavior. [Selected Figure] Figure 1

Description

Technical Field

[0001] Description of research and development funded by the federal government. This invention was made with government support under contract number VA118 - 12 - C - 0051. The government has certain rights in this invention.

[0002] The subject matter disclosed herein relates to task planning and navigation of robots in a living environment.

Background Art

[0003] Various entities can use a robot or other autonomously controlled device in an indoor environment or other environment where humans are also present. For example, such a device can be used to move or deliver articles within the environment, clean or inspect a part of the environment, or operate appliances or equipment within the environment.

[0004] In view of such a situation, in conventional application examples, real - time detection is used to identify in real - time the presence and / or position of humans in the environment by a robot or a similar device, and to respond so as to avoid contact with the detected humans. However, such real - time detection and avoidance can be computationally intensive and may affect the computational resources of the device (for example, the robot). Moreover, the method of responding after detection is suitable for dealing with the immediate surrounding environment, but may not be very suitable for task and / or operation planning in the context of the overall plan, rather than a post - hoc response.

Summary of the Invention

Means for Solving the Problems

[0005] In one embodiment, a robot navigation system is provided. According to this embodiment, the robot navigation system comprises a sensor network configured to generate historical data of human behavior in a given environment over time, and one or more processor-based systems configured to receive this historical data of human behavior and to derive route data between multiple locations in the environment based on this historical data of human behavior. The robot navigation system further comprises a robot configured to perform one or more tasks in the environment, the robot including a processing component configured to receive route data generated from one or more sensors and historical data of human behavior, to use this route data to generate multiple waypoints for each task, to move the robot sequentially along these waypoints to perform specified actions at some or all of these waypoints, and to adjust the robot's movement along these waypoints in accordance with real-time sensing data acquired by one or more sensors mounted on the robot.

[0006] In a further embodiment, a method is provided for navigating a robot in an environment shared with humans. According to this method, historical data of human behavior in a given environment is acquired over time. Route data between multiple locations in the environment is derived based on this historical data of human behavior. Next, the robot receives the route data generated from the historical data of human behavior, and a number of waypoints corresponding to a task to be performed are generated based on this route data. The robot then moves sequentially along these waypoints, performing specified actions at some or all of these waypoints, and the robot's movement along these waypoints is adjusted according to real-time sensing data acquired by one or more sensors mounted on the robot.

[0007] In another embodiment, a robot is provided. According to this embodiment, the robot comprises one or more sensors, a drive unit including motors, and a processing component. The processing component is configured to receive path data from an external source, which is generated from historical data of human behavior, to use this path data to generate multiple waypoints for each task, to transmit commands to the motors to move the robot sequentially along these waypoints, and to adjust the robot's movement along these waypoints in accordance with real-time sensing data acquired by one or more sensors mounted on the robot. [Brief explanation of the drawing]

[0008] These and other features, aspects, and advantages of the present invention will be better understood by reading the following detailed description with reference to the accompanying drawings, where similar letters throughout the drawings represent similar parts.

[0009] [Figure 1] This is a schematic diagram of a robot and network environment according to the aspects of this disclosure. [Figure 2] This is a block diagram of a processor-based system suitable for use in the task planning and navigation system of Figure 1, according to aspects of this disclosure. [Figure 3] This figure shows an example of a heat map that displays human behavior according to the aspects of this disclosure. [Figure 4] This figure shows an example of a hidden Markov model that describes human behavior according to the aspects of this disclosure. [Figure 5] This figure shows an example of a human behavior map according to the aspects of this disclosure. [Figure 6] This figure shows an example of a cost map according to the aspects of this disclosure. [Figure 7] This diagram shows the process flow of steps when navigating a robot in a shared environment according to the aspects of this disclosure. [Modes for carrying out the invention]

[0010] One or more specific embodiments are described below. For the sake of brevity, not all features of the actual implementations are described herein. In developing any such actual implementation, it should be understood that any engineering or design project will require numerous implementation-specific decisions to achieve developer-specific goals that may vary from implementation to implementation, such as compliance with system-related and business-related constraints. It should also be understood that while such development efforts can be complex and time-consuming, they are still routine tasks of design, fabrication, and manufacturing for those skilled in the art who benefit from this disclosure.

[0011] When describing elements of the various embodiments of this disclosure, the articles “a,” “an,” “the,” and “said” mean that there is one or more of these elements. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that additional elements other than those listed may exist. Furthermore, all numerical examples in the following description are intended to be non-limiting, and additional numbers, ranges, and percentages are within the scope of the embodiments disclosed.

[0012] As described herein, a robot or robotic device may be operated in an indoor environment and move along a path to perform an action or task, such as delivering or moving goods, performing a management task, or performing a security or monitoring function. Specifically, aspects of this method facilitate the navigation (e.g., path planning and movement) of such robots in indoor environments shared with humans in a safe, robust, and efficient manner. This method addresses several challenges, including those related to steps of detecting human behavior, including but not limited to human movement; modeling human behavior and incorporating this modeled human behavior into the planning of robot action or movement; planning robot action or movement in complex indoor environments, including environments with multiple floors or levels; and combining global path planning with local response planning or coordination.

[0013] With the above in mind, this method can be characterized as having two aspects: modeling human behavior and planning hybrid actions. Regarding the modeling of human behavior, in one implementation, a network or internet-connected server (e.g., a cloud server) is connected to one or more sensors positioned to detect human behavior in the indoor environment, such as within a building, thereby continuously or periodically collecting human behavior data. Such a sensor network may be adjusted or reconfigured based on the data obtained while the system is running. Based on this collected data, routines executed on the server can generate a model of the environment that incorporates the observed human behavior and / or trends identifiable in such human behavior (e.g., trends or correlations observed in human behavior performed at a given location over different periods and / or times, or directional or flow characteristics of human movements or actions performed at each location and / or over different periods). In one implementation, such a model may incorporate the concept of "cost," and candidate robot paths or movements may be evaluated, proposed, and / or selected based on their costs determined by the model. For example, the higher the cost value associated with a particular region or location, the less desirable it may be to move a robot to that region. In other words, in such an implementation, a route may be selected or suggested based on cost minimization. In one embodiment, a graph may be created showing the “cost” for all floors in a building, and this graph may be used to plan the movements of one or more robots.

[0014] In a second aspect, the robot or a server communicating with the robot may use this cost graph to plan actions for the robot such that it can perform one or more tasks at an acceptable operating cost (such as the minimum operating cost) determined from the graph. In such a scenario, the robot can obtain information from the graph about the global environment and the probabilistic expectations related to human traffic and movement. When executing the action plan determined using the graph, real-time sensing using sensors on the robot may be employed to detect or sense the surrounding environment. In this way, local response actions and / or avoidance actions may be performed. Connecting to a cloud server to obtain both historical graph data and the latest, i.e., most recent sensor data helps the robot optimize its plan both locally and globally.

[0015] As used herein, the terms robot or robotic device include, but are not limited to, ground-based mobile robots, including those equipped with legs, wheels, tracks, etc. Such devices may further include actuators or other movable components (e.g., grippers or arms) that are programmable and movable to perform tasks. It should be understood that the devices characterized as robots herein encompass any suitable autonomous mobile device that can perform programmable movement without human supervision or with limited human supervision, or can be remotely controlled by an automation system or controller. Such programmable movement, if employed, may be based on locally generated path waypoints or path guidance, or path guidance or path waypoints generated by a remote system (e.g., a server or controller) and transmitted to the robot. Thus, as used herein, such devices move entirely or primarily without direct human intervention or control, or with limited or partial human intervention or supervision, during the operational phase or period. Although this specification primarily describes ground robots, the method may also relate to navigation and / or task planning in the context of unmanned aerial vehicles (UAVs), including fixed-wing and rotary-wing vehicles, in shared monitoring environments or enclosed environments, as well as unmanned submersible vehicles (USVs) that can move along or along the seabed of the body of water, as taught in this description.

[0016] Furthermore, to the extent that the term “path” is used herein, it should be understood that this term encompasses one-dimensional (1D) (e.g., along a trajectory), two-dimensional (2D) (e.g., defined or undefined planar paths), three-dimensional (3D) (e.g., in the air, or generally along a plane, but in places where vertical elements such as stairs, elevators, or ducts are incorporated), or four-dimensional (4D) (e.g., places with defined temporal aspects that can characterize the velocity, acceleration, or dwell time at a waypoint ground station). As used herein, paths and the like relate to planning the route or other movement plan on which a robot will travel as part of performing one or more tasks as part of an automated or semi-automated process. Thus, as used herein, “path” can be characterized as any 1D, 2D, 3D, or 4D route or path on which a device such as a robot will travel to perform one or more assigned tasks. Such paths may be adaptive, such as dynamically adapting to sensor data acquired in real time or near real time by the robot in its local environment, and may consist of one or more waypoints that the robot follows in a specific order, with the order and positions of these waypoints defining the path or route. It should be understood that such paths may further incorporate not only temporal and / or spatial positions, but also action commands to be taken at defined points and / or times along the path, such as pressing an elevator button or opening a door.

[0017] For ease of explanation and to illustrate useful real-world scenarios, various examples of indoor environments such as offices, schools, hospitals, or factories may be referred to herein. However, it should be understood that the present method can be widely applied to enclosed spaces and / or monitored spaces where both humans and robots are present and sensors can be used to monitor patterns of movement and behavior over time. For this reason, any examples described herein are merely presented for ease of explanation, and the present method is suitable for use in the indoor environments or similar environments described herein. Accordingly, the present method is not intended to be limited to the circumstances of this example.

[0018] With the above in mind, referring to the figures, FIG. 1 shows an aspect of a robot navigation system 10 that uses one or more robots 12 suitable for performing tasks in an indoor environment shared with humans. In the illustrated example, a remote server or server group (e.g., cloud server 16) accessible via a network interface is also shown. To perform storage, computing, or other functions described herein, such network or cloud-based computing may be used to access one or more remote servers, virtual machines, etc. Such a server 16 may communicate with one or more robots 12 to coordinate the operation of the one or more robots 12, such as navigating an environment shared with humans to perform one or more tasks.

[0019] Although Figure 1 shows only a single remote server 16, it should be understood that the functions performed by this remote server 16 may be performed by multiple remote servers 16 and / or by virtualized instances of the server environment. In various implementations, the remote server 16 may include a data processing unit that includes a memory component and a processor that processes data received from the robot 12 and / or instructions transmitted to the robot 12. As will be further detailed below, in some embodiments, the robot 12 may provide sensor data to the remote server 16 to transmit information about the environment currently perceived by the robot 12, which may be related to the navigation of the robot 12 and / or task execution by the robot 12.

[0020] In one embodiment, one or more robots 12 have onboard cellular or network connectivity and can communicate with a remote server 16 before, during, and / or after performing one or more tasks. In a particular implementation, the cellular or network connectivity of one or more robots 12 enables communication during task execution, allowing data acquired during the execution of each task (e.g., sensor data) to be transmitted to the remote server 16, and / or the remote server 16 to transmit commands to a given robot 12. In some cases, such as when a robot 12 is operating outside the communication range of the remote server 16, or in response to other communication interruptions, a processor inside the robot 12, i.e., an onboard processor, may determine changes to the path.

[0021] Returning to FIG. 1, certain specific details regarding the task execution and navigation system 10 will be described in more detail. For example, FIG. 1 shows a schematic diagram of one embodiment of the robot 12. However, it should be understood that in other embodiments of the robot 12, additions, reductions, and / or combinations of different components are expected to occur. As shown, the robot 12 includes a power source 20 that supplies power to operate the robot 12. The power source 20 may include a replaceable or rechargeable battery, a combustion engine, a generator, and an electric motor, a solar panel, a chemical reaction power generation system, etc., or some combination thereof.

[0022] This robot may include a user interface that can be used by a user to set or adjust various setting values of the robot 12. This user interface may include one or more input devices (such as knobs, buttons, switches, dials, etc.), and in some cases, may include a display (such as a screen, an array of LEDs, etc.) that provides feedback to the user.

[0023] The network interface 22 can communicate with a remote server 16 or other devices (such as a docking station, a remote controller, a smartphone, a computing device, a tablet, etc.) (such as via the cloud). For example, the network interface 22 can communicate via a wireless network connection, a wired network connection, a cellular data service, Bluetooth, Near Field Communication (NFC), ZigBee, ANT+, or some other communication protocol.

[0024] The detection device 26 may include one or more sensors 28 (e.g., tactile sensors, chemical sensors, methane sensors, temperature sensors, laser / LIDAR, sonar, cameras, red, blue, green, and depth (RGB-D) cameras, inertial measurement units (IMUs), etc.) configured to detect various qualities and collect data during the navigation of the robot 12 and / or during the execution of tasks by the robot 12. The sensors 28 may be used to acquire detection data corresponding to the sensor type and observation range, which conveys information about the environment in which the robot 12 is located.

[0025] The drive unit 34 can drive the robot 12 to move, such as along the ground or through the air. As shown in the figure, the drive unit 34 may include one or more motors 36 and one or more encoders 38. The one or more motors 36 may drive propellers, legs, wheels, tracks, etc. The one or more encoders 38 may detect one or more parameters (e.g., rotational speed) of the one or more motors 36 and provide the data to the control unit.

[0026] This control device may include one or more memory components and one or more processors. The motion control device may receive signals from one or more encoders 38 of the drive device 34 and output control signals to one or more motors 36 to control the movement of the robot 12. Similarly, the data acquisition control device may control the operation of the sensing device 26 and receive data from the sensing device 26. The communication interface between the sensor 28 and the onboard processor can be a standard industrial interface including a parallel bus, serial bus (I2C, SPI), and USB. The data processing and analysis device may receive the data collected by the sensing device 26 and process or analyze this collected data. In some embodiments, this data processing and analysis device may process or analyze the data in different ranges, such as based on the sensor modality and / or based on task or navigation-specific considerations.

[0027] In the examples of sensors and drive components described above, in one such example, a LiDAR and / or encoder may be used for self-localization of the robot 12 (i.e., to accurately estimate the position of the robot 12 in the environment), while other sensors (e.g., cameras including RGB and infrared) are used to collect information about the environment. In certain implementations, an RGB-D sensor is used to improve the operational efficiency of the sensing device 26 and reduce computational costs.

[0028] In the illustrated example, the robot 12 is further shown to include subsystems for robot navigation planning 40, task planning 42, SLAM (simultaneous localization and mapping) 44, and / or human detection and environmental analysis 46, and / or for executing processor-implemented algorithms. Such subsystems and / or algorithms may be provided as part of the control unit or to communicate with the control unit, as further detailed below. As shown in Figure 1, external factors or parameters such as task requirements 50 and communication with one or more remote servers 16 may also be included in the system 10.

[0029] As shown in Figure 1, another aspect of the robot navigation system 10, besides the robot 12, is a sensor network 70, which may be managed by and / or communicate with one or more remote servers 16 (e.g., one or more cloud servers). In one implementation, the sensor network 70 collects human behavior data (represented by reference numeral 76) in the environment navigated by the robot 12 and transmits this data to the remote servers 16. The sensor network 70 is run and updated continuously or periodically, thereby enabling the collection of human behavior data 76 in the environment over time. In one implementation, the sensor units 74 within the sensor network 70 are not fixed and may be moved, adjusted, added, or removed as needed.

[0030] As an example, in one embodiment, some or all of the sensors 74 may be RGB depth (RGB-D) cameras that acquire and provide both RGB and depth information to generate a point cloud of the observed environment. An RGB texture may be mapped to this point cloud to generate a 3D model of the detected environment. In such a scenario, human behavior 76 in the shared environment can be continuously tracked using the RGB-D sensors, and the position and orientation of the human skeletal joints can also be acquired over time. Based on gesture and / or other motion data that can be identified from this data using the derived skeleton, human motion behavior can be identified in the shared environment over a period of time. In addition to or instead of the RGB-D sensors, other types of sensors 74 may be used to track human behavior 76. For example, sensors 74 such as Lidar, sonar, radar, Doppler sensors, or RGB cameras may be used in addition to or instead of the RGB-D cameras, but such sensors may be limited to providing human behavior data in the form of human position information over time, as opposed to motion data derived from derived human skeletal data.

[0031] As shown in Figure 1, the data acquired by the sensor network 70 may be transferred to one or more remote servers 16. As described herein, sensor data acquired over time may be used to model human behavior in a monitored environment (i.e., an environment shared by humans and at least one robot 12). By modeling human behavior in this way, it may be possible to characterize human behavior based on one or more of the following temporal trend patterns: location, time, or other patterns (e.g., weekly patterns or monthly patterns).

[0032] With the above in mind, various embodiments of System 10 may relate to components or devices that process sensor data and / or execute routines or algorithms to model such data, or perform navigation functions in a shared environment. Before providing additional details about the operation of System 10, it may be useful to briefly show examples of embodiments of processor-based System 90 that may correspond to components or functions found in one or both of the robot 12 and / or the remote server 16. For example, referring to Figure 2, such a device or system may include components shown in the figure illustrating a block diagram of exemplary components of a computing device 90 that can be installed in the remote server 16, the robot 12, or a device communicating with a workstation or other device of System 10. As used herein, the computing device 90 may be implemented as the robot 12, or as one or more computing devices including a laptop computer, note computer, desktop computer, tablet computer, or workstation computer, and a server-type device, or a portable communication device such as a mobile phone, and / or other suitable computing device.

[0033] As shown in the figures, the computing device 90 may include various hardware components such as one or more processors 92, one or more buses 94, memory 96, input structure 98, power supply 100, network interface 102, user interface 104, and / or other computer components useful for performing the functions described herein.

[0034] These one or more processors 92 are microprocessors (e.g., CPUs, or GPUs) configured in a particular implementation to execute instructions stored in memory 96 or other accessible locations. Alternatively, these one or more processors 92 may be implemented as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and / or other devices designed exclusively to perform the functions described herein. As understood herein, multiple processors 92 or processing components may be used to perform the functions described herein in a distributed or parallel manner.

[0035] Memory 96 may contain any tangible non-temporary medium for storing data or executable routines (including routines and / or algorithms for performing the steps of modeling human behavior, pathfinding, and / or cost minimization described herein), including volatile memory, non-volatile memory, or any combination thereof. Although shown as a single block in Figure 2 for convenience, memory 96 may actually contain various distinct media or media forms located in the same or different physical locations. One or more processors 92 may access the data in memory 96 via one or more buses 94.

[0036] An input structure 98 is used to allow the user to input data and / or commands into the device 90, and this input structure 98 may include a mouse, touchpad, touchscreen, keyboard, VR controller, motion sensor, optical sensor, or microphone. The power supply 100 can be any suitable power supply for supplying power to various components of the computing device 90, including line power and battery power. In the illustrated example, the device 90 includes a network interface 102. Such a network interface 102 allows communication with other devices on the network using one or more communication protocols. In the illustrated example, the device 90 includes a user interface 104, such as a display configured to display images or data provided by one or more processors 92, and / or a speaker configured to play audio data or sound data provided by one or more processors 92.

[0037] As can be understood here, in real-world situations, some or all of this method may be performed using a processor-based system such as the computing device 90 in Figure 2 to perform functions of other processing units used to operate or monitor the remote server 16 and / or robot 12 shown in Figure 1, or the robot 12 navigating in a shared environment.

[0038] With the above description in mind, we will now return to Figure 1 to explain the interrelationships and operation of the various components introduced above. As described herein, System 10 is a navigation system used to navigate the mobile robot 12 between different work locations in an environment shared with humans. One or more computers may be used to provide onboard computing resources for the robot 12 to process sensing data (e.g., from the sensing device 26), plan and decide on movements, and otherwise control the robot and any manipulators (e.g., manipulator arms) attached to the robot 12.

[0039] To facilitate such navigation, a first aspect of this method uses a sensor network 70 to detect human movement and behavior. In one embodiment, one or more heatmaps are generated (on a remote server 16, etc.) using human behavior data acquired by the sensor network 70.

[0040] In one such implementation, the heatmap describes long-term human behavior, allowing for the identification of human movement and traversal patterns. These patterns may then be used to provide the robot 12 with navigation commands that facilitate the robot 12's execution of its assigned task. For example, these patterns may be used to determine a path that the robot 12 travels along while performing its task, limiting or minimizing the possibility of human intervention or presence.

[0041] With this in mind, in one embodiment, the sensor 74 is placed in an environment where navigation is to take place for a sufficient period of time (e.g., one day, one week, two weeks, one month, six months, or one year) to record the location of humans using time information detailed enough to identify human movement and traffic patterns. The time required for this process will vary depending on the situational environment (e.g., heavy traffic versus light traffic, or 12 hours of use versus 24 hours of use) and application requirements such as the extent to which human presence is permissible for the robot 12 to perform its tasks. For example, in one implementation, the sensor 74 can be placed in the environment for a week to collect weekly data, thereby identifying variations specific to the time of day or the day of the week. Such a scheme may be particularly useful for human behavior that has several traceable patterns that can be identified within a weekly time frame.

[0042] Referring to Figure 3, an example of a heatmap 120 generated based on sensor data acquired over time (such as from a single remote server 16) is shown. The heatmap 120 conveys the relative movements of people at different spatial locations by color, grayscale, or intensity. For example, a heatmap 120 as shown in Figure 3 can describe human behavior in a given environment, and this human behavior data is derived from skeletal data generated from detection data acquired using one or more RGB-D cameras within that environment.

[0043] Based on the detected human behavior and / or actions, a model of human behavior in the environment may be generated on a remote server 16 or the like. In one implementation, to simplify this modeling process, a Gaussian Mixture Model (GMM) is used to model locations corresponding to human behavior. "Hot" or high-activity locations are described as Gaussian models centered on the peak points of human behavior occurring daily. For example, such peak points of human behavior may correspond to public or common spaces in the environment, such as elevators, stairs, entrances and exits, corridors, conference rooms, lobbies, restrooms, or cafeterias. Mathematically, the locations in each Gaussian model are described as follows: JPEG0007848988000001.jpg740 Here, k represents the kth order, j is the index number of the kth-order Gaussian model, (x, y) is the position of the center of the kth-order Gaussian model relative to the local coordinate system, and σ is the variance.

[0044] In one implementation, the Hidden Markov Model (HMM) is generated based on temporal information (e.g., time or time related to data and the location data of each human action). This HMM model determines the locations between which a person will move if they are within an area described using a Gaussian model. An example of an HMM140 constructed according to this method is shown in Figure 4. As shown in Figure 4, indexed Gaussian models Gk(j) (shown as reference no. 142) corresponding to various locations where human actions take place are illustrated along with directional arrows 144 indicating the relationships between each model 142 based on the flow or movement of people between locations.

[0045] Based on the human activity detection process described herein, once areas of high activity are identified, a map or graph may be generated based on this information, describing information about the environment over a certain period for each modeled location and time data. In one such embodiment, the map or graph conveys the "cost" associated with movement between different modeled locations at a given time, such as the current time. Such costs may then be used in pathfinding, for example, by minimizing a cost function, thereby determining the path for the robot 12 to navigate and restricting movement through areas where human movement or presence is significant.

[0046] In the scenario of the above embodiment, a Gaussian model may be used to describe human behavior in high-activity areas within the environment. However, it is not necessary to use all models (i.e., modeled locations) in a global shortest path search operation. Instead, in cost-based methods, the important parameter is the cost between major locations, not all locations. For example, in one implementation, these major locations could include (but are not limited to) (1) elevators and stairs connecting two floors, and (2) target locations that are not high-activity areas detected as described herein, but to which the robot must move in order to perform its assigned task.

[0047] One approach is to divide the free space of a static map into a grid. The high-activity areas identified as described herein may then be overlaid or displayed on a static grid-based map (i.e., overlay map 150), as shown in Figure 5. In one such example, the cost of moving from one grid to an adjacent free grid is 1, and the cost of moving from one grid to an adjacent impedance (i.e., high-activity) grid (i.e., c(i, j)) is defined using the following equation: JPEG0007848988000002.jpg883 Here, impedance 現在 It is defined by the number of people present in that location. Also, impedance 過去 This is defined by the number of people detected at that location at time t, based on historical data.

[0048] Based on these costs, a cost map, such as a corresponding major cost map, may be generated for each floor. An example of such a cost map 160, generated from the overlay map 150 in Figure 5, is shown in Figure 6. As mentioned earlier, routes between locations may not always be necessary. However, route costs between major locations need to be identified.

[0049] In many cases, the target location for a navigation task is a public space (e.g., a conference room or office), which may not be close to a primary location. With this in mind, another primary location may be included in cost map 160 before creating the cost graph. Once cost map 160 is generated, the nearest neighbors are connected to the target node, and the cost is calculated using the Gaussian impedance region. JPEG0007848988000003.jpg868 Here, x is the target position, x iis the center of the Gaussian impedance region, and Σ is the dispersion matrix. Next, a pathfinding algorithm is applied to find the shortest path between all major locations on each floor. In one implementation, since this map is known and the heuristic cost and current cost are all known, A * You may use an algorithm. Specifically, this A * The pathfinding algorithm selects the path that minimizes the following: JPEG0007848988000004.jpg949 Here, n is the last node on the path, g(n) is the cost of the path from the starting node to n, and h(n) is a heuristic that estimates the cheapest path cost from n to the destination.

[0050] The path thus determined may be communicated to the robot 12 to follow in order to perform the assigned task, or, in other embodiments, steps or actions along the path, rather than the complete path, may be communicated to the robot 12 to facilitate the execution of the task.

[0051] In addition to the global pathfinding process described above, and returning to Figure 1, the robot 12 may provide or perform other functions to facilitate the execution of tasks and / or navigation in an environment shared with humans. For example, self-localization of the robot 12 within the environment is also a form of robot 12 navigation. In one implementation, a self-localization / SLAM function may be provided that uses information detected by Lidar and IMU (as shown by reference block 44). In this example, a self-localization algorithm is executed to continuously or periodically localize the robot 12's pose in the environment, including its position and orientation. In one such implementation, an environment model is built simultaneously. Depending on the circumstances, mapping can be performed before the robot 12 moves to perform tasks or begins to reduce computational costs. As shown in the example in Figure 1, the output of the self-localization algorithm (e.g., robot pose information or environment model) may be provided to one or both of the task planning function and / or the robot navigation planning function.

[0052] Furthermore, as shown in Figure 1, the robot 12 may be equipped with functions for performing human detection and / or environmental analysis 46. Unlike the human detection processes described herein, which contribute to global behavioral mapping based on time-lapse data or historical data (such as data that can be derived using the sensor network 70), the human detection 46 process performed on the robot 12 can be a real-time and / or local process that detects a human (or other object) currently in the immediate vicinity of the robot 12. For example, and also as shown in Figure 1, various sensors 28 of the detection system 26 (e.g., cameras, sonar, or Lidar) may be used to detect a human or object in the immediate vicinity of the robot 12 as it moves along a set path. Corrective actions (e.g., stopping or changing direction) may then be performed to avoid contact with the human or other environmental feature thus detected by the local detection system 26. In another embodiment, the information thus detected by the detection system 26 may be provided to a remote server 16 to update the cost map 160, which may update or modify the path that the robot 12 is instructed to use when performing a given task.

[0053] Regarding the task planning function 42 provided in the robot 12, in this embodiment, it plans a series of actions that the robot 12 will perform when executing an assigned task. This plan is based on task requirements 50 set by a human operator or the system. In one implementation, the output of this task planning function is an array of class instances, each instance representing one action. The description of each action may include one or more of the following: the specifications or selection of the robot 12's manipulator or arm or the base of the robot 12, the target configuration, the target action, or the determination or specification of one or more environmental constraints, and / or an indicator of operational performance. The output of the task planning function may be provided to a robot navigation planning routine or algorithm 40.

[0054] In the navigation planning function 40, which can be triggered by the task plan 42, waypoints along a path are planned to correspond to or facilitate operational tasks performed at the work site, generating a set of waypoints. As shown in Figure 1, the navigation plan of the robot 12 may receive input from the task plan 42, human detection and environmental analysis 46, self-localization / SLAM 44, one or more remote servers 16, or encoder 38, etc. As an example, a sonar detection modality may be used to detect obstacles in the environment and send a signal to the motor 36 at the base of the drive unit 34 to avoid obstacles within or outside the context of the navigation plan (for example, in response to the sudden appearance of a human in the vicinity of the robot 12). Although Figure 1 shows the navigation plan occurring at the robot 12, it should be understood that some or all of the navigation plan may occur at one or more remote servers 16, and then the waypoints may be transmitted to the robot 12.

[0055] In one example, after waypoints have been generated and validated, the task plan 42 triggers the robot navigation plan 40, as described herein. *The robot 12 (for example, the robot base) is planned using a path selection algorithm. In this example, this planning process provides a plan for the robot 12 to move through all planned waypoints and perform the corresponding desired operational task at each processing location. Commands or instructions may then be transmitted to the motors 36 of the drive unit 34 via a motion control unit or the like. This motion control unit may be configured to control the position, velocity, and / or acceleration of the robot base. Feedback to this motion control unit may be provided from one or more of the self-localization / SLAM subsystem, sensing device 26, human detection and environment analysis 46, and / or encoder 38. Furthermore, the motion planning algorithm can handle not only the movement of the robot base via the drive unit 34, but also the movement of one or more manipulators, arms, or grippers of the robot 12. Examples of algorithms that can be used for motion planning include, but are not limited to, potential field algorithms and RRT connection algorithms.

[0056] With the above in mind, Figure 7 shows an exemplary process flow for one example of this method. In the illustrated flow, processes executed by the robot 12 itself are shown on the left, and processes occurring outside the robot 12, such as on the remote server 16, are shown on the right. In this example, the navigation plan is generally illustrated as occurring on the robot 12, but as mentioned above, some or all of the navigation plan may occur outside the robot 12, in which case the waypoints 190 or other path parameters calculated as described above are transmitted to the robot 12.

[0057] Referring to the illustrated process flow example, with regard to the explanations related to Figures 1 to 6, a waypoint 190 corresponding to a certain path and tasks to be performed by the robot 12 on that path are generated and used as part of a set of instructions given to at least the drive unit 34 of the robot 12 (step 192). In this example, the waypoint 190 is generated by the robot 12 (step 194). The generation of the waypoint 190 can be based on one or more tasks 200 to be performed by the robot 12, the current position 204 of the robot 12, and the real-time or current proximity 208 of an object or person to the robot 12, which may be determined by monitoring local sensors 28 on the robot 12 such as a camera, sonar, or Lidar (step 212).

[0058] In one embodiment, the generation of waypoints 190 also uses route information, including the shortest paths between various key locations on each floor (identified based on a cost-minimizing algorithm), as described above. In this example, the process includes a step (step 220) of monitoring a sensor network 70 provided within the environment in which the robot 12 is navigating. Monitoring of the sensor network 70 may be performed over a period of time (for example, one day, seven days (for example, weekly working hours), one week, or one month).

[0059] Using the sensor data 224 thus acquired, indicators 228 of human behavior observed over the period in question are derived or identified (step 226), and / or useful trends corresponding to time, day of the week, etc. For example, as described herein, one of the figures showing the human behavior thus derived may be a heatmap 120 or a similar creation.

[0060] Next, the observed human behavior 228 may be used to model human behavior in various locations, such as areas of peak activity or areas of activity exceeding a certain threshold level for a subject (step 232). Such modeled locations 236 may further provide information about the relationships between each location, such as traffic flow information. As described herein, an example of such a model may be a hidden Markov model (140).

[0061] Using a location model, one or more cost maps 160 may be generated (step 240) that show an index of the cost (based on human behavior) of traveling between parts of all the locations in question. Such cost maps 160 may be time- and / or day-specific, for example, the cost of traveling between two locations may vary based on time and / or days. The cost maps 160 may be used together with a route determination algorithm to determine the shortest or most appropriate route 254 between locations (e.g., major locations) at a given time and / or day (step 250). These routes 254 may then be provided to the robot 12 in this example and used in navigation processes such as the waypoint generation process 194 shown in the diagram.

[0062] As can be understood from the above explanation, conventional online detection, planning, and execution algorithms may also be used during the movement of robot 12, for example, to avoid unexpected human actions. However, this method significantly reduces the computing workload on the robot. For example, robot 12 has a model of what is happening in the environment based on a model (e.g., an HMM model) built from historical data. In such an example, this model may be used to search for an optimal or suitable path based on historical information, under the assumption that the current situation is similar to a past situation that occurred at the same time every day or week.

[0063] The technical effects of the present invention include facilitating the navigation of robots (e.g., path planning and movement) in indoor environments shared with humans. The method includes the steps of detecting human behavior over time, including but not limited to human movements; modeling human behavior using the human behavior history; and planning robot movements or movements using the modeled human behavior.

[0064] This specification discloses the present invention, including its best mode, using various examples, and enables any person skilled in the art to practice the invention, including by manufacturing and using any apparatus or system and by performing any incorporated method. The patentable scope of the present invention is defined by the claims and may include other examples that a person skilled in the art could conceive. Such other examples are intended to be within the scope of the claims if they have structural elements that are not different from the literal wording of the claims, or if they include equivalent structural elements that are not substantially different from the literal wording of the claims.

Claims

1. A sensor network configured to generate historical data of human behavior in a given environment over time, One or more processor-based systems configured to receive the aforementioned human behavior history data and to derive route data between multiple locations in the environment based on the aforementioned human behavior history data, A step of creating one or more figures illustrating human behavior using the aforementioned historical data of human behavior. A step of modeling human behavior at the aforementioned multiple locations based on the aforementioned diagram of human behavior, The steps include creating one or more cost maps based on the modeled human behavior, and A step of deriving the route data between the multiple locations using the one or more cost maps. Thus, one or more processor-based systems are configured to derive the route data between the multiple locations, The step of deriving the route data includes the steps of deriving the route cost between each of the plurality of locations and identifying the lowest route cost between each of the plurality of locations, wherein the plurality of locations are modeled as locations corresponding to human behavior using a Gaussian Mixture Model (GMM) in the step of modeling human behavior. The aforementioned one or more processor-based systems, A robot configured to perform one or more tasks within the aforementioned environment, wherein the robot is One or more sensors, and A processing component configured to receive the route data generated from the history data of the aforementioned human behavior, use the route data to generate multiple waypoints for each task, move the robot sequentially along the waypoints to perform specified actions at some or all of the waypoints, and adjust the robot's movement along the waypoints in accordance with real-time detection data acquired by one or more sensors mounted on the robot. robots and A robot navigation system equipped with the following features.

2. The robot navigation system according to claim 1, wherein the sensor network includes a plurality of RGB-D cameras arranged to monitor human behavior in the environment.

3. The robot navigation system according to claim 1, wherein the one or more processor-based systems include one or more servers communicating with the sensor network and the robot via a network.

4. The robot navigation system according to claim 1, wherein the historical data of human behavior includes human behavior data acquired over at least one week, and also includes either or both of the daily trends and the hourly trends in human behavior.

5. The robot navigation system according to claim 1, wherein the route data between the plurality of locations includes route data between locations with high traffic volume in the environment.

6. The robot navigation system according to claim 5, wherein the aforementioned high-traffic area includes one or more of the following: stairs, elevators, conference rooms, lobbies, restrooms, or entrances.

7. The robot navigation system according to claim 1, wherein one or more figures showing the aforementioned human behavior include a heat map.

8. The robot navigation system according to claim 1, wherein the step of modeling human behavior at the multiple locations includes the step of generating a hidden Markov model.

9. The robot navigation system according to claim 1, wherein the one or more cost maps include corresponding main cost maps for each floor of the environment.

10. A method of navigating robots in an environment shared with humans, The steps include acquiring historical data of human behavior in a given environment over time, The steps include: using one or more processor-based systems to derive route data between multiple locations in the environment based on the historical data of human behavior; On the robot side, The steps include receiving the route data generated from the historical data of the aforementioned human behavior, The steps include generating multiple waypoints corresponding to the tasks to be executed based on the route data, The steps include sequentially moving the robot along the waypoints and performing a specified action at some or all of the waypoints, A step of adjusting the movement of the robot along the waypoint in accordance with real-time detection data acquired by one or more sensors mounted on the robot. A method including, The step of deriving the route data between the plurality of locations in the environment is The steps include creating one or more figures illustrating human behavior using the aforementioned historical data of human behavior, A step of modeling human behavior at the aforementioned multiple locations based on the aforementioned diagram of human behavior, The steps include creating one or more cost maps based on the modeled human behavior, The step of deriving route data between the plurality of locations using the one or more cost maps, wherein the step of deriving route data includes the steps of deriving the route cost between each of the plurality of locations and identifying the lowest route cost between each of the plurality of locations. A method comprising the step of modeling the aforementioned locations as locations corresponding to human behavior using a Gaussian Mixture Model (GMM) in the step of modeling the aforementioned human behavior.

11. The method according to claim 10, wherein the historical data of the human behavior is generated by a sensor network external to the robot.

12. The method according to claim 10, wherein the historical data of human behavior includes human behavior data acquired over a period of at least one week, and also includes either or both of the daily trends and the hourly trends in human behavior.

13. The method according to claim 10, wherein the route data includes route data between locations with high traffic volume within the environment.

Images