Method, Apparatus and Non-Transitory Computer-Readable Medium for Controlling a Robotic Device - Patent application

Abstract

To provide a method for controlling a robotic device based on observed object locations.SOLUTION: The method includes: observing objects in an environment; generating a probability distribution for locations of the observed objects; and controlling a robotic device to perform an action when an object is at a location in the environment with a location probability that is less than a threshold.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] Certain aspects of the present disclosure relate generally to object detection, and more particularly to an apparatus and method for detecting out-of-place objects based on spatial and temporal knowledge of objects in an environment. [Background technology]

[0002] The robotic device may use one or more sensors, such as a camera, to identify objects in the environment. The location of each identified object may be estimated. Additionally, a label may be assigned to each localized object. In conventional object localization systems, the estimated location of each object may be stored in a database along with an estimated timestamp. Summary of the Invention [Problem to be solved by the invention]

[0003] Conventional object detection systems may be limited to detecting objects within an environment. It is an object of the present invention to improve the ability of object detection devices to detect when an object is out of place. It is also desirable for the robotic system to take action when it detects that an object is out of place. [Means for solving the problem]

[0004] In one aspect of the present disclosure, a method for controlling a robotic device based on observed object positions is disclosed. The method includes observing objects in an environment. The method also includes generating a probability distribution for the observed object positions. The method further includes controlling the robotic device to perform an action when the object is at a position in the environment with a position probability less than a threshold.

[0005] In another aspect of the present disclosure, a non-transitory computer readable medium having recorded thereon a non-transitory program code for controlling a robotic device based on positions of observed objects, the program code being executed by a processor and including program code for observing objects in an environment, the program code also including program code for generating a probability distribution of positions of the observed objects, the program code further including program code for controlling the robotic device to perform an action when the object is at a position in the environment with a position probability less than a threshold.

[0006] Another aspect of the present disclosure is directed to an apparatus for controlling a robotic device based on a position of an observed object. The apparatus has a memory and one or more processors coupled to the memory. The processor is configured to observe objects in an environment. The processor is also configured to generate a probability distribution of positions of the observed objects. The processor is further configured to control the robotic device to perform an action when the object is at a position in the environment with a position probability less than a threshold.

[0007] This has outlined rather broadly the features and technical advantages of the present disclosure in order that the following detailed description may be better understood. Additional features and advantages of the present disclosure are set forth below. It should be understood by those skilled in the art that the present disclosure may be readily utilized as a basis for modifying or designing other structures for carrying out the same purposes of the present disclosure. It should also be understood by those skilled in the art that such equivalent constructions do not depart from the teachings of the present disclosure as set forth in the appended claims. The novel features believed to be characteristic of the present invention, both as to its structure and method of operation, as well as its objects and advantages, will be better understood from the following description when considered in connection with the accompanying drawings. It should be expressly understood, however, that each of the figures is provided for the purpose of illustration and description only and is not intended as a limiting definition of the present disclosure.

[0008] The features, nature and advantages of the present disclosure will become more apparent from the detailed description set forth below in conjunction with the drawings in which like reference characters identify correspondingly throughout. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 illustrates a robot in an environment according to one embodiment of the present disclosure. [Diagram 2] FIG. 1 illustrates an example of a clustering analysis map according to an aspect of the present disclosure. [Diagram 3] FIG. 1 illustrates an example of generating a clustering analysis map of object positions in an environment by observing the object positions over time. [Figure 4A] FIG. 1 illustrates an example environment according to one aspect of the present disclosure. [Figure 4B] FIG. 1 illustrates an example of a cost map according to one aspect of the present disclosure. [Figure 4C] FIG. 13 illustrates an example of a cost map having object positions integrated over a period of time, according to one aspect of the present disclosure. [Diagram 5] FIG. 2 is a diagram illustrating a hardware configuration of an object location device according to an embodiment of the present disclosure. [Figure 6] FIG. 1 illustrates a flowchart for controlling a robotic device based on observed object positions according to one aspect of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The detailed description set forth below in conjunction with the accompanying drawings is intended as a description of various configurations and is not intended to represent the only configurations in which the concepts described herein may be practiced. The detailed description includes specific details for the purpose of providing a thorough understanding of the various concepts. However, it will be apparent to those skilled in the art that these concepts may be practiced without these specific details. In some instances, well-known configurations and components are shown in block diagram form to avoid obscuring such concepts.

[0011] Conventional systems that perform visual object classification (e.g., object detection networks) generate false positive and false negative detections. False positive and false negative detections reduce the accuracy of the detection. A system or method for visual object detection may be combined with a system or method for estimating detected object locations to analyze object locations. Detected visual objects may be identified by bounding boxes. The analysis of object locations may be referred to as cluster analysis.

[0012] Errors may be detected from the cluster analysis. Furthermore, true object locations may be estimated based on the cluster analysis. Cluster analysis assumes a stationary environment and utilizes this assumption to count the number of objects in the environment, refine the estimated true location of each object, and discard detections that are not part of a spatial cluster (e.g., false positive detections).

[0013] Additionally or alternatively, the objective location estimate may be analyzed using a probability density function. A probability density function (e.g., probability density analysis) does not assume a static environment. Probability density analysis does not use cluster analysis, object counting, or false positive estimation. In one configuration, spatial and temporal searching of objects may be performed based on the probability density analysis.

[0014] Spatial and temporal search of objects may be used to detect out-of-place objects. For example, the robotic device may detect a human lying in a location. The location may be an area with a high probability of a human lying (e.g., a bed) or a low probability area (e.g., a floor near a staircase). If the location is in a low probability area, the robotic device may assist the human.

[0015] The robotic device may use one or more sensors to identify objects in the environment. The sensors may include a red-green-blue (RGB) camera, a radio detection and ranging (RADAR) sensor, a light detection and ranging (LiDAR) sensor, or another type of sensor. In images captured by the sensors, the location of one or more identified objects may be estimated. The estimate of each object's location may be stored along with an estimated timestamp. The estimated timestamp identifies the time the object was at the estimated location. In one configuration, three-dimensional (3D) object locations are determined by registering the detection bounding box centroid with a point cloud generated by a depth camera.

[0016] In most systems, images are generated at a number of frames per second, such as 10 frames per second. Furthermore, each frame may contain multiple objects. The number of frames and the number of objects per frame increases the amount of storage resources used by conventional object detection systems. Furthermore, conventional object detection devices may use more processing resources to process the images.

[0017] Conventional object detection systems can also produce false positive and false negative position estimates, i.e., for each object detected in a frame, some detections may be valid and some may be invalid.

[0018] One aspect of the present disclosure is to use a density-based cluster function to estimate the number of objects in an environment. This estimation is based on the premise that the observed density for true object locations is greater than a high density threshold. Furthermore, the observed density for false positive object locations is less than a low density threshold.

[0019] In this disclosure, for simplicity, the robotic device may be referred to as a robot. Furthermore, the objects may include stationary and moving objects in the environment. The objects may include man-made objects (e.g., chairs, desks, cars, books, etc.), natural objects (e.g., rocks, trees, animals, etc.), and humans.

[0020] FIG. 1 illustrates a robot 100 in an environment 102 according to one embodiment of the present disclosure. In FIG. 1, the robot 100 is a humanoid robot and the environment 102 is a kitchen. Aspects of the present disclosure are not limited to humanoid robots. The robot 100 may be any type of autonomous or semi-autonomous device, such as a drone or a vehicle. Furthermore, the robot 100 may be in any kind of environment.

[0021] In one configuration, the robot 100 acquires one or more images of the environment 102 via one or more sensors of the robot 100. The robot 100 can localize one or more objects per image. Localization refers to determining the location (e.g., coordinates) of an object in an image. In conventional object detection systems, bounding boxes can be used to indicate the location of an object in an image. The localized objects may be one or more specific classes of objects, such as chair 104, or all objects in the image. The objects may be localized via an object detection device, such as a pre-trained object detection neural network.

[0022] As described above, one or more identified objects within an image (e.g., a frame) may be localized. In one configuration, an object localization estimator is observed in an environment over time. In particular, cluster analysis may be applied to the object localization estimator over different time windows (e.g., different time periods).

[0023] FIG. 2 illustrates an example of a cluster analysis map 200 according to an aspect of the disclosure. In the example of FIG. 2, an object position estimator is observed in an environment over time. Each node 202, 208 identifies a location of an estimated position of an object in the environment. As shown in FIG. 2, some nodes 202 (e.g., non-clustered nodes 202) may not be associated with a cluster. Other nodes 208 (e.g., in-cluster nodes 208) may be clustered around a centroid 204. The non-clustered nodes 202 may be considered false positives. For simplicity, labels have not been provided for each of the non-clustered nodes 202 and in-cluster nodes 208 in FIG. 2.

[0024] In the example of FIG. 2, the intra-cluster nodes 208 may be edge nodes 210 or core nodes 212. The size (e.g., diameter) of the edge intra-cluster nodes 210 is smaller than the size of the core intra-cluster nodes 212. Each edge intra-cluster node 210 is associated with one neighboring node, and the distance between the neighboring nodes is less than a distance threshold. Each core intra-cluster node 212 is associated with one neighboring node, and the distance between the neighboring nodes is less than a distance threshold. In one configuration, the environment is a 3D environment, so the centroid 204 is a 3D centroid. The centroid 204 may be associated with a true location of the object.

[0025] Typically, the nodes 202 in a cluster are estimates of the location of the same object or object of the same type. For example, in an office, the location of office furniture may be less dynamic than the locations of office supplies such as pens and paper. Nodes 202 within an area of ​​locations may be clustered. For example, nodes 202 corresponding to the locations of chairs in an office may form a cluster if the locations are within a particular area over the course of observations. A 3D centroid 204 (e.g., average centroid) may be determined for each cluster.

[0026] In the case of density-based clustering, the accuracy of the estimation of the object location may be determined based on the 3D centroid 204 of each cluster. Estimating the object location based on the 3D centroid 204 may improve the accuracy of the estimation. Furthermore, the accuracy of the object localization may be determined based on the 3D standard deviation of each cluster. The 3D mean of each cluster may be used to estimate the true location of the object. The standard deviation may be used to determine the variance of the cluster. If the standard deviation is greater than a threshold, there may be uncertainty in the true location of the object. The uncertainty of the object location decreases as the variance of the standard deviation decreases. Thus, the standard deviation correlates to the accuracy of the object location.

[0027] Figure 3 illustrates an example of generating a cluster analysis map 350 according to an aspect of the present disclosure. As shown in Figure 3, the cluster analysis map 350 is based on observations of objects 302, 304 in an environment 300. In Figure 3, a robot may observe the positions of objects (e.g., chair 304 and table 302) in the environment 300 over a period of time. The robot may determine the positions of the chair 304 and table 302 for each observation.

[0028] After multiple observations, the robot may perform a cluster analysis to generate a cluster analysis map 350. The cluster analysis map 350 includes in-cluster nodes 314 and non-clustered nodes 316. The non-clustered nodes 316 are observations that are isolated from the environment 300. The in-cluster nodes 314 may be considered as repeated observations of the same object 304, 302. Each centroid 318, 320, 322 may be considered as a true object location. For example, the centroids 318, 322 correspond to the estimated true location of the chair 304. The centroid 320 corresponds to the estimated true location of the table 302. The centroids 318, 320, 322 may be 3D centroids.

[0029] False positives are common in conventional convolutional neural network detectors. Cluster analysis (e.g., density-based clustering) improves the estimation of multiple objects in a spatial environment, thereby reducing false positives. Density-based clustering can be used to set a density criterion for multiple detections in similar or identical locations. Thus, repeated observations of the same object can be taken into account. That is, in one configuration, density-based clustering is used to take multiple observations into account and discard isolated observations. Furthermore, density-based clustering can fuse repeated observations into unified object instances. Each cluster represents an example of an object, such as a bottle or a plant.

[0030] In the example of Figure 2, each cluster corresponds to an object based on multiple observations of the object. For example, hundreds or thousands of individual detections of the object may be used per cluster. Furthermore, the clustering analysis may be used to determine the quality of the location estimate. That is, the cluster analysis will determine how well the location estimator is performing.

[0031] In one configuration, a probability distribution of object locations in an environment can be generated by observing object locations over time. That is, a probability density function (e.g., probability density analysis) estimates a continuous distribution using a set of observations to represent the probability of finding an object at any given location on a map. Thus, a probability density function may be used to represent spatial and temporal knowledge about objects in a spatial environment.

[0032] The memory footprint of the data corresponding to the estimated probability density function is less than the memory footprint of the data corresponding to a conventional object localization system. For example, a conventional object localization system may generate 1 GB of object observation logs. In contrast, the estimated probability density function may be 50 KB. Because of the reduced memory footprint, the kernel density estimates may be transferred between robotic devices. Thus, the robotic devices may share knowledge about object locations in the environment.

[0033] As mentioned above, probability density function analysis estimates a continuous distribution using all forms of observation. A probability distribution of object locations in the environment is generated by observing object locations over time. Random samples may be drawn from the probability distribution. Random samples from the distribution may be more likely to be in areas of high probability as opposed to areas of low probability. Additionally, evaluating the distribution across the map grid may provide a heat map (e.g., a cost map) indicating areas of the map that are more likely to contain objects.

[0034] The robotic device may sample high probability regions to reduce the object search period. If the object is not in the high probability region (e.g., the most likely location), the object search may sample low probability regions. This probability distribution may reduce search time by distinguishing between high and low probability regions. In one configuration, if the object is found in an unlikely location (e.g., a low probability region), the robotic device may return the object to the most likely location (e.g., a high probability region).

[0035] As another example, certain objects may correspond to certain high probability and low probability regions. For example, a first set of chairs is for a kitchen dining table and a second set of chairs is for a formal dining table. In this example, the kitchen is a high probability region for the first set of chairs and the region outside the kitchen is a low probability region. For example, the formal dining room may be a low probability region for the first set of chairs. Furthermore, the formal dining room is a high probability region for the second set of chairs and the kitchen is a low probability region for the second set of chairs.

[0036] FIG. 4A shows an image of an environment 400 according to an embodiment of the invention. In the example of FIG. 4A, the environment 400 includes a road 402 and a sidewalk 424. In one configuration, a robot observes the environment 400 over time. For example, the robot may observe the environment 400 over a 24-hour period. Based on the observations, the robot generates a probability distribution of object locations within the environment 400. The objects may be humans or other types of objects. The probability distribution may be used to generate a cost map.

[0037] 4B, the cost map 450 may be generated based on a probability density analysis of objects in the environment. The cost map 450 may be based on a 2D or 3D probability density function. Therefore, any type of map may be generated, such as an ego view or a top-down view.

[0038] In the cost map 450, low probability regions may be distinguished from high probability regions. Each region may be assigned a probability value. The probability value or probability value range may be visualized by color or other visual representation. For example, the color or shading of low probability regions may be different than high probability regions. Different granularity levels may be assigned to the occupancy probability. For example, as shown in FIG. 4B, the probability regions may be low 404, light 406, medium 408, high 410, or very high 412. The likelihood of an object being present at a location may be determined from a numerical value assigned to the location in the cost map. The probability that an object is present at a location may be referred to as an object probability.

[0039] Figure 4C illustrates an example of a cost map 450 with object positions integrated over a period of time 452. In the example of Figure 4C, the period may be 24 hours. Other periods such as hours, days, weeks, months, etc. are also contemplated.

[0040] In the example of Figure 4C, each object location 452 represents the location of an object, such as a human. The cost map 450 is generated based on a probability density analysis. As described above, the probability density analysis estimates a continuous distribution using all the observations to represent the probability of finding an object at any given location on the cost map 450. The cost map may be viewed as a Gaussian mixture model with several two-dimensional Gaussian functions that are fit to the data.

[0041] Each object or type of object may correspond to a specific high probability region and a low probability region. For example, for kitchen utensils in a home environment, the high probability region may be the kitchen and the low probability region may be the bathroom. As another example, for clothes in a home environment, the high probability region may be one or more closets and the low probability region may be the kitchen. According to aspects of the present disclosure, the robotic device may return an object in the environment to a "home" location (e.g., a high probability region) without user direction or user intervention.

[0042] As described above, the cost map may be generated from a probability distribution. The cost map may be overlaid on a map of the environment. An accurate spatio-temporal probability density function for where objects are likely to be found in the environment allows for automatic detection of objects in unlikely locations. For example, an object detection system may automatically detect when a human is found in an unlikely location.

[0043] Detection is based on the position and / or pose of the object in the environment. For example, a human is more likely to be standing in a kitchen and less likely to be lying in a kitchen. As another example, a human is standing at the bottom of a staircase and less likely to be lying at the bottom of a staircase.

[0044] As mentioned above, in one configuration, an object sensing system generates a probability distribution for the location of one or more objects, such as a human, by observing the location of the objects over time. For simplicity, a human is exemplified as the object. Furthermore, aspects of the present disclosure are not limited to detecting humans and are contemplated for other objects.

[0045] The object detection system may be a component of a robot or an environmental monitoring system such as a home security camera. If an object is identified in a location other than one of the acceptable locations, the device or robot may generate an alarm. For example, the alarm may be a call to emergency services. Furthermore, if an object is not identified in an acceptable location, the robot may initiate a search for a human.

[0046] For example, based on the observed location of a human in an environment, the system generates a probability distribution for acceptable locations (e.g., likely locations), where the human may be immobile for longer than a threshold. Acceptable locations may correspond to probability regions (e.g., location probabilities) that are greater than a threshold, such as medium 408, high 410, and / or very high 412 (see FIGS. 4B-C). Unacceptable locations (e.g., unlikely locations) may correspond to probability regions (e.g., location probabilities) that are less than a threshold, such as low 404 and / or light 406.

[0047] As mentioned above, a location that differs from an acceptable location may be referred to as an unlikely location. An acceptable location may also be referred to as a possible location. For example, a hallway floor is an unlikely location for an immobile human. After determining the acceptable locations, the device may monitor the environment.

[0048] Upon detecting an object in an unlikely location, the system may generate an alert. In one configuration, the duration that the object is in an unexpected location is also determined. If the duration is greater than a threshold, the device may generate an alert. For example, an alert may be generated if a human remains motionless in an unlikely location for a period longer than a threshold.

[0049] The alert may trigger one or more actions. In one configuration, the alert may trigger the robot to initiate a verbal dialogue with the human. Additionally or alternatively, the robot may call emergency services for a telepresence assessment. The robot may attempt to move the human to an acceptable position. The alert may also trigger the robot to perform emergency services, such as chest compressions.

[0050] According to one aspect of the disclosure, the system observes the patterns of people and the spaces they occupy at any given time. If the human is not in an expected location, the system may actively search for the human. If the human is found unresponsive, the system or robot may contact emergency services.

[0051] 5 is a diagram illustrating a hardware embodiment for an object location system 500 according to one aspect of the disclosure. The object location system 500 may be a component of a vehicle, a robotic device, or another device. For example, as shown in FIG. 5, the object location system 500 is a component of a robot 528 (e.g., a robotic device).

[0052] An aspect of the present disclosure is not limited to the object location system 500 being a component of a robot 528. Other systems, such as a bus, boat, drone, or vehicle, may also use the object location system 500. The robot 528 may operate in at least an autonomous mode of operation and a manual mode of operation.

[0053] Object location system 500 may be implemented with a bus architecture, generally represented by bus 550. Bus 550 may include any number of interconnected buses and bridges, depending on the particular application and overall design constraints of object location system 500. Bus 550 links together various circuits including one or more processors and / or hardware modules, represented by processor 520, communications module 522, position module 518, sensor module 502, movement module 526, navigation module 525, and computer readable medium 515. Bus 550 may link various other circuits, such as timing sources, peripherals, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further.

[0054] The object location system 500 includes a transceiver 516 coupled to a processor 520, a sensor module 502, an object location analysis module 508, a communication module 522, a location module 518, a movement module 526, a navigation module 525, and a computer readable medium 515. The transceiver 516 is coupled to an antenna 555. The transceiver 516 communicates with various other devices via a transmission medium. For example, the transceiver 516 may receive commands via transmission from a user or a remote device. As another example, the transceiver 516 may transmit statistics and other information from the object location analysis module 508 to a server (not shown).

[0055] Object location system 500 includes a processor 520 coupled to a computer readable medium 515. Processor 520 performs processes including executing software stored on computer readable medium 515 that provides functionality according to the present disclosure. The software, when executed by processor 520, causes object location system 500 to perform various functions described for a particular device, such as robot 528 or any of modules 502, 508, 515, 516, 518, 520, 522, 525, 526. Computer readable medium 515 may also be used to store data that is manipulated by processor 520 when executing the software.

[0056] The sensor module 502 may be used to obtain measurements via different sensors, such as a first sensor 506 and a second sensor 505. The first sensor 506 may be a visual sensor, such as a stereo camera or a red-green-blue (RGB) camera for capturing 2D images. The second sensor 505 may be a ranging sensor, such as a light detection and ranging (LIDAR) sensor or a radio detection and ranging (RADAR) sensor. Of course, aspects of the present disclosure contemplate other types of sensors for either of the sensors 505, 506, such as, for example, thermal sensors, sonar sensors, and / or laser sensors, and are not limited to the aforementioned sensors.

[0057] Measurements of the first sensor 506 and the second sensor 505 are processed by one or more of the processor 520, the sensor module 502, the object location analysis module 508, the communication module 522, the position module 518, the movement module 526, the navigation module 525, along with the computer readable medium 515 to perform functions described herein. In one configuration, data captured by the first sensor 506 and the second sensor 505 may be transmitted to an external device via the transceiver 516. The first sensor 506 and the second sensor 505 may be coupled to or in communication with the robot 528.

[0058] The location module 518 may be used to determine a location of the robot 528. For example, the location module 518 may determine the location of the robot 528 using a global positioning system. The communication module 522 may be used to facilitate communication via the transceiver 516. For example, the communication module 522 may be configured to provide communication capabilities via different wireless protocols such as WiFi, LTE (long term evolution), 5G, etc. The communication module 522 may also be used to communicate with other components of the robot 528 that are not modules of the object location system 500.

[0059] The locomotion module 526 may be used to facilitate locomotion of the robot 528. As another example, the locomotion module 526 may be in communication with one or more power sources of the robot 528, such as motors and / or batteries. Locomotion may be performed via wheels, movable limbs, propellers, treads, fins, jet engines, and / or other locomotion sources.

[0060] Object location system 500 also includes a navigation module 525 for planning a path or controlling movement of a robot 528 via a movement module 526. A path may be planned based on information provided via object location analysis module 508. The modules may be executed within processor 520, reside / stored within computer readable medium 515, one or more hardware modules coupled to processor 520, or a combination thereof.

[0061] The object location analysis module 508 may be in communication with the sensor module 502, the transceiver 516, the processor 520, the communication module 522, the position module 518, the movement module 526, the navigation module 525, and the computer-readable medium 515. In one configuration, the object location analysis module 508 may receive sensor data from the sensor module 502. The sensor module 502 may receive sensor data from the first sensor 506 and the second sensor 505. According to aspects of the present disclosure, the sensor module 502 may filter the data, remove noise, encode the data, decode the data, merge the data, extract frames, or perform other functions. In an alternative configuration, the object location analysis module 508 may receive sensor data directly from the first sensor 506 and the second sensor 505.

[0062] In one configuration, the object location analysis module 508 determines a probability region for the object based on information from the processor 520, the location module 518, the computer readable medium 515, the first sensor 506, and / or the second sensor 505. The probability region may be determined by generating a probability distribution of object positions for the observed object. Based on the probability region, the object location analysis module 508 may control one or more actions of the robot 528.

[0063] For example, an action can be providing assistance to an object, such as lifting a stationary object or communicating with the object. Additionally or alternatively, an action can include contacting emergency services. The object location analysis module 508 can execute the action via the processor 520, the location module 518, the communication module 522, the computer readable medium 515, the movement module 526, and / or the navigation module 525.

[0064] FIG. 6 illustrates a flow chart 600 for controlling a robotic device based on the location of an observed object according to an embodiment of the present disclosure. For simplicity, the robotic device will be referred to as a robot. As shown in FIG. 6, in step 602, the robot observes an object in the environment. The object may be observed via one or more sensors of the robot, such as a LiDAR, RADAR, and / or an RGB camera. The object may be observed over a period of time, such as hours, days, etc.

[0065] In step 604, the robot generates a probability distribution for the positions of the observed objects. In one configuration, the robot estimates a continuous distribution using observations of the objects over a period of time. In this configuration, the probability distribution is based on a continuous distribution.

[0066] The cost map may be generated from a probability distribution. This cost map may also be called a heat map. The cost map may be overlaid on a map of the environment. The cost map may be a 2D or 3D cost map.

[0067] In step 606, an action is taken when the object is in a position in the environment with a position probability below a threshold. The robot may take an action based on the cost map. For example, the robot may assist the object and / or contact emergency services. For example, if the robot identifies a human in an unlikely position (e.g., with a position probability below a threshold), the robot may contact emergency services.

[0068] Based on the present teachings, one skilled in the art should understand that the scope of the present disclosure is intended to cover any aspect of the present disclosure, whether implemented independently of or in combination with any other aspect of the present disclosure. For example, an apparatus or method that may be implemented may be performed using any number of the aspects shown. Furthermore, the scope of the present disclosure is intended to cover such an apparatus or method that is implemented using other structure, functionality, or structure and functionality in addition to or other than the various aspects of the present disclosure described. It should be understood that any aspect of the present disclosure may be embodied by one or more elements of a claim.

[0069] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration," and any aspect described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects.

[0070] Although some benefits and advantages of the preferred aspects are mentioned, the scope of the disclosure is not intended to be limited to any particular benefit, use, or purpose. Rather, the aspects of the disclosure are intended to be broadly applicable to different technologies, system configurations, networks, and protocols, some of which are illustrated by way of example in the drawings and the following description of the preferred aspects. The detailed description and drawings are merely illustrative rather than limiting of the disclosure, and the scope of the disclosure is defined by the scope of the appended claims and their equivalents.

[0071] The term "determining" as used herein encompasses a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, database, or another data structure), ascertaining, etc., and "determining" may also include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), etc., and "determining" may also include resolving, selecting, choosing, establishing, etc.

[0072] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items, including single members. As an example, "at least one of a, b, or c" is intended to encompass a, b, c, ab, ac, bc, and abc.

[0073] Various exemplary logic blocks, units, and circuits described in connection with this disclosure may be implemented or performed using a processor specially configured to perform the functions described in this disclosure. The processor may be a neural network processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array signal (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. Alternatively, the processing system may include one or more neuromorphic processors for implementing the neural models and models of neural systems described herein. The processor may be a microprocessor, controller, microcontroller, or state machine specially configured as described herein. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or other such specialized configurations, as described herein.

[0074] The steps of a method or algorithm described in connection with the present disclosure may be embodied directly in hardware, or in a software module executed by a processor, or in a combination of the two. The software module may reside in a random access memory, a read-only ROM, a flash memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, a register, a hard disk, a removable disk, a CD-ROM or other optical disk storage, a magnetic disk storage or other magnetic storage, or any other medium that can be used to carry or store a desired program code in the form of instructions or data structures accessible by a computer. A software module may comprise a single instruction, or many instructions, and may be distributed among several different code segments, among different programs, and across multiple storage media. A storage medium may be coupled to the processor such that the processor can read information from, and write information to, the storage medium. Alternatively, the storage medium may be integral to the processor.

[0075] The methods disclosed herein may include one or more steps or actions for achieving the described method. The method steps and / or actions may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims.

[0076] The described functions may be implemented in hardware, software, firmware, or any combination thereof. When implemented in hardware, the hardware configuration may include a processing system in the device. The processing system may be implemented using a bus architecture. The bus may include any number of interconnected buses and bridges depending on the particular application and overall design constraints of the processing system. The bus may couple various circuits together, including the processor, the machine-readable medium, and the bus interface. The bus interface may be used to connect a network adapter to the processing system, among other things, via the bus. The network adapter may be used to perform signal processing functions. In some aspects, a user interface (e.g., keypad, display, mouse, joystick, etc.) may also be connected to the bus. The bus may link various other circuits, such as timing sources, peripherals, voltage regulators, power management circuits, etc., known in the art, and therefore will not be described further.

[0077] The processor may be responsible for managing the bus and processing, including executing software stored on a machine-readable medium. Software shall be construed to mean instructions, data, or any combination thereof, whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0078] In hardware embodiments, the machine-readable medium may be part of a processing system separate from the processor. However, as one skilled in the art will readily appreciate, the machine-readable medium, or any portion thereof, may be external to the processing system. For example, the machine-readable medium may include a transmission line, a carrier wave modulated by data, and / or a computer product separate from the device, all of which may be accessed by the processor via a bus interface. Alternatively, or in addition, the machine-readable medium, or any portion thereof, may be integrated into the processor, such as in the case of having a cache file and / or specialized register files. While the various components described may be described as having a specific location, such as a local component, they may be configured in various ways, such as a particular component being configured as part of a distributed computing system.

[0079] The machine-readable medium may include a number of software modules. The software modules may include a transmitting module and a receiving module. Each software module may reside in a single storage device or may be distributed across multiple storage devices. As an example, a software module may be loaded into RAM from a hard drive when a triggering event occurs. During execution of a software module, the processor may load a portion of instructions into a cache to improve access speed. One or more cache lines may then be loaded into a special purpose register file for execution by the processor. When referring to functionality of a software module below, it will be understood that such functionality is implemented by the processor when executing instructions from that software module. It will be further understood that aspects of the present disclosure provide for improvements in the functionality of a processor, computer, machine, or other device implementing such aspects.

[0080] If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any storage medium that facilitates transfer of a computer program from one place to another.

[0081] Further, it should be appreciated that modules and / or other suitable means for performing the methods and techniques described herein can be downloaded and / or otherwise obtained by an applicable user terminal and / or base station. For example, such an apparatus can be coupled to a server to facilitate the transfer of means for performing the methods described herein. Alternatively, the various methods described herein can be provided via a storage means such that the user terminal and / or base station can obtain the various methods upon coupling or providing the storage means to the device. Moreover, any other suitable technique for providing the methods and techniques described herein to the device can be utilized.

[0082] Of course, it is to be understood that the claims are not limited to the precise configuration and components illustrated above. Various modifications, changes and variations can be made in the arrangement, operation and details of the methods and apparatus described above without departing from the scope of the claims.

[0083] The present disclosure includes the following aspects. (Appendix 1) 1. A method for controlling a robotic device based on an observed position of an object, comprising: Observing objects in an environment; generating a probability distribution for the positions of the observed objects; controlling the robotic device to perform an action when an object is at a position in the environment with a position probability below a threshold; The method includes: (Appendix 2) The method of claim 1, further comprising observing the object over a period of time. (Appendix 3) 3. The method of claim 2, further comprising estimating a continuous distribution using the observations of the object over the period of time. (Appendix 4) The method described in (Appendix 3), wherein the probability distribution is based on the continuous distribution. (Appendix 5) generating a cost map from the probability distribution; overlaying the cost map onto the environment; and controlling the robotic device based on the cost map; The method according to claim 1, further comprising: (Appendix 6) The method described in (Appendix 5), wherein the action includes at least one of providing assistance to the object, contacting emergency services, or a combination thereof. (Appendix 7) The method described in (Appendix 1), wherein the object is a human. (Appendix 8) 1. An apparatus for controlling a robotic device based on an observed position of an object, comprising: Memory, at least one processor coupled to the memory, Observe objects in the environment; generating a probability distribution for the positions of the observed objects; at least one processor configured to control the robotic device to perform an action when the object is at a position in the environment with a position probability below a threshold; An apparatus having the above configuration. (Appendix 9) The at least one processor is further configured to observe the object over a period of time. (Appendix 10) The apparatus of claim 9, wherein the at least one processor is further configured to estimate a continuous distribution using the observations of the object over the period of time. (Appendix 11) The apparatus of claim 10, wherein the probability distribution is based on the continuous distribution. (Appendix 12) The at least one processor: generating a cost map from the probability distribution; overlaying said cost map on said environment; The apparatus of claim 8, further configured to control the robotic device based on the cost map. (Appendix 13) The apparatus of claim 12, wherein the action includes at least one of providing assistance to the object, contacting emergency services, or a combination thereof. (Appendix 14) The apparatus of claim 8, wherein the object is a human. (Appendix 15) 1. A non-transitory computer readable medium having program code recorded thereon for controlling a robotic device based on an observed position of an object, the non-transitory computer readable medium comprising: The program code is executed by a processor, program code for observing objects in an environment; program code for generating a probability distribution for the locations of the observed objects; program code for controlling a robotic device to perform an action when the object is at a position in the environment with a position probability below a threshold; A non-transitory computer readable medium, which is program code comprising: (Appendix 16) 16. The non-transitory computer-readable medium of claim 15, wherein the program code further comprises program code for observing the object over a period of time. (Appendix 17) 17. The non-transitory computer-readable medium of claim 16, wherein the program code further comprises program code for estimating a continuous distribution using observations of the object over a period of time. (Appendix 18) 18. The non-transitory computer-readable medium of claim 17, wherein the probability distribution is based on the continuous distribution. (Appendix 19) The program code comprises: program code for generating a cost map from the probability distribution; program code for overlaying the cost map onto the environment; and program code for controlling the robotic device based on the cost map. (Appendix 20) 20. The non-transitory computer-readable medium of claim 19, wherein the action includes at least one of providing assistance to the object, contacting emergency services, or a combination thereof.

Claims

1. 1. A method for controlling a robotic device based on an observed position of an object, comprising: Observing a set of objects in an environment for a period of time prior to a current time; generating a probability distribution for a position of each object of the set of objects within the environment based on observations of the set of objects over the period of time; identifying an object of the set of objects that is at a position within the environment at the current time; determining a probability that the object is at the location in the environment based on an identification of the object at the location, the probability being based on a probability distribution associated with the object at the location; controlling the robotic device to perform an action with the probability less than a threshold; The method includes:

2. The method of claim 1 , further comprising estimating a continuous distribution using the observations of the object over the period of time.

3. The method of claim 2 , wherein the probability distribution is based on the continuous distribution.

4. generating a cost map from the probability distribution; overlaying the cost map onto the environment; and controlling the robotic device based on the cost map; The method of claim 1 further comprising:

5. The method of claim 4 , wherein the action includes at least one of providing assistance to the object, contacting emergency services, or a combination thereof.

6. The method of claim 1 , wherein the object is a human.

7. 1. An apparatus for controlling a robotic device based on an observed position of an object, comprising: Memory, at least one processor coupled to the memory, Observing a set of objects in an environment over a period of time prior to a current time; generating a probability distribution for the location of each object of the set of objects within the environment based on observations of the set of objects over the period of time; identifying an object of the set of objects that is at a position within the environment at the current time; determining a probability that the object is at the location in the environment based on an identification of the object at the location, the probability being based on a probability distribution associated with the object at the location; at least one processor configured to control the robotic device to perform an action with the probability less than a threshold; An apparatus having the above configuration.

8. The apparatus of claim 7 , wherein the at least one processor is further configured to estimate a continuous distribution using observations of the object over the period of time.

9. 1. A non-transitory computer readable medium having program code recorded thereon for controlling a robotic device based on an observed position of an object, the non-transitory computer readable medium comprising: The program code is executed by a processor, program code for observing a set of objects in an environment over a period of time prior to a current time; program code for generating a probability distribution for a position of each object of the set of objects in the environment based on observations of the set of objects over the period of time; program code for identifying an object of the set of objects that is located at a position in the environment at the current time; program code for determining, based on an identification of the object at the location, a probability that the object is at the location in the environment based on a probability distribution associated with the object at the location; program code for controlling a robotic device to perform an action with the probability less than a threshold; A non-transitory computer readable medium, which is program code comprising:

10. 10. The non-transitory computer readable medium of claim 9, wherein the program code further comprises program code for observing the object over the period of time.

11. 11. The non-transitory computer readable medium of claim 10, wherein the program code further comprises program code for estimating a continuous distribution using observations of the object over the period of time.

12. The non-transitory computer-readable medium of claim 11 , wherein the probability distribution is based on the continuous distribution.

13. The program code comprises: program code for generating a cost map from the probability distribution; program code for overlaying the cost map onto the environment; and program code for controlling the robotic device based on the cost map.

14. The non-transitory computer-readable medium of claim 13 , wherein the action includes at least one of providing assistance to the object, contacting emergency services, or a combination thereof.

Images