Object Location Analysis

Abstract

To provide a method for controlling a robotic device based on observed object locations.SOLUTION: The method includes observing objects in an environment. The method also includes generating a probability distribution for locations of the observed objects. The method further includes controlling the robotic device to perform an action in the environment based on the generated probability distribution.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] Certain aspects of the present disclosure relate generally to object detection, and more particularly to systems and methods for detecting objects based on spatial and temporal knowledge of objects in an environment. [Background technology]

[0002] A robotic device can use one or more sensors, such as a camera, to identify objects in the environment. The location of each identified object can be estimated. Additionally, a label can be assigned to each localized object. In a traditional object localization system, each estimated object location can be stored in a database along with an estimated timestamp.

[0003] In most systems, images are generated at a large number of frames per second, such as 10 frames per second. Each frame can contain multiple objects. The number of frames and the number of objects in each frame increases the amount of storage resources used by conventional object detection systems. Furthermore, conventional object localization systems can use an increased amount of processing resources to process images to localize objects.

[0004] Conventional object detection systems can generate false positive and false negative location estimates, i.e., for each estimated object location, some estimates may be valid while some others may be invalid.

[0005] It is desirable to improve object detection systems to reduce the amount of resources used and to reduce the number of false positives and false negatives. Summary of the Invention

[0006] In one aspect of the present disclosure, a method for controlling a robotic device based on observed object positions is disclosed. The method includes observing objects in an environment. The method also includes generating a probability distribution for the positions of the observed objects. The method further includes controlling the robotic device to perform an action in the environment based on the generated probability distribution.

[0007] In another aspect of the present disclosure, a non-transitory computer-readable medium having non-transitory program code recorded thereon is disclosed. The program code is for controlling a robotic device-based observed object position. The program code, when executed by a processor, includes program code for observing objects in an environment. The program code also includes program code for generating a probability distribution for the positions of the observed objects. The program code further includes program code for controlling the robotic device to perform an action in the environment based on the generated probability distribution.

[0008] Another aspect of the present disclosure is directed to an apparatus for controlling a robotic device based on observed object positions. The apparatus includes a memory and one or more processors coupled to the memory. The processor is configured to observe objects in an environment. The processor is also configured to generate a probability distribution for the positions of the observed objects. The processor is further configured to control the robotic device to perform an action in the environment based on the generated probability distribution.

[0009] This has outlined rather broadly the features and technical advantages of the present disclosure in order that the detailed description that follows may be better understood. Additional features and advantages of the present disclosure are set forth below. It should be understood by those skilled in the art that this disclosure may be readily utilized as a basis for modifying or designing other structures for carrying out the same purposes of the present disclosure. It should also be understood by those skilled in the art that such equivalent constructions do not depart from the teachings of the present disclosure as set forth in the appended claims. The novel features believed characteristic of the present disclosure, both as to its organization and method of operation, together with further objects and advantages, will be better understood from the following description when considered in connection with the accompanying drawings. It is to be expressly understood, however, that each of the figures is provided for the purpose of illustration and description only and is not intended as a definition of the limits of the present disclosure. [Brief explanation of the drawings]

[0010] The features, nature, and advantages of the present disclosure will become more apparent from the detailed description set forth below when taken in conjunction with the drawings in which like reference characters identify correspondingly throughout. [Figure 1] FIG. 1 illustrates an example of a robot in an environment according to an embodiment of the present disclosure. [Figure 2] FIG. 2 illustrates an example of a clustering analysis map according to an embodiment of the present disclosure. [Figure 3] FIG. 3 shows an example of generating a clustering analysis map of object positions in an environment by observing temporal object positions. [Figure 4A] FIG. 4A illustrates an example environment according to an embodiment of the present disclosure. [Figure 4B] FIG. 4B illustrates an example of a cost map according to an aspect of the present disclosure. [Figure 4C] FIG. 4C illustrates an example of a cost map with object positions integrated over time, according to an embodiment of the present disclosure. [Figure 5] FIG. 5 illustrates an example of a hardware implementation for an object localization system according to an aspect of the present disclosure. [Figure 6] FIG. 6 illustrates a flow diagram for a method for controlling a robotic device based on observed object positions according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0011] The detailed description set forth below in connection with the accompanying drawings is intended as a description of various configurations and is not intended to represent the only configurations in which the concepts described herein may be practiced. The detailed description includes specific details for the purpose of providing a thorough understanding of the various concepts. However, it will be apparent to those skilled in the art that these concepts may be practiced without these specific details. In some instances, well-known structures and components are shown in block diagram form to avoid obscuring such concepts.

[0012] Conventional systems that perform visual object classification (e.g., object detection networks) generate false positive and false negative detections. False positive and false negative detections reduce the accuracy of detection. A system or method for visual object detection can be combined with a system or method for estimating detected object positions to analyze object positions. Detected visual objects can be identified by bounding boxes. The analysis of object positions can be referred to as clustering analysis.

[0013] Errors can be detected from clustering analysis. Furthermore, true object locations can be estimated based on clustering analysis. Clustering analysis assumes a static environment and utilizes this assumption to count the number of objects in the environment, refine the estimated true location of each object, and reject detections that are not part of a spatial cluster (e.g., false positive detections).

[0014] Additionally or alternatively, a probability density function can be used to analyze the object position estimate. A probability density function (e.g., probability density analysis) does not assume a static environment. Probability density analysis does not use clustering analysis, object counting, or false detection estimation. In one configuration, spatial and temporal search for objects can be performed based on probability density analysis.

[0015] The robotic device can identify objects in an environment using one or more sensors. The sensors can include a red-green-blue (RGB) camera, a radio detection and ranging (RADAR) sensor, a light detection and ranging (LiDAR) sensor, or another type of sensor. In images captured by the sensors, the location of one or more identified objects can be estimated. Each object's location estimate can be stored along with an estimation timestamp. The estimation timestamp identifies the time the object was at the estimated location. In one configuration, three-dimensional (3D) object location is determined by registering a detection bounding box centroid with a point cloud generated by a depth camera.

[0016] In most systems, images are generated at a large number of frames per second, such as 10 frames per second. Furthermore, each frame can contain multiple objects. The number of frames and the number of objects in each frame increases the amount of storage resources used by conventional object detection systems. Furthermore, conventional object detection systems can use an increased amount of processing resources to process the images.

[0017] Conventional object detection systems can also generate false positive and false negative position estimates: for each object detected in a frame, some detections may be valid and some may be invalid.

[0018] Aspects of the present disclosure are directed to using a density-based clustering function to estimate the number of objects in an environment. This estimation is based on the assumption that the observation density for true object locations is greater than a high-density threshold. Furthermore, the observation density for false-positive object locations is less than a low-density threshold.

[0019] For simplicity, in this disclosure, robotic devices will be referred to as robots. Furthermore, objects can include stationary and dynamic objects in the environment. Objects can include man-made objects (e.g., chairs, desks, cars, books, etc.), natural objects (e.g., rocks, trees, animals, etc.), and humans.

[0020] FIG. 1 illustrates an example of a robot 100 in an environment 102 according to an embodiment of the present disclosure. In the example of FIG. 1, the robot 100 is a humanoid robot and the environment 102 is a kitchen. Aspects of the present disclosure are not limited to humanoid robots. The robot 100 may be any type of autonomous or semi-autonomous device, such as a drone or a vehicle. Furthermore, the robot 100 may be in any type of environment.

[0021] In one configuration, the robot 100 acquires one or more images of the environment 102 via one or more sensors of the robot 100. The robot 100 can locate one or more objects in each image. Localization refers to determining the location (e.g., coordinates) of an object within an image. In conventional object detection systems, bounding boxes can be used to indicate the location of an object within an image. The localized objects can be one or more specific classes of objects, such as chair 104, or all objects in the image. The objects can be localized via an object detection system, such as a pre-trained object detection neural network.

[0022] As described, one or more identified objects within an image (e.g., a frame) can be localized. In one configuration, an object position estimator is observed over time within an environment. Specifically, clustering analysis can be applied to the object position estimator over various time windows (e.g., different time periods).

[0023] FIG. 2 illustrates an example of a clustering analysis map 200 according to an embodiment of the present disclosure. In the example of FIG. 2, an object position estimator was observed in an environment over time. Each node 202, 208 identifies the location of the estimated position of the object within the environment. As shown in FIG. 2, some nodes 202 (e.g., unclustered nodes 202) may not be associated with a cluster. Other nodes 208 (e.g., clustered nodes 208) may be clustered around a centroid 204. The unclustered nodes 202 may be considered false positives. For simplicity, labels have not been provided for each of the unclustered nodes 202 and 208 in FIG. 2.

[0024] In the example of FIG. 2, the in-cluster nodes 208 may be edge nodes 210 or core nodes 212. The size (e.g., diameter) of the edge in-cluster nodes 210 is smaller than the size of the core in-cluster nodes 212. Each edge in-cluster node 210 is associated with one neighboring node, and the distance between the neighboring nodes is less than a distance threshold. Each core in-cluster node 212 is associated with one neighboring node, and the distance between the neighboring nodes is less than a distance threshold. In one configuration, the environment is a 3D environment, so the centroid 204 is a 3D centroid. The centroid 204 can be associated with the true location of the object.

[0025] Generally, clustered nodes 202 are location estimates of the same object or class of objects. For example, in an office, the location of office furniture is less dynamic than the locations of office supplies such as pens and paper. Nodes 202 within a certain location region can be clustered. For example, nodes 202 corresponding to the location of office chairs can form a cluster if their locations are within a particular area over the course of observation. A 3D centroid 204 (e.g., mean centroid) can be determined for each cluster.

[0026] In the case of density-based clustering, the accuracy of the object position estimation can be determined based on the 3D centroid 204 of each cluster. Estimating the object position based on the 3D centroid 204 can improve the accuracy of the estimation. Furthermore, the accuracy of the object localization function can be determined based on the 3D standard deviation of each cluster. The 3D mean of each cluster can be used to estimate the true position of the object. The standard deviation can be used to determine the variance of the cluster. If the standard deviation is greater than a threshold, there may be uncertainty in the true position of the object. The uncertainty of the object position decreases as the variance of the standard deviation decreases. Therefore, the standard deviation correlates to the accuracy of the object position.

[0027] 3 illustrates an example of generating a clustering analysis map 350 according to an embodiment of the present disclosure. As shown in FIG. 3, the clustering analysis map 350 is based on observations of objects 302, 304 in an environment 300. In FIG. 3, a robot may observe the positions of objects (e.g., chair 304 and table 302) in the environment 300 over a period of time. The robot may determine the positions of chair 304 and table 302 for each observation.

[0028] After multiple observations, the robot can perform clustering analysis to generate a clustering analysis map 350. The clustering analysis map 350 includes in-cluster nodes 314 and non-cluster nodes 316. The non-clustered nodes 316 are observations that are isolated from the environment 300. The in-cluster nodes 314 can be considered as repeated observations of the same object 304, 302. Each centroid 318, 320, 322 is considered a true object location. For example, the centroids 318, 322 correspond to the estimated true location of the chair 304. The centroid 320 corresponds to the estimated true location of the table 302. The centroids 318, 320, 322 may be 3D centers of gravity.

[0029] False positives are common in traditional convolutional neural network detectors. Clustering analysis (e.g., density-based clustering) improves estimates of multiple objects in a spatial environment, thereby reducing false positives. Density-based clustering can be used to set a density criterion for multiple detections in similar or identical locations. Thus, repeated observations of the same object can be considered. That is, in one configuration, density-based clustering is used to consider multiple observations and discard isolated observations. Furthermore, density-based clustering can fuse repeated observations into a unified object instance. Each cluster represents an instance of an object, such as a bottle or a plant.

[0030] In the example of Figure 2, each cluster corresponds to one object based on multiple observations of the object. For example, hundreds or thousands of individual detections of the object can be used for each cluster. Furthermore, clustering analysis can be used to determine the quality of the location estimation. That is, clustering analysis can determine how well the location estimator is performing.

[0031] In one configuration, a probability distribution of object locations in an environment can be generated by observing object locations over time. That is, a probability density function (e.g., probability density analysis) estimates a continuous distribution using a series of observations to represent the probability of finding an object at any given location on a map. Thus, the probability density function can be used to represent spatial and temporal knowledge about objects in a spatial environment. The memory footprint of the data corresponding to the estimated probability density function is smaller than that of data corresponding to conventional object localization systems. For example, a conventional object localization system can generate a 1 GB object observation log. In contrast, an estimated probability density function may be 50 KB. Due to the reduced memory footprint, kernel density estimates can be transferred between robotic devices. Thus, robotic devices can share knowledge about object locations in an environment.

[0032] As mentioned above, probability density function analysis estimates a continuous distribution using all observation configurations. A probability distribution of object locations in an environment is generated by observing object locations over time. Random samples can be drawn from the probability distribution. Random samples from the distribution are more likely to be in high probability regions as opposed to low probability regions. Furthermore, evaluating the distribution on a map grid can provide a heat map (e.g., cost map) that indicates areas of the map that are more likely to contain the object.

[0033] The robotic device can sample high probability areas to reduce object search time. If the object is not in a high probability area (e.g., the most likely location), the object search can sample low probability areas. The probability distribution can reduce search time by distinguishing between high and low probability areas. In one configuration, if an object is found in an unlikely location (e.g., a low probability area), the robotic device can return the object to its most likely location (e.g., a high probability area).

[0034] As another example, certain objects may correspond to certain high probability and low probability regions. For example, a first set of chairs may be for a kitchen dining table, and a second set of chairs may be for a formal dining table. In this example, the kitchen is a high probability region for the first set of chairs, and areas outside the kitchen are low probability regions. For example, the formal dining room may be a low probability region for the first set of chairs. Furthermore, the formal dining room is a high probability region for the second set of chairs, and the kitchen is a low probability region for the second set of chairs.

[0035] FIG. 4A shows an image of an environment 400 according to an embodiment of the present disclosure. In the example of FIG. 4A, the environment includes a road 402 and a sidewalk 424. In one configuration, a robot observes the environment 400 over time. For example, the robot may observe the environment 400 over a 24-hour period. Based on the observations, the robot generates a probability distribution of object locations within the environment 400. The objects may be humans or other types of objects. The probability distribution can be used to generate a cost map.

[0036] As shown in Figure 4B, the cost map 450 can be generated based on a probability density analysis of objects in the environment. The cost map 450 can be based on a 2D or 3D probability density function. Therefore, any type of map can be generated, such as an ego view or a top-down view.

[0037] In the cost map 450, low probability regions can be distinguished from high probability regions. Each region can be assigned a probability value. The probability value or probability value range can be visualized by color or other visual representation. For example, the color or shading of low probability regions can be different from high probability regions. Different granularity levels can be assigned to occupancy probabilities. For example, as shown in FIG. 4B, probability regions may be low (404), light (406), medium (408), high (410), or very high (412). The probability that an object is present at a location can be determined from the value assigned to that location in the cost map. The probability that an object is present at a location can be referred to as the object probability.

[0038] Figure 4C shows an example of a cost map 450 with object positions integrated over a period of time 452. In the example of Figure 4C, the period may be 24 hours. Other periods such as hours, days, weeks, months, etc. are also possible.

[0039] In the example of Figure 4C, each object location 452 represents the location of an object, such as a human. The cost map 450 is generated based on probability density analysis. As described above, probability density analysis uses all observations to estimate a continuous distribution to represent the probability of finding an object at any given location on the cost map 450. The cost map can be thought of as a Gaussian mixture model with some number of 2D Gaussian functions that fit the data.

[0040] Each object or object type may correspond to specific high probability and low probability regions. For example, for kitchen utensils in a home environment, the high probability region is the kitchen and the low probability region is the bathroom. As another example, for clothing in a home environment, the high probability region is one or more closets and the low probability region is the kitchen. According to aspects of the present disclosure, a robotic device can return an object in an environment to its "home" location (e.g., a high probability region) without user direction or user intervention.

[0041] As described above, a cost map can be generated from a probability distribution. The cost map can be overlaid on a map of the environment. In one configuration, a path planning function can use the overlaid cost map to minimize or maximize the probability of encountering a particular object. As shown in Figures 4B and 4C, low-probability regions can be distinguished from high-probability regions.

[0042] In one configuration, a robot uses a cost map overlay for route planning. For example, a robot's path can be planned to avoid areas where the robot is likely to encounter human-like objects. The robot may also use the inverse of the cost map for paths that increase the likelihood of encountering human-like objects. That is, a probability density function can be explored to maximize or minimize the chance of encountering one or more objects.

[0043] In one example, to avoid one or more objects, a robot can plan a route that avoids probability regions greater than a threshold. For example, the robot can avoid medium probability region 408, high probability region 410, and very high probability region 412 (see FIGS. 4B-C). The probability regions can be thresholded based on, for example, the robot's task, fuel level, cost, and / or other factors.

[0044] In another example, to encounter one or more objects, the robot may plan a route that includes probability regions greater than a threshold. For example, the robot may plan a route that includes medium probability region 408, high probability region 410, and very high probability region 412 (see FIGS. 4B-C). Additionally or alternatively, in this example, the robot may plan a route that avoids probability regions that are less than a threshold. For example, the robot may plan a route that avoids low probability region 404 and / or low probability region 406 (see FIGS. 4B-C). The aforementioned probability regions may be threshold-determined based on, for example, the robot's task, fuel level, cost, and / or other factors.

[0045] According to aspects of the present disclosure, stationary objects such as sofas, tables, and potted plants have stationary position probability density functions. Dynamic objects such as people, coffee mugs, pets, and chairs are dynamic and have non-stationary probability density functions. For objects with non-stationary distributions, the probability density functions may be estimated at different frequencies. For example, the probability density functions may be estimated during one or more different times of the day, different days of the week, different months, etc.

[0046] In one configuration, the probability of finding a non-dynamic object in the environment can vary based on time. In one configuration, the robot can search for dynamic objects based on a time-related distribution. The search can be limited to a specific object or any other object that the robot can recognize and estimate the location of.

[0047] In some embodiments, based on the probability distribution of people in a space, an algorithm can determine furniture placement to minimize impact on pedestrian traffic flow or identify unused furniture in a space. Based on the probability distribution of people in a space, an algorithm can determine optimal advertisement placement in an environment. Furthermore, based on the probability distribution of people in a space, an algorithm can determine how long people engage with advertisements in different locations.

[0048] 5 is a diagram illustrating an example of a hardware implementation for an object location system 500 according to an embodiment of the present disclosure. The object location system 500 may be a component of a vehicle, a robotic device, or another device. For example, as shown in FIG. 5, the object location system 500 is a component of a robot 528 (e.g., a robotic device).

[0049] Aspects of the present disclosure are not limited to the object location system 500 being a component of a robot 528. Other devices, such as a bus, boat, drone, or vehicle, are also contemplated for using the object location system 500. The robot 528 may operate in at least an autonomous mode of operation and a manual mode of operation.

[0050] The object location system 500 may be implemented using a bus architecture, generally represented by bus 550. The bus 550 may include any number of interconnected buses and bridges, depending on the particular application and overall design constraints of the object location system 500. The bus 550 links together various circuits, including one or more processors and / or hardware modules, represented by the processor 520, the communications module 522, the position module 518, the sensor module 502, the movement module 526, the navigation module 525, and the computer-readable medium 515. The bus 550 may also link various other circuits, such as timing sources, peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further.

[0051] The object location system 500 includes a processor 520, a sensor unit 502, an object location analysis unit 508, a communication unit 522, a location unit 518, a movement module 526, a navigation module 525, and a transceiver 516 coupled to a computer-readable medium 515. The transceiver 516 is coupled to an antenna 555. The transceiver 516 communicates with various other devices via a transmission medium. For example, the transceiver 516 can receive commands via transmission from a user or a remote device. As another example, the transceiver 516 can transmit statistics and other information from the object location analysis module 508 to a server (not shown).

[0052] Object location system 500 includes a processor 520 coupled to a computer-readable medium 515. Processor 520 performs processes, including executing software stored on computer-readable medium 515, that provide functionality according to the present disclosure. The software, when executed by processor 520, causes object location apparatus 500 to perform the various functions described for a particular device, such as robot 528, or any of parts 502, 508, 515, 516, 518, 520, 522, 525, 526. Computer-readable medium 515 can also be used to store data that is manipulated by processor 520 when executing the software.

[0053] The sensor module 502 can be used to obtain measurements via different sensors, such as a first sensor 506 and a second sensor 505. The first sensor 506 may be a visual sensor, such as a stereoscopic camera or a red-green-blue (RGB) camera, for capturing 2D images. The second sensor 505 may be a ranging sensor, such as a light detection and ranging (LIDAR) sensor or a radio detection and ranging (RADAR) sensor. Of course, aspects of the present disclosure are not limited to the aforementioned sensors, as other types of sensors are also contemplated for either of the sensors 505, 506, such as, for example, thermal sensors, sonar sensors, and / or lasers.

[0054] Measurements of the first sensor 506 and the second sensor 505 can be processed by one or more of the processor 520, the sensor unit 502, the object position analysis unit 508, the communication unit 522, the localization unit 518, the movement module 526, and the navigation module 525, along with the computer-readable medium 515, to perform the functions described herein. In one configuration, data captured by the first sensor 506 and the second sensor 505 can be transmitted to an external device via the transceiver 516. The first sensor 506 and the second sensor 505 can be coupled to or in communication with the robot 528.

[0055] The position module 518 may be used to determine the position of the robot 528. For example, the positioning unit 518 may use a global positioning system (GPS) to determine the position of the robot 528. The communication module 522 may be used to facilitate communication via the transceiver 516. For example, the communication module 522 may be configured to provide communication capabilities via different wireless protocols, such as WiFi, LTE (long term evolution), 5G, etc. The communication module 522 may also be used to communicate with other components of the robot 528 that are not modules of the object localization system 500.

[0056] The locomotion module 526 can be used to facilitate locomotion of the robot 528. As another example, the locomotion module 526 can be in communication with one or more power sources of the robot 528, such as motors and / or batteries. Locomotion can be evidenced via wheels, movable limbs, propellers, treads, fins, jet engines, and / or other sources of locomotion.

[0057] The object location system 500 also includes a navigation module 525 for planning paths and controlling the movement of a robot 528 via a movement module 526. The paths can be planned based on data provided via the object location analysis module 508. The modules can be software modules executing within the processor 520, software modules residing / stored within the computer readable medium 515, one or more hardware modules coupled to the processor 520, or some combination thereof.

[0058] The object position analysis module 508 may be in communication with the sensor module 502, the transceiver 516, the processor 520, the communication module 522, the position module 518, the movement module 526, the navigation module 525, and the computer-readable medium 515. In one configuration, the object position analysis module 508 receives sensor data from the sensor module 502. The sensor module 502 may receive sensor data from the first sensor 506 and the second sensor 505. According to aspects of the present disclosure, the sensor module 502 may filter the data, remove noise, encode the data, decode the data, merge the data, extract frames, or perform other functions. In an alternative configuration, the object position analysis module 508 may receive sensor data directly from the first sensor 506 and the second sensor 505.

[0059] In one configuration, the object position analysis module 508 determines a probability region of the object based on information from the processor 520, the position module 518, the computer-readable medium 515, the first sensor 506, and / or the second sensor 505. The probability region can be determined by generating a probability distribution of object positions for the observed object. Based on the probability region, the object position analysis module 508 can control one or more actions of the robot 528.

[0060] For example, an operation may be planning a route for the robot 528. The path may be determined based on a given task, such as encountering or avoiding one or more objects. The object position analysis module 508 may plan the path via the processor 520, the position module 518, the computer-readable medium 515, the movement module 526, and / or the navigation module 525.

[0061] FIG. 6 illustrates a flow diagram 600 for controlling a robotic device based on observed object positions, according to an embodiment of the present disclosure. For simplicity, the robotic device will be referred to as a robot. As shown in FIG. 6, in block 602, the robot observes objects in an environment. The objects may be observed via one or more sensors of the robot, such as a LiDAR, a RADAR, and / or an RGB camera. The objects may be observed over a period of time, such as hours, days, etc.

[0062] In block 604, the robot generates a probability distribution for the locations of the observed objects. In one configuration, the robot uses observations of the objects over a period of time to estimate a continuous distribution. In this configuration, the probability distribution is based on the continuous distribution.

[0063] The cost map can be generated from a probability distribution. The cost map is also called a heat map. The cost map can be overlaid on a map of the environment. The cost map can be a 2D or 3D cost map.

[0064] In block 606, the robot performs an action in the environment based on the generated probability distribution. The robot can be controlled based on the cost map. As an example, the robot can avoid a first region in the environment with a first object probability greater than a first threshold. Additionally or alternatively, the robot can navigate to a second region in the environment with a second object probability greater than a second threshold.

[0065] In another configuration, the robot can identify out-of-position objects based on a probability distribution. For example, the robot can identify objects in low-probability regions. The robot can return misplaced objects to their intended (e.g., correct) positions.

[0066] Based on the teachings, those skilled in the art should understand that the scope of the present disclosure is intended to cover any aspect of the present disclosure, whether implemented independently of or in combination with any other aspect of the present disclosure. For example, an apparatus may be implemented or a method may be practiced using any number of the described aspects. Furthermore, the scope of the present disclosure is intended to cover such an apparatus or method implemented using other structure, functionality, or structure and functionality in addition to or other than the various aspects of the present disclosure described. It should be understood that any aspect of the present disclosure may be embodied by one or more elements of a claim.

[0067] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration," and any aspect described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects.

[0068] While specific aspects are described herein, many variations and permutations of these aspects are within the scope of the present disclosure. Although some benefits and advantages of the preferred aspects are mentioned, the scope of the present disclosure is not intended to be limited to particular benefits, uses, or purposes. Rather, aspects of the present disclosure are intended to be broadly applicable to different technologies, system configurations, networks, and protocols, some of which are illustrated by way of example in the drawings and the following description of the preferred aspects. The detailed description and drawings are merely illustrative rather than limiting of the present disclosure, the scope of which is defined by the appended claims and their equivalents.

[0069] As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" can include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, database, or another data structure), ascertaining, etc., and "determining" can include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), etc., and "determining" can include resolving, selecting, choosing, establishing, etc.

[0070] As used herein, a phrase referring to "at least one of" a list of items includes a single item and refers to any combination of those items. As an example, "at least one of a, b, or c" is intended to encompass a, b, c, ab, ac, bc, and abc.

[0071] The various illustrative logic blocks, modules, and circuits described in connection with this disclosure may be implemented or performed using a processor specially configured to perform the functions described herein. The processor may be a neural network processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. Alternatively, the processing system may include one or more neuromorphic processors for implementing the neuron models and models of the nervous system described herein. The processor may be a microprocessor, controller, microcontroller, or state machine specially configured as described herein. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors associated with a DSP core, or other such special configurations, as described herein.

[0072] The steps of a method or algorithm described in connection with this disclosure may be embodied directly in hardware, in a software module executed by a processor, or a combination of the two. A software module may reside in a storage or machine-readable medium, including random-access memory, read-only memory, flash memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, registers, a hard disk, a removable disk, a CD-ROM or other optical disk storage, a magnetic disk storage or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. A software module may comprise a single instruction, or many instructions, and may be distributed among several different code segments, among different programs, and across multiple storage media. A storage medium may be coupled to a processor such that the processor can read information from, and write information to, the storage medium. In an alternative embodiment, the storage medium may be integral to the processor.

[0073] The methods disclosed herein comprise one or more steps or actions for achieving the described method. Method steps and / or actions may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims.

[0074] The described functions may be implemented in hardware, software, firmware, or any combination thereof. When implemented in hardware, an example hardware configuration may include a processing system within an apparatus. The processing system may be implemented using a bus architecture. The bus may include any number of interconnected buses and bridges, depending on the particular application and overall design constraints of the processing system. The bus may link various circuits together, including a processor, a machine-readable medium, and a bus interface. A bus interface may be used to connect a network adapter to the processing system, among other things, via the bus. The network adapter may be used to perform signal processing functions. In some embodiments, a user interface (e.g., a keypad, a display, a mouse, a joystick, etc.) may also be connected to the bus. The bus may also link various other circuits, such as timing sources, peripherals, voltage regulators, power management circuits, etc., which are well known in the art and therefore will not be described further.

[0075] The processor may be responsible for managing the bus and processing, including executing software stored on a machine-readable medium. Software shall be taken to mean instructions, data, or any combination thereof, whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0076] In a hardware implementation, the machine-readable medium may be part of a processing system separate from the processor. However, as will be readily understood by those skilled in the art, the machine-readable medium, or any portion thereof, may be external to the processing system. By way of example, the machine-readable medium may include a transmission line, a carrier wave modulated by data, and / or a computer product separate from the device, all of which may be accessed by the processor via a bus interface. Alternatively, or in addition, the machine-readable medium, or any portion thereof, may be integrated into the processor, such as with cache and / or specialized register files. While the various components described may be described as having a particular location, such as a local component, they may also be configured in various ways, such as with a distributed computing system, where certain components are configured as part of a distributed computing system.

[0077] The machine-readable medium may include multiple software modules. The software modules may include a transmitting module and a receiving module. Each software module may reside in a single storage device or may be distributed across multiple storage devices. As an example, a software module may be loaded into RAM from a hard drive when a triggering event occurs. During execution of a software module, the processor may load some of the instructions into a cache to improve access speed. One or more cache lines may then be loaded into a special-purpose register file for execution by the processor. When referring to the functionality of a software module below, it will be understood that such functionality is implemented by the processor when executing instructions from that software module. Furthermore, it will be understood that aspects of the present disclosure provide improved functionality for a processor, computer, machine, or other system implementing such aspects.

[0078] If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any storage medium that facilitates transfer of a computer program from one place to another.

[0079] Furthermore, it should be appreciated that modules and / or other suitable means for performing the methods and techniques described herein may be downloaded and / or otherwise obtained by a user terminal and / or base station, where applicable. For example, such a device may be coupled to a server to facilitate transmission of means for performing the methods described herein. Alternatively, the various methods described herein may be provided via storage means, such that the user terminal and / or base station may obtain the various methods upon coupling or providing the storage means to the device. Furthermore, any other suitable technique for providing the methods and techniques described herein to a device may be utilized.

[0080] It is to be understood that the claims are not limited to the precise configuration and components described above. Various modifications, changes and variations can be made in the arrangement, operation and details of the methods and apparatus described above without departing from the scope of the claims.

Claims

1. 1. A method for controlling a robotic device based on a position of an observed object, comprising: observing objects in an environment at different times, said environment remaining the same at said different times; generating, for the object, a location cluster including a plurality of nodes, each node of the plurality of nodes being associated with an observation of a plurality of observations, each observation of the plurality of observations corresponding to a different time zone of the plurality of observations; localizing the object in the environment based on the centroids of the location clusters; generating a probability distribution for observed locations of the object based on observations of the object at the different times; overlaying a cost map onto a map of the environment, the cost map being associated with a probability distribution to identify the probability distribution of the objects associated with the localized object in the environment; controlling the robotic device to perform an action in the environment based on the cost map and the map of the environment; The method includes:

2. The method of claim 1 , further comprising estimating a continuous distribution using the observations of the object at the different times.

3. The method of claim 2 , wherein the probability distribution is based on the continuous distribution.

4. The method of claim 1 , further comprising generating the cost map from the probability distribution.

5. 5. The method of claim 4, further comprising controlling the robotic device to avoid a first region in the environment having a first object probability greater than a first threshold or to navigate to a second region in the environment having a second object probability greater than a second threshold.

6. The method of claim 1 , further comprising controlling the robotic device to place an object in the environment based on the probability distribution.

7. 1. An apparatus for controlling a robotic device based on observed object positions, the apparatus comprising: a memory; and at least one processor coupled to the memory; the at least one processor observes objects in an environment at different times, the environment remaining the same at the different times; generating a location cluster for the object, the location cluster including a plurality of nodes, where each node in the plurality of nodes is associated with an observation of a plurality of observations, each observation of the plurality of observations corresponding to a time period of the different time periods; localizing the object in the environment based on the centroids of the location clusters; generating a probability distribution for observed locations of the object based on observations of the object at the different times; overlaying a cost map onto the map of the environment, where the cost map is associated with the probability distribution to identify the probability distribution of the objects associated with the localized object in the environment; configured to control the robotic device to perform an action in the environment based on the cost map and the map of the environment. Device.

8. the at least one processor is further configured to estimate a continuous distribution using the observations of the object at the different time periods.

8. The apparatus of claim 7.

9. The probability distribution is based on the continuous distribution.

9. The apparatus of claim 8.

10. The at least one processor is further configured to generate the cost map from the probability distribution.

8. The apparatus of claim 7.

11. the at least one processor is further configured to control the robotic device to avoid a first region in the environment having a first object probability greater than a first threshold or to navigate to a second region in the environment having a second object probability greater than a second threshold.

11. The apparatus of claim 10.

12. The at least one processor is further configured to control the robotic device to place an object in the environment based on the probability distribution.

8. The apparatus of claim 7.

13. 1. A non-transitory computer readable medium having program code recorded thereon for controlling a robotic device based on observed object positions, the non-transitory computer readable medium comprising: the program code, when executed by a processor, for observing objects in an environment at different time periods, the environment remaining in the same state at the different time periods; generating a location cluster for the object comprising a plurality of nodes, each node in the plurality of nodes being associated with an observation of a plurality of observations, each observation of the plurality of observations corresponding to a time period of the different time periods; Program code, program code for localizing the object in the environment based on the centroids of the location clusters; program code for generating a probability distribution for observed locations of the object based on observations of the object at the different times; program code for overlaying a cost map onto a map of the environment, the cost map relating to the probability distribution to identify the probability distribution of the objects associated with the localized object in the environment; program code for controlling the robotic device to perform an action within the environment based on the cost map and the map of the environment; Including, Non-transitory computer-readable medium.

14. the program code further comprising program code for estimating a continuous distribution using the observations of the object at the different time periods; 14. The non-transitory computer-readable medium of claim 13.

15. The probability distribution is based on the continuous distribution.

15. The non-transitory computer-readable medium of claim 14.

16. The program code further includes program code for generating a cost map from the probability distribution.

14. The non-transitory computer-readable medium of claim 13.

17. the program code further comprising program code for controlling the robotic device to avoid a first region in the environment having a first object probability greater than a first threshold or to navigate to a second region in the environment having a second object probability greater than a second threshold.

20. The non-transitory computer-readable medium of claim 16.

Images