Method and system for detecting navigation conditions for unobservable regions - Patents.com

Abstract

The method includes maintaining a map of an occupied area of ​​the facility; controlling a sensor of the mobile device to capture sensor data within a field of view (FOV); identifying unobservable occupied areas in the map that are located outside the FOV based on a current location of the mobile device in the facility; selecting a first reference occupied area having a first reference identifier and a second reference occupied area having a second reference identifier from the map; generating a first connectivity score associating the unobservable occupied area with the first reference occupied area and a second connectivity score associating the unobservable occupied area with the second reference occupied area; selecting a processing operation for the unobservable occupied area based on the first and second connectivity scores; updating the map according to the selected processing operation; and controlling a mobile assembly of the mobile device according to the updated map.

Description

[Background technology]

[0001]

[0001] Autonomous or semi-autonomous mobile devices may include navigation sensors, such as depth sensors, to detect obstacles near the device for navigation purposes. Such sensors may have a limited field of view, such that as the device traverses, areas in which obstacles were previously observed fall out of the sensor's field of view. The inability of the mobile device to observe those areas creates ambiguity as to whether previously observed obstacles are still present, which may lead to inaccurate and / or inefficient navigation of the device. Summary of the Invention

[0002]

[0002] The accompanying drawings, in which like reference numbers refer to the same or functionally similar elements throughout the separate views, and which, together with the following detailed description, are incorporated in and form a part of this specification, further illustrate embodiments of the concepts that comprise the claimed inventions and serve to explain various principles and advantages of those embodiments. [Brief description of the drawings]

[0003] [Figure 1]

[0003] FIG. 1 illustrates a mobile device and a portion of an environment in which the device is deployed for navigation. [Diagram 2]

[0004] FIG. 2 is a diagram illustrating detection of an obstacle by the device of FIG. 1. [Diagram 3]

[0005] FIG. 1 illustrates the unobservable occupancy area resulting from the movement of the device and / or other obstacles during navigation. [Figure 4]

[0006] 1 is a flow chart of a method for navigation state detection for an unobservable occupancy area. [Diagram 5]

[0007] 5 illustrates an example implementation of blocks 405 and 410 of the method of FIG. 4. [Figure 6]

[0008] 5 illustrates a further exemplary implementation of blocks 405 and 410 of the method of FIG. [Figure 7]

[0009] 5 illustrates an example implementation of block 420 of the method of FIG. 4. [Figure 8]

[0010] 5 illustrates an example implementation of block 425 of the method of FIG. 4. [Figure 9]

[0011] 5 illustrates the results of a further exemplary execution of the method of FIG. 4. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0004]

[0012] Those skilled in the art will appreciate that elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale For example, the dimensions of some of the elements in the figures may be exaggerated relative to other elements to help improve understanding of embodiments of the present invention.

[0005]

[0013] Components of the apparatus and methods have been represented, where appropriate, by conventional symbols in the drawings, showing only those specific details that are relevant to understanding the embodiments of the invention, so as not to obscure the present disclosure with details that will be readily apparent to those of ordinary skill in the art having the benefit of the descriptions herein.

[0006]

[0014] Examples disclosed herein are directed to a method including maintaining a map of an occupied area of ​​a facility; controlling a navigation sensor of a mobile device to capture sensor data within a sensor field of view (FOV); identifying an unobservable occupied area in the map that is located outside the sensor FOV based on a current location of the mobile device in the facility; selecting from the map a first reference occupied area having a first reference identifier and a second reference occupied area having a second reference identifier; generating a first connectivity score associating the unobservable occupied area with the first reference occupied area and a second connectivity score associating the unobservable occupied area with the second reference occupied area; selecting a processing operation for the unobservable occupied area based on the first and second connectivity scores; updating the map according to the selected processing operation; and controlling a locomotive assembly of the mobile device according to the updated map.

[0007]

[0015] Further examples disclosed herein are directed to a computing device including a sensor having a sensor field of view (FOV) and a processor, the processor being configured to maintain a map of an occupied area of ​​a facility, control the sensor to capture sensor data within the sensor field of view (FOV), identify unobservable occupied areas in the map that are located outside the sensor FOV based on a current location of a mobile device in the facility, select from the map a first reference occupied area having a first reference identifier and a second reference occupied area having a second reference identifier, generate a first connectivity score associating the unobservable occupied area with the first reference occupied area and a second connectivity score associating the unobservable occupied area with the second reference occupied area, select a processing operation for the unobservable occupied area based on the first and second connectivity scores, update the map in accordance with the selected processing operation, and control a locomotive assembly of the mobile device in accordance with the updated map.

[0008]

[0016] 1 illustrates an autonomous or semi-autonomous mobile device 100, also referred to herein simply as device 100. Device 100 may be located in any of a variety of environments, such as a material handling facility (e.g., a warehouse, a retail facility, etc.). The facility may include a number of support structures, such as, for example, shelving modules 104 that support items 108.

[0009]

[0017] As will be apparent, the facility in which the device 100 is located can take a variety of other forms, such as a manufacturing facility, an outdoor work environment, etc. The device 100 includes certain components and implements certain functionality that enables the device 100 to at least partially autonomously navigate the facility in which the device 100 is located. For example, the device 100 can be configured to navigate between aisles formed by the shelf modules 104, collect images or other data corresponding to the items 108, receive and transport the items 108 for order fulfillment operations, etc. Accordingly, the device 100 may also be referred to as a transport device 100.

[0010]

[0018] The device 100 includes a chassis 112 that supports various other components of the device 100. Among other things, the chassis 112 supports a locomotive assembly 116, such as one or more electric motors that drive a set of wheels, tracks, and the like. The chassis 112, in the illustrated example, supports one or more containers 120, such as, for example, large boxes or bags, for receiving and transporting items 108 placed therein by personnel at a facility. In other examples, the containers 120 can be omitted, depending on the specific conditions of the device 100's deployment (e.g., whether the device 100 is deployed to transport items 108).

[0011]

[0019] The components supported by chassis 112 also include a processor 124, e.g., in the form of one or more central processing units (CPUs), graphics processing units (GPUs), or dedicated hardware controllers such as application specific integrated circuits (ASICs). Processor 124 is communicatively coupled to memory 128, e.g., a suitable combination of volatile and non-volatile memory elements. Processor 124 may also be coupled to a communication interface 132, such as a wireless transceiver that enables device 100 to communicate with other computing devices over a suitable network infrastructure. However, in other examples, communication interface 132 may be omitted.

[0012]

[0020] The memory 128 stores various data used for autonomous or semi-autonomous navigation by the device 100, including applications 136 executable by the processor 124 to perform navigation functions. In some examples, the above functions may be performed via multiple separate applications stored in the memory 128. In further examples, some or all such functions may be performed by a separate computing device remote from the device 100. Results of such remote navigation functions may be communicated to the device 100 via the communication interface 132. The processor 124 configured through execution of the applications 136 may also be referred to as a navigation controller. Thus, as will be apparent to one skilled in the art, the navigation controller may also be implemented by a separate computing device as described above.

[0013]

[0021] The chassis 112 also supports navigation sensors 140, such as one or more cameras and / or depth sensors (e.g., lidar or depth cameras) communicatively coupled to the processor 124. The sensors 140, in the illustrated example, are configured to capture sensor data, such as depth measurements indicative of at least a portion of the physical environment of the device 100. In particular, the sensor data represents an area encompassed by a field of view (FOV) 144 of the sensors 140. In some examples, the FOV 144 can be defined by the individual FOVs of multiple sensors. For example, the device 100 can include a depth camera with a substantially pyramidal FOV, such as that shown in FIG. 1, as well as a lidar with a substantially planar FOV. As seen in FIG. 1, the sensors 140 are disposed on the base of the chassis. In other examples, the sensors 140 (or any subset of the sensors, if multiple sensors are implemented) can be disposed in other suitable locations on the chassis 112, including on a rack supporting the container 120.

[0014]

[0022] As discussed in further detail below, the processor 124, through execution of the application 136, is configured to process sensor data captured by the sensors 140 to detect obstacles proximate the device 100 for use in navigating a facility, e.g., to navigate to a target location while avoiding such obstacles.

[0015]

[0023] 2, for example, an overhead view 200 and a side view 204 show the device 100 passing along a shelf module 104 or set of shelf modules 104 in a direction 206, with a person 208 standing proximate to the shelf module 104 or set of shelf modules 104. As seen in the side view 204, the field of view 144 encompasses a portion of the shelf module 104, as well as the person 208. Thus, the sensor data 212 (e.g., a frame of depth measurements) captured by the sensor 140 shows the entire person 208. As will be apparent, certain portions of the person 208 may not be visible to the sensor 140 if they are obscured by other portions of the person 208, but the entire person 208 is shown for ease of illustration. Once the sensor data 212 is captured, the processor 124 is capable of detecting obstacles, such as the shelf module 104 and the person 208 therein, and generating navigation commands to control the mobile assembly 116 to continue traveling toward the target location while avoiding the detected obstacles.

[0016]

[0024] The sensor data 212, or a processed form thereof, may be stored in the device 100 for use in the navigation control operations described above, and may be referred to as an obstacle map. In some implementations, the obstacle map takes the form of an occupancy grid, such as a grid of three-dimensional voxels (e.g., 1 cubic cm, although various other resolutions are contemplated). Each voxel may be marked as occupied or unoccupied, and the device 100 is configured to select travel paths and / or other navigation actions to avoid contact with occupied voxels.

[0017]

[0025] Further sets of sensor data, e.g., further frames of depth measurements, can be captured periodically at various frequencies. For example, the processor 124 can be configured to control the sensor 140 to capture updated sensor data several times per second. As would be apparent to one skilled in the art, movement of either or both of the device 100 and the person 208 (or any other dynamic, i.e., movable, obstacles in the facility, including shopping carts, boxes, etc.) changes the position of the FOV 144 relative to such obstacles. As a result, areas of the obstacle map generated from one or more prior frames of sensor data may later fall outside the FOV 144. That is, those areas may become unobservable in subsequent frames of sensor data.

[0018]

[0026] With reference to FIG. 3, for example, the device 100 is shown having traveled some distance along the shelf module 104 in the direction 206 from a previous position 300 (corresponding to the position shown in FIG. 2). In addition, the person 208 shown in FIG. 2 has moved away from the position shown in FIG. 2 (shown in dashed lines in FIG. 3), and thus the FOV 144 encompasses only a portion of the shelf module 104. Thus, in generating the updated obstacle map 304, the processor 124 clears some areas of the map 304 that previously showed the person 208. Those areas can be cleared because they were observable, fell within the FOV 144, and are clearly no longer occupied. However, certain parts of the previously observed person 208 are out of the FOV 144. Those parts were marked as occupied areas of the map from the sensor data 212, but are no longer observable. Thus, the map 304 includes unobservable occupied areas 308 whose occupancy status cannot be updated by direct observation. Thus, the unobservable occupancy may be navigationally significant in the immediate vicinity of the chassis 112, as shown in Figure 3. The volume in the immediate vicinity of the chassis 112 that is unobservable may depend on the number of sensors, their placement on the chassis 112, and the angle and direction of their respective FOVs.

[0019]

[0027] In some systems, device 100 can handle such unobservable occupancies by assuming that they remain occupied. However, as shown in FIG. 3, that assumption may be inaccurate in some scenarios, which may lead to inefficient navigation by device 100, for example, in the form of waiting for non-existent obstacles to clear or generating a new travel path that bypasses non-existent obstacles. In other systems, device 100 can instead simply clear the unobservable occupancies. However, that approach may lead to conflicts if the areas are actually still occupied.

[0020]

[0028] In further systems, as outlined above, the navigational challenges posed by unobservable occupancies can be addressed by evaluating the strength of connections between unobservable and observable occupancies. Such evaluation can be based, for example, on the proximity and / or presence of unoccupied space between the unobservable and observable regions. Thus, an unobservable occupancy that is close to an observed (observed) occupancy and separated from the observed occupancy with little or no unoccupied space can still be assumed to be occupied. However, the above approach may lead to inaccurate retention of an unobservable occupancy, such as region 308, due to the close proximity between the person 208 and the shelf module 104 (as seen in the overhead view 200 of FIG. 2).

[0021]

[0029] Thus, the device 100 implements further functionality through execution of the application 136, as described in further detail below, to generate individual connectivity assessments between unobserved occupancies and multiple individual categories of reference occupancies (e.g., observed obstacles and / or predefined static obstacles such as shelf modules 104). Thus, the device 100 is enabled to detect erroneous connections, such as the one between the person 208 and shelf modules 104 described above, resulting in improved accuracy in processing unobserved occupancies.

[0022]

[0030] 4, a method 400 of navigation state detection for unobservable regions is shown. Method 400 is described below with reference to, among other things, execution of method 400 by apparatus 100 via execution of application 136 by processor 124 and resulting control of other components of apparatus 100 by processor 124. As will be apparent, method 400 may also be performed by other apparatuses, and in some examples, may be performed by a computing device that is physically separate from apparatus 100, as discussed above.

[0023]

[0031] In block 405, the processor 124 is configured to control the sensor 140 to capture sensor data. In this example, the sensor 140 is assumed to be a depth camera, and thus in block 405 the sensor 140 is controlled to capture a frame of depth measurements (with or without color data). In other examples, as previously discussed, the device 100 can include multiple sensors, such as additional depth cameras or lidars. Such additional sensors can also be controlled in block 405 to capture sensor data.

[0024]

[0032] In block 410, upon capturing the sensor data in block 405, processor 124 is configured to detect obstacles in the sensor data and to update an obstacle map maintained in memory 128 to represent such obstacles. As will be apparent in the discussion below, the updated map generated in block 410 may not be a completely updated version of the map upon which navigation decisions are made; that is, further updates to the map may be made prior to use for navigation.

[0025]

[0033] FIG. 5 illustrates an example obstacle map 500 generated from the sensor data 212 shown in FIG. 2. That is, in an example implementation of block 405, when the device 100 and the person 208 are positioned as shown in FIG. 2, the processor 124 obtains the sensor data 212 and processes the sensor data 212 to identify an occupancy area 504 corresponding to the person 208 and an occupancy area 508 corresponding to the shelf module 104. The specific state of the occupancy areas 504 and 508 varies according to the implementation. For example, in some implementations, the processor 124 may perform an object segmentation operation to identify the shelf module 104 and the person 208 as separate objects having separate object identifiers. In other examples, as shown in FIG. 5, the processor 124 instead implements the map 500 as an occupancy grid, e.g., a three-dimensional grid of voxels registered to a coordinate system in which the device 100 also tracks its position. In such implementations, it is not necessary for the objects to be segmented or otherwise recognized. Instead, each voxel is assigned an occupied or unoccupied (empty) state based on whether the sensor data 212 indicates the presence of an obstacle in that voxel.

[0026]

[0034] In addition, the processor 124 is configured to label each occupied region (e.g., each occupied voxel observed in the sensor data 212) with one of a plurality of reference identifiers. The reference identifiers correspond to distinct categories of obstacles. In some examples, such as examples where object segmentation is performed, each reference identifier may correspond to a single particular obstacle (e.g., person 208). That is, each obstacle may be assigned a unique identifier, and thus each category includes only one obstacle. In other examples, each reference identifier may be assigned to multiple physical objects according to particular attributes of the objects detectable from the sensor data 212.

[0027]

[0035] Specifically, in this example, the reference identifiers include a first reference identifier corresponding to a static obstacle and a second reference identifier corresponding to a dynamic obstacle. A static obstacle is an obstacle represented in a predefined map of the facility, for example, stored in the memory 128 when the device 100 is located. The predefined map may indicate the location of fixed or infrequently reconfigured structures in the facility, such as the shelf module 104, walls, doorways, etc. A dynamic obstacle is any obstacle not represented in the predefined map. In this example, the memory 128 stores a predefined map 512 of the facility, in which the current location 516 of the device 100 is shown. To label the occupied area in the map 500, the processor 124 may determine whether the corresponding area in the map 512 is occupied. If the corresponding area in the map 512 is occupied, the occupied area in the map 500 is marked as a static obstacle. Otherwise, the occupied area in the map 500 is marked as a dynamic obstacle.

[0028]

[0036] Thus, in the illustrated example, the occupied regions in area 520, shown as voxels 524, are labeled with the reference identifier "S" to indicate that those voxels correspond to static obstacles in a given map 512. Area 520 coincides with the location of a portion of shelf module 104, as shown by the dashed line connecting area 520 with map 512. However, the occupied regions in area 528 correspond to areas of map 512 that include free space. Thus, certain voxels 532 in area 528 can be marked with an indication that those voxels 532 include free space (e.g., a value of zero, etc.). Other voxels 536 in area 528, which in this example correspond to the shoulders of person 208, are labeled with the reference identifier "D" to indicate that those voxels correspond to dynamic obstacles.

[0029]

[0037] Returning to FIG. 4, in block 415, the processor 124 is configured to determine whether the map from block 410 includes an unobservable occupied region. An unobservable occupied region is a region that has previously been marked as occupied (and thus associated with a reference identifier as described above) but is outside the FOV 144 of the current sensor data from block 405. A region may be outside the FOV 144, for example, by being outside the maximum range of the FOV 144, shown by the dashed line in FIG. 3, or by being within those ranges but occluded by another object. The position of the FOV 144, and therefore the distinction between observable and unobservable regions, such as the voxels mentioned above, is made by the device 100 based on the current position of the device 100 and calibration parameters corresponding to the sensor 140 held in the memory 128. The calibration parameters may include, for example, camera-intrinsic parameters and / or extrinsic parameters corresponding to a depth camera.

[0030]

[0038] 5 is the result of an initial execution of block 405, and therefore, it is assumed that there is no unobservable occupied area. Thus, the determination at block 415 is negative. In case of a negative determination at block 415, the processor 124 proceeds to block 450. At block 450, the processor 124 is configured to control the mobile assembly 116 to navigate the device 100 within the facility according to the obstacle map. Thus, in this example, the processor 124 can determine that the person 208 is sufficiently far away from the device 100 that forward movement in the direction 206 can continue.

[0031]

[0039] 6, in a second execution of block 405, device 100 is traveling in direction 206 from location 300 towards person 208. However, person 208 remains in the same location, in contrast to the scenario shown in FIG. 3, and thus device 100 is approaching person 208 such that a portion of person 208 is no longer observable within FOV 144.

[0032]

[0040] In block 410, the processor 124 is configured to update the map 500 based on the sensor data captured from the location of the device 100 shown in Figure 6. The processor 124 thus generates an updated map 600 in which the area 604 corresponding to the observable part of the person 208 is marked with a reference identifier representing a dynamic obstacle. Additionally, the area 608 corresponding to the observable part of the shelf module 104 is marked with a reference identifier representing a static obstacle. In this example, the area 612 corresponding to the currently unobservable part of the shelf module 104 remains marked with a reference identifier representing a static obstacle, and since they match the predefined map 512, they do not need to be treated as an unobservable occupied area according to the mechanism presented below.

[0033]

[0041] Further, region 616 corresponds to a portion of person 208 that is currently unobservable. Moreover, region 616 is not shown as occupied in given map 512. Thus, the current status of region 616 is unclear. The presence of region 616 in map 600 leads to a positive determination at block 415, and therefore processor 124 proceeds to block 420.

[0034]

[0042] In block 420, the processor 124 is configured to select one or more reference regions. In this example, where the maps 500, 600 are voxel-based occupancy grids, the processor 124 is configured to select a set of reference voxels from the map 600. The reference regions correspond to observed (i.e., within the FOV 144 for the current execution of block 405) and / or static obstacles from the predefined map 512. In other words, the reference regions represent obstacles whose presence and location are known with a high degree of confidence.

[0035]

[0043] In some examples, the selection of the reference region is limited to regions within a threshold distance from any unobservable occupancy region. Limiting the selection of the reference region based on proximity to the unobservable occupancy region can reduce the computational burden associated with processing the unobservable occupancy region. For example, the processor 124 can be configured to select as the reference region only voxels that are static or currently observed and that are also within a threshold distance from any of the regions 616. The selection of the reference region can be further limited to only voxels that are directly adjacent to the unobservable occupancy region and that are separated from the unobservable occupancy region only by empty space.

[0036]

[0044] Referring to Figure 7, an exemplary process for selecting a reference region at block 420 is shown. In particular, a portion 700 of map 600 is shown in a top view rather than the side view of Figure 6. In portion 700, two sets of voxels 704 and 708 are highlighted. Each set 704 and 708 has a depth of one voxel and is shown in detail in the bottom half of Figure 7 as viewed from corresponding directions 712 and 716. For clarity, the location of set 708 is also shown in side view 720.

[0037]

[0045] Set 704 includes voxels 724 and 728 corresponding to shelf module 104, voxel 732 representing empty space, unobservable occupied voxel 736 corresponding to a previously observed portion of person 208, and voxel 740 representing a currently observed portion of person 208 (i.e., labeled as a dynamic obstacle). Voxels 724, 728, and 740 are reference voxels, but need not all be selected in block 420. Instead, as described above, processor 124 is configured to select only reference voxels that are directly adjacent to unobservable occupied voxel 736 or that are separated from unobservable occupied voxel 736 only by empty space. Thus, processor 124 selects voxels 728 and 740 from set 704.

[0038]

[0046] Meanwhile, set 708 includes voxels 740 (described above) and 744 that correspond to currently observed portions of person 208, as well as unobservable occupied voxel 736. By the same logic as above, processor 124 selects voxel 740 from set 708 in block 420. As will now be apparent, processor 124 may also select various further voxels from map 600 according to the same criteria described above.

[0039]

[0047] In other examples, such as those in which map 600 includes coordinates or other attributes of segmented obstacles rather than an occupancy grid, selecting the reference area in block 420 may include selecting any segmented obstacles currently observed or in map 512 based on the proximity of the surfaces of such obstacles to the unobservable occupancy area.

[0040]

[0048] In block 425, once the reference regions are selected in block 420, the processor 124 is configured to generate at least one connectivity score for each unobservable occupied region (e.g., each unobservable occupied voxel 736). More specifically, the processor 124 is configured to generate a separate connectivity score for each reference identifier according to the selected reference region from block 420. Thus, in this example, the processor 124 is configured to generate two connectivity scores for each unobservable occupied voxel 736, a first score corresponding to the reference regions labeled as static and a second score corresponding to the reference regions labeled as dynamic.

[0041]

[0049] The connectivity score generated for a given unobservable region in block 425 associates the unobservable region with the corresponding reference identifier. In particular, the connectivity score indicates the strength of the association between the relevant unobservable region and the corresponding reference identifier. A stronger association (e.g., a higher connectivity score) indicates that the unobservable region is more likely to be physically connected to the object having the reference identifier. Then, based on the resulting connectivity score for a given unobservable occupied region and the originally assigned reference identifier for that region from block 410, processor 124 can determine whether to retain the occupied status of the region.

[0042]

[0050] In this exemplary implementation where the obstacle map is an occupancy grid, a connectivity score can be determined for each voxel in block 425 based on a flood-fill operation. For example, starting with a selected reference region from block 420, processor 124 can apply a score component to any adjacent unobservable occupancy regions. For example, processor 124 can be configured to apply a percentage (e.g., 80%, although various other percentages are contemplated) of the selected reference region's own connectivity score to any adjacent unobservable occupancy regions. The reference region is assigned the maximum connectivity score (e.g., 100% or any other suitable notation).

[0043]

[0051] 8, the sets of voxels 704 and 708 shown in FIG. 7 are shown during a particular stage of the flood-fill operation described above. In particular, as shown in the upper portion of FIG. 8, each selected reference region 740 that shares a side with an unobservable occupied region 736 contributes 80% of its connectivity score to that unobservable region 736. Thus, the unobservable occupied regions 800 each receive two 80% contributions since they each border two separate reference regions 740. If the connectivity score is limited to a maximum value (e.g., 100%), as in this example, the score for voxel 800 is therefore set to 100%. However, in other examples, setting an upper limit is not necessary. Furthermore, voxel 800 is assigned a reference identifier "D" that corresponds to the reference identifier of the selected voxel 740.

[0044]

[0052] 8, the flood-fill operation continues by selecting the next "unfilled" voxel, in this case a subset 804 of unobservable occupied voxels 736 that are adjacent to voxel 800 that has already been processed as discussed above. The subset 804 of voxels is filled as described above, with voxel 800 thereby serving as a seed voxel. As will now be apparent, the subset 804 of voxels also receives a connectivity score of 100% and a dynamic reference identifier, respectively, as does the last voxel 808 in the set 704.

[0045]

[0053] Referring to the set 708 shown in the lower third of FIG. 8, the selected reference voxels 728 are separated from the unobservable occupied voxels 736 by empty voxels. When flood-filling empty voxels rather than occupied voxels, the percentage donated to the empty voxels may be lower than the percentage donated to the occupied voxels. For example, the selected reference voxels 728 may donate 50% (rather than 80% as above, although various other percentages may be employed in other embodiments) of their connection scores to the adjacent empty voxels. Thus, in the three filling steps, the unobservable occupied voxels 736 receive a 20% contribution from the corresponding reference voxels 728. Specifically, the first adjacent empty voxel 732 receives 50%, the next adjacent empty voxel 732 receives 25% (50% of the 50% allocated to the first empty voxel 732). The unobservable occupied voxel then receives 80% of the 25%, or 20%. Thus, each of the three unobservable occupied voxels 736 shown receives a connectivity score of 20% associated with the static reference identifier in addition to the aforementioned 80% connectivity score associated with the dynamic reference identifier. The above process is repeated for any other unobservable occupied regions in the map.

[0046]

[0054] In other implementations, generation of the connectivity scores need not be performed via a flood-fill operation as described above. For example, a connectivity score for a given unobservable occupancy region or set of unobservable occupancy regions can be generated based on a calculated distance between the unobservable occupancy region and one or more reference regions (e.g., the centroid of the reference region). Such scores can be modified according to the presence or absence of free space between the reference region and the unobservable region.

[0047]

[0055] Having generated a connectivity score for each unobservable region of occupation, processor 124 is configured to complete execution of block 425 by selecting a current reference identifier for each unobservable region of occupation. Specifically, the current reference identifier is the reference identifier associated with the highest connectivity score among the reference identifiers generated in block 425. Thus, in the example of FIG. 8, a dynamic reference identifier is selected for unobservable region of occupation 736.

[0048]

[0056] Processor 124 is then configured to select a processing action for each unobservable occupancy region based on the connection score and the selected reference identifier from block 425 and on the previously selected reference identifier originally for the currently unobservable region. The processing action may include, for example, retaining the occupancy status and current reference identifier for the unobservable region, or clearing the unobservable occupancy region, i.e., discarding its reference identifier and marking the region as free space.

[0049]

[0057] In block 430, processor 124 is configured to determine whether the reference identifier selected in block 425 for each unobservable occupied region is different from the reference identifier originally assigned to the region when it was detected in block 410 (i.e., when the region was observable). If the determination in block 430 is positive, indicating that the current reference identifier does not match the original reference identifier, processor 124 proceeds to block 440. In block 440, processor 124 is configured to clear the unobservable occupied region and mark the unobservable occupied region as free space.

[0050]

[0058] However, in this example, the determination in block 430 is negative because the unobservable occupied voxels 736 were assigned dynamic reference identifiers. The unobservable occupied voxels 736 were also initially assigned dynamic reference identifiers (given that they represented parts of the person 208), and therefore the initial reference identifiers and the current reference identifiers match.

[0051]

[0059] In response to a negative determination at block 430, the processor 124 is configured to proceed to block 435. At block 435, the processor 124 is configured to determine whether the connectivity score associated with the selected current reference identifier from block 425 exceeds a threshold. The threshold at block 435 is selected (e.g., at the time of placement of the device 100) such that a connectivity score that meets the threshold is likely to indicate a true physical connection between an object in the unobservable region and an object that is observed. The threshold may be 15% in this example (although various other thresholds may be employed). Thus, as can be seen from the discussion of FIG. 8, the determination at block 435 is positive for each of the unobservable occupancy regions 736 in this exemplary implementation.

[0052]

[0060] Thus, a negative determination at block 430 and a positive determination at block 435 indicate that the detected connections between the unobservable occupancies and the still observable portions of the same category of obstacles originally assigned to those unobservable occupancies are strong enough to conclude that the obstacles have remained in substantially the same locations, and therefore that the currently unobservable portions are likely to remain in their previously observed locations. Thus, in block 445, the processor 124 retains the current reference identifiers (which match the initial reference identifiers) in association with those unobservable occupancies. That is, the unobservable occupancies are retained as occupancies with the same category of obstacles originally assigned to them. In other examples, the determinations at blocks 430 and 435 can be performed in the reverse order from that shown in FIG. 4.

[0053]

[0061] Following block 445, processor 124 proceeds to block 450. The dashed line from block 445 back to block 425 indicates that the above process is repeated for each unobservable occupancy, although such iterations need not be performed sequentially. Instead, all unobservable occupancies can be processed together, as described above in connection with the flood-fill operation.

[0054]

[0062] In block 450, the processor 124 is configured to control the mobile assembly 116 according to the updated obstacle map resulting from execution of blocks 410 to 445. For example, the processor 124 may issue a command to the mobile assembly 116 to pause forward movement of the device 100 until the person 208 moves out of the path of the device 100. The processor 124 then returns to block 405.

[0055]

[0063] In a further exemplary execution of the method 400, it is assumed that the person 208 has moved out of the FOV 144 of the sensor 140, as shown in FIG. 3. The obstacle map obtained in block 410 thus appears as the map 304 in FIG. 3, where the observable regions that previously contained parts of the person 208 have been cleared, but the unobservable occupancy regions 308 remain. As will now be apparent, since there are no longer any observable dynamic regions in the map from block 410, the subsequent execution of block 420 results in the selection of only the reference regions corresponding to the shelf module 104. Thus, the voxels that define the unobservable occupancy regions 308 are respectively assigned a connectivity score as described above only with respect to the static reference identifiers. In other words, the connectivity score corresponding to the dynamic reference identifiers for such voxels is zero. Thus, in block 425, static reference identifiers are selected for the unobservable occupancy regions 308. The determination in block 430 is therefore positive, since those regions were previously assigned dynamic reference identifiers. Thus, region 308 is cleared in block 440 resulting in the updated map 900 shown in Figure 9. Updated map 900 indicates the presence of only shelf module 104. Thus, upon further execution of block 450, device 100 is able to step along shelf module 104.

[0056]

[0064] In the foregoing specification, specific embodiments have been described. However, those skilled in the art will understand that various modifications and changes can be made without departing from the scope of the invention as set forth in the following claims. Accordingly, the specification and figures are to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present teachings.

[0057]

[0065] Benefits, advantages, solutions to problems, and any elements that may cause any benefit, advantage, or solution to occur or become more pronounced should not be construed as critical, required, or essential features or elements of any or all claims. The invention is defined solely by the appended claims, including any amendments made during the pendency of this application, and all equivalents of those claims as issued.

[0058]

[0066] Moreover, in this document, relationship terms, such as first and second, upper and lower, etc., may be used only to distinguish one entity or action from another and do not necessarily require or imply any actual such relationship or order between such entities or actions. The terms "comprises," "comprising," "has," "having," "includes," "including," "contains," "containing," or any other variations thereof, are intended to cover a non-exclusive inclusion, whereby a process, method, article, or apparatus that comprises, has, includes, or contains a list of elements may include other elements not expressly listed or inherent to such process, method, article, or apparatus, rather than including only those elements. An element preceded by "comprises a...", "has a...", "includes a...", or "contains a..." does not, without further constraints, preclude the presence of additional identical elements in the process, method, article, or apparatus that comprises, has, includes, or encompasses the element. The terms "a" and "an" are defined as one or more, unless expressly stated otherwise herein. The terms "substantially", "essentially", "approximately", "about", or any other variation thereof, are defined as "close to", as understood by one of ordinary skill in the art, and in one non-limiting embodiment, the term is defined as within 10%, in another embodiment within 5%, in another embodiment within 1%, and in another embodiment within 0.5%. The term "coupled", as used herein, is defined as connected, but not necessarily directly, and not necessarily mechanically.A device or structure that is "configured" in a particular way is configured in at least that way, but may be configured in ways not listed.

[0059]

[0067] Certain phrases may be employed herein to list combinations of elements. Examples of such phrases include "at least one of A, B, and C," "one or more of A, B, and C," "at least one of A, B, or C," "one or more of A, B, or C." Unless otherwise specified, the above phrases encompass any combination of A and / or B and / or C.

[0060]

[0068] It will be appreciated that some embodiments may be comprised of one or more dedicated processors (or "processing devices"), such as microprocessors, digital signal processors, customized processors, and field programmable gate arrays (FPGAs), and unique stored program instructions (including both software and firmware) that control the one or more processors to implement some, most, or all of the functions of the methods and / or apparatus described herein in conjunction with specific non-processor circuitry. Alternatively, some or all functions may be implemented by state machines that do not have stored program instructions, or in one or more application specific integrated circuits (ASICs) in which each function or some combination of some of those functions is implemented as custom logic. Of course, a combination of the two approaches may be used.

[0061]

[0069] Moreover, the embodiments can be implemented as a computer-readable storage medium having stored thereon computer-readable code for programming a computer (including, for example, a processor) to perform the methods described and claimed herein. Examples of such computer-readable storage media include, but are not limited to, hard disks, CD-ROMs, optical storage devices, magnetic storage devices, Read Only Memory (ROM), Programmable Read Only Memory (PROM), Erasable Programmable Read Only Memory (EPROM), Electrically Erasable Programmable Read Only Memory (EEPROM), and flash memory. Moreover, it is expected that a person skilled in the art would be readily able to generate such software instructions and programs and ICs with minimal experimentation when guided by the concepts and principles disclosed herein, despite significant effort and numerous design choices motivated, for example, by available time, current technology, and economic considerations.

[0062]

[0070] The Abstract of the present disclosure is provided to enable the reader to quickly ascertain the nature of the technical disclosure. The Abstract is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, it can be understood that in the foregoing Detailed Description, various features have been grouped together in various embodiments for the purpose of streamlining the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Thus, the following claims are incorporated into the Detailed Description, with each claim standing on its own as individually claimed subject matter.

Claims

1. maintaining a map of the facility's footprint; Controlling a navigation sensor of the mobile device to capture sensor data within a sensor field of view (FOV); identifying unobservable occupied areas in the map that are located outside the sensor field of view based on a current location of the mobile device in the facility; selecting from the map a first reference occupation region having a first reference identifier and a second reference occupation region having a second reference identifier; generating a first connectivity score relating the unobservable occupancy to the first reference occupancy and a second connectivity score relating the unobservable occupancy to the second reference occupancy; selecting a processing operation for the unobservable occupancy based on the first and second connectivity scores; updating the map according to the selected processing operations; controlling a moving assembly of the mobile device according to the updated map; The method includes:

2. processing the captured sensor data to detect observed occupancies located in the sensor field of view; updating the map according to the observed occupancy area; Further comprising: The method of claim 1 , wherein the first reference occupancy comprises the observed occupancy.

3. the map includes a static occupancy area; The method of claim 2 , wherein the second reference occupancy comprises the static occupancy.

4. The method of claim 2 , wherein the first and second reference identifiers are respective obstacle identifiers.

5. The method of claim 2 , wherein the first and second reference identifiers are respective obstacle categories.

6. 6. The method of claim 5, further comprising the step of, in response to detecting the observed occupied area, assigning an obstacle category to the observed occupied area based on a comparison of the observed occupied area to a set of predetermined static obstacles.

7. The step of generating the first and second connectivity scores comprises at least (i) the proximity between the reference occupancies; and (ii) the presence of an unoccupied region separating the reference occupied region from the unobservable occupied region; The method of claim 1 , based on

8. The method of claim 1 , wherein selecting the first and second reference occupancies comprises selecting an occupancy in the map that is within a threshold distance from the unobservable occupancy.

9. selecting the processing operations for the unobservable occupancy region, determining that neither of the first and second connectivity scores exceeds a threshold; clearing the unobservable occupancy in the map; The method of claim 1 , comprising:

10. selecting the processing operations for the unobservable occupancy region, selecting a larger of the first and second connectivity scores; associating the corresponding reference identifier with the unobservable domain of occupation; The method of claim 1 , comprising:

11. selecting the processing operations for the unobservable occupancy region, comparing said corresponding reference ID with a previous reference ID associated with said unobservable occupancy; clearing the unobservable occupancy in the map if the corresponding reference ID and the previous reference ID are different; The method of claim 10, comprising:

12. The mobile device a chassis base supporting the mover assembly and the sensor; a rack extending from the chassis for supporting one or more containers; The method of claim 1 , comprising:

13. 1. A computing device comprising: a sensor having a sensor field of view (FOV); a processor; The processor, Maintain a map of the facility's occupied area; Controlling the sensor to capture sensor data within a sensor field of view (FOV); identifying unobservable occupied areas in the map that are located outside the sensor field of view based on a current location of a mobile device in the facility; selecting from the map a first reference occupation region having a first reference identifier and a second reference occupation region having a second reference identifier; generating a first connectivity score relating the unobservable occupancy to the first reference occupancy and a second connectivity score relating the unobservable occupancy to the second reference occupancy; selecting a processing operation for the unobservable occupancy based on the first and second connectivity scores; updating the map according to the selected processing operations; controlling a moving assembly of the mobile device according to the updated map; 1. A computing device configured to:

14. The processor, processing the captured sensor data to detect observed occupancies located in the sensor field of view; updating the map according to the observed occupancy; It is further structured as follows: The method of claim 13 , wherein the first reference occupancy comprises the observed occupancy.

15. the map includes a static occupancy area; The computing device of claim 13 , wherein the second reference footprint comprises the static footprint.

16. The computing device of claim 13 , wherein the first and second reference identifiers are respective obstacle identifiers.

17. The computing device of claim 13 , wherein the first and second reference identifiers are respective obstacle categories.

18. The processor, In response to detecting the observed occupancy, assigning an obstacle category to the observed occupancy based on a comparison of the observed occupancy to a set of predetermined static obstacles.

20. The computing device of claim 17, further configured to:

19. The processor includes at least (i) the proximity between the reference occupancies; and (ii) the presence of an unoccupied region separating the reference occupied region from the unobservable occupied region; The computing device of claim 13 , configured to generate the first and second connectivity scores based on:

20. 14. The computing device of claim 13, wherein the processor is configured to select the first and second reference occupancies by selecting occupancies in the map that are within a threshold distance from the unobservable occupancy.

21. The processor, determining that neither of the first and second connectivity scores exceeds a threshold; clearing the unobservable occupancy in the map; and 14. The computing device of claim 13, configured to select the processing operations for the unobservable occupancy by:

22. The processor, selecting a greater one of the first and second connectivity scores; associating said corresponding reference identifier with said unobservable domain of occupancy; 14. The computing device of claim 13, configured to select the processing operations for the unobservable occupancy by:

23. The processor, comparing the corresponding reference ID with a previous reference ID associated with the unobservable occupancy; clearing the unobservable occupancy in the map if the corresponding reference ID and the previous reference ID are different; 22. The computing device of claim 21, configured to select the processing operations for the unobservable occupancy by:

24. The mobile device a chassis base supporting the mover assembly and the sensor; a rack extending from the chassis for supporting one or more containers; The computing device of claim 13 , comprising:

Images