Method and system for detecting navigation state in unobservable regions

Abstract

The method includes maintaining a map of an occupied area of ​​the facility; controlling a sensor of the mobile device to capture sensor data within a field of view (FOV); identifying unobservable occupied areas in the map that are located outside the FOV based on a current location of the mobile device in the facility; selecting a first reference occupied area having a first reference identifier and a second reference occupied area having a second reference identifier from the map; generating a first connectivity score associating the unobservable occupied area with the first reference occupied area and a second connectivity score associating the unobservable occupied area with the second reference occupied area; selecting a processing operation for the unobservable occupied area based on the first and second connectivity scores; updating the map according to the selected processing operation; and controlling a mobile assembly of the mobile device according to the updated map.

Description

Background Art

[0001]

[0001] Autonomous or semi-autonomous mobile devices may for navigation purposes comprise navigation sensors such as depth sensors for detecting obstacles in the vicinity of the device. Such sensors may have a limited field of view, whereby as the device travels, areas in which obstacles were previously observed move out of the sensor's field of view. The inability of mobile devices to observe these areas introduces ambiguity as to whether previously observed obstacles are still present, which may lead to inaccurate and / or inefficient navigation of the device. Summary of the Invention

[0002]

[0002] The accompanying drawings, in which like reference numerals refer to the same or functionally similar elements throughout the separate views, are incorporated in and form a part of this specification together with the following detailed description, and further illustrate embodiments of the concepts comprising the claimed invention and serve to explain various principles and advantages of those embodiments. Brief Description of the Drawings

[0003] [Figure 1]

[0003] Fig. 1 is a diagram showing a mobile device and a portion of an environment in which the device is positioned for navigation. [Figure 2]

[0004] Fig. 2 is a diagram showing detection of an obstacle by the device of Fig. 1. [Figure 3]

[0005] Fig. 3 is a diagram showing unobservable occupied areas resulting from movement of the device and / or other obstacles during navigation. [Figure 4]

[0006] Fig. 4 is a flowchart of a method for detecting a navigation state relating to an unobservable occupied area. [Figure 5]

[0007] Fig. 5 is a diagram showing exemplary implementation of blocks 405 and 410 of the method of Fig. 4. [Figure 6]

[0008] This figure shows a further exemplary execution of blocks 405 and 410 of the method in Figure 4. [Figure 7]

[0009] This figure shows an exemplary execution of block 420 of the method in Figure 4. [Figure 8]

[0010] This figure shows an exemplary execution of block 425 of the method in Figure 4. [Figure 9]

[0011] This figure shows the results of a further exemplary implementation of the method shown in Figure 4. [Modes for carrying out the invention]

[0004]

[0012] Those skilled in the art will understand that the elements in the figures are shown for conciseness and clarity and are not necessarily drawn to scale. For example, the dimensions of some elements in the figures may be exaggerated compared to others in order to help improve the understanding of embodiments of the invention.

[0005]

[0013] Components of the apparatus and methods are represented in the drawings by conventional symbols where appropriate, and only specific details relevant to understanding embodiments of the invention are shown so as not to obscure this disclosure with details that would be readily apparent to those skilled in the art who benefit from the description herein.

[0006]

[0014] Examples disclosed herein include a method comprising: maintaining a map of the occupied areas of a facility; controlling the navigation sensors of a mobile device to capture sensor data within the sensor field of view (FOV); identifying unobservable occupied areas located outside the sensor FOV on the map based on the current position of the mobile device in the facility; selecting a first reference occupied area having a first reference identifier and a second reference occupied area having a second reference identifier from the map; generating a first connection score associating the unobservable occupied areas with the first reference occupied areas and a second connection score associating the unobservable occupied areas with the second reference occupied areas; selecting a processing operation for the unobservable occupied areas based on the first and second connection scores; updating the map according to the selected processing operation; and controlling the locomotive assembly of the mobile device according to the updated map.

[0007]

[0015] Further examples disclosed herein relate to a computing device comprising a sensor having a field of view (FOV) and a processor, wherein the processor maintains a map of the occupied areas of a facility, controls the sensor to acquire sensor data within the sensor's field of view (FOV), identifies unobservable occupied areas located outside the sensor's FOV on the map based on the current position of a mobile device in the facility, selects a first reference occupied area having a first reference identifier and a second reference occupied area having a second reference identifier from the map, generates a first connection score associating the unobservable occupied areas with the first reference occupied areas and a second connection score associating the unobservable occupied areas with the second reference occupied areas, selects a processing operation for the unobservable occupied areas based on the first and second connection scores, updates the map according to the selected processing operation, and controls the locomotive assembly of the mobile device according to the updated map.

[0008]

[0016] Figure 1 shows an autonomous or semi-autonomous mobile device 100, which is also referred to herein simply as device 100. Device 100 can be deployed in any of a variety of environments, such as an article handling facility (e.g., a warehouse, a retail facility, etc.). The facility may include multiple support structures, such as a shelf module 104 for supporting articles 108.

[0009]

[0017] As will be apparent, the facility in which the device 100 is installed can take various other forms, such as a manufacturing facility or an outdoor work environment. The device 100 includes certain components that enable the device 100 to navigate the facility in which it is installed at least partially autonomously, and implements certain functionalities that enable it to navigate at least partially autonomously. For example, the device 100 can be configured to navigate between aisles formed by shelf modules 104, collect images or other data corresponding to articles 108, and receive and transport articles 108 for order fulfillment operations. Thus, the device 100 may also be referred to as a transport device 100.

[0010]

[0018] The apparatus 100 includes a chassis 112 that supports various other components of the apparatus 100. In particular, the chassis 112 supports a locomotive assembly 116, such as a set of wheels, tracks, and one or more electric motors that drive them. In the example shown, the chassis 112 supports one or more containers 120, such as a large box or large bag, for receiving and transporting articles 108 placed inside by workers in a facility. In other examples, the containers 120 may be omitted depending on the specific configuration of the apparatus 100 (for example, whether the apparatus 100 is deployed to transport articles 108).

[0011]

[0019] The components supported by the chassis 112 also include a processor 124 in the form of a dedicated hardware controller, such as one or more central processing units (CPUs), graphics processing units (GPUs), or application-specific integrated circuits (ASICs). The processor 124 is communicatively coupled to memory 128, for example, a suitable combination of volatile and non-volatile memory elements. The processor 124 may also be coupled to a communication interface 132, such as a wireless transceiver, which enables the device 100 to communicate with other computing devices via a suitable network infrastructure. However, in other examples, the communication interface 132 may be omitted.

[0012]

[0020] Memory 128 stores various data used for autonomous or semi-autonomous navigation by the device 100, including applications 136 that can be executed by the processor 124 to perform navigation functions. In some examples, the above functions can be performed through multiple separate applications stored in memory 128. In further examples, some or all such functions can be performed by a separate computing device located away from the device 100. The results of such remote navigation functions can be communicated to the device 100 via the communication interface 132. The processor 124, configured through the execution of applications 136, is sometimes referred to as the navigation controller. Thus, as will be apparent to those skilled in the art, the navigation controller can also be performed by a separate computing device as described above.

[0013]

[0021] The chassis 112 also supports one or more cameras and / or depth sensors (e.g., a lidar or depth camera) and a navigation sensor 140, which are communicatively coupled to the processor 124. The sensor 140 is configured to capture sensor data, such as depth measurements, which represent at least a portion of the physical environment of the device 100, in the example shown. In detail, the sensor data represents the area encompassed by the field of view (FOV) 144 of the sensor 140. In some examples, the FOV 144 can be defined by the individual FOVs of multiple sensors. For example, the device 100 may include a depth camera with a substantially pyramidal FOV, as shown in Figure 1, and a lidar with a substantially planar FOV. As seen in Figure 1, the sensor 140 is located on the base of the chassis. In other examples, the sensor 140 (or any subset of sensors if multiple sensors are implemented) may be located in other suitable locations on the chassis 112, including on a rack supporting the container 120.

[0014]

[0022] As will be discussed in more detail below, the processor 124 is configured to process sensor data acquired by the sensor 140 through the execution of application 136 to detect obstacles near the device 100 and to use when navigating within the facility, for example, to pass through to a target location while avoiding such obstacles.

[0015]

[0023] For example, referring to Figure 2, the overhead view 200 and the side view 204 show the device 100 moving along shelf module 104 or a set of shelf modules 104 in direction 206, with a person 208 standing very close to the shelf module 104 or set of shelf modules 104. As seen in the side view 204, the field of view 144 encompasses a portion of shelf module 104, as well as person 208. Therefore, the sensor data 212 captured by sensor 140 (e.g., a frame of depth measurements) shows the entire person 208. As will be obvious, certain parts of person 208 may not be visible to sensor 140 if they are obscured by other parts of person 208, but the entire person 208 is shown for the sake of illustration simplicity. Once sensor data 212 is acquired, the processor 124 can detect obstacles such as the shelf module 104 and the person 208 inside it, and generate navigation commands to control the mobile assembly 116 to continue moving toward the target position while avoiding the detected obstacles.

[0016]

[0024] Sensor data 212, or its processed form, can be stored in the device 100 for use in the navigation control operations described above and may be referred to as an obstacle map. In some implementations, the obstacle map takes the form of an occupied grid, such as a grid of three-dimensional voxels (e.g., 1 cubic cm, but various other resolutions are also possible). Each voxel can be marked as occupied or unoccupied, and the device 100 is configured to select a path and / or other navigation action to avoid contact with occupied voxels.

[0017]

[0025] Further sets of sensor data, such as additional frames of depth measurements, can be periodically acquired at varying frequencies. For example, the processor 124 can be configured to control the sensor 140 to acquire updated sensor data several times per second. As will be apparent to those skilled in the art, the movement of either or both of the apparatus 100 and / or the person 208 (or any other dynamic, i.e., movable obstacle in the facility, including shopping carts, boxes, etc.) changes the position of the FOV 144 relative to such obstacles. As a result, areas of the obstacle map generated from one or more preceding frames of sensor data may later fall outside the FOV 144; that is, those areas may become unobservable in subsequent frames of sensor data.

[0018]

[0026] Referring to FIG. 3, for example, the device 100 is shown as having traveled some distance along the shelf module 104 in direction 206 from the previous position 300 (corresponding to the position shown in FIG. 2). In addition, the person 208 shown in FIG. 2 has moved away from the position shown in FIG. 2 (indicated by a broken line in FIG. 3), and therefore the FOV 144 only covers a part of the shelf module 104. Therefore, when generating the updated obstacle map 304, the processor 124 clears some regions of the map 304 that previously indicated the person 208. Those regions are observable, fall within FOV 144, and are clearly no longer occupied, so they can be cleared. However, certain portions of the previously observed person 208 are outside of FOV 144. These portions were marked as occupied regions of the map from the sensor data 212, but are no longer observable. Therefore, the map 304 includes an unobservable occupied region 308 having an occupation status that cannot be updated by direct observation. Therefore, as shown in FIG. 3, the unobservable occupied region may be in the immediate vicinity of the chassis 112 and important for navigation. The volume in the immediate vicinity of the chassis 112 that is unobservable may depend on the number of sensors, their placement positions on the chassis 112, and the angle and direction of their respective FOVs.

[0019]

[0027] In some systems, device 100 can handle such unobservable occupied areas by assuming they remain occupied. However, as shown in Figure 3, this assumption is inaccurate in some scenarios, which can lead to inefficient navigation by device 100, for example, by waiting for non-existent obstacles to disappear or by generating new paths that bypass non-existent obstacles. In other systems, device 100 can instead simply clear the unobservable occupied areas. However, this approach can lead to collisions if those areas are actually still occupied.

[0020]

[0028] In further systems, as outlined above, navigational challenges posed by unobservable occupied regions can be addressed by evaluating the strength of connectivity between unobservable and observable occupied regions. Such evaluation can be based, for example, on proximity and / or the presence or absence of unoccupied space between the unobservable and observable regions. Thus, an unobservable occupied region that is close to an observed occupied region and separated from it with little or no unoccupied space can still be assumed to be occupied. However, the above approach may lead to inaccurate identification of unobservable occupied regions, such as region 308, due to the extremely close proximity between person 208 and shelf module 104 (as seen in the overhead view 200 in Figure 2).

[0021]

[0029] Accordingly, as will be described in further detail below, the apparatus 100 implements additional functionality via execution of the application 136 to generate individual connectivity assessments between an unobservable occupied area and reference occupied areas of a plurality of individual categories (e.g., observed obstacles and / or predetermined static obstacles such as the shelf module 104). Accordingly, the apparatus 100 is enabled to detect false connections such as that between the person 208 and the shelf module 104 described above, and as a result, accuracy in processing unobserved occupied areas is improved.

[0022]

[0030] Referring to FIG. 4, there is shown a method 400 of navigation state detection for unobservable areas. The method 400 is described below in connection with execution of the method 400 by the apparatus 100, via execution of the application 136 by the processor 124 among other things, and the resulting control of other components of the apparatus 100 by the processor 124. As will be appreciated, the method 400 may also be executed by other apparatuses, and in some examples, may be executed by a computing device physically separate from the apparatus 100, as noted above.

[0023]

[0031] At block 405, the processor 124 is configured to control the sensor 140 to capture sensor data. In this example, the sensor 140 is assumed to be a depth camera, and thus at block 405, the sensor 140 is controlled to capture one frame of depth measurements (values that include color data or do not include color data). In other examples, as noted above, the apparatus 100 may include a plurality of sensors such as additional depth cameras or lidar. Such additional sensors may also be controlled to capture sensor data at block 405.

[0024]

[0032] In block 410, once sensor data is acquired in block 405, the processor 124 is configured to detect obstacles in the sensor data and to update the obstacle map held in memory 128 to represent such obstacles. As will become clear in the following discussion, the updated map generated in block 410 may not be a fully updated version of the map on which navigation decisions are made. That is, further updates to the map may be made prior to its use for navigation.

[0025]

[0033] Figure 5 shows an exemplary obstacle map 500 generated from the sensor data 212 shown in Figure 2. That is, in an exemplary execution of block 405, if the device 100 and person 208 are arranged as shown in Figure 2, the processor 124 obtains the sensor data 212 and processes it to identify the occupied area 504 corresponding to person 208 and the occupied area 508 corresponding to shelf module 104. The specific states of occupied areas 504 and 508 vary depending on the implementation. For example, in some implementations, the processor 124 can perform an object segmentation operation to identify shelf module 104 and person 208 as separate objects with separate object identifiers. In other examples, as shown in Figure 5, the processor 124 instead implements the map 500 as an occupied grid, for example, a three-dimensional grid of voxels registered in a coordinate system in which the device 100 also tracks its own position. In such implementations, it is not necessary for objects to be segmented or otherwise recognized. Instead, each voxel is assigned an occupied or unoccupied (empty) state based on whether the sensor data 212 indicates the presence of an obstacle in that voxel.

[0026]

[0034] In addition, the processor 124 is configured to label each occupied area (for example, each occupied voxel observed in the sensor data 212) with one of several reference identifiers. The reference identifiers correspond to individual categories of obstacles. In some examples, such as when object segmentation is performed, each reference identifier may correspond to a single specific obstacle (e.g., a person 208). That is, each obstacle can be assigned a unique identifier, and therefore each category contains only one obstacle. In other examples, each reference identifier may be assigned to multiple physical objects according to specific characteristics of the objects detectable from the sensor data 212.

[0027]

[0035] Specifically, in this example, the reference identifier includes a first reference identifier corresponding to a static obstacle and a second reference identifier corresponding to a dynamic obstacle. Static obstacles are, for example, obstacles represented in a predetermined map of the facility, which is stored in memory 128 when the device 100 is placed. The predetermined map can show the locations of fixed or not frequently reconfigured structures in the facility, such as shelf modules 104, walls, and doorways. Dynamic obstacles are any obstacles not represented in the predetermined map. In this example, memory 128 stores a predetermined map 512 of the facility, which shows the current location 516 of the device 100. To label occupied areas in map 500, the processor 124 can determine whether the corresponding area in map 512 is occupied. If the corresponding area in map 512 is occupied, the occupied area in map 500 is marked as a static obstacle. Otherwise, the occupied area in map 500 is marked as a dynamic obstacle.

[0028]

[0036] Therefore, in the example shown, the occupied area in area 520, shown as voxel 524, is labeled with reference identifier "S" to indicate that those voxels correspond to static obstacles in a given map 512. Area 520 coincides with the location of a portion of shelf module 104, as indicated by the dashed line connecting area 520 to map 512. However, the occupied area in area 528 corresponds to an area of ​​map 512 that includes empty space. Therefore, a particular voxel 532 in area 528 can be marked with an indication (e.g., a value of zero) that those voxels 532 include empty space. In this example, another voxel 536 in area 528, corresponding to the shoulder of person 208, is labeled with reference identifier "D" to indicate that those voxels correspond to dynamic obstacles.

[0029]

[0037] Returning to Figure 4, in block 415, processor 124 is configured to determine whether the map from block 410 contains unobservable occupied regions. Unobservable occupied regions are regions that were previously marked as occupied (and therefore associated with a reference identifier as described above) but are outside the FOV 144 of the current sensor data from block 405. Regions may be outside the FOV 144 by being outside the maximum range of the FOV 144, indicated by the dashed line in Figure 3, or by being within that range but occupying it by another object. The distinction between the observable and unobservable regions of the FOV 144, and therefore the voxels described above, is made by the device 100 based on the device's current position and calibration parameters corresponding to the sensor 140 held in memory 128. Calibration parameters may include, for example, camera-specific parameters and / or external parameters corresponding to a depth camera.

[0030]

[0038] In this example, the map 500 shown in Figure 5 is the result of the initial execution of block 405, and therefore it is assumed that there are no unobservable occupied areas. Thus the determination in block 415 is negative. In the case of a negative determination in block 415, the processor 124 proceeds to block 450. In block 450, the processor 124 is configured to control the mobile assembly 116 to navigate the device 100 within the facility according to the obstacle map. Therefore, in this example, the processor 124 can determine that the person 208 is far enough away from the device 100 that it is possible to continue forward movement in direction 206.

[0031]

[0039] In the second execution of block 405, referring to Figure 6, the device 100 is moving from position 300 towards person 208 in direction 206. However, person 208 remains in the same location, in contrast to the scenario shown in Figure 3, and therefore the device 100 is approaching person 208, thereby making part of person 208 no longer observable within FOV 144.

[0032]

[0040] In block 410, the processor 124 is configured to update map 500 based on sensor data acquired from the location of device 100 shown in Figure 6. Thus, the processor 124 generates an updated map 600 in which the region 604 corresponding to the observable portion of person 208 is marked with a reference identifier representing a dynamic obstacle. Furthermore, the region 608 corresponding to the observable portion of shelf module 104 is marked with a reference identifier representing a static obstacle. In this example, the region 612 corresponding to the currently unobservable portion of shelf module 104 remains marked with a reference identifier representing a static obstacle, and since they coincide with the given map 512, they do not need to be treated as unobservable occupied areas according to the mechanism shown below.

[0033]

[0041] Furthermore, region 616 corresponds to the portion of person 208 that is currently unobservable. Moreover, region 616 is not shown as occupied in the given map 512. Therefore, the current state of region 616 is unclear. The presence of region 616 in map 600 leads to a positive determination in block 415, and therefore processor 124 proceeds to block 420.

[0034]

[0042] In block 420, processor 124 is configured to select one or more reference regions. In this example, where maps 500, 600 are voxel-based occupancy grids, processor 124 is configured to select a set of reference voxels from map 600. Those reference regions are observed (i.e., within the FOV 144 for the current execution of block 405) and / or correspond to static obstacles from a given map 512. In other words, those reference regions represent obstacles whose existence and location are known with high confidence.

[0035]

[0043] In some examples, the selection of reference regions is limited to regions within a threshold distance from any unobservable occupied regions. Limiting the selection of reference regions based on proximity to unobservable occupied regions can reduce the computational burden associated with processing unobservable occupied regions. For example, processor 124 can be configured to select only voxels as reference regions that are static or currently observed and also within a threshold distance from any of the regions 616. The selection of reference regions can be further limited to voxels directly adjacent to unobservable occupied regions, and voxels separated from unobservable occupied regions only by free space.

[0036]

[0044] Referring to Figure 7, an exemplary process for selecting a reference region in block 420 is shown. In detail, a portion 700 of map 600 is shown in the top view, rather than the side view in Figure 6. In portion 700, two sets of voxels 704 and 708 are highlighted. Each set 704 and 708 has a depth of one voxel and is shown in detail in the lower half of Figure 7 as seen from the corresponding directions 712 and 716. For clarity, the location of set 708 is also shown in the side view 720.

[0037]

[0045] Set 704 includes voxels 724 and 728 corresponding to shelf module 104, voxel 732 representing free space, unobservable occupied voxel 736 corresponding to previously observed portion of person 208, and voxel 740 representing currently observed portion of person 208 (i.e., labeled as a dynamic obstacle). Voxels 724, 728, and 740 are reference voxels, but not all of them need to be selected in block 420. Instead, as described above, processor 124 is configured to select only reference voxels that are directly adjacent to the unobservable occupied voxel 736, or that are separated from the unobservable occupied voxel 736 only by free space. Thus, processor 124 selects voxels 728 and 740 from set 704.

[0038]

[0046] On the other hand, set 708 includes voxels 740 (mentioned above) and 744, which correspond to the currently observed portion of person 208, as well as the unobservable occupied voxel 736. By the same logic as above, processor 124 selects voxel 740 from set 708 in block 420. As will be apparent here, it is also possible for processor 124 to select various further voxels from map 600 according to the same criteria as above.

[0039]

[0047] In other examples, for instance, where map 600 includes coordinates or other attributes of segmented obstacles rather than an occupied grid, the selection of a reference region in block 420 may include selecting any currently observed or present segmented obstacle in map 512 based on the proximity of the surface of such obstacles to an unobservable occupied region.

[0040]

[0048] In block 425, once a reference region is selected in block 420, the processor 124 is configured to generate at least one connection score for each unobservable occupied region (for example, each unobservable occupied voxel 736). More specifically, the processor 124 is configured to generate separate connection scores for each reference identifier according to the selected reference region from block 420. Thus, in this example, the processor 124 is configured to generate two connection scores for each unobservable occupied voxel 736: a first score corresponding to a statically labeled reference region and a second score corresponding to a dynamically labeled reference region.

[0041]

[0049] The connection score generated in block 425 for a given unobservable region associates that unobservable region with a corresponding reference identifier. More specifically, the connection score indicates the strength of the association between the relevant unobservable region and the corresponding reference identifier. A stronger association (e.g., a higher connection score) indicates a higher probability that the unobservable region is physically connected to the object having the reference identifier. Then, based on the resulting connection score for a given unobservable occupied region and the reference identifier initially assigned to that region from block 410, the processor 124 can determine whether to maintain the occupied status of that region.

[0042]

[0050] In this exemplary execution where the obstacle map is an occupancy grid, the connectivity score can be determined for each voxel in block 425 based on a flood fill operation. For example, starting with a selected reference region from block 420, the processor 124 can apply the score component to any adjacent unobservable occupancy region. For example, the processor 124 can be configured to apply a certain percentage of the connectivity score of the selected reference region itself (e.g., 80%, but various other percentages are also possible) to any adjacent unobservable occupancy region. The reference region is then allocated the maximum connectivity score (e.g., 100% or any other appropriate notation).

[0043]

[0051] Referring to Figure 8, the sets of voxels 704 and 708 shown in Figure 7 are shown during a particular stage of the flood fill operation described above. More specifically, as shown in the upper part of Figure 8, each selected reference region 740 that shares a side with the unobservable occupied region 736 grants 80% of its connection score to the unobservable region 736. Thus, the unobservable occupied regions 800 each receive two 80% grants, since each of them is adjacent to two separate reference regions 740. In this example, where the connection score is limited to a maximum value (e.g., 100%), the score for voxel 800 is therefore set to 100%. However, in other examples, it is not necessary to set an upper limit. Furthermore, voxel 800 is assigned the reference identifier "D" corresponding to the reference identifier of the selected voxel 740.

[0044]

[0052] As shown in the central part of Figure 8, the flood fill operation continues by selecting a subset 804 of the unobservable occupied voxel 736 adjacent to the next “unfilled” voxel, in this case voxel 800, which has already been processed as discussed above. The subset 804 of voxels is filled as described above, and consequently voxel 800 serves as the seed voxel. As will be apparent here, the subset 804 of voxels also receives a 100% connection score and a dynamic reference identifier, respectively, as does the last voxel 808 in set 704.

[0045]

[0053] Referring to set 708 shown in the lower third of Figure 8, the selected reference voxel 728 is separated from the unobservable occupied voxel 736 by empty voxels. When flood-filling into empty voxels instead of occupied voxels, the proportion granted to those empty voxels may be lower than the proportion granted to occupied voxels. For example, the selected reference voxel 728 may grant 50% of its connection score to adjacent empty voxels (not 80% as above, and in other embodiments, various other proportions may be adopted). Therefore, in the three satisfying steps, the unobservable occupied voxel 736 receives a 20% grant from the corresponding reference voxel 728. Specifically, the first adjacent empty voxel 732 receives 50%, and the next adjacent empty voxel 732 receives 25% (50% of the 50% allocated to the first empty voxel 732). The unobservable occupied voxel then receives 80% of 25%, i.e., 20%. Thus, each of the three unobservable occupied voxels 736 shown receives the aforementioned 80% connection score associated with the dynamic reference identifier, in addition to the 20% connection score associated with the static reference identifier. The above process is repeated for any other unobservable occupied regions in the map.

[0046]

[0054] In other implementations, the generation of connectivity scores does not necessarily have to be performed via a flood-fill operation as described above. For example, a connectivity score for a given unobservable occupied region or set of unobservable occupied regions can be generated based on the calculated distance between that unobservable occupied region and one or more reference regions (e.g., the centroids of reference regions). Such a score can be modified according to the presence or absence of free space between the reference regions and the unobservable regions.

[0047]

[0055] Once a connection score has been generated for each unobservable occupied region, the processor 124 is configured to complete the execution of block 425 by selecting the current reference identifier for each unobservable occupied region. Specifically, the current reference identifier is the reference identifier associated with the highest connection score among the reference identifiers generated in block 425. Therefore, in the example in Figure 8, a dynamic reference identifier is selected for the unobservable occupied region 736.

[0048]

[0056] The processor 124 is then configured to select a processing action for each unobservable occupied region based on the connection score and selected reference identifier from block 425, as well as the past reference identifier initially selected for the currently unobservable region. The processing actions include, for example, retaining the occupied status and current reference identifier for the unobservable region, or clearing the unobservable occupied region, i.e., discarding its reference identifier and marking the region as free space.

[0049]

[0057] In block 430, the processor 124 is configured to determine whether the reference identifier selected in block 425 for each unobservable occupied region is different from the reference identifier initially assigned to that region when it was detected in block 410 (i.e., when the region was observable). If the determination in block 430 is positive and indicates that the current reference identifier does not match the original reference identifier, the processor 124 proceeds to block 440. In block 440, the processor 124 is configured to clear the unobservable occupied region and mark it as free space.

[0050]

[0058] However, in this example, a dynamic reference identifier was assigned to the unobservable occupied voxel 736, so the determination in block 430 is negative. The unobservable occupied voxel 736 was also initially assigned a dynamic reference identifier (assuming that those unobservable occupied voxels 736 represented the portion of person 208), and therefore the initial reference identifier matches the current reference identifier.

[0051]

[0059] In response to a negative determination in block 430, processor 124 is configured to proceed to block 435. In block 435, processor 124 is configured to determine whether the connection score associated with the currently selected reference identifier from block 425 exceeds a threshold. The threshold in block 435 is selected such that connection scores satisfying the threshold are more likely to indicate a true physical connection between an object in the unobservable region and an observed object (for example, when placing device 100). In this example, the threshold can be 15% (however, various other thresholds may be adopted). Thus, as can be seen from the discussion in Figure 8, the determination in block 435 is positive with respect to each of the unobservable occupied regions 736 in this exemplary execution.

[0052]

[0060] Therefore, the negative determination in block 430 and the positive determination in block 435 indicate that the detected connection between the unobservable occupied regions and the still observable portion of the same category of obstacle initially allocated to those unobservable occupied regions is strong enough that the obstacle remains substantially in the same location, and therefore it is likely that the currently unobservable portion remains in its previously observed location. Thus, in block 445, processor 124 retains a current reference identifier (matching the initial reference identifier) ​​in relation to those unobservable occupied regions. That is, those unobservable occupied regions are retained as occupied regions having the same category of obstacle to which they were initially allocated. In other examples, the determinations in blocks 430 and 435 can be performed in the reverse order of those shown in Figure 4.

[0053]

[0061] Following block 445, processor 124 proceeds to block 450. The dashed line returning from block 445 to block 425 indicates that the above process is repeated for each unobservable occupied region, but such iterations do not need to be performed sequentially. Instead, all unobservable occupied regions can be processed together, as described above in relation to flood fill operations.

[0054]

[0062] In block 450, the processor 124 is configured to control the mobile assembly 116 according to the updated obstacle map resulting from the execution of blocks 410 through 445. For example, the processor 124 can issue a command to the mobile assembly 116 to pause the forward movement of the device 100 until person 208 moves out of the path of the device 100. The processor 124 then returns to block 405.

[0055]

[0063] In a further exemplary execution of Method 400, it is assumed that person 208 has moved outside the FOV 144 of sensor 140, as shown in Figure 3. Thus, the obstacle map obtained in block 410 appears as map 304 in Figure 3, where the observable region that previously contained the portion of person 208 has been cleared, but the unobservable occupied region 308 remains. As will be apparent here, in the map from block 410 there are no longer any observable dynamic regions, so the subsequent execution in block 420 results in the selection of only the reference region corresponding to shelf module 104. Thus, the voxels defining the unobservable occupied region 308 are each assigned the connection score described above, but only with respect to the static reference identifier. In other words, the connection score corresponding to the dynamic reference identifier for such voxels is zero. Thus, in block 425, the static reference identifier is selected for the unobservable occupied region 308. Thus, the determination in block 430 is affirmative, because those regions were previously assigned dynamic reference identifiers. Therefore, area 308 is cleared in block 440, resulting in the updated map 900 shown in Figure 9. The updated map 900 shows only the presence of shelf module 104. Thus, during further execution of block 450, the device 100 can proceed along shelf module 104.

[0056]

[0064] The aforementioned specification describes specific embodiments. However, those skilled in the art will understand that various modifications and changes can be made without departing from the scope of the invention as described in the following claims. Accordingly, this specification and the figures should be considered illustrative rather than restrictive, and all such modifications are intended to be within the scope of this teaching.

[0057]

[0065] No benefit, advantage, solution to a problem, or any element that may cause any benefit, advantage, or solution to occur or become more prominent should be construed as an essential, required, or indispensable feature or element of any or all of the claims. The present invention is defined solely by the appended claims, including any amendments made during the pendency of this application, and all equivalents of those claims issued.

[0058]

[0066] Furthermore, in this document, relational terms such as first and second, upper and lower may be used only to distinguish one entity or action from another entity or action, and do not necessarily require or imply any actual relationship or order between such entities or actions. The terms “comprises,” “comprising,” “has,” “having,” “includes,” “including,” “contains,” and “containing,” or any other variations thereof, are intended to cover non-exclusive inclusion, thereby allowing a process, method, article, or apparatus that comprises, has, includes, or contains a list of elements to include not only those elements but also other elements not expressly listed or specific to such process, method, article, or apparatus. An element preceded by “comprises a…”, “has a…”, “includes a…”, or “contains a…” does not, unless further restricted, preclude the presence of further identical elements in processes, methods, articles, or apparatus that comprise, have, include, or contain that element. The terms “a” and “an” are defined as one or multiple, unless expressly stated herein otherwise. The terms “substantially”, “essentially”, “approximately”, “about”, or any other variation thereof are defined as “close to”, as understood by those skilled in the art, in one non-limiting embodiment, this term is defined as being within 10%, in another embodiment within 5%, in another embodiment within 1%, and in another embodiment within 0.5%. As used herein, the term “combined” is defined as “connected,” but not necessarily direct or mechanical.A device or structure that is "configured" in a particular way is configured in at least that way, but may also be configured in ways not listed.

[0059]

[0067] In this specification, certain expressions may be used to list combinations of elements. Examples of such expressions include: “at least one of A, B, and C,” “one or more of A, B, and C,” “at least one of A, B, or C,” and “one or more of A, B, or C.” Unless otherwise specified, the above expressions encompass any combination of A and / or B and / or C.

[0060]

[0068] It will be understood that some embodiments may consist of one or more dedicated processors (or “processing devices”), such as microprocessors, digital signal processors, customized processors, and field-programmable gate arrays (FPGAs), and a set of unique stored program instructions (including both software and firmware) that control the one or more processors to perform some, most, or all of the functions of the methods and / or apparatus described herein in conjunction with specific non-processor circuits. Alternatively, some or all of the functions may be performed by a state machine that does not have stored program instructions, or in one or more application-specific integrated circuits (ASICs) in which some of the functions, or some combinations of some of those functions, are performed as custom logic. Of course, combinations of these two approaches may be used.

[0061]

[0069] Furthermore, embodiments can be implemented as computer-readable storage media storing computer-readable code for programming a computer (including, for example, a processor) to perform the methods described and claimed herein. Examples of such computer-readable storage media include, but are not limited to, hard disks, CD-ROMs, optical storage devices, magnetic storage devices, ROMs (Read Only Memory), PROMs (Programmable Read Only Memory), EPROMs (Erasable Programmable Read Only Memory), EEPROMs (Electrically Erasable Programmable Read Only Memory), and flash memory. Moreover, those skilled in the art will anticipate that, guided by the concepts and principles disclosed herein, it will be readily possible to generate such software instructions and programs and ICs with minimal experimentation, regardless of the considerable effort and numerous design choices motivated, for example, by available time, current technology, and economic considerations.

[0062]

[0070] This abstract of the disclosure is provided to enable readers to quickly grasp the characteristics of the technical disclosure. The abstract is presented with the understanding that it is not used to interpret or limit the scope or meaning of the claims. In addition, it can be understood that in the aforementioned “Modes for Carrying Out the Invention,” various features are grouped together in various embodiments for the purpose of simplifying the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the claimed embodiments require more features than are explicitly listed in each claim. Rather, as reflected in the claims below, the inventive subject matter lies in fewer features than all the features of a single disclosed embodiment. Therefore, the claims below are incorporated into the “Modes for Carrying Out the Invention,” with each claim standing independently as individually claimed subject matter.

Claims

1. The steps include: maintaining a map of the facility's occupied area, The steps include controlling the navigation sensor of a mobile device to acquire sensor data within the sensor's field of view (FOV), The steps include identifying an unobservable occupied area located outside the sensor's field of view in the map, based on the current location of the mobile device in the facility, The steps include selecting a first reference occupied region having a first reference identifier and a second reference occupied region having a second reference identifier from the aforementioned map, The steps include generating a first connection score that associates the unobservable occupied region with the first reference occupied region, and a second connection score that associates the unobservable occupied region with the second reference occupied region, The steps include selecting a processing operation for the unobservable occupied area based on the first and second connection scores, The steps include updating the map according to the selected processing operation, The steps include controlling the mobile assembly of the mobile device according to the updated map, and A method that includes this.

2. The steps include processing the acquired sensor data to detect the observed occupied area located within the sensor's field of view, The steps include updating the map according to the observed occupied area and It further includes, The method according to claim 1, wherein the first reference occupied region includes the observed occupied region.

3. The aforementioned map includes a statically occupied region, The method according to claim 2, wherein the second reference occupied region includes the static occupied region.

4. The method according to claim 2, wherein the first and second reference identifiers are respective obstacle identifiers.

5. The method according to claim 2, wherein the first and second reference identifiers are respective obstacle categories.

6. The method according to claim 5, further comprising the step of assigning an obstacle category to the observed occupied area based on a comparison of the observed occupied area with a predetermined set of static obstacles, in response to the detection of the observed occupied area.

7. The step of generating the first and second connection scores is at least (i) the degree of proximity between the reference occupied areas, and (ii) The existence of an unoccupied region separating the reference occupied region from the unobservable occupied region. The method according to claim 1, based on the present invention.

8. The method according to claim 1, wherein the step of selecting the first and second reference occupied regions includes the step of selecting occupied regions in the map that are within a threshold distance from the unobservable occupied regions.

9. The step of selecting the processing operation for the unobservable occupied area is: The steps include determining that neither of the first and second connection scores exceeds a threshold, A step of clearing the unobservable occupied area in the aforementioned map. The method according to claim 1, including the method described in claim 1.

10. The step of selecting the processing operation for the unobservable occupied area is: The steps include selecting the larger of the first and second connection scores, The steps include associating the corresponding reference identifier with the unobservable occupied region. The method according to claim 1, including the method described in claim 1.

11. The step of selecting the processing operation for the unobservable occupied area is: The steps include comparing the corresponding reference identifier with a previous reference identifier associated with the unobservable occupied area, If the corresponding reference identifier is different from the previous reference identifier, the steps include clearing the unobservable occupied area in the map. The method according to claim 10, including the method described in claim 10.

12. Mobile devices A chassis base that supports the mobile assembly and the sensor, A rack extending from the chassis and supporting one or more containers The method according to claim 1, including the method described in claim 1.

13. A computing device, A sensor having a sensor field of view (FOV), Equipped with a processor, The aforementioned processor, The facility maintains a map of its occupied area. The sensor is controlled to acquire sensor data within the sensor's field of view (FOV). Based on the current location of the mobile device in the facility, the unobservable occupied area located outside the sensor's field of view is identified in the map. From the aforementioned map, select a first reference occupied region having a first reference identifier and a second reference occupied region having a second reference identifier. A first connection score is generated that associates the unobservable occupied region with the first reference occupied region, and a second connection score is generated that associates the unobservable occupied region with the second reference occupied region. Based on the first and second connection scores, a processing operation is selected for the unobservable occupied area. The map is updated according to the selected processing operation. Control the mobile assembly of the mobile device according to the updated map. A computing device configured in such a way.

14. The aforementioned processor, The acquired sensor data is processed to detect the observed occupied area located within the sensor's field of view. The map is updated according to the observed occupied area. It is further structured in the following way: The computing device according to claim 13, wherein the first reference occupied area includes the observed occupied area.

15. The aforementioned map includes a statically occupied region, The computing device according to claim 13, wherein the second reference occupied area includes the static occupied area.

16. The computing device according to claim 13, wherein the first and second reference identifiers are obstacle identifiers, respectively.

17. The computing device according to claim 13, wherein the first and second reference identifiers are respective obstacle categories.

18. The aforementioned processor, In response to the detection of the observed occupied area, an obstacle category is assigned to the observed occupied area based on a comparison of the observed occupied area with a predetermined set of static obstacles. The computing device according to claim 17, further configured as follows.

19. The aforementioned processor, at (i) the degree of proximity between the reference occupied areas, and (ii) The existence of an unoccupied region separating the reference occupied region from the unobservable occupied region. The computing device according to claim 13, configured to generate the first and second connection scores based on the above.

20. The computing device according to claim 13, wherein the processor is configured to select the first and second reference occupied regions by selecting an occupied region in the map that is within a threshold distance from the unobservable occupied region.

21. The aforementioned processor, It is determined that neither of the first and second connection scores exceeds the threshold, To clear the unobservable occupied area in the aforementioned map and The computing device according to claim 13, configured to select the processing operation relating to the unobservable occupied area by performing the above.

22. The aforementioned processor, Selecting the larger of the first and second connection scores, Associating the corresponding reference identifier with the unobservable occupied area The computing device according to claim 13, configured to select the processing operation relating to the unobservable occupied area by performing the above.

23. The aforementioned processor, This involves comparing the corresponding reference identifier with a previous reference identifier associated with the unobservable occupied area, If the corresponding reference identifier is different from the previous reference identifier, the unobservable occupied area in the map is cleared. The computing device according to claim 21, configured to select the processing operation relating to the unobservable occupied area by performing the above.

24. Mobile devices A chassis base that supports the mobile assembly and the sensor, A rack extending from the chassis and supporting one or more containers The computing device according to claim 13, including the following:

Images