Generating mappings of physical space from point cloud data

Abstract

The invention relates to generating a mapping of a physical space from point cloud data. In particular, this specification describes systems and methods for generating a mapping of a physical space from point cloud data of the physical space. The method may include: receiving the point cloud data of the physical space; filtering the point cloud data to at least remove sparse points from the point cloud data; aligning the point cloud data along x, y, and z dimensions corresponding to an orientation of the physical space; and classifying the points in the point cloud data as corresponding to one or more types of physical surfaces. The method may also include identifying a particular physical structure in the physical space based at least in part on a classification of the points in the point cloud data; and generating the mapping of the physical space to identify the particular physical structure and a corresponding profile of the particular physical structure within the orientation of the physical space.

Description

[0001] Information about divisional applications

[0002] This case is a divisional application of the invention patent application with the application date of April 13, 2022, application number 202280037806.0, and invention name "Generating a mapping of physical space from point cloud data". Technical Field

[0003] This document generally describes the use of point clouds to generate maps of physical spaces, such as to generate blueprints. Background Art

[0004] Physical spaces, such as warehouses and storage facilities, can be large and navigating therein can be challenging. Mapping such physical spaces can assist users, such as warehouse workers, in understanding the floor plan of the physical space and moving around the physical space without getting lost. Generating a blueprint or map of the physical space can be performed by a user walking around the physical space with a scanner to obtain a 3D scan and / or image of the entire physical space. The 3D scan and / or image can then be manually transformed into a blueprint or map of the physical space. Whenever changes are made to the physical space, the user can manually update such a blueprint or map.

[0005] Manual mapping can be a time-consuming process and may contain inaccuracies due to human error. For example, a user may not be able to obtain images of every area of ​​a physical space. Thus, a map may include certain areas of the physical space but not other areas. As another example, a user may not be able to obtain updated images of areas of the physical space in which changes are made. Thus, maps may be outdated, and users may rely on outdated maps. Current blueprints and maps of physical spaces may be inaccurate, lack details, or lack all details. In addition, due to the amount of time and manpower required to generate blueprints and maps, such blueprints and maps of physical spaces may not be easily updated to reflect daily changes in the actual physical space. Summary of the invention

[0006] This document generally describes computer-based techniques and methods for generating a map (e.g., a blueprint) of a physical space using point cloud data. For example, a physical space map may include a series of nodes and vertices in 3D space that define the outlines of physical objects, such as buildings, terrain features, shelves, pallets, vehicles, and other physical objects. As mentioned above, physical maps are traditionally generated using scanning techniques, such as LIDAR and other scanning techniques, which transmit signals to physical objects and record the signals when they are reflected by these objects to determine their outlines. Scanning techniques may be time-consuming and expensive to implement, which may hinder their use. On the other hand, image analysis (e.g., stereo image analysis) can be used to generate point clouds, which can be generated using cameras. Cameras can be more common and cheaper than scanning techniques (e.g., LIDAR). Point cloud data may include, for example, points in 3D space that correspond to features of physical objects, such as edges, surfaces, corners, and / or other physical features. Although point clouds may be noisy, filtering techniques can be applied to point clouds to more accurately discern the outlines of physical objects.

[0007] The disclosed techniques can take advantage of using point cloud data to determine mappings of physical objects and / or spaces through various techniques described throughout this document. Advantages of point cloud data include that implementations can be less expensive than traditional mapping techniques described above, point cloud data can be dynamically generated, and mappings formed from point cloud data can be dynamically updated. Furthermore, the disclosed techniques can understand different point cloud data, such that physical objects and their outlines can be easily identified from only a series of points.

[0008] The disclosed technology can provide for receiving 3D scans and / or images from one or more different types of devices in a physical space. In some embodiments, the physical space can be a warehouse. The physical space can also be any type of storage facility, building, house, other structure, or outdoor environment, such as a part of a town and / or some other environment that can be surveyed. Using the 3D scans and / or images, the disclosed technology can use a point cloud to generate an accurate representation of the physical space, including (but not limited to) the footprint (e.g., blueprints, maps, floor plans, etc.), cubic feet, wall locations, bracket locations, aisles, corridors, etc. of the physical space. Point cloud data can also be used to determine the coordinates and orientation of items within the physical space, such as bracket locations and bracket shelves in a warehouse environment. In addition, the disclosed technology can provide real-time, periodic and / or automatic dynamic updates or otherwise updates (renew) blueprints and mappings of physical spaces.

[0009] Certain embodiments described herein include systems and methods for generating a mapping of a physical space from point cloud data of the physical space. The embodiments may include: receiving, by a computing system, the point cloud data of the physical space, the point cloud data including a plurality of points in a three-dimensional space that approximate locations of physical surfaces within the physical space; and filtering the point cloud data to remove at least sparse points from the point cloud data. The embodiments may include: aligning the point cloud data along x, y, and z dimensions corresponding to an orientation of the physical space; classifying the points in the point cloud data as corresponding to one or more types of physical surfaces; identifying a particular physical structure in the physical space based at least in part on the classification of the points in the point cloud data; and generating the mapping of the physical space to identify the particular physical structure and a corresponding contour of the particular physical structure within the orientation of the physical space.

[0010] In some implementations, the embodiment may optionally include one or more of the following features. For example, classifying points in the point cloud data may include: selecting a reference point from the point cloud data; identifying k nearest neighbor points for the reference point; calculating spherical coordinates relative to the reference point for each of the k nearest neighbor points; determining a spherical feature of the reference point based on the spherical coordinates of each of the k nearest neighbor points; and classifying the reference point based on determining the spherical feature of the reference point. The reference point may be classified as belonging to at least one of a floor, a wall, a vertical rod, a support beam, a pallet, and noise. In addition, classifying the points in the point cloud data may further include outputting at least one of (i) a classification of each of the points in the point cloud data, (ii) a spherical feature of each of the points in the point cloud data, and (iii) an object represented by the point cloud data based on the classification of each of the points in the point cloud data.

[0011] As another example, aligning the point cloud data may include identifying one or more reference points around at least one of a door or a window in the physical space. The one or more reference points indicate an axis around which to rotate the point cloud data. Aligning the point cloud data may also include detecting a bounding box around the physical space. The bounding box may indicate an axis around which to rotate the point cloud data.

[0012] In some implementations, the embodiment may further include: detecting vertical poles from the point cloud data; detecting bracket sections from the point cloud data and based on the detected vertical poles; determining a pallet footprint based on the detected bracket sections; determining a height of each shelf section in the detected bracket sections; and identifying a pallet location of each shelf section of the detected bracket sections based on the pallet footprint and the height of each shelf section. In addition, detecting vertical poles may include: classifying points in the point cloud data associated with vertical poles; locating the classified points; and straightening the located points into vertical poles. In addition, detecting bracket sections may include: clustering the vertical poles into bracket groups; classifying the clustered vertical poles by bracket type and bracket orientation; interpolating missing bracket poles for each of the clustered vertical poles based on classifying the clustered vertical poles by bracket type and bracket orientation; and detecting bracket sections based on the clustered vertical poles.

[0013] In some embodiments, classifying the points in the point cloud data associated with the vertical pole may include: voxelizing the point cloud data into a first predetermined grid size; for each voxel, clustering the points of the point cloud data in the voxel; for each cluster of points in the voxel, voxelizing the cluster of points into a second predetermined grid size; for each point in each voxelized cluster of points, classifying the point as a vertical pole point; for each cluster of points in the voxel, normalizing the classification of each point; and determining whether each cluster of points in the voxel is associated with the vertical pole based on the normalization of the classification of each point.

[0014] For each point in each voxelized point cluster, classifying the point as a vertical rod point may also include: applying a covariance matrix to the voxelized point cluster; determining a standard deviation value for each point in the voxelized point cluster; identifying a high standard deviation in the Z direction; identifying a low standard deviation in the X and Y directions; and for each point in the voxelized point cluster and based on the identified high standard deviation and the identified low standard deviation, assigning a score to a one-dimensional or two-dimensional point extension of the point.

[0015] In some implementations, for each point in each voxelized point cluster, classifying the point as a vertical rod point may include applying a spherical covariance matrix to the voxelized point cluster. Classifying the point as a vertical rod point may also include applying a histogram filter to the voxelized point cluster. Classifying the point as a vertical rod point may also include applying a neural network to the voxelized point cluster.

[0016] As another example, clustering the vertical rods into bracket groups may include: receiving bracket information; and for each cluster, determining size information of the cluster. The size information may include a bounding box, a minimum X, Y, and Z coordinate of the bounding box, and a maximum X, Y, and Z coordinate of the bounding box. Clustering the vertical rods may include: determining whether the size information of the cluster is consistent with bracket information; based on determining that the size information of the cluster is inconsistent with the bracket information, returning points in the cluster to a point pool in the point cloud data; based on determining that the size information of the cluster is consistent with the bracket information, identifying orientation information of the cluster; and generating a bracket list containing the clusters, the size information of the clusters, and the orientation information of the clusters.

[0017] In some embodiments, clustering the vertical rods may also include: selecting a bracket from the bracket list, determining the distance between each of the vertical rods in the selected bracket; determining the orientation information of each of the vertical rods in the selected bracket; and determining the bracket type of the selected bracket based on the distance and the orientation information of each of the vertical rods.

[0018] As another example, determining a pallet footprint based on the detected rack segments may also include: determining, for each of the detected rack segments, whether the detected rack segment has a selected rack type; based on determining that the detected rack segment has the selected rack type, calculating two pallet footprint centers in the detected rack segment; and based on determining that the detected rack segment does not have the selected rack type, calculating a pallet footprint center in the detected rack segment.

[0019] In some embodiments, determining the height of each shelf segment in the detected support segments may include: for each of the detected support segments, filtering out vertical objects from the detected support segments; detecting a horizontal plane in the filtered support segments; identifying whether the horizontal plane has characteristics of a shelf segment; determining a high shelf score based on determining that the horizontal plane has the characteristics of the shelf segment; and determining a low shelf score relative to the high shelf score based on determining that the horizontal plane does not have the characteristics of the shelf segment, wherein the low shelf score is closer to 0 and the high shelf score is closer to 1.

[0020] The techniques and methods described throughout this disclosure provide many advantages. For example, the disclosed techniques can automate the process of generating blueprints and maps of physical spaces. Automating such processes can be used to reduce the time and amount of resources required to map the physical space. Because cameras can be positioned throughout the physical space to capture images of the physical space, users (e.g., warehouse workers) may not need to walk around the entire physical area and capture images of the physical area using traditional scanning techniques and technologies. Instead, the user's time can be allocated to performing other tasks in the warehouse or other physical space, thereby improving the user's efficiency.

[0021] As another example, the map of the physical space may be more accurate. Since the user does not manually map the physical space, human error may be removed or otherwise mitigated using the described techniques. Thus, the disclosed techniques may improve the accuracy of blueprints and maps of the physical space. The disclosed techniques may also allow for continuous updating and / or updating of blueprints and maps without human intervention.

[0022] As yet another example, the disclosed technology can be used to more accurately determine process flows, movements, and / or other characteristics specific to the physical space when the physical space is occupied and / or used for its intended purpose. For example, in a warehouse environment, point cloud data can be used to identify exact rack segments and shelf locations. These identified locations can then be used to determine the placement of pallets in the warehouse. As a result, pallets can be efficiently collected, stored, and moved within the space. This can improve and optimize warehouse efficiency. Similarly, the disclosed technology can be used to improve forklift tracking in a warehouse. Forklifts can be tracked in real time. In addition, forklifts can dynamically interact with blueprints and maps generated by point clouds to improve navigation, reduce bottlenecks and traffic, and improve overall warehouse efficiency.

[0023] The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1A Depicted is an exemplary mapping system using point clouds.

[0025] Figure 1B Depicts an exemplary system diagram of the disclosed invention.

[0026] Figure 2 is a flow chart of the process used to process a point cloud.

[0027] Figure 3 is a flow chart of the process used to filter a point cloud.

[0028] Figure 4A 1 to 2 depict exemplary results of filtering the point cloud.

[0029] Figure 5A 1 to 2 depict exemplary results of filtering the point cloud.

[0030] Fig. 6A is a flow chart of the process used for the spherical feature algorithm.

[0031] Figure 6B 1 to C are graphical depictions of the spherical feature algorithm during implementation.

[0032] Fig. 7A 1 to 2 are exemplary uses of the disclosed system in a warehouse environment.

[0033] Figure 8 is a flow chart of the process used to classify vertical pole points.

[0034] Fig. 9 is a flow chart of the process used to position the vertical rod.

[0035] Fig.10 is a flow chart of a process for cleaning vertical rod positioning results.

[0036] Fig.11 is a flow chart of the process used to cluster vertical rods into bracket groupings.

[0037] Fig.12 is a flow chart of the process used to classify bracket rods.

[0038] Fig.13 is a flow chart of a process for interpolating missing stent rods.

[0039] Fig.14A 1 to B are flow charts of a process for detecting stent segments.

[0040] Fig.15 is a flow chart of the process used to calculate the location of a pallet footprint.

[0041] Fig.16 is a flow chart of a process for positioning a rack shelf for a rack section.

[0042] Fig.17 is a flow chart of a process for detecting rack sections and pallet levels. DETAILED DESCRIPTION

[0043] This document generally describes the use of point clouds to generate blueprints and / or maps of physical spaces.

[0044] Figure 1AAn example mapping system is depicted that includes a physical space 100, a point cloud 102, a computer system 104, and a map 106 of the physical space. As depicted, the physical space 100 may include one or more supports 108A-N. In other embodiments, the physical space 100 may include one or more other types of physical items in addition to the supports 108A-N. For example, the physical space 100 may include walls, doorways, doors, tables, support beams, etc. In some embodiments, the physical space 100 may be a warehouse environment. In other embodiments, the physical space 100 may be any type of storage facility (e.g., a freezer). In still other embodiments, the physical space 100 may be a building, a house, other structure, or an outdoor space / landscape. Still referring to Figure 1A , the physical space 100 further includes a drone 110 and a stereo camera 112. In some implementations, the physical space 100 may include a plurality of drones 110 and / or stereo cameras 112. The drones 110 and stereo cameras 112 may be configured to capture one or more 3D scans and / or images of the entire physical space 100. In some implementations, instead of at least one of the drones 110 and / or stereo cameras 112, the physical space 100 may employ a person, a forklift, and / or some type of device to capture a 3D scan and / or image of the physical space 100.

[0045] Once one or more 3D scans and / or images of the physical space 100 are captured, they may be used to generate one or more point clouds 102. The one or more point clouds 102 may be generated by the drone 110, a device in the physical space 100 used to capture 3D scans / images, and / or any other computer system / computing device in communication with the drone 110, the stereo camera 112, and / or the computer system 104. The one or more point clouds 102 may then be transmitted / communicated to the computer system 104. The communications described throughout this disclosure may occur via a network and / or wireless / wired communications (e.g., BLUETOOTH, WIFI, Ethernet, etc.).

[0046] At computer system 104, point cloud 102 may be processed (e.g., filtered, cleaned) before being used to generate map 106 of physical space. Map 106 of physical space may be generated by computer system 104 and / or another computer system / computing device in communication with computer system 104. Map 106 of physical space may be an accurate map and / or blueprint that is updated in real time, periodically, and / or automatically. For example, as depicted, map 106 of physical space includes replicas of supports 108A to N, which are mapped supports 114A to N. Map 106 of physical space further includes a replica of stereo camera 112, which is mapped camera 116. In other embodiments, map 106 of physical space may include replicas of any other physical items and / or permanent structures within physical space 100.

[0047] Still refer to Figure 1A , the described processes may be performed in real time and / or at different times. For example, a 3D scan and / or image of the physical space 100 may be captured at time 1. A point cloud 102 may be generated at time 2. Processing of the point cloud at a computer system 104 may be performed at time 3. And finally, generating a map 106 of the physical space may be performed at time 4. Furthermore, the described processes may be performed more than once, and may be performed automatically / periodically to ensure that an updated map and / or blueprint of the physical space 100 is maintained.

[0048] Figure 1B As depicted, the 3D scanning device 118 and the point cloud mapping system 120 can communicate via a network 122, as previously described in Figure 1A One or more 3D scanning devices may communicate with the point cloud mapping system 120. The 3D scanning device 118 may be the previously referenced Figure 1A At least one of the drone 110 and the stereo camera 112 discussed. In addition, the point cloud mapping system 120 may be Figure 1A The computer system 104 disclosed in and throughout this disclosure.

[0049] The 3D scanning device 118 may include a communication interface 124, at least one stereo camera 126, and an optional point cloud generator 128. The communication interface 124 may facilitate communication between the 3D scanning device 118 and the point cloud mapping system 120 via a network 122, as discussed. The at least one stereo camera 126 may capture one or more 3D scans and / or other images of a physical space. In some implementations, the 3D scanning device 118 may have other types of image capture sensors, cameras, and / or devices configured to capture 3D scans and / or images of a physical space in place of or in combination with the stereo camera 126. As mentioned, the 3D scanning device 118 may optionally include a point cloud generator 128, which is configured to generate a point cloud from the captured 3D scan / image. If the 3D scanning device 118 includes a point cloud generator 128 and generates a point cloud, these point clouds 130 may be transmitted / communicated to the point cloud mapping system 120 via the network 122. The point cloud mapping system 120 may then store such point clouds 130 in a point cloud database 152. The point cloud database 152 stores the generated point clouds, as indicated by exemplary point cloud 156. If the 3D scanning device 118 does not generate a point cloud, the 3D scan 132 may be transmitted / communicated from the device 118 to the point cloud mapping system 120 via the network 122.

[0050] The point cloud mapping system 120 may use the received 3D scans to generate one or more point clouds via a point cloud generator 148. Thus, the system 120 may optionally include a point cloud generator 148. The point cloud mapping system 120 may include a communication interface 134, a point cloud refinement module 136, a physical structure classification module 142, and a confidence score module 146. The system 120 may further and optionally include a bracket detection module 150. The system 120 may communicate with the 3D scanning device 118 and a point cloud database 152 and a generated map database 154 via the communication interface 134 and through the network 122.

[0051] Once the system 120 generates a point cloud (via the point cloud generator 148) or receives the point cloud 130 from the 3D scanning device 118, the point cloud may be refined at the point cloud refinement module 136. The point cloud refinement module 136 may optionally include a histogram filter 138 and / or a k-guide filter 140, both of which may be used alone and / or in combination to purify and clean the point cloud. Purifying and cleaning the point cloud involves more clearly defining physical structures by upsampling and / or downsampling points in the point cloud. Purifying and cleaning the point cloud involves making physical structures more clearly defined by upsampling and / or downsampling points in the point cloud. Once refined, the point cloud may be passed through a physical structure classification module 142 to identify and classify any physical structures / items / objects in the point cloud. The module 142 may include a spherical filter 144 for identifying and classifying physical structures. A confidence score module 146 may further be used to determine the score of the identified physical structure and the overall accuracy of the refined point cloud. Finally, the bracket detection module 150 can be used to identify vertical poles and brackets from the point cloud. The module 150 is useful for determining brackets in a warehouse environment or other type of storage facility. Each of the modules and / or filters comprising the point cloud mapping system 120 is discussed in further detail below.

[0052] Once any determinations are made (e.g., refined point clouds, final maps, classifications and scores of physical structures, etc.), such determinations may be stored in a generated map database 154. The database 154 may store a plurality of generated maps 158. The generated maps 158 may include a refined point cloud 160, at least one identified and / or classified physical structure information 162, and at least one confidence score 164 associated with the accuracy of the refined point cloud 160 and / or at least one physical structure 162. The generated maps 158 may further and optionally include at least one vertical pole information 166 and at least one bracket information 168.

[0053] Figure 22 is a flow chart of a process 200 for processing a point cloud. The process 200 may be performed by any computing device, such as the computer system 104 (see FIG. 1 ). In an exemplary warehouse environment, a point cloud may be used to map the warehouse environment. Therefore, identifying objects in the warehouse, such as vertical poles for racks, helps determine the layout of the warehouse. First, in step 202, the computer system receives a point cloud. As previously mentioned, the computer system may receive a point cloud from a drone that performs a 3D scan of a physical environment. In a warehouse, a new point cloud scan includes at least one of a wall, floor, ceiling, rack, pallet, door, lamp, vertical support pole, person, forklift, noise, random variation, occlusion, etc. For example, in an outdoor space, a new point cloud scan may include at least one of a tree, bush, building, residence, parking lot, person, highway, overpass, bridge, lake, river, etc. The new point cloud scan is randomly aligned in space, and the floor of the physical environment is typically nearly parallel to the XY plane. Typically, a point cloud scan includes noise, which may include several points of physical objects from other nearby locations and / or rooms. This noise therefore increases the bounding box of the physical environment beyond walls or other existing perimeters.

[0054] Next, the computer system refines the point cloud (step 204). The computer system filters and cleans the point cloud data in step 206 (e.g., by upsampling and / or downsampling the points in the point cloud data), then aligns the point cloud data in step 208, and finally classifies the point cloud in step 210. In step 206, the computer system uses several filters, such as guide filters, which can reduce the variation in point positions. Guide filters can make surfaces cleaner, make edges sharper, and remove abnormal noise. Using this type of filter as a first step can improve the accuracy of all point cloud processes discussed below. Exemplary filters used by the computer system include histogram filters and k-guide filters. Alternatively, if guide filters are not used in the step (206) of filtering and cleaning the point cloud data, then anomaly filters can be used to remove some noise in the point cloud scan. For each noise detection, the confidence score / value associated with the identified physical object is reduced accordingly.

[0055] In step 208, the point cloud data may be aligned with the X, Y, and Z axes. In other words, the point cloud will be rotated and oriented in the correct direction. For example, a warehouse room scan may be received in an arbitrary space (step 202), not aligned with any of the axes. Then, the scan may be rotated and aligned with the appropriate space of the warehouse. In some instances, the point cloud may be aligned with the width of the wall in the warehouse to determine the proper orientation of the point cloud. The computer system may look for reference points around doorways, door frames, windows, and / or other objects that may appear in the point cloud to properly orient the point cloud. In yet other instances, the computer system may detect a bounding box of the walls of a physical space (e.g., a warehouse room) that surrounds the physical space, and then rotate the physical space so that the bounding box is aligned with the X, Y, and Z axes. A tightly fitting bounding box may also be used to more easily identify noise points outside the walls of the physical space. Performing this alignment step is important because it reduces the amount of time required to perform certain calculations later in the point cloud processing.

[0056] Once the point cloud data has been aligned and initially cleaned, object detection and localization can begin (step 210). In this step, a spherical filter can be used to classify the point cloud data (see below). Figures 6A to 6C In this step, physical objects, perimeters, and other items that appear in the point cloud can be identified, classified, and scored. In some embodiments, some of the identified, classified, and scored items can be removed from the point cloud. For example, floors and ceilings can be eliminated. Points that are unlikely to be vertical poles in a warehouse environment can also be eliminated.

[0057] Based on the purpose and use of the point cloud, different physical objects can be identified for elimination and / or retention. For example, in a land survey, the computer system can be configured to identify and eliminate trees and shrubs. The computer system can be further configured to identify buildings and parking lots and keep them in the point cloud. Typically, the easiest thing to detect in a point cloud scan is a floor or other horizontal plane. For example, in a warehouse environment where rack detection may be most important, points that may be part of the floor can be identified and removed from the point cloud. In most embodiments, ceilings are also relatively easy to detect, as are light fixtures, light beams, evaporation coils, and other objects that may be suspended from the ceiling. Wall detection can be more challenging, but using a classifier, the confidence scores for vertical poles and other physical items can be adjusted accordingly so that walls, ceilings, etc. are removed from the point cloud.

[0058] Once the point cloud refinement is complete (204), the computer system can identify specific physical structures in the remaining point cloud data (step 212). For example, vertical poles, racks, and / or aisles in a warehouse can be identified and assigned confidence scores. In a land survey example, the system can identify residences, buildings, parking lots, and / or roads. Thus, such identification can be used to generate an accurate blueprint and / or map of the associated physical space.

[0059] Figure 3 300 is a flow chart of a process 300 for filtering a point cloud. The process 300 may be performed by a computer system similar to the system 104 (see FIG. 1 ) or any other computing device. First, the computer system may determine the density of points in the point cloud (step 302). The computer system may remove sparse points from the point cloud (step 304). Then, in step 306, the computer system may upsample and / or downsample the remaining points in the point cloud. Upsampling may require the computer system to add points in the location of the point cloud where the physical structure is located when the points are sparse. Downsampling may require the computer system to remove points in the location of the point cloud when there are too many points. In some embodiments, the computer system may upsample and downsample the same point cloud in different locations. In other embodiments, the computer system may only upsample or downsample. In order to reduce noise and draw out physical structures in the point cloud, the process 300 is advantageous. Once the point cloud is cleaned and the noise is removed, the point cloud data can be more efficiently used in point cloud machine learning and the techniques described throughout the present disclosure.

[0060] Figure 4A B depicts an exemplary result of filtering the point cloud (refer to Figure 3 ). Figure 4A Depicting a 3D scan of the outdoor physical space before detailing. Figure 4A The 3D scan of FIG. 1 is a street with buildings, parking lots, houses, etc. As shown, the scan is not refined or detailed enough to identify what each of the objects and / or shapes in the scan are. Figure 4B Depicts the same 3D scan after being refined with filtering techniques such as removing sparse points, upsampling, downsampling, k-guide filtering, and histogram filtering. As a result of such filtering / refining, each of the objects / shapes in the 3D scan may be more easily identified.

[0061] The different colors in the scan indicate the confidence score for each of the identified objects / shapes. Figure 4A The purpose of the 3D scan into B may be to identify buildings and houses for a land survey project. When an object / shape in this scan is red, it is more likely to be one of the things identified (e.g., 90% accurate identification after filtering). Figure 4BIn the example of , red squares and rectangles represent buildings and homes. Objects, shapes, and / or spaces shown in green are less likely to be one of the things identified (e.g., 80% accurate after filtering). And, objects, shapes, and / or spaces shown in blue are even less likely to be one of the things identified (e.g., 50% accurate). Therefore, streets, parking lots, and other open spaces are Figure 4B The images are shown in blue because they are least likely to represent buildings and residences.

[0062] By using the 3D scanning and filtering techniques discussed throughout this disclosure in different settings, it is possible to obtain Figure 4A To similar results shown in B. For example, a 3D scan of a room in a building, warehouse, and / or any other physical environment may be captured to achieve comparable results.

[0063] Figure 5A B depicts exemplary results of thinning and / or filtering the point cloud (refer to Figure 3 ). Figure 5A Depicting a 3D scan of the outdoor physical space before detailing. Figure 4A The 3D scan of the point cloud is a curved street with buildings, parking lots, trees, houses, etc. As shown, the scan is not refined or detailed enough to identify what each of the objects and / or shapes in the scan is. In other words, the point cloud has sparse points, which results in a lower density of points in the point cloud. Because of the lower density, points can be added to areas of the point cloud to increase the density, thereby mapping out physical structures / objects from the original point cloud. Therefore, Figure 5B Depicted in use Figure 3 The same 3D scan after being refined using the techniques described in . Figure 5B In order to increase the point density, points are added. Upsampling, downsampling, k-guided filtering and / or histogram filtering techniques may also be used to further refine the point cloud, reduce noise and delineate physical structures / objects.

[0064] As about Figure 4A As described in B, the different colors in the scan indicate the confidence score for each of the identified objects / shapes. Figure 5A In B, the greener objects / shapes are more likely to be defined / identified objects, such as trees, streets, parking lots, and / or buildings. These objects are more accurately identified after filtering, such as Figure 5B The bluer objects / shapes are less likely to be the things being defined / identified. These objects will be less accurately identified after filtering, such as Figure 5B By using the 3D scanning and filtering techniques discussed throughout this disclosure, Figure 4A Results were similar to those shown in 5A to B and 5A to B.

[0065] Fig. 6A is a flow chart of a process 600 for a spherical feature algorithm. Figures 6B to 6C is a graphical depiction of a spherical feature algorithm during implementation. Process 600 may be performed by any computing device, such as computer system 104 (see FIG. 1 ). Process 600 facilitates classification of objects in physical space / point cloud. This process 600 may be performed after thinning and / or filtering. In other implementations, process 600 may be performed without thinning and / or filtering, and / or may be performed while thinning and / or filtering.

[0066] The process 600 begins by selecting an origin (e.g., a reference point, a point of reference) from a point cloud at step 602. The selected point becomes the focus of reference / orientation and the center or origin of the spherical coordinate system. Therefore, all of its neighboring points are viewed relative to (or from the perspective of) the selected point. At 604, the nearest neighbors of the origin may be located by some known techniques.

[0067] In 606, a nearest neighbor point of the reference point may be selected. Once the neighbor point is selected, the computer system may identify the elevation, radius, and azimuth coordinates of the selected nearest neighbor using the reference point as the origin (step 608). The radius may be the distance from the neighbor point to the reference point. The elevation indicates the inclination of the neighbor point relative to the reference point, measured from -90° to 90°. The azimuth is a horizontal angle, ranging from 0° to 360°, measured from the direction of the reference point to a point on a horizontal line that intersects the neighbor's latitude line. These are the spherical coordinates of the neighbor point relative to the reference point. The spherical coordinates of all neighbor points may be calculated in order to calculate the spherical features of the central origin.

[0068] In step 610, the computer system can determine whether there are more neighbors of the origin. If there are more neighbors, the computer system can return to step 606 and repeat the previously discussed steps for the next neighbor. If there are no more neighbors for the current origin, the spherical features of the origin can be calculated in step 612.

[0069] In step 612, the computer system can calculate the spherical features of the origin. For example, the computer system can use known techniques to identify the minimum, maximum, mean, and standard deviation of the elevation, radius, and azimuth coordinates; these 12 numbers can constitute the spherical features of the currently selected origin (e.g., the reference point). These features contain information about the local neighborhood points around a given reference point and can be used to determine what type of object the point belongs to or is part of. These features can be stored for each point in the point cloud and used in subsequent calculations.

[0070] In step 614, the computer system may use the spherical features of the most recently determined origin point to classify it as belonging to one type of object or another. For example, a point may belong to a floor, a wall, a vertical pole, a support beam, a pallet, noise, etc. The classification of the reference point may be determined based on the values ​​of the spherical coordinates and the X, Y, and Z coordinates of the point itself. Different types of objects may contain points with different distributions of values ​​for these 15 point features. For example, a floor point may have any X and Y values, but its Z value may fall within a small range and may be centered at the height of the floor, which may be near 0. The elevation parameter of the floor point may also have a small standard deviation, and the minimum, maximum, and average values ​​of the elevation parameter may all be close to 0. The minimum azimuth of the floor point may be near 0, while the maximum azimuth may be near 360, and the average azimuth is near 180. The standard deviation of the azimuth parameter may be large because the points are scattered around the reference point in the horizontal direction.

[0071] The points that make up the vertical support pole can have different characteristic values ​​for 15 features. The X, Y, and Z coordinate values ​​of the points can be any numbers, and the minimum, maximum, and average values ​​of the azimuth angle can also be any numbers, but the standard deviation of the azimuth angle can be small because the poles are not spread out horizontally. In addition, due to the large vertical dispersion of the pole points, the standard deviation of the elevation parameter can be large. The minimum, maximum, and average values ​​of the radius feature can contain information about how dense or sparse the point cloud is around a given reference point. This can indicate whether the point is noise or whether the point belongs to an object. A low standard deviation of the radius can indicate that the points are concentrated in a spherical shape or ring, while a large radial standard deviation can indicate that the points are scattered across the entire space from the reference center point. A heuristic algorithm can be created to check the value of each of these parameters in order to classify each specific type of object expected (e.g., floor, wall, door, support pole, etc.). Machine learning algorithms can also be used to classify based on feature values.

[0072] After classifying the reference point in step 614, the computer system may determine whether there are more points in the point cloud in step 616. If so, the computer system may return to step 602 and repeat the previously disclosed steps for the next point in the point cloud.

[0073] Finally, once the computer system determines that there are no more points to analyze in the point cloud, the computer system may return a list of point classifications in step 618. The list may include, for example, the final classifications for each point in the point cloud, with such classifications being modified / adjusted throughout process 600 (e.g., when the computer system returns to step 602, selects a new reference point, and repeats the steps discussed previously), as well as the spherical features themselves (e.g., the minimum, maximum, average, and standard deviation of the spherical coordinates of each neighbor point). In some implementations, the list may include, instead of or in conjunction with a classified list of several types of items / objects and / or planes that exist in the point cloud based on the point classifications for each point in the point cloud. Additionally, a user at a computing device in communication with the computer system may determine what information should be returned in the list.

[0074] Figure 6B 600 during an implementation. Specifically, Figure 6B The selected reference point 628 and its nearest neighbors 626A through N are depicted as spherical coordinates. Figure 6B Further included is an azimuth 622, an elevation 620, and a radius 624. As discussed above, the azimuth, elevation, and radius values ​​for each of the nearest neighbors 626A-N may be identified. The standard deviation of the radius values ​​for reference point 628 may be small because these points are generally close to each other. Therefore, reference point 628 may not be an outlier. Further, there is a high standard deviation of elevations between the nearest neighbors 626A-N, which indicates that reference point 628 may be part of a particular object / item (e.g., a vertical plane).

[0075] Figure 6C The selected reference point 638 and its nearest neighbors 636A through N are depicted as spherical coordinates. Figure 6C Further includes azimuth 632, elevation 630, and radius 634. In this example, the standard deviation of the radius of nearest neighbors 636A to N is higher. Therefore, it is more likely that there are outliers and / or points that do not include items / objects in the physical space that reference reference point 638. As previously described in Fig. 6A As discussed in , analysis of the standard deviation of all points in the point cloud in which different reference points are selected can indicate whether points 636A to N are actually part of an item / object in physical space. Figure 6C The exemplary depiction in confirms that reference point 638 may be an outlier. The standard deviation of elevation is also small, indicating that the points are spread out horizontally rather than vertically. Therefore, reference point 638 may be associated with a horizontal plane rather than an item / object in physical space (e.g., a vertical pole). Finally, the standard deviation of azimuth is somewhat small, indicating that the points may include a corner in physical space (e.g., a warehouse environment, a room, etc.).

[0076] Fig. 7A B is an exemplary use of the described system in a warehouse environment. Process 700 may be performed by any computing device, such as computer system 104 (see FIG. 1 ). Fig. 7A In process 700 of both B, the computer system receives a setup point cloud (step 702) (refer to Figure 2 6). The computer system may then detect the vertical rods in step 704. Detecting the vertical rods includes multiple steps. For example, the computer system classifies and scores the points associated with the vertical rods (step 706), locates the classified and scored vertical rods (step 708), and straightens the located vertical rods (step 710). Once the computer system completes the detection of the vertical rods, the computer system may detect the brackets in step 712. Detecting the brackets includes multiple steps. For example, in step 714, the computer system clusters the vertical rods into bracket groups. Then, in step 716, the computer system may classify the bracket rods by bracket type and orientation. The computer system further interpolates the missing bracket rods (step 718) and finally detects or otherwise locates the bracket segments from the classified bracket rods in step 720. Once the computer system completes the step of detecting the brackets, the computer system may identify the tray position of each of the detected bracket segments (step 722). Detecting the tray position may also include multiple steps. For example, in step 724, the computer system may first identify the pallet footprint (the X and Y location within the rack where the pallet may be placed). Next, the computer system may examine the entire rack point cloud to detect the height of the shelf levels in the rack (step 726). Once the shelf levels and pallet footprint locations are known, the computer system may examine the rack specifically at each pallet footprint location to find a specific shelf height available at that XY location (step 728). Step 728 may produce a complete set of X, Y, and Z coordinates for the rack's pallet location.

[0077] Process 700 is beneficial because it can allow accurate and real-time generation of a map / blueprint of a warehouse based on existing vertical poles, racks, and / or other physical structures in the warehouse environment. The maps generated from these point clouds can be used to track forklifts and / or other devices / human users in real-time as they operate / navigate through the warehouse environment.

[0078] Figure 8 8 is a flow chart of a process 800 for classifying vertical pole points. The process 800 may be performed by any computing device, such as the computer system 104 (see FIG. 1 ). Figure 8 The process 800 depicted in FIG. 8 involves Fig. 7AThe vertical pole points in the 3D scan / point cloud are classified and scored (step 706). Several features in the 3D scan / point cloud may indicate the presence of vertical support poles or any type of vertical pole that may be present in a warehouse environment or other building / structure. For example, if the shapes present in the point cloud are tall and / or thin, then they may be indicative of vertical poles. Classifying points as vertical poles may be accomplished by calculating features of the point cloud, such as curvature, covariance, etc. Using these values, the computer system may score the points, thereby classifying the points as being part of a vertical pole.

[0079] First, the computer system receives a setup point cloud in step 802, as previously described with reference to Fig. 7A As discussed in step B. The computer system may then voxelize the point cloud into a predetermined grid size (step 804). Voxelization means that the computer system breaks up the point cloud into smaller blocks and then analyzes each of the blocks. For example, the predetermined grid size may be 1m x 1m x 1m. Therefore, in step 804, the computer system may break up the point cloud into blocks (e.g., segments) of 1m x 1m x 1m. Next, in step 806, the computer system may select a voxel (e.g., a 1m x 1m x 1m segment). The computer system may determine whether the voxel is occupied by a point (step 808). If there are not enough points in the voxel, the computer system may determine whether there are more voxels in the point cloud (step 810). If so, the computer system may select a different voxel in step 806 and repeat the following steps. If there are no more voxels, the computer system returns to step 804 and voxelizes the point cloud using a different predetermined grid size in step 804 and repeats the following steps. Using different grid sizes may produce different results, which may be averaged and used to more accurately identify vertical poles, especially if the vertical poles are resolved or discontinuous in the received point cloud. Therefore, in one embodiment, the computer system may determine that the point cloud should be voxelized twice into 1m x 1m x 1m segments and 1m x 1m x 2m segments. In some embodiments, the computer system may not repeat the process of performing the voxelization through the vertical poles. Figure 8 Instead of following the steps described in , the point cloud can be voxelized once.

[0080] Returning to step 808, if the points occupy the selected voxel, the computer system may cluster the points in step 812. The computer system may use a known clustering algorithm, such as HDBSCAN. The spherical features calculated in technique 700 may be used as input to the clustering algorithm here to obtain optimal clustering results. The computer system may then determine that if there is a ball / grouping of points next to a plane, then the points may be clustered into two clusters, one cluster representing the plane and the other cluster representing any object next to the plane. For example, the plane may be a floor. In addition, when clustering the points in step 812, the computer system and / or a user at a device communicating with the computer system may determine how many clusters should be generated within the selected voxel. For example, having multiple clusters may result in more accurate classification and identification of all objects within the selected voxel. On the other hand, multiple clusters with small voxel sizes may result in separation of points that would otherwise indicate singular objects. If the computer system determines that the points of a 1m x 1m x 1m voxel should be clustered into 10 clusters, then the computer system may cluster the points indicating a vertical pole into more than one cluster because the points are separated by a certain distance. Therefore, it may be more difficult for the computer system to classify a cluster of points as a vertical pole when another part of the vertical pole is in a separate cluster.

[0081] Once the points are clustered in step 812, the computer system selects one of the clusters in step 814. The clusters are further voxelized into one or more predetermined grid sizes (step 816). Doing so can result in a more accurate classification of the vertical rods, as well as a score of whether the point is associated with a vertical rod. Once the clusters are voxelized, the computer system can perform at least one of four methods to classify, identify, and score the vertical rod points within the clusters. Those four methods include covariance matrices (refer to steps 818 to 820), histogram filters (steps 822 to 824), spherical covariance matrices (steps 826 to 828), and / or neural networks (steps 830 to 832). The computer system can optionally perform these methods in any order. In other embodiments, the computer system can only perform one of these four methods. In other embodiments, the computer system can selectively perform a combination of any of these four methods.

[0082] Referring first to the covariance matrix method, in step 818, the computer system may apply the covariance matrix to the cluster and find the standard deviation value. In this step, the computer system may eliminate horizontal planes, such as floors, ceilings, and walls. The covariance matrix is ​​useful for comparing points and seeing how they relate to each other in all X, Y, and Z axes. Thus, the computer system can distinguish between planes and vertical rods. For example, if the points extend in one dimension, then this indicates that there is a vertical rod - for example, the vertical rod will extend in the Z direction. On the other hand, if the points extend in two dimensions, then this is more indicative of planes, such as floors, ceilings, or walls. Once the standard deviation is completed, the computer system can identify high standard deviations in the Z direction and low standard deviations in the X and Y directions. This will indicate points representing vertical rods. Therefore, in step 820, the computer system may assign scores to one-dimensional or two-dimensional point extensions. The score may be determined by the computer system and / or a user on a device communicating with the computer system. For example, the computer system may determine that the score should be in the range of 0 to 1, where 1 indicates a high probability that the point represents a vertical rod, and 0 indicates a low probability that the point represents a vertical rod. Therefore, in step 820, the computer system may assign scores to one-dimensional or two-dimensional point extensions. The score may be determined by the computer system and / or a user on a device communicating with the computer system. For example, the computer system may determine that the score should be in the range of 0 to 1, where 1 indicates that the point represents a high probability of a vertical rod, and 0 indicates that the point represents a low probability of a vertical rod. Therefore, in step 820, the computer system may assign scores to one-dimensional or two-dimensional point extensions. The score may be determined by the computer system and / or a user on a device communicating with the computer system. For example, the computer system may determine that the score should be in the range of 0 to 1, where 1 indicates that the point represents a high probability of a vertical rod, and Figure 8 In the example of , in step 820 , the computer system may assign a score of 0 to points that extend in the X and / or Y directions or indicate a plane, and a score of 1 to points that extend only in the Z direction or indicate a vertical pole.

[0083] Similarly, the computer system can apply the spherical covariance matrix to the cluster and determine the standard deviation (step 826). Applying the spherical covariance matrix is ​​similar to the method described above about the covariance matrix. The difference is that in step 826, the computer system identifies the standard deviation of the spherical coordinates (that is, the elevation coordinates), rather than finding the standard deviation of the Cartesian coordinates. Then, the computer system can assign scores to the elevation coordinates (step 828). Applying both the covariance matrix and the spherical covariance matrix may be redundant, however, both produce slightly different scores. Therefore, the computer system can identify and classify points from more than one angle, which is conducive to eliminating / removing points that do not indicate vertical rods. For further discussion of the spherical covariance matrix, refer to the disclosure of Figure 6 above.

[0084] The computer system may also apply a histogram filter to the cluster (step 822). For further discussion of the histogram filter, refer to the disclosure above. Using a histogram filter is beneficial to identifying repeated points within each cluster. Once a vertical point pattern is identified, the computer system may determine that those point patterns indicate vertical objects. For example, if the points are stacked on top of each other along the Z axis, they more indicate vertical rods. On the other hand, if the points are distributed in more than one dimension / along more than one axis, the points more indicate planes. In the case where the points are distributed in more than one dimension, a low score such as 0 may be assigned to the cluster (step 824). In the case where the points are stacked on top of each other and / or appear in a repeated vertical point pattern, in step 824, a higher score (such as 1) may be assigned to the cluster.

[0085] Finally, as mentioned, in step 830, the computer system may apply a neural network to the clusters. Basic neural networks known in the art may be applied to the entire point cloud and / or selected clusters. The neural network may receive all clusters and determine whether a vertical pole exists based on all clusters together. In addition, a classification score may be generated for the point cloud, for each point in the point cloud, and / or for each cluster (step 832). The higher the score, the more indicative of a vertical pole.

[0086] Once the computer system applies one or more of the methods described above, the computer system can weight and average the scores (step 834). In other words, the computer system can normalize each of the scores to find the final score of the cluster. The computer system may not combine the scores. In the implementation scheme of the scoring output different scale scores in each of the methods, the computer system can transform or normalize each of the scores to a 0 to 1 scoring scale or any other appropriate scoring scale. Using a 0 to 1 scoring scale, a score of 1 can be associated with 100% certainty that the point is a part of a vertical rod. A score of 0 can be associated with 100% certainty that the point is not a part of a vertical rod. In addition, a score of 0.5 can be associated with uncertainty about the score.

[0087] In other embodiments of step 834, the computer system may determine which scores it will retain. For example, the computer system may apply all four methods, thereby generating four scores. However, the computer system may determine that it will only keep the scores above a predetermined threshold (e.g., assuming that all scores are on the same rating scale and / or have been normalized). In another example, the computer system may determine that it will keep the scores with the highest certainty that the points are associated with the vertical rod, thereby ignoring other scores. For example, if the covariance matrix (step 818), the histogram filter (step 822), and the spherical covariance matrix (step 826) all return a score of 0.3, but the neural network (step 830) returns a score of 0.8, then the computer system may determine that the score of 0.8 should be kept because it indicates the highest certainty of the vertical rod. 0.8 represents a higher certainty than 0.3 because it is farther away from 0.5, which represents uncertainty. Therefore, in step 834, the computer system does not combine the scores from all the methods mentioned above-instead, the computer system identifies the most certain score and keeps the score.

[0088] Once the scores are weighted and normalized, the computer system may determine in step 836 whether there are more clusters. If so, the computer system returns to step 814, selects a cluster, and repeats the steps described previously. If and / or once there are no more clusters, the computer system returns to step 810, determines whether there are more voxels, and repeats the steps described previously. Repeating the voxelization is optional. In other embodiments, if the computer system determines at step 836 that no more clusters exist, the computer system may stop selecting voxels for each cluster. Figure 8 Process 800 is described.

[0089] Fig. 9 is a flow chart of a process 900 for positioning a vertical pole. The process 900 may be performed by any computing device, such as the computer system 104 (see FIG. 1 ). Fig. 9 The process 900 depicted in FIG. Fig. 7A The classified and scored vertical poles are located in the process 900 (step 708). Once the poles are scored, the computer system can determine which points are most likely to contain information about the location of the vertical pole. At this point, the computer system can process thousands or millions of scored points. The localization process results in the creation of a condensed point cloud with fewer points than the input number of point cloud points. Therefore, a point in the output point cloud generated from the process 900 can represent a single vertical pole. Therefore, the localization can produce a list of pole predictions, which consists of at least an xy position and optionally a confidence score associated with the likelihood that the point is a vertical pole.

[0090] In step 902, the computer system may receive the scored points. The points are flattened onto a plane where z=0 (step 904). Once flattened, the points may be clustered in step 906. Clustering may be performed by a known clustering algorithm. Clustering is important for determining which points belong to the same vertical rod. A cluster may be selected in step 908, and the center of the selected cluster may be found, representing the center of the vertical rod. Once a cluster is selected, the computer system may identify a bounding box in step 910. The computer system may then determine whether the bounding box is greater than a predetermined threshold (step 912). If the bounding box is greater than the threshold, the bounding box extends too much in either direction, and the cluster is unlikely to be a vertical rod. Therefore, the computer system may locate the rod within the bounding box in step 914, and then expand the rod and score in step 916. Once step 916 is completed, the computer system may determine in step 930 whether there are more clusters. If so, another cluster may be selected in step 908, and the steps described previously may be repeated. If no more clusters exist, then in step 932 the computer system may return a list of pole locations and their associated confidence scores.

[0091] If the bounding box is less than the threshold, the bounding box does not extend too much in either direction, and therefore the cluster is more likely to indicate a vertical pole. Therefore, the computer system can calculate the centroid of the cluster in step 918, and calculate the center of the bounding box in step 920. In step 922, using those values, the computer system can find the magnitude or difference between the centroid of the cluster and the center of the bounding box. If the distance between the centroid and the center is the smallest, then it is more indicative that the cluster represents a vertical pole. However, if the distance between the centroid and the center is greater than a predetermined threshold (step 924), then the cluster most likely does not represent a vertical pole. Therefore, the cluster can receive a score of 0. Therefore, if the magnitude is greater than the threshold in step 924, then the computer system can record the position and assign a low score of 0 in step 926. Then, in step 930, the computer system can determine whether there are more clusters, as previously described.

[0092] In step 924, in the case where the difference (e.g., magnitude) is less than a threshold, the position of the pole may be recorded and assigned a score in step 928. In some implementations, a score greater than 0 may be assigned, thereby indicating a higher confidence / likelihood that the cluster is a vertical pole. The computer system and / or a user at a device communicating with the computer may determine a range of scores to be used and / or a predetermined threshold. For example, the computer system may determine that, on a range of 0 to 1, 0 indicates that the cluster represents a vertical pole with no confidence. On the other hand, a score of 0.5 indicates that the cluster represents a 50% confidence of a vertical pole, and a score of 1 indicates that the cluster represents a 100% confidence of a vertical pole.

[0093] Once step 928 is complete, the computer system may determine in step 930 whether there are more clusters. If so, the computer system may return to step 908, select another cluster, and repeat the previously described steps. If the computer system determines that there are no more clusters to analyze, the computer system may return a list of the locations of the poles and their associated confidence scores (step 932). This list may be communicated to a user computer device and / or any type of display screen / computing device in communication (e.g., wired and / or wireless) with the computer system performing technique 900.

[0094] Fig.10 is a flow chart of a process 1000 for cleaning positioning results. The process 1000 can be performed by any computing device, such as the computer system 104 (see FIG. 1). Fig.10 The process 1000 depicted in Fig. 7A The computer system then performs a straightening operation to locate the vertical rods in the image (step 710). Since it is known that the vertical rods should be straight and at 90 degrees to each other, the computer system can use this information to clean up the vertical rod predictions, add missing rod portions, realign the rods into rows, etc. Thus, the computer system can identify the dominant lines and determine whether points near such lines should be on those lines. If the point should be on those lines, then it indicates a vertical rod and should therefore be moved to those lines. Cleaning the vertical rod location results consists of adjusting the xy rod predictions and their associated confidence scores.

[0095] like Fig.10As depicted in , in step 1002, the computer system receives the located and scored points. The computer system can then detect the dominant vertical line by identifying the vertical line in the points (step 1004). The area occupied by the points that appear in the vertical line can be identified as the dominant vertical line. Next, in step 1006, the computer system can select the dominant vertical line. In step 1008, the computer system can determine whether there are points near the line. In other words, the computer system can determine whether the points are within a threshold distance from the line. In some embodiments, the computer system can make this determination based on identifying the nearest neighbors and whether those neighbors are at a predetermined distance from the dominant vertical line. If the points are close enough to the line so that they are within a predetermined distance (e.g., a threshold), the computer system can move those points to the line (step 1010), thereby cleaning and forming a straighter vertical line representing the vertical rod. Once the points are moved to the line, the computer system can adjust the vertical rod confidence score in step 1012. If the computer system moves the points to the vertical line, thereby adjusting its xy position, the confidence score increases because it is more indicative of an accurate vertical rod. The further the point is from the line, the lower its final score. A point that is already directly on top of the line may have a score of 1. As previously mentioned, the computer system may assign scores in the range of 0 to 1, so that if the computer system moves the point onto the line in step 1010, a score close to 1 may be assigned to the vertical bar in step 1012 because it indicates the highest confidence that the vertical bar exists. Once the computer system adjusts the confidence score in step 1012, the computer system may determine in step 1018 whether there is a more dominant vertical bar. If so, the computer system may return to step 1006, select a vertical line, and repeat steps 1008 to 1018 until there are no more dominant vertical lines to check.

[0096] Fig.11 1 is a flow chart of a process 1100 for clustering vertical poles into bracket groups. The process 1100 can be performed by any computing device, such as the computer system 104 (see FIG. 1 ). Fig.11 The process 1100 depicted in FIG. 1 involves Fig. 7A The computer system may implement HDBSCAN and / or k-means clustering to identify groups of racks separated by aisles. The rack groups may generally have elongated bounding boxes for identifying the orientation of the racks. In addition, the bounding boxes may generally contain two rows of racks. Depending on the requirements of the physical environment, these racks may be of the same / different types. Fig.11 As a result of the process in , there may be useful clustering results that make it easier and faster to identify the brackets before classifying the bracket types in the physical environment. Fig.11In embodiments where the clustering in the process depicted in is not so accurate, the computer system may repeat this process later by using classification information obtained from other processes described throughout the present disclosure.

[0097] refer to Fig.11 In step 1102, the computer system receives the vertical pole point (reference Fig.10 ). Clusters may be generated for these points using the techniques previously described (step 1104). The point cloud may be decomposed into a predetermined number of maximum clusters. The computer system may select a cluster in step 1106. Next, in step 1108, the computer system may determine size information for the selected cluster. This may include finding a bounding box for the selected cluster, identifying the minimum x, y, and z coordinates of the bounding box, and the maximum x, y, and z coordinates. The computer system may identify the length of the bounding box, which may be the difference between the maximum x coordinate and the minimum x coordinate. The computer system may also repeat these calculations for the length of the y coordinate. The computer system may also determine the aspect of the bounding box, where the computer system divides the length of the y coordinate by the length of the x coordinate. Once the size information of the cluster is determined in step 1108, the computer system may determine in step 1110 whether the size information is consistent with the support information. The support information may be entered into the computer system by a user at a computing device and / or any other type of computer / system in communication with the computer system performing the technique 1100. If the size information is inconsistent with the bracket information, then in step 1112, the computer system can return the points from the selected cluster to the pool of vertical pole points. The computer system can then generate a new pole point cluster to determine whether a different point cluster indicates a bracket grouping. Thus, the computer system can repeat the steps described previously.

[0098] Referring back to step 1110, if the computer system determines that the size information is consistent with the bracket information, then the orientation information can be determined in step 1114 based on, for example, the bounding box in the size information. Therefore, the bracket group generally has an elongated bounding box, which indicates a vertical orientation. Once the orientation information is determined, the computer system can add the selected cluster and its associated size information and orientation information to the bracket list (step 1116). Then, in step 1118, the computer system can determine whether there are more clusters. If so, another cluster can be selected in step 1108, and the steps described previously can be repeated. If there are no more clusters, the computer system can return to the bracket list in step 1120. This list can be displayed and / or sent to a user computing device, a display, and / or any other device / computer / system in communication with the computer system performing process 1100.

[0099] Fig.12is a flow chart of a process for classifying stent poles. Process 1200 may be performed by any computing device, such as computer system 104 (see FIG. 1 ). Fig.12 The process 1200 depicted in FIG. 1 involves Fig. 7A Step 716 in. There are many types of brackets that can be used in a physical environment. For example, one common type of bracket is called a drive-in bracket. This bracket is typically characterized by a vertical rod spacing pattern of 1.2mx 1.4m, so that a forklift or other warehouse vehicle can drive into / under the bracket to store / pick up physical objects / articles. Another common bracket type is called a selection bracket. The orientation of each type of bracket can be different. Some bracket types can also have different angles. The x and y distances to adjacent rods can contain information that can be matched with a bracket specification to then identify the type of bracket used. In addition, a user at a device that communicates with the computer system can provide a bracket specification to the computer system, such as the type of bracket in a warehouse environment, and therefore should be identified.

[0100] When good clustering results are obtained in the previous process, the stent orientation information can be used in process 1200 to reduce processing time. After all, to determine the type of stent used, the computer system uses / relies on only one orientation, not two. In embodiments where the clustering results are not so good, the computer system can consider all orientations to classify the stent type.

[0101] In step 1202, the computer system may receive a list of brackets (see Fig.11 ). In step 1204, a rack may be selected from the list. The computer system may then determine a distance value between each of the rack poles in the selected rack (step 1206). For example, the nearest neighbor poles may be identified, and then the distance between each of the nearest neighbor poles and the poles in the selected rack may be determined. The computer system may also determine orientation information for the poles in the selected rack (step 1208). Next, a rack type may be determined based on the distance value and the orientation information (step 1210). For example, a larger distance value between poles may indicate a drive-in rack type, while a smaller distance value between poles may indicate a selected rack.

[0102] Once the bracket type is determined in step 1210, the computer system may determine whether there are more brackets in the bracket list in step 1212. If so, the computer system may select another bracket from the list in step 1204 and repeat the previously described steps until there are no more brackets in the list. If there are no more brackets in step 1212, the computer system returns the list of bracket types in step 1214. As discussed, the list may be displayed and / or sent to a user computing device, a display, and / or any other device / computer / system in communication with the computer system performing process 1200.

[0103] Fig.13 is a flow chart of a process 1300 for interpolating missing stent rods. The process 1300 may be performed by any computing device, such as the computer system 104 (see FIG. 1 ). Fig.13 The process 1300 depicted in FIG. 1 involves Fig. 7A 718 in step 719. When there is a series of xy vertical support rod positions and associated support types and orientations, the computer system can look for missing rod positions that match the detected pattern. If a new rod prediction is predicted, the computer system can look for a vertical rod in the original point cloud at that location. When a vertical rod is predicted based on a specific support type and orientation, the predicted rod can inherit that type and orientation and the associated confidence score.

[0104] refer to Fig.13 , the computer system may receive the bracket information generated in the process discussed previously (step 1302). This bracket information may include, but is not limited to, the pole, the type of pole, the orientation of each pole, and a confidence score. Next, in step 1304, a bracket may be selected from the received bracket information. Once a bracket is selected, the pole that comprises the bracket may be selected (step 1306). Then, in step 1308, the computer system may predict the position of the other poles in the bracket based on the position of the selected pole. In step 1310, the computer system may determine whether the other poles are in the predicted position. This may be achieved by known techniques, and in some embodiments, the computer system may analyze the information of the selected bracket to identify the positions of the other poles in the bracket that the computer system did not select in step 1306. The computer system may also analyze all bracket information received from step 1302 to determine whether any pole in the information falls within the predicted position of the other poles from step 1308.

[0105] If the computer system determines in step 1310 that the other poles are at the predicted positions, a high confidence value may be assigned to the poles in the selected bracket (step 1312). This indicates that the computer system accurately refines, filters, and / or processes the point cloud to determine the positions, bracket groupings, and bracket types of the vertical poles in the environment. If the other poles are not at the predicted positions in step 1310, the computer system analyzes the point cloud at the predicted positions (step 1314). The computer system may obtain all data collected, received, and analyzed from the previous process to determine whether the computer system is missing poles that are in the point cloud and should be considered as part of the selected bracket. In step 1316, based on the determination of the computer system, the computer system may update / add its analysis to the bracket information. For example, if the computer system determines that there is actually a pole at one of the predicted positions in the point cloud, the computer system may add information about the pole to the bracket information. In that manner, when the computer system loops back to step 1304 or 1306 through process 1300, the newly added bracket information may be analyzed, and the confidence value may be adjusted accordingly.

[0106] The computer system may then determine if there are more rods within the selected bracket that need to be analyzed (step 1318). If there are more rods, the computer system returns to step 1306 and repeats the steps previously described. If there are no more rods, then in step 1320, the computer system determines if there are more brackets in the bracket information. In addition, referring back to step 1312, once the computer system assigns a high confidence value to a rod, the computer system also enters step 1320 and must determine if there are more brackets. If there are, the computer system returns to step 1304 and repeats the steps previously described. If there are no more brackets in the bracket information, the computer system returns the bracket information in step 1322. The bracket information may include, but is not limited to, an updated high confidence value for the rod and / or an analysis of whether the rod is not at one of the predicted locations but should be there. The bracket information may be displayed and returned as previously discussed throughout this disclosure.

[0107] Fig.14A 1 to B is a flow chart of a process 1400 for detecting bracket segments in a vertical bracket rod. The process 1400 may be performed by any computing device, such as the computer system 104 (see FIG. 1 ). Fig.14A The process 1400 depicted in B involves Fig. 7AStep 720 in. Using the location, rack type, and orientation information of each vertical rod, the computer system can identify sections of the rack, where those sections may consist of four adjacent rods of the same type and orientation. Rack sections typically have a type, orientation, and location. In some embodiments, for example, two adjacent rack sections may share two rods. This indicates that two adjacent rack sections are parts of the same rack. Rack groups that may be separated by aisles may consist of different types of racks, and may even consist of one rack, two racks, or multiple complete racks. In addition, vertical support rods that extend from the floor to the ceiling of a physical environment typically cause the rack to be broken down into several smaller racks. Therefore, the purpose of process 1400 is to locate each rack section in a group of classified vertical rods and identify its rack type and orientation. Once the rack section information is determined, the pallet center can be calculated (reference Fig.15 ).

[0108] refer to Fig.14A Continuing to process 1400 in B, the computer system may receive pole information from the previous process (refer to FIGS. 7 to 13 ) in step 1402. The received information may include, but is not limited to, pole location, support type, support and pole orientation, confidence score, and pole grouping. In step 1404, a pole is selected from the pole information. The computer system then predicts the location of the support segment center in the four diagonal directions from the selected pole (step 1406). Thus, the support segment center may be predicted along the NW, NE, SW, and SE directions. Once the support segment center is predicted, the support segment center may be selected in step 1408.

[0109] In step 1410, the computer system may determine whether the selected support segment is within the bounding box of the support grouping. If it is within the bounds, the computer system may continue processing the support segment in step 1412. If the support segment is not within the bounds, the computer system may jump to step 1424. Once the support segment is selected and determined to be within the bounds of the support grouping, the computer system may predict four pole positions, each in a diagonal direction (e.g., NW, NE, SW, and SE) from the center of the support segment (step 1412). The computer system may select the predicted pole position in step 1414. In step 1416, the computer system may determine whether the pole is at the predicted position. This determination may be made using the techniques discussed previously (see FIG. 14A ). Fig.13 ). If a pole is found at the predicted location, a high confidence value is assigned to the bracket segment (step 1418). A high confidence value indicates that the pole is actually part of the bracket segment that was accurately captured in the point cloud and refined through the process discussed previously.

[0110] If the computer system determines in step 1416 that there is no pole at the predicted position, then the predicted pole position may be added to the pole information in step 1420. Once either step 1418 or 1420 is completed, the computer system may determine if there are more predicted pole positions to analyze for this support segment prediction (step 1422). If so, the computer system may return to step 1414 and repeat the previously described steps. If there are no more pole predictions to analyze for this support segment, then the computer system may determine if there are more support segment predictions to analyze (step 1424). If so, the computer system may return to step 1408 and repeat the previously described steps.

[0111] If there are no more stent segment predictions to analyze, then in step 1426 the computer system determines if there are more rods in the rod information. If so, then the computer system returns to step 1404 and repeats the steps previously described. If there are no more rods in the rod information, then in step 1428 the computer system returns the rod information and stent segment information. The stent segment information may include, but is not limited to, the location and confidence score of the predicted stent segment center, the stent segment orientation, and the stent segment type. The rod information may include, but is not limited to, the predicted locations of other rods. The stent segment and rod information may be displayed and returned as previously discussed throughout this disclosure.

[0112] Fig.15 is a flow chart of a process 1500 for calculating a pallet footprint location. The process 1500 may be performed by any computing device, such as the computer system 104 (see FIG. 1 ). Fig.15 The process 1500 depicted in FIG. 1 involves Figure 7B The number of pallets held in a rack depends on the type of racks utilized in the physical environment and the number of shelves on each rack. For example, a rack location may typically hold two pallets side by side in each rack section. On the other hand, a drive-in rack location may typically hold one pallet. Regardless of how many pallets the rack type holds, once the rack location is known, the same algorithm / technique may be used to calculate a list of pallet footprint centers.

[0113] refer to Fig.15, the computer system may receive the rack section information from the previous process (refer to Figures 7 to 14) (step 1502). The rack section information may include (but is not limited to) rack section location, rack type, rack and / or rod orientation, and rack and / or rod grouping. The computer system may then select a rack section in step 1504. The computer system may then determine whether the selected rack has a "selected" rack type (step 1506). If the rack section has a selected rack type, then in step 1510 the computer system may calculate the centers of the two pallet footprints in the rack section. If the rack type is not "selected", then it is likely to be a single wide type rack, such as a "drive-in" rack, and a single pallet footprint position may be calculated at the center of the rack section. Known techniques may be used for the calculated center of the rack. In step 1510, the computer system may also calculate the pallet direction using known techniques and the boundaries and types of the rack groupings.

[0114] Once the calculations are completed in step 1508 or 1510, the computer system may add such calculations to the rack section information in step 1512. Once the rack section information is updated, the computer system may determine in step 1514 whether there are more rack sections in the rack section information. If so, the computer system may return to step 1504 in process 1500 and repeat the previously described steps. If there are no more rack sections in the rack section information, the computer system may return the rack section information in step 1516. The rack section information may include, but is not limited to, rack center calculations, pallet footprint predictions, and / or pallet orientation. The rack section information may be displayed and returned in a manner similar to the pole information previously discussed throughout this disclosure.

[0115] Fig.16 1 is a flow chart of a process 1600 for locating a rack shelf for a rack section. Specifically, the height of the shelf can be determined so as to know how many trays the rack can hold and where the trays are located in space. Process 1600 can be performed by any computing device, such as computer system 104 (see FIG. 1 ). Fig.16 The process 1600 depicted in FIG. 1 involves Figure 7B Step 726 in .

[0116] refer to Fig.16 , the computer system may receive the rod information from the previous process (refer to FIGS. 7 to 14 ) (step 1602). The rod information may include, but is not limited to, rod position, support type, support and / or rod orientation, and support and / or rod grouping. The computer system may then select a support grouping in step 1604.

[0117] The computer system may filter out all vertical objects from the rack data, such as vertical bars or vertical planes that make up the pallet face (step 1606). In step 1608, the computer system may locate large horizontal planes by known techniques. These large horizontal planes may contain information about the height of the shelves on the rack.

[0118] Once the computer system has identified several large horizontal planes, it can go through a scoring process of the planes using known clustering and classification algorithms to determine whether each plane has the characteristics of a shelf, or whether each plane looks more like the top of a tray (step 1610). This scoring process can reveal information about the confidence of the predicted shelf height, which can be useful in later calculations. Thus, the calculated shelf scores and confidences can be stored in a remote data repository or other data storage system.

[0119] In step 1612, the computer system may update the rod information using the identified shelf height. In step 1614, the computer system may determine whether there are more bracket groupings to analyze. If there are more bracket groupings, the computer system may return to step 1604 and repeat the steps described above. If there are no more bracket groupings, the computer system may return updated rod information in step 1616. The rod information may include, but is not limited to, the newly predicted shelf height and confidence score. The rod information may be displayed and returned in a manner similar to the rod information previously discussed throughout this disclosure.

[0120] Fig.17 1 is a flow chart of a process 1700 for detecting rack sections and tray levels. In other words, the process 1700 can be used to locate rack shelves for individual rack sections. The process 1700 can be performed by any computing device, such as the computer system 104 (see FIG. 1 ). Fig.17 The process 1700 depicted in FIG. 1700 involves Figure 7B In step 728 of . In process 1700, the actual height of the individual trays may be located. Locating the actual height may be helpful because a rack may have several different shelf levels that are not constant along the entire length of the rack.

[0121] refer to Fig.17 , the computer system may receive the rod and rack section information from the previous process (refer to FIGS. 7-16 ) (step 1702). The rod information may include, but is not limited to, rack type, rack and / or rod orientation, and rack and / or rod grouping. The rack section information may include, but is not limited to, rack section location, pallet footprint location, rack section type, and rack section orientation. The computer system may select a rack section in step 1704.

[0122] The computer system can then filter out all vertical objects from the rack data, such as vertical rods or vertical planes that make up the pallet surface (step 1706). In step 1708, the computer system can locate horizontal planes by known techniques. These horizontal planes can contain information about the height of the shelves on the rack.

[0123] In step 1710, the computer system may calculate a confidence score for the detected rack section plane. The confidence score may be calculated based on comparing the heights of these rack section shelves to the rack heights found in process 1600. The detected rack section plane may be compared to the rack grouping plane. If the height information for this rack section is consistent with the overall rack level height information for the rack, then the pallet height may be scored with a high confidence score. A high confidence score may be close to 1, while a low confidence score may be close to 0. If the pallet height is found in step 1710 to be inconsistent with any rack height identified in technique 1600, then this prediction may have a low confidence score unless it can be verified by some other algorithm or by a user.

[0124] In step 1712, the computer system may update the rod and support section information using the identified shelf height. The information may also be updated using a confidence score. In step 1714, the computer system may determine if there are more support sections to analyze in the support section information. If so, the computer system may return to step 1704. If there are no more support sections, the computer system may return updated rod and support section information. The rod and support section information may include, but is not limited to, the newly predicted shelf height and the confidence scores belonging to the individual rods and individual pallet positions, respectively. The rod and support section information may be displayed and returned in a manner similar to the rod information previously discussed throughout this disclosure.

[0125] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of the disclosed technology or what may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular disclosed technology. Certain features described in the context of a single embodiment in this specification may also be implemented in combination in a single embodiment, in part or in whole. On the contrary, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although features may be described herein as working in certain combinations and / or initially claimed, in some cases, therefore, one or more features from the claimed combination may be cut out from the combination, and the claimed combination may involve a sub-combination or a variant of a sub-combination. Similarly, although operations may be described in a particular order, this should not be understood as requiring such operations to be performed in a particular order or sequence, or to perform all operations, to achieve the desired result. Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims.

Claims

1. A method for determining occupancy information of a physical space from point cloud data, the method comprising: receiving, by a computing system, point cloud data of a physical space, the point cloud data comprising a plurality of points in a three-dimensional space approximating locations of physical structures within the physical space; identifying, by the computing system, the physical structure in the physical space based at least in part on classifying the points in the point cloud data that correspond to a contour of the physical structure; The identified characteristic of the physical structure is determined by the computing system based at least in part on: detecting vertical poles from the point cloud data; as well as detecting a bracket segment in the physical space based on the vertical rod; generating, by the computing system, a map of the physical space using the physical structures and corresponding characteristics of the physical structures; generating, by the computing system, occupancy information for the physical space based at least in part on the mapping of the physical space and the detected bracket segments, wherein generating the occupancy information includes determining process flow and movement of objects in the physical space when the physical space is occupied; as well as The occupancy information of the physical space is returned by the computing system. 2 . The method of claim 1 , wherein generating the occupancy information comprises tracking facility vehicles and facility workers in the physical space. 3 . The method of claim 1 , wherein generating the occupancy information comprises determining placement of trays in the physical space based on the mapping of the physical space. 4 . The method of claim 1 , wherein generating the occupancy information comprises determining available space in the detected rack section for placement of an incoming pallet in the physical space.

5. The method of claim 1, wherein the physical structure comprises at least one of a floor, a wall, a vertical pole, a shelf, a pallet, a door, a rack, a facility worker, and a facility vehicle. 6 . The method of claim 1 , wherein determining the characteristic of the physical structure comprises determining a pallet footprint based on the detected bracket segments.

7. The method of claim 1, wherein determining the characteristic of the physical structure comprises determining a number of pallet locations detected in the rack section.

8. The method according to claim 7, further comprising: determining, by the computing system, a height of each shelf segment in the detected rack segments; as well as A pallet location for each shelf section of the detected rack sections is identified by the computing system based on the pallet footprint and the height of each shelf section.

9. The method of claim 8, wherein identifying the tray position of each shelf section of the detected rack sections comprises: identifying a pair of X and Y locations within the shelf section at which at least one pallet can be placed; detecting the height of the shelf section; determining a specific shelf height at each of the X and Y position pairs; and A corresponding pair of X and Y positions and a corresponding shelf height is returned for each of the tray positions.

10. The method of claim 8, wherein detecting the stent segment comprises: clustering the vertical rods into bracket groups; classifying the clustered vertical rods by bracket type and bracket orientation; as well as Stent segments are detected based on the clustered vertical rods.

11. The method according to claim 10, further comprising: The clustered vertical rods are classified based on the bracket type and bracket orientation, and the missing bracket rods are interpolated for each of the clustered vertical rods.

12. The method of claim 10, wherein clustering the vertical rods into bracket groups comprises: Receive bracket information; For each cluster of the vertical rods, determining size information of the cluster; Determining that the size information of the cluster is consistent with the scaffold information; as well as A scaffold list including the cluster and the size information of the cluster is generated based on the consistency between the cluster and the scaffold information.

13. The method according to claim 12, further comprising: selecting, by the computing system, a bracket from the bracket list; determining, by the computing system, a distance between each of the vertical rods in the selected bracket; as well as A bracket type of the selected bracket is determined by the computing system based on the distance for each of the vertical rods.

14. The method of claim 1, wherein detecting a vertical pole comprises: classifying a subset of the points in the point cloud data that are associated with vertical poles; locating the classified subset of the points; as well as The located point is straightened into a vertical rod.

15. The method of claim 1, wherein detecting a vertical pole comprises: iteratively voxelizing the point cloud data into at least one predetermined grid size; iteratively clustering points of the point cloud data in each voxel; as well as It is determined that each cluster of points in each voxel corresponds to at least one vertical rod.

16. A computerized system for generating a map of a physical space from point cloud data of the physical space, the system comprising: one or more processors; and One or more computer readable devices comprising instructions that, when executed by the one or more processors, cause the computerized system to perform operations comprising: receiving point cloud data of a physical space, the point cloud data comprising a plurality of points in a three-dimensional space approximating locations of physical structures within the physical space; identifying the physical structure in the physical space based at least in part on classifying the points in the point cloud data that correspond to the outline of the physical structure; The identified characteristic of the physical structure is determined based at least in part on: Detecting vertical poles from the point cloud data; and detecting a bracket segment in the physical space based on the vertical rod; generating a map of the physical space using the physical structures and corresponding characteristics of the physical structures; generating occupancy information for the physical space based at least in part on the mapping of the physical space and the detected bracket segments, wherein generating the occupancy information includes determining process flow and movement of objects in the physical space when the physical space is occupied; and The occupancy information of the physical space is returned.

17. The computerized system of claim 16, wherein generating the occupancy information includes tracking facility vehicles and facility workers in the physical space.

18. The computerized system of claim 16, wherein generating the occupancy information comprises determining placement of trays in the physical space based on the mapping of the physical space.

19. The computerized system of claim 16, wherein generating the occupancy information comprises determining available space in the detected rack section for placement of an incoming pallet in the physical space.

20. The computerized system of claim 16, wherein determining the characteristic of the physical structure comprises: (i) determining a pallet footprint based on the detected rack section, and (ii) determining a number of pallet locations in the detected rack section.

Images