Point cloud filtering

Abstract

This application relates to point cloud filtering. This specification describes systems and methods for refining point cloud data. A method can include receiving point cloud data of a physical space; iteratively selecting points along x, y, and z dimensions; clustering the selected points into 2D histograms; determining a slope value for each 2D histogram; and removing points from the point cloud data based on the slope value exceeding a predetermined value. The method can also include iteratively voxelizing each 2D histogram into a predetermined grid size; summing points in each voxelized 2D histogram; removing points from the point cloud data based on determining a total sum is below a predetermined sum value; retaining a center point based on determining a number of points in each voxelized 2D histogram exceeds a threshold value; selecting a point for each histogram; identifying nearest neighbor points in the point cloud data; removing the identified nearest neighbor points from the data; and returning remaining points.

Description

[0001] Information about divisional applications

[0002] This application is a divisional application. The parent application is an invention patent application filed on April 13, 2022, with application number 202280034287.2 and the title of invention being “Point Cloud Filtering.” Technical Field

[0003] This document generally describes techniques for thinning a point cloud, which may comprise a collection of points in three-dimensional space. Background Art

[0004] Physical spaces, such as warehouses and storage facilities, can be large and challenging to navigate. Mapping such physical spaces can help users, such as warehouse workers, learn the floor plan of the physical space and navigate around it without getting lost. Generating a blueprint or map of the physical space can be performed by a user walking around the physical space with a scanner to obtain a 3D scan and / or image of the entire physical space. The 3D scan and / or image can then be manually converted into a blueprint or map of the physical space. The user can manually update such a blueprint or map whenever the physical space changes.

[0005] Manual mapping can be a time-consuming process and can contain inaccuracies due to human error. For example, a user may not obtain images of every area of ​​a physical space. As a result, a map may include some areas of the physical space but not others. As another example, a user may not be able to obtain updated images of areas that have changed in the physical space. As a result, maps may be outdated, and users may rely on outdated maps. Blueprints and maps of modern physical spaces may be inaccurate, lack detail, or be missing entirely. Summary of the Invention

[0006] This document generally describes techniques for refining a point cloud so that the point cloud more accurately represents physical structures within a 3D space. The refined point cloud can be used to more effectively and efficiently generate a blueprint and / or map of the physical space. The point cloud can be generated from an image and / or other representation of the physical space. The point cloud can include points that provide a rough approximation of physical structures and surfaces within the physical space and their corresponding positions relative to each other. The point cloud can include some noise and inaccurate points, which can hinder the use of the point cloud for accurately mapping physical structures and their corresponding outlines within the physical space. The disclosed techniques can remove most, if not all, of the noise to provide a refined point cloud that more accurately corresponds to and highlights the outlines and relative positions of physical structures within the physical space.

[0007] In some embodiments, the disclosed technology can provide for selecting an axis of point cloud data for initial operation. From each axis, the entire segment of the point cloud can be compressed into a series of two-dimensional (2D) histograms. The histograms can be examined like kernels, which can result in voxels defined in 3D space being evaluated. A predefined threshold can be used to determine whether individual points within a voxel should be added to the final filtered point cloud output. For example, if the center of the examined voxel is an existing point, the points within the voxel can be summed and compared to the threshold. If the sum of the points within the voxel is above the threshold, the point can be added to the final filtered point cloud output. If the sum of the points within the voxel is not above the threshold, the point can be omitted from the final filtered cloud output. After the filtering is completed for each of the three axes (x-axis, y-axis, and z-axis), the nearest neighbor can be used to remove the selected point and N distance neighbors from the original cloud. The filter can then be applied (using the same and / or different parameters) to the resulting simplified cloud. This process can be repeated multiple times, tuning each pass to extract specific features. The resulting point clouds can be combined to form a subsampled point cloud. This final subsampled cloud can be input into, for example, a k-means filter to enhance the physical features discovered by the initial filtering process.

[0008] The point cloud described herein can be generated from a 3D scan and / or image of the physical space. The physical space can be a warehouse environment, a storage facility, a building, a house, and / or other structure. In some embodiments, the physical space can be an outdoor environment, such as a portion of a town and / or some other environment being surveyed. As described herein, the point cloud can be refined using techniques such as histogram filtering, k-means filtering, and slice filtering. The disclosed techniques can apply histogram filtering to the point cloud to remove noise (e.g., outliers). The disclosed techniques can analyze the resulting point cloud and determine the distribution of point density in the point cloud to determine whether to upsample portions of the point cloud and / or downsample portions of the point cloud. Upsampling and / or downsampling the point cloud can help create a more uniform distribution of points within the physical environment and a more refined depiction of objects.

[0009] In situations where regions of the point cloud have a low point density (e.g., the density is below a predetermined threshold of a desired density), it may be advantageous to upsample. Thus, upsampling requires filling portions of the point cloud with additional points. Upsampling may be achieved by applying a k-means filter to portions of the point cloud. In situations where regions of the point cloud have a high point density (e.g., the density exceeds the predetermined threshold of the desired density), downsampling may be advantageous. Thus, downsampling requires removing points from the filled portions of the point cloud. Downsampling may be achieved by applying a slicing filter to portions of the point cloud. Applying the filtering techniques disclosed herein may refine the point cloud so that the point cloud has an optimal (e.g., minimum, necessary) number of points and point density to identify features and objects within the physical space.

[0010] Once the point cloud is filtered and refined, it can be used to identify features and objects in the physical space (e.g., poles and racks in the warehouse environment) and / or buildings and roads in the outdoor environment. The disclosed technology can also assign confidence values / scores to identified items and use such values ​​to resolve missing objects, missing portions of identified objects, and / or generate a more accurate and updated map of the physical space.

[0011] Thus, using the refined point cloud, a more accurate representation of the physical space can be generated, including, but not limited to, the physical space's footprint (e.g., blueprint, map, floor plan, etc.), cubic feet, wall locations, rack locations, aisles, corridors, etc. The disclosed technology can also be used to update and / or restart a map of the physical space in real-time, periodically, and / or automatically.

[0012] Certain embodiments described herein may include systems and methods for refining point cloud data. The certain embodiments may include: receiving, by a computing system, point cloud data of a physical space, the point cloud data including a plurality of points in three-dimensional space that approximate locations of physical surfaces within the physical space; iteratively selecting points from the plurality of points along an x-dimension, a y-dimension, and a z-dimension; and clustering the selected points into one or more two-dimensional ("2D") histograms. The embodiments may also include: determining a slope value for each of the 2D histograms; removing points from the point cloud data based on a determination that the slope value exceeds a predetermined slope value; iteratively voxelizing each of the 2D histograms into a predetermined grid size; summing the points in each of the voxelized 2D histograms based on identifying a center point in each of the voxelized 2D histograms; and removing points from the point cloud data based on a determination that the sum of the points in each of the voxelized 2D histograms is below a predetermined sum value. The embodiment further includes: determining the number of points in each of the voxelized 2D histograms; determining whether the number of points in each of the voxelized 2D histograms exceeds a predetermined threshold; and in response to determining that the number of points in each of the voxelized 2D histograms exceeds the predetermined threshold, retaining the center point in each of the voxelized 2D histograms. The embodiment includes: selecting a point from the center points for each of the voxelized 2D histograms; identifying a nearest neighbor point in the point cloud data for the point; removing the identified nearest neighbor point from the point cloud data; and returning, by the computing system, the remaining points in the point cloud data.

[0013] In some implementations, a particular embodiment may optionally include one or more of the following features. For example, the embodiment may further include: performing a standard distribution on the density of each point in the voxelized 2D histogram; and determining whether the standard distribution of the density of each point in the voxelized 2D histogram exceeds a predetermined density distribution value. The embodiment may include: removing points from the voxelized 2D histogram of the point cloud data based on determining that the standard distribution of the density of each point in the voxelized 2D histogram exceeds the predetermined density distribution value; and adding points to the voxelized 2D histogram of the point cloud data based on determining that the standard distribution of the density of each point in the voxelized 2D histogram is below the predetermined density distribution value. In some implementations, adding points to the voxelized 2D histogram of the point cloud data may include applying a k-means filter. Removing points from the voxelized 2D histogram of the point cloud data may include applying a slicing filter.

[0014] As another example, points removed from the point cloud data based on determining that the sum of points in each of the voxelized 2D histograms is below a predetermined sum value can be outliers. The outliers may not include the physical surface within the physical space. The predetermined grid size can also be determined at least in part based on the number of points in the point cloud data and the type of the physical surface within the physical space.

[0015] In some implementations, the embodiment may include: aligning the point cloud data along an x-dimension, a y-dimension, and a z-dimension corresponding to the orientation of the physical space represented by the point cloud data; classifying the points in the point cloud data as corresponding to one or more types of physical surfaces; identifying specific physical structures in the physical space based at least in part on the classification of the points in the point cloud data; and generating, by the computing system, a map of the physical space based on the specific physical structures within the orientation of the physical space and the corresponding contours of the specific physical structures.

[0016] The embodiment may also include: for each of the 2D histograms, identifying a number of nearest neighboring points; and determining whether the number of nearest neighboring points includes a predetermined number of recurring points. The recurring points may be located in the same position along the x-dimension, the y-dimension, and the z-dimension. The embodiment may also include: based on determining that the number of nearest neighboring points includes the predetermined number of recurring points, generating a point near the recurring points of the nearest neighboring points.

[0017] In some implementations, the physical surfaces within the physical space may include pallets, racks, shelves, buildings, houses, trees, plants, and roads. The physical space may also include warehouses, storage facilities, and outdoor environments.

[0018] The disclosed techniques may provide numerous advantages. For example, the disclosed techniques may provide filtering methods that result in more accurate and refined point clouds. These more accurate and refined point clouds may be used to generate more accurate blueprints and / or maps of physical spaces. These more accurate and refined blueprints and / or maps may be used to identify objects and features within the physical space.

[0019] As another example, the disclosed technology can automate the process of generating a blueprint and / or map of the physical space by using filtered and refined point clouds. Automating the filtering, refinement, and generation processes can help mitigate or otherwise eliminate human error in manually performing these processes. Automation can also reduce the amount of time required to perform the above processes. The disclosed technology can further allow for continuous updating and / or restarting of the blueprint and / or map without the need for human intervention. Thus, the disclosed technology can be used to more accurately determine process flow, movement, and / or other characteristics unique to the physical space when the physical space is occupied and / or used for its intended purpose.

[0020] Furthermore, the disclosed techniques can fill gaps in missing data, which can be beneficial in subsequent calculations attempting to classify and locate objects in the point cloud. Filling the gaps can also benefit rendering software or other tools used to create 3D images or simulations from the point cloud data. The disclosed techniques can provide interpolation of point clouds, as gaps can be filled with missing data points. As described herein, interpolation can extend points into areas of the point cloud where no points previously existed. This interpolation can be performed on a small scale to avoid creating spurious or unrealistic surfaces. Thus, using the disclosed techniques, point clouds can be refined and used to more accurately identify objects and features within the physical space.

[0021] The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1A Depicting an exemplary point cloud filtering system.

[0023] Figure 1B Depicts an exemplary system diagram of the disclosed invention.

[0024] Figure 2A is a flow chart of the process for filtering a point cloud.

[0025] Figures 2B to 2C Depicts exemplary results of filtering a point cloud using a histogram filter and a k-means filter.

[0026] Figure 3A It is a flow chart of the process of histogram filtering algorithm.

[0027] Figures 3B to 3C is a graphical depiction of the histogram filter during implementation.

[0028] Figure 4A It is a flow chart of the process of k-means filtering algorithm.

[0029] Figure 4Bis a graphical depiction of the k-means filter during implementation.

[0030] Figure 5 is a flow chart of the process for filtering a point cloud.

[0031] Figures 6A to 6B is a flow chart of the process used to analyze voxels of points within a point cloud.

[0032] Figure 7 is a flow chart of a process for analyzing density distribution within a point cloud.

[0033] Figures 8A to 8B It is a graphical depiction of the density distribution within the analyzed point cloud.

[0034] Figure 9 is a flow chart of the process used to process point clouds.

[0035] Figures 10A to 10E This is an exemplary depiction of voxelizing a point cloud.

[0036] Figure 11 Depicts example point cloud data with k-means centroid points. DETAILED DESCRIPTION

[0037] This document generally describes filtering and thinning point clouds using techniques such as k-means filtering, histogram filtering, and slicing. The refined point clouds can then be used to more accurately identify objects and features within a physical space. Furthermore, the refined point clouds can be used to generate more accurate 3D images or simulations of physical spaces, as well as blueprints and / or maps of such physical spaces.

[0038] Refer to the figures, Figure 1A An exemplary point cloud filtering system is depicted that includes a physical space 100, a point cloud 102, a computer system 104, and a refined point cloud 106. As depicted, the physical space 100 may be a warehouse environment. The physical space 100 may include one or more racks 108A-108N. In other embodiments, the physical space 100 may include one or more other types of physical items in addition to the racks 108A-108N. For example, the physical space 100 may include walls, doorways, doors, tables, support beams, etc. In some embodiments, the physical space 100 may be any type of storage facility (e.g., a freezer). In still other embodiments, the physical space 100 may be a building, a house, other structure, or an outdoor space / landscape. Still referring to Figure 1A, physical space 100 further includes an unmanned aerial vehicle 110 and a stereo camera 112. In some embodiments, physical space 100 may include multiple unmanned aerial vehicles 110 and / or stereo cameras 112. Unmanned aerial vehicles 110 and stereo cameras 112 may be configured to capture one or more 3D scans and / or images of the entire physical space 100. In some embodiments, instead of at least one of unmanned aerial vehicles 110 and / or stereo cameras 112, physical space 100 may employ a person, a forklift, and / or some other type of device to capture 3D scans and / or images of physical space 100.

[0039] Once one or more 3D scans and / or images of physical space 100 are captured, the scans and / or images may be used to generate one or more point clouds 102. One or more point clouds 102 may be generated by UAV 110, a device in physical space 100 used to capture 3D scans / images, and / or any other computer system / computing device in communication with UAV 110, stereo camera 112, and / or computer system 104. One or more point clouds 102 may then be transmitted / communicated to computer system 104. The communications described throughout this disclosure may occur via a network and / or wireless / wired communications (e.g., Bluetooth, WIFI, Ethernet, etc.). In some embodiments, one or more point clouds 102 may be generated by computer system 104.

[0040] At computer system 104, the point cloud 102 may be processed (e.g., filtered, cleaned, refined) using techniques described throughout this disclosure to generate a refined point cloud 106 (see Figures 2-8). Computer system 104 and / or another computer system / computing device in communication with computer system 104 may use the refined point cloud 106 to generate a map of the physical space. The map of the physical space may be an accurate map and / or blueprint that is updated in real time, periodically, and / or automatically.

[0041] Still refer to Figure 1A , the process of scanning physical space 100, generating point cloud 102, processing point cloud 102, and outputting refined point cloud 106 can be performed in real time and / or at different times. For example, a 3D scan and / or image of physical space 100 can be captured at time 1. Point cloud 102 can be generated at time 2. Processing the point cloud at computer system 104 to generate refined point cloud 106 can be performed at time 3. And finally, generating a map of the physical space can be performed at time 4 (not depicted). Moreover, the described process can be performed more than once and can be performed automatically / periodically to ensure that an accurate and updated point cloud of physical space 100 is maintained.

[0042] Figure 1BAs depicted, the 3D scanning device 118, the point cloud filtering system 120, and the point cloud mapping system 170 can communicate via a network 122, such as Figure 1A In some embodiments, the point cloud mapping system 170 may be optionally integrated into the systems described throughout this disclosure. In other words, an exemplary system having the 3D scanning device 118 and the point cloud filtering system 120 may not have the point cloud mapping system 170.

[0043] One or more 3D scanning devices may communicate with the point cloud filtering system 120 and / or the point cloud mapping system 170 via the network 122. The 3D scanning device 118 may be the previously referenced Figure 1A At least one of the UAV 110 and the stereo camera 112 discussed. Furthermore, the filtering system 120 may be Figure 1A In some embodiments, the mapping system 170 may be a computer system 104 disclosed in Figure 1A In yet other embodiments, the filtering system 120 and the mapping system 170 may be integrated into one system, such as Figure 1A The computer system 104 disclosed in .

[0044] The 3D scanning device 118 may include a communication interface 124, at least one stereo camera 126, and optionally a point cloud generator 128. As discussed, the communication interface 124 may facilitate communication between the 3D scanning device 118 and the filtering system 120 and / or mapping system 170 via the network 122. The at least one stereo camera 126 may capture one or more 3D scans and / or other images of a physical space. In some implementations, instead of or in combination with the stereo camera 126, the 3D scanning device 118 may include other types of image capture sensors, cameras, and / or devices configured to capture 3D scans and / or images of a physical space. As mentioned, the 3D scanning device 118 may optionally include a point cloud generator 128 configured to generate a point cloud from the captured 3D scans / images. If the 3D scanning device 118 includes the point cloud generator 128 and generates point clouds, those point clouds 130 may be transmitted / passed to the point cloud filtering system 120 and / or mapping system 170 via the network 122. The point cloud filtering system 120 may then store such point clouds 130 in a point cloud database 152. The point cloud database 152 stores the generated point clouds, as indicated by exemplary point cloud 156. If the 3D scanning device 118 does not generate a point cloud, the 3D scan 132 may be transmitted / passed from the device 118 to the point cloud filtering system 120 via the network 122. In some implementations, the device 118 may transmit the point cloud 130 and the 3D scan 132 to the point cloud filtering system 120.

[0045] Point cloud filtering system 120 can use the received 3D scan to generate one or more point clouds via point cloud generator 148. Therefore, filtering system 120 may optionally include point cloud generator 148. In some implementations, filtering system 120 can receive point cloud 130 and 3D scan 132 from 3D scanning device 118 and generate additional point clouds based on received 3D scan 132. Doing so can increase the accuracy of the point cloud, which can further reduce the amount of filtering / refinement required to generate an accurate blueprint and / or map of the physical space.

[0046] Still refer to Figure 1B The point cloud filtering system 120 may include a communication interface 134, a point cloud refinement module 136, and the previously mentioned optional point cloud generator 148. The filtering system 120 may communicate with the 3D scanning device 118, the mapping system 170, the point cloud database 152, and / or any additional computer systems via the communication interface 134 and over the network 122.

[0047] Once the point cloud is generated by the filtering system 120 (via the point cloud generator 148) and / or the point cloud 130 is received from the 3D scanning device 118, the point cloud may be refined at the point cloud refinement module 136, as further described throughout this disclosure. The point cloud refinement module 136 may include a histogram filter 137 (see Figures 3A to 3C ), density determination module 138 (refer to Figures 6 to 7), k-means filter 139 (e.g., k-guided filter) (refer to Figures 4A to 4B , 8A to 8B) and slice filter 140 (refer to Figure 7 8). Filters and modules 137, 138, 139, and 140 may be used individually and / or in combination to clarify and clean the point cloud. A histogram filter 137 may be used to cluster / voxelize points in the point cloud and remove outlier points from the point cloud. A density determination module 138 may be used to perform a statistical distribution of the density of points in the point cloud to determine whether portions of the point cloud need to be upsampled and / or downsampled. Based on the determination made by module 138, a k-means filter 139 may be applied to portions of the point cloud to upsample such portions, and / or a slicing filter 140 may be applied to portions of the point cloud to downsample such portions. The refined point cloud may then be stored in a point cloud database 152. The database 152 may store a plurality of refined point clouds 156.

[0048] An optional point cloud mapping system 170 can communicate with the filtering system 120 via the network 122 to receive the refined point cloud. The mapping system 170 can include a communication interface 172, a physical structure classification module 174, a confidence score module 178, an optional point cloud generator 180, and an optional rack detection module 182. In some implementations, such as when the mapping system 170 communicates directly with the 3D scanning device 118, the point cloud generator 180 can generate the point cloud based on the 3D scan 132 received from the device 118. In other implementations, the mapping system 170 can receive the generated and refined point cloud from the filtering system 120. The mapping system 170 can also communicate with the generated map database 154. Maps / blueprints generated by the mapping system 170 and based on the point cloud can be stored in the database 154. An exemplary generated map 158 can include the refined point cloud 160, the physical structure 162, and the confidence score 164.

[0049] Still referring to mapping system 170, upon receiving the refined point cloud from filtering system 120, the refined point cloud may be processed by physical structure classification module 174 to identify and classify any physical structures / items / objects in the point cloud. Module 174 may include a spherical filter 176 for identifying and classifying physical structures. A confidence score module 178 may further be used to determine the scores of the identified physical structures and the overall accuracy of the refined point cloud. Finally, a rack detection module 182 may be used to identify vertical poles and racks from the point cloud. Module 182 is advantageous for identifying racks in warehouse environments or other types of storage facilities. In other embodiments, module 182 may be directed to identifying other types of items / objects located within a physical space, such as trees or roads in outdoor environments. Once any determinations are made (e.g., the refined point cloud, the final map, the classification and scoring of physical structures, etc.), such determinations may be stored in generated map database 154. Database 154 may store multiple generated maps 158.

[0050] Figure 2A is a high-level flow chart of a process 200 for filtering a point cloud. The process 200 may be performed by, for example, the computer system 104 (see Figure 1A ) and / or point cloud filtering system 120 (reference Figure 1B ) is executed by a computing device. First, the computer system converts the histogram filter ( Figures 3A to 3B The histogram filter is useful for removing any noise or outliers from the point cloud. The computer system may also apply a k-means filter ( Figures 4A to 4B ) is applied to the output of the histogram filter (step 204). The k-means filter is useful for further refining the point cloud by adding points to portions of the point cloud that indicate objects / items within the physical space.

[0051] In some implementations, steps 202 and 204 may be performed in any order and / or simultaneously. In some implementations, only one of steps 202 and 204 may be performed. For example, as shown in FIG. Figure 5 、 7 As described in Figures 2A and 2B, a histogram filtering technique may be used to analyze the point cloud (step 202), and then a k-means filter (step 204) and / or a slicing filter may be applied to upsample and / or downsample the point cloud, respectively. In some implementations, the k-means filter may be applied to portions of the point cloud, while the slicing filter may be applied to other portions of the point cloud. Using both the k-means filter and the slicing filter is beneficial in ensuring that the point cloud has an optimal distribution of points and that those points accurately represent objects / items within the physical space.

[0052] Figures 2B to 2C Describe the use of histogram filters and k-means filters (refer to Figure 2A ) Demonstration results of filtering the point cloud. Figure 2B Depicting a 3D scan of the outdoor physical space before detailing. Figure 2B The 3D scan of FIG. 1 is of a street with buildings, parking lots, houses, etc. As shown, the scan is not fine or detailed enough to identify what each of the objects and / or shapes in the scan is. Figure 2C Depicted is the same 3D scan after being refined using filtering techniques such as histogram filtering and k-means filtering. As a result of this filtering / refining, objects in the 3D scan / each of the scans can be more easily identified.

[0053] The different colors in the scan represent the confidence score for each of the recognized objects / shapes. For example, Figures 2B to 2C The purpose of a 3D scan in may be to identify buildings and houses for a land survey project. When an object / shape in this scan is red, it is more likely to be one of the things being identified (e.g., 90% accurate identification after filtering). Figure 2C In the example of , red squares and rectangles represent buildings and homes. Objects, shapes, and / or spaces that appear in green are less likely to be one of the things being recognized (e.g., 80% accurate recognition after filtering). And, objects, shapes, and / or spaces that appear in blue are even less likely to be one of the things being recognized (e.g., 50% accurate). Thus, streets, parking lots, and other open spaces are Figure 2C The image is shown in blue because it is least likely to represent buildings and residences.

[0054] By using the 3D scanning and filtering techniques discussed throughout this disclosure in different settings, it is possible to obtain Figures 2B to 2CFor example, a 3D scan of a room in a building, warehouse, and / or any other physical environment can be captured to obtain comparable results.

[0055] Figure 3A is a flow chart of a process 300 of a histogram filtering algorithm. Figures 3B to 3C is a graphical depiction of the histogram filter during implementation. Specifically, Figure 3B is a graphical depiction of histogram filtering before downsampling the point cloud. Figure 3C is a graphical depiction of histogram filtering after downsampling. As described herein, a histogram filter is a noise reduction filter that can be used to remove points in a point cloud that may not be part of a larger, expanded object. Thus, a histogram filter can be used to remove blur and random noise while retaining points found in objects and features in the environment. Applying a histogram filter may not result in a significant reduction in the number of points in the point cloud.

[0056] Process 300 may be performed by, for example, computer system 104 (see Figure 1A ) and / or point cloud filtering system 120 (reference Figure 1B ) can be performed by any computing device. Histogram filtering facilitates downsampling of point cloud data. In other words, histogram filtering facilitates removing noise from the point cloud and points that are most likely not associated with the features / physical objects that the computer system is trying to identify. In some embodiments, histogram filtering is separate from slice filtering (see Figure 5 、 7 That is because after removing noise using the histogram filtering technique disclosed herein, the slice filter can be applied to the point cloud.

[0057] As an example, in a warehouse, a histogram filter can be applied to the point cloud to remove points that do not appear frequently and therefore do not indicate racks and / or vertical poles. The more points that fill a particular area in the point cloud, the more likely all of those points will remain in the point cloud. The histogram filter does not add points. Then, as described throughout this disclosure, a k-means filter (see Figures 4A to 4B ) can be applied to the remaining part of the point cloud where objects are located, but there are not enough points to indicate such objects. Slice filter (refer to Figure 7 、 Figures 8A to 8B ) can also be applied to portions of the remaining point cloud where objects are located, but too many points occupy the area representing such objects. Using all of the techniques described in this disclosure can result in the generation of a refined point cloud having a predetermined number and / or density of points, rather than having too many and / or too few points to represent an object in the point cloud.

[0058] Return Reference Figure 3AFirst, the computer system selects one of the X-axis, Y-axis, and Z-axis (step 302). Figure 3B As depicted in FIG, the X axis is selected. Once the axis is selected, the points in the point cloud associated with the selected axis are isolated (step 304). Figure 3B As shown in FIG, the plurality of points isolated on the graph are associated with the X axis. Next, in step 306, the computer system generates segments / groups for the points associated with the selected axis. Figure 3B , resulting in three segments / groups / clusters: Group 350A, Group 350B, and Group 350C. Group 350A is associated with any point that appears in the first 1 / 3 of the graph. Group 350B is associated with points in the second 1 / 3. Group 350C is associated with points in the third 1 / 3 of the graph. Groups 350A through 350C create a Figure 3B The overall curve is depicted below the graph in . Figure 3B As shown in , groups with more points in an associated region of the graph have higher peaks in the overall curve than other groups. Thus, group 350B associated with the second third of the graph has the highest peak because the largest amount / number of points (e.g., density) is located in the second third of the graph. If a region has more points, then those points are less likely to be removed from that region of the point cloud.

[0059] Return Reference Figure 3A Once the segments (e.g., groups, clusters) are generated along the selected axis (step 306), a segment can be selected in step 308. Figure 3B In the example of , group 350B may be selected. The computer system then determines the slope value of the selected segment (step 310). Figure 3B In the example of , the slope of group 350B is likely to be zero or close to zero because there are many points in that area of ​​the graph, thereby producing the highest, flattest peak in the entire curve. In step 312, after determining the slope, the computer system immediately determines whether the magnitude of the slope is below a predetermined threshold. A slope magnitude below the threshold indicates that the associated area on the graph has a high concentration / large number of points that should not be removed from the point cloud. On the other hand, a slope magnitude above the threshold indicates that the associated area on the graph has a low concentration / smaller number of points that may be more closely associated with noise rather than physical objects (e.g., vertical poles / racks in a warehouse).

[0060] exist Figure 3B In the example of , the threshold value may be 1. As a result, since the slope of 350B is 0, the slope is lower than the threshold value. Therefore, returning to the reference Figure 3A, the computer system will designate the points along that segment (group 350B) for filtering (step 314). In other words, the points in group 350B should not be removed. On the other hand, if the slope is greater than the threshold, then the segment / group has fewer, potentially scattered, and more likely noisy points. Regardless of whether the computer system determines in step 312 that the slope is greater than the threshold, or whether points along the segment / group are designated for filtering in step 314, the next step is to determine whether there are more segments / groups along the selected axis (step 316). If there are more segments / groups along the selected axis, then the computer system repeats steps 308 to 314 for the remaining segments / groups of the selected axis. Figure 3B In step 310, the computer system may identify the slope values ​​for groups 350A and 350C. With respect to both groups 350A and 350C, the computer system may determine that the associated slope values ​​are greater than a threshold value (step 312), thereby indicating that both groups 350A and 350C do not have many points and do not have points that indicate physical objects, such as vertical poles / racks. Therefore, the points in groups 350A and 350C are not designated for filtering (step 314), but rather indicate noise to be removed.

[0061] Once steps 308 to 316 are completed for each segment / group, and no more segments / groups remain, the computer system determines if there are more axes (step 318). If there are more axes, then steps 302 to 318 are repeated for each axis identified. Figure 3B In the example of , the computer system performs steps 302 to 318 for the Y axis and the Z axis. Once there are no more axes in step 318, the computer system identifies the points that have been assigned for all axes in step 320. The assigned points are retained. Therefore, in step 322, the computer system removes all unidentified / assigned points. Figure 3B and 3C In the example of , the computer system removes points 360A to 360C, all of which are associated with groups 350A and 350C having slope values ​​above the threshold. Figure 3B The histogram filter before downsampling the image in

[15] demonstrates scanning a point cloud before removing any points. Figure 3C The histogram filter after downsampling the image in FIG. 5 illustrates the point cloud scan after removing points (eg, points 360A to 360C).

[0062] The histogram filter (process 300) facilitates the removal of points from the point cloud that do not belong to or are not associated with a common grouping / segment. In the example of a land survey, points can be grouped based on height (steps 306 to 310). As a result, some buildings and / or trees above a certain height (threshold) may not belong to a cluster or common group with many points (step 312). Those buildings and / or trees above the threshold can be marked as noise and removed from the point cloud (step 322). On the other hand, buildings of the same height can be grouped together into a common group / cluster and thus designated as points to be filtered, especially if their height is below the threshold (steps 312 to 314). In the example of a land survey, homes, buildings, parking lots, and / or roads at or near the same height or other value are most likely to appear in a common group / cluster and are retained for further filtering, while physical objects such as trees and / or overpasses are most likely to be outliers marked as noise and therefore removed from the point cloud. As described throughout this disclosure, once noise is removed from a point cloud, the remaining points may be further refined using k-means filtering and / or slicing techniques disclosed herein.

[0063] Figure 4A is a flow diagram of a process 400 of a k-means (eg, k-guided) filtering algorithm. Figure 4B is a graphical depiction of a k-means filter during implementation. The k-means filter may be applied to a point cloud to enhance features or objects that are left alone after the histogram filter is applied. Thus, the k-means filter may include generating points to enhance such features or objects. Process 400 may be performed by, for example, computer system 104 (see Figure 1A ) and / or point cloud filtering system 120 (reference Figure 1B ) to be executed by any computing device.

[0064] The k-means filter is useful for cleaning up the point cloud and adding / interpolating points between existing points (e.g., upsampling), so that the point cloud can then be used for more accurate detection of physical items and mapping of physical space. In some embodiments, the k-means filter can be applied to the entire point cloud. In other embodiments (e.g., see Figures 8A to 8B ), the k-means filter can be applied to only part of the point cloud. As mentioned throughout this disclosure, the k-means filter can be used in conjunction with the histogram filter described above (refer to Figures 3A to 3C ) and slice filters (reference Figure 7 、 8A 8B). In other embodiments, the k-means filter can be the only filter used. As described, the k-means filter is a form of upsampling, and the k-means method can be implemented to complete the interpolation of the filter.

[0065] In an exemplary embodiment of land surveying, a k-means filter can be used to fill in points that may be missing from the original 3D / Lidar scan. Thus, buildings, roads, houses, and / or other structures in the scan can be made clearer by adding points where those points may appear. Furthermore, in any embodiment, the k-means filter can have parameters that allow the user and / or computer system to determine how many points to add to the point cloud. For example, the parameters may require filling the point cloud with an additional 10 million points, where those 10 million new points are not placed above existing points in the scan. 10 million points can be added around the most populated area in the point cloud scan.

[0066] refer to Figure 4A In step 402, an axis is selected. A one-dimensional projection of the selected axis is presented. Figure 4B As demonstrated in FIG, the X axis is selected. Next, the computer system isolates all points in the point cloud with respect to the selected axis (step 404). Then, in step 406, the points are clustered into a threshold number of groups using a k-means mathematical method. The threshold number of groups may be determined by a user at a computing device, a computer system, and / or a device in communication with the computer system. Figure 4B In the example of , the threshold number of groups / clusters is 3. There are three clusters: cluster 450A is associated with the first 1 / 3 of the graph, cluster 450B is associated with the second 1 / 3, and cluster 450C is associated with the third 1 / 3. Figure 4B As depicted in the first graph in , three clusters are also identified along the Y-axis.

[0067] Next, in step 408, a cluster may be selected. The computer system then identifies a threshold number of nearest neighbors to the selected cluster (step 410). This threshold number may be determined by the computer system and / or a user at a device communicating with the computer system. In other embodiments, the computer system may define the number of nearest neighbors to be identified. Once the nearest neighbors are identified, the computer system determines whether the number of points that are repeated as nearby neighbors is greater than a threshold (step 412). When the same point is located at the same position on the X-axis, Y-axis, and Z-axis, it may be considered that the point is repeated. In other words, the same point is generated over and over again, which not only indicates that the point is repeating, but also indicates that the point is common and should be retained in the point cloud. Points may then be interpolated around that repeated point on at least one of the X-axis, Y-axis, and Z-axis.

[0068] Still referring to step 410, a threshold value can be associated with a plurality of best repeated points. For example, in a point cloud of 100 points, the 100 points can be clustered into 3 clusters, and then the 10 nearest neighbors for each of the 3 clusters can be identified. When reviewing the 10 nearest neighbors for a cluster, the computer system may only need to retain the 5 best, repeated nearest neighbors (of the 10 nearest neighbors). Retaining not all nearest neighbors is beneficial because some neighbors may not reoccur, repeat, or be close enough to indicate a vertical pole or some other identifiable physical object / item. Therefore, overlapping, reoccurring, and / or repeated points are more likely to indicate a physical object / item to be retained in the point cloud.

[0069] Referring back to step 412, if there are recurring points that are near neighboring points and are above a threshold, the computer system generates (e.g., interpolates) points near those recurring points (step 414). Points are generated at or around the cluster that is the most common grouping. If the number of recurring points that are near neighboring points in step 412 is not above the threshold, the points are too scattered to be physical items / objects, and the computer system will not add / interpolate points. Instead, it will determine whether there are more clusters to investigate (step 416). If there are more clusters, the computer system returns to step 408 and repeats the steps until no more clusters exist for the selected axis. Once no more clusters exist, the computer system determines whether there are more axes in step 418. If there are clusters, the computer system returns to step 402 and repeats the steps discussed previously. Once no more axes exist, the computer system identifies the points generated from step 414 (step 420). In step 406, the identified generated points may be clustered into a threshold number of groups, and steps 406 through 420 may be repeated until all points are interpolated and / or analyzed.

[0070] like Figure 4B As demonstrated in , each of the points pulled from the graph and placed outside the X-axis and Y-axis lines represents the central grouping of each cluster. Cluster 450B has the highest peak on the curve depicted below the X-axis line because most of the points in this point cloud are concentrated in the second 1 / 3 of the graph. In other words, the second 1 / 3 of the graph (cluster 450B) has the most recurring points and is therefore more likely to indicate a vertical pole or another physical item / object in a warehouse environment. Cluster 450B is a common grouping and the points of the point cloud can be interpolated (refer to Figure 4B A portion of the added points 460A to 460C) in the k-means filter after the interpolated image is shown in FIG. Then, as shown in FIG. Figure 4A As mentioned, via the k-means filter, the interpolation process may be repeated for each point in the second 1 / 3 of the graph added to the X-axis.

[0071] Figure 5 is a flow chart of a process 500 for filtering a point cloud. The process 500 may be performed by, for example, the computer system 104 (see Figure 1A ) and / or point cloud filtering system 120 (reference Figure 1B First, in step 502, the computer system may receive a point cloud (refer to Figures 1A to 1B ). Next, in step 504, the computer system may apply a histogram filter, as described throughout this disclosure (see Figures 3A to 3C ). As mentioned, the purpose of step 504 is to clean up the point cloud by removing excess / outlier points in the point cloud that do not constitute one or more objects / items of interest. Using the described histogram filtering techniques can make the application of additional refined filters, such as k-means filters and / or slicing filters, more accurate and facilitate the identification of objects / items within the point cloud. In some embodiments, step 504 may not be performed. In other embodiments, step 504 may be performed simultaneously or after either the k-means filter or the slicing filter.

[0072] In step 506, a k-means filter may be applied to the upsampled portion of the point cloud (see Figures 4A to 4B ). In step 508, a slicing filter may be applied to the downsampled portion of the point cloud (refer to Figure 7 、 8A 8B). In some embodiments, the k-means filter and the slice filter may be applied to the point cloud simultaneously. The k-means filter may be applied to a specific portion of the point cloud, while the slice filter may be applied to other portions of the point cloud (see Figures 8A to 8B ). In other embodiments, only one of the k-means filter and the slice filter may be applied to the point cloud. The computer system may determine which filter should be applied to the point cloud based on statistical distribution analysis, as described herein (with reference to Figures 6-7).

[0073] Figures 6A to 6B 6 is a flow chart of a process 600 for analyzing voxels (eg, groups, clusters) of points within a point cloud. The process 600 may be performed by, for example, computer system 104 (see Figure 1A ) and / or point cloud filtering system 120 (reference Figure 1B ) can be performed by any computing device. Process 600 can be a form of histogram filtering implemented to remove noise and analyze point density in a point cloud (see Figure 5 In some embodiments, the process 300 (see step 504) may be combined with Figures 3A to 3C) to perform process 600. In other embodiments, process 600 may be performed in place of process 300. In still other embodiments, process 600 and process 300 may be combined into a technique performed by a computer system to reduce redundancy and improve efficiency in analyzing and refining point clouds.

[0074] In order for the computer system to determine whether to apply the k-means filter and / or the slice filter, the computer system must analyze the density of the points in the point cloud. Analyzing the point density can be done by analyzing the statistical distribution of the voxel density of the points in the point cloud (see Figure 7 Statistical distribution analysis is performed on voxels that have a sufficient number of points that are sufficiently far from each other and from the center of the voxel. Statistical distribution analysis is not performed on voxels that have too few, overly dispersed points, because such points are more likely to be outliers or not indicate an object / item within the point cloud. Before the statistical distribution analysis can be performed, the computer system must determine which voxels of points to retain in the point cloud, as described with reference to process 600.

[0075] refer to Figures 6A to 6B In both processes 600, first, in step 602, the computer system receives a point cloud. In some embodiments, before applying the reference Figures 3A to 3C Following the described histogram filtering process 300, the computer system may receive a point cloud. In other embodiments, the computer system may receive a point cloud that has not yet been refined using any of the techniques disclosed herein. Next, in step 604, the computer system selects an axis. In this step, the entire segment / group of points is compressed into a 2D histogram. Once the axis is selected, the computer system voxelizes the resulting point cloud into a predetermined grid size (step 606). In some embodiments, the grid size may be determined by the computer system. The computer system may make this determination using predictive analysis based at least in part on the number of points in the point cloud, the size of the point cloud, and / or other characteristics of the point cloud. In other embodiments, the computer system may receive a user-determined grid size from a user at a user input device.

[0076] Furthermore, in some embodiments, step 606 may be repeated a predetermined number of times based on how many different grid sizes are determined and applied to the point cloud. Repeating process 600 with different grid sizes may result in more accurate refinement of the point cloud. This is because different grid sizes may be used to identify or detect objects / items of different sizes / shapes within a physical space. For example, a smaller grid size may be used to detect whether a point in a point cloud represents a house, while a larger grid size may be used to detect whether a point in a point cloud represents a large commercial building.

[0077] Still refer to Figures 6A to 6B In step 608, the computer system selects a voxel. Then, in step 610, the computer system determines the number of points within the voxel.

[0078] In step 612, the computer system may determine whether the number of points in the voxel is greater than or equal to a predetermined threshold. In step 614, if the number of points is greater than or equal to the predetermined threshold, the computer system may retain the center voxel point. In step 616, if the number of points is less than the predetermined threshold, the computer system may determine whether more voxels exist.

[0079] If there are more voxels, the computer system may return to step 608 and repeat the steps described above. If there are no more voxels, the computer system may determine whether there are more axes (step 618). If there are more axes, the computer system may return to step 604 and repeat steps 604 to 618 until there are no more axes to analyze. In some embodiments, the computer system may repeat step 606 by iteratively increasing the predetermined grid size. The grid size may increase with each pass through steps 604 to 624 per axis. In other embodiments, the grid size may be increased a predetermined number of times per specific axis. As previously discussed, iteratively increasing the grid size helps ensure that the computer system retains points that identify objects / items within the point cloud. Iteratively increasing the grid size also helps ensure that the computer system does not mistakenly remove points that are actually related to the object / item of interest within the identified point cloud.

[0080] If no more axes exist, the computer system can proceed to step 620. In step 620, the computer system can determine whether more grid sizes exist. At this point, the computer system can perform the step of downsampling the point cloud y by a tuned threshold. In other words, if a point exceeds the voxel size threshold, that point can be used to remove its nearest neighbors from the original point cloud. If the tile filter requires additional grid sizes, the new downsampled point cloud can be used. Therefore, if no more grid sizes exist at 620, process 600 can stop.

[0081] In step 622, a point can be selected from the filtered points (e.g., see step 614). The nearest neighboring points of the selected point in the original, unfiltered point cloud can be identified (step 624). In step 626, the identified points can be removed. Such points can be removed to remove features from the point cloud that have been captured by previous executions of the slicing filter and the techniques described herein. For example, each iteration of the slicing filter can be tuned to identify one or more specific features. When one or more specific features are identified, such features can be removed from the point cloud so that the next, potentially less dense, feature can be identified.

[0082] In step 628, the computer system may determine whether there are more points to select from the filtered points. If there are more points, the computer system may return to step 622 and repeat steps 622 through 628. Repeating steps 622 through 628 facilitates identifying and extracting specific features within the point cloud. With each pass through steps 622 through 628, additional features may be extracted from the point cloud and / or the features may become more distinct. Furthermore, the number of points used to identify specific features may be reduced or refined so that an excessive number of points does not fill the point cloud. In steps 622 through 628, all points that have been filtered using the previously described slice filter may be used as nearest neighbor calculation points.

[0083] Once there are no more points to analyze, the computer system may return the remaining points at 630. The remaining points may constitute a subsampled point cloud.

[0084] Figure 7 7 is a flow chart of a process 700 for analyzing density distribution in a point cloud. The process 700 may be performed by, for example, the computer system 104 (see Figure 1A ) and / or point cloud filtering system 120 (reference Figure 1B Process 700 facilitates a computer system to determine whether to apply a k-means filter and / or a slicing filter to upsample and / or downsample points in a point cloud, as described throughout this disclosure.

[0085] First, in step 702, the computer system receives a subsampled point cloud (refer to Figures 6A to 6B ). This point cloud has been refined using the histogram filtering techniques described throughout this disclosure. Thus, noise (e.g., sparse / outlier points) has been removed, and only points within the point cloud that are relevant to the identified object / item remain.

[0086] In step 704, the computer system performs a statistical distribution analysis on the voxel density (e.g., referring to Figures 6A to 6B Performing statistical distribution analysis on the voxel density results in the creation of a graph (see process 600 in FIG. Figures 8A to 8B ), where the Y axis of the graph represents the density of points and the X axis represents the number of voxels with that density. Furthermore, the direction of the axis being viewed can change which axis is used for density and which axis is used for voxels. Thus, the graph axes can be determined relative to the user's viewing angle. In step 706, after performing the statistical distribution analysis, the computer system determines whether the resulting density distribution is greater than a predetermined threshold. The threshold can be predetermined by the computer system and / or user input, as described throughout this disclosure. In other embodiments, the threshold can be determined when the statistical distribution is performed. For example, the results of the statistical distribution can be analyzed to identify an optimal minimum density level in a particular point cloud. As Figures 8A to 8BAs depicted in , the flat upper portion of the bell-shaped curve (e.g., desired density 806) can indicate an optimal minimum density level for the point cloud. Thus, the flat upper portion of the bell-shaped curve can be identified as a threshold density value. That threshold can then be used to determine which portions of the curve (e.g., the point cloud) need to be upsampled and / or downsampled in order to achieve a uniform minimum density level.

[0087] Still refer to Figure 7 If the density distribution is greater than a predetermined threshold, then portions of the point cloud with a larger distribution may be downsampled using a slicing filter (step 708). In other words, portions of the point cloud with too many points, or portions with a number of points exceeding the number required for the identified optimal minimum density level, may be further refined by removing points and / or compressing points within those portions of the point cloud. On the other hand, if the density distribution is less than a predetermined threshold, then portions of the point cloud with a smaller distribution may be upsampled using a k-means filter (step 710). In other words, portions of the point cloud with too few points, or portions with a number of points not equal to the number required for the identified optimal minimum density level, may be further refined by adding points to fill gaps within those portions of the point cloud.

[0088] Figures 8A to 8B It is a graphical depiction of the density distribution within the analyzed point cloud. Figure 8A A point cloud 800A before upsampling and downsampling is depicted, along with an associated subsampled density distribution graph 802A for point cloud 800A. As depicted in point cloud 800A, points are scarcely distributed along the X-axis and the Y-axis. This sparse distribution of points is further depicted in density distribution curve 804 of graph 802A. The X-axis of graph 802A represents the number of voxels having each of the determined voxel densities. The Y-axis of graph 802A represents the density value.

[0089] In graph 802A, the desired density 806 is the reference Figure 7 The predetermined threshold discussed in step 706. The desired density 806 is at the peak of the density distribution curve 804 because the desired density 806 indicates the optimal minimum number of points (e.g., point density) within the point cloud 800A. The desired density 806 represents the removal of high-density regions of the point cloud 800A, thereby helping to make the points of the point cloud 800A uniform. As depicted, the k-means filter performs an upsampling filter on the low-density regions (reference 810), and the slice filter performs a downsampling filter on the high-density regions (reference 820) to generate the fitted curve 830 (reference 830). Figure 8B ). Applying these filters may move the points in point cloud 800A to a certain density, which allows point cloud 800A to become more uniform.

[0090] The portion of curve 804 to the left of desired density 806 indicates a portion of point cloud 800A where points are scarcely distributed (e.g., less densely populated). In such a portion, k-means filter 810 is applied for upsampling. The portion of curve 804 to the left of desired density 806 indicates a portion of point cloud 800A where points are densely distributed (e.g., more densely populated). In such a portion, slicing filter 820 is applied for downsampling.

[0091] Now refer to Figure 8B , Figure 8B A graph 802B depicts a point cloud and its corresponding filtered density distribution after upsampling and downsampling 800B. Graph 802B includes a filtered curve 830, shown as a solid line, and the original density distribution curve 804, shown as a dashed line. Applying a k-means filter 810 to the portion of curve 804 to the left of the desired density 806 results in upsampling (e.g., adding points) of that portion of curve 804, as depicted in filtered curve 830. Consequently, the left portion of curve 830 has a density closer to the left of the desired density 806 (e.g., graphically, the left portion of curve 830 is raised). Applying a slicing filter 820 to the portion of curve 804 to the right of the desired density results in downsampling (e.g., removing points) of that portion of curve 804, as depicted in filtered curve 830. Consequently, the right portion of curve 830 has a density less than that to the right of the desired density 806 (e.g., graphically, the right portion of curve 830 is lowered).

[0092] High density areas may escape the k-means filter 810 because too many points may be selected in too small a area. Therefore, it may be necessary to upsample small density areas of the point cloud and downsample large density areas. Low density areas and high density areas may be detected based on the error of the k-means filter, the distance of the centroid point to the points within the group of centroid points, and whether the group of centroid points is lacking or rich in points. Therefore, in certain areas of the point cloud, more points may be created when they are needed, and fewer points may be created when they are not needed. Also, points may not be added to the edge of the point cloud because a group of points may not exceed a certain distance indicating an edge (see Figure 11 ).

[0093] Finally, point cloud 800B after upsampling and downsampling depicts a more refined point cloud, where points have been added, removed, and compressed together to produce a cleaner version of the point cloud than that depicted in 800A.

[0094] Figure 9 is a high-level flow chart of a process 900 for processing a point cloud. The process 900 may be performed by, for example, the computer system 104 (refer to Figure 1A), point cloud filtering system 120 and / or point cloud mapping system 170 (refer to Figure 1B ) can be performed by any computing device. In some embodiments, some steps (e.g., step 904) can be performed by the point cloud filtering system 120, as depicted and described throughout this disclosure (see Figure 1B In some embodiments, some steps (e.g., steps 906 to 912) may be performed by the point cloud mapping system 170, as described in reference to FIG. Figure 1B In yet other embodiments, process 900 can be performed by a computer system.

[0095] In an exemplary warehouse environment, a point cloud can be used to map the warehouse environment. As a result, identifying objects in the warehouse (e.g., vertical bars for shelves) can help determine the layout of the warehouse. First, in step 902, the computer system receives a point cloud. As previously mentioned (refer to Figures 1A to 1B ), a computer system may receive a point cloud from an unmanned aerial vehicle capturing a 3D scan of a physical environment. In a warehouse, the new point cloud scan may include at least one of walls, floors, ceilings, racks, pallets, doors, light fixtures, vertical support poles, people, forklifts, noise, random variations, obstructions, and the like. For example, in an outdoor space, the new point cloud scan may include at least one of trees, shrubs, buildings, residences, parking lots, people, highways, overpasses, bridges, lakes, rivers, and the like. The new point cloud scan is randomly aligned in space, and typically the floor of the physical environment is nearly parallel to the XY-plane. Typically, the point cloud scan includes noise, which may include several points from other nearby locations and / or physical objects in the room. As a result, this noise increases the bounding box of the physical environment beyond the walls or other existing perimeters.

[0096] Next, the computer system refines and / or filters (e.g., cleans) the point cloud (step 904), as described throughout this disclosure (see Figures 2-8). The computer system uses several filters that can reduce the variation in point positions, such as guided filters (e.g., k-means filters, slicing filters). Guided filters can make surfaces cleaner, sharper edges, and remove outlier noise. Using this type of filter as a first step can improve the reference Figure 9 The accuracy of all point cloud processing discussed. Exemplary filters used by the computer system include histogram filters ( Figures 3A to 3B ) and the k-means filter ( Figures 4A to 4B Alternatively, if guided filters are not used in the step of filtering and cleaning the point cloud data (904), an outlier filter can be used to remove some of the noise in the point cloud scan. For each noise detection, the confidence score / value associated with the identified physical object is reduced accordingly.

[0097] Then, in step 906, the computer system may map the point cloud. Mapping the point cloud may include aligning the point cloud data in step 908, classifying the point cloud data in step 910, and optionally identifying specific physical structures in the point cloud data in step 912. In step 908, the point cloud data may be aligned with the X-axis, Y-axis, and Z-axis. In other words, the point cloud is rotated and oriented in the correct direction. For example, a warehouse room scan may be received (step 902) in an arbitrary space without being aligned with any axis. The scan may then be rotated and aligned with the appropriate space in the warehouse. In some examples, the point cloud may be aligned with the width of a wall in the warehouse to determine the proper orientation of the point cloud. The computer system may find reference points around doorways, doorframes, windows, and / or other objects that may appear in the point cloud to properly orient the point cloud. In other examples, the computer system may detect a bounding box of the walls of a physical space (e.g., a warehouse room) and then rotate the physical space so that the bounding box is aligned with the X-axis, Y-axis, and Z-axis. Tightly fitting bounding boxes can also be used to more easily identify noise points outside the walls of the physical space. Performing this alignment step is important because it reduces the time required to perform certain calculations later in the point cloud processing.

[0098] Once the point cloud data has undergone alignment and initial cleaning, object detection and localization can begin (step 910). In this step, the point cloud data can be classified using a spherical filter. In this step, physical objects, perimeters, and other items present in the point cloud can be identified, classified, and scored. In some embodiments, some identified, classified, and scored items can be removed from the point cloud. For example, floors and ceilings can be removed. Points that are unlikely to be vertical poles in a warehouse environment can also be removed.

[0099] Based on the purpose and use of the point cloud, different physical objects can be identified for removal and / or retention. For example, in a land survey, the computer system can be configured to identify and remove trees and shrubs. The computer system can be further configured to identify buildings and parking lots and retain them in the point cloud. Typically, the easiest thing to detect in a point cloud scan is a floor or some other horizontal plane. For example, in a warehouse environment where rack detection may be most important, points that may be part of the floor can be identified and removed from the point cloud. In most embodiments, ceilings are also relatively easy to detect, as are light fixtures, light beams, evaporation coils, and other objects that may be suspended from the ceiling. Walls can be more difficult to detect, but using classifiers, vertical poles, and other physical item confidence scores can be adjusted accordingly so that walls, ceilings, etc. are removed from the point cloud.

[0100] Next, the computer system may optionally identify specific physical structures within the remaining point cloud data (step 912). For example, vertical poles, racks, and / or aisles in a warehouse may be identified and assigned confidence scores. In a land survey example, the system may identify residences, buildings, parking lots, and / or roads. Consequently, such identification can be used to generate accurate blueprints and / or maps of the associated physical space.

[0101] Figures 10A to 10E This is an exemplary depiction of voxelizing a point cloud. Figure 10A A voxel 1000 depicts a point cloud in three-dimensional ("3D") space. The points that make up the point cloud may be located within the voxel 1000. Moreover, the voxel 1000 may be further voxelized into different grid sizes. Voxelizing the voxel 1000 may provide more accurate identification of physical structures from the point cloud data. Although the process and technique of voxelizing the point cloud occurs in 3D space, for illustration purposes, Figures 10B to 10E The depiction generates voxel 1000 in two-dimensional ("2D") space. Figure 10B and 10C As depicted in , slices 1002 and 1004 of the point cloud may be selected along the x-axis and the y-axis (as mentioned, slices of the point cloud may also be selected along the z-axis in 3D space). Next, Figure 10D A bounded region 1006 is depicted that includes the selected point along the x-axis, the y-axis, and the z-axis. The volume of the bounded region 1006 can be determined. This bounded region 1006 in 3D space creates a voxel or cube (e.g., Figure 10A voxels 1000 in ), which may then be analyzed using the techniques described throughout this disclosure. Figure 10E The analysis of points outside voxel 1000 is depicted. Figure 6A As described, the center point of voxel 1000 can be identified. Then, the distance between the points and the center point can be determined. Points that are more than a predetermined distance from the center point can be removed from the point cloud. These points are Figure 10E Points removed from the point cloud marked with an "X" in the figure may indicate noise. If a sufficient number of points remain within a voxel, those remaining points may be further analyzed to determine relationships among the points, such as whether the points indicate physical structure in physical space and whether the points in the voxel need to be upsampled and / or downsampled, as described throughout this disclosure. The process described herein may be repeated a predetermined number of times and / or with different voxel grid sizes.

[0102] Figure 11Depicted is example point cloud data with k-means centroid points. As shown, point cloud 1100 may include k-means centroid points 1102A through 1102N and point cloud points 1104A through 1104N. k-means centroid point 1102A is located in a low-density region of point cloud 1100. Therefore, k-means centroid point 1102A requires upsampling. There is a greater distance in all directions from point 1102A to surrounding point cloud points 1104A through 1104N. Therefore, the larger distances in all directions can be filled with new points.

[0103] K-means centroid point 1102B is located in a high-density region of point cloud 1100. Therefore, k-means centroid point 1102B needs to be downsampled. The distances from point 1102B to surrounding point cloud points 1104A through 1104N are small in all directions. Therefore, during downsampling, the density of points 1104A through 1104N may be thinned.

[0104] Furthermore, k-means centroid point 1102C is located in a low-density region of point cloud 1100. Therefore, upsampling is necessary. There are larger distances in some, but not all, directions from point 1102C to the surrounding point cloud points 1104A through 1104N. Therefore, the large distances in some directions can be filled with points, leaving some gaps between k-means centroid point 1102C and the nearby point cloud points 1104A through 1104N.

[0105] k-means centroid point 1102D lies within the average density region of point cloud 1100. Therefore, neither upsampling nor downsampling is required. The density of point cloud 1100 can be preserved. After all, there can be an average distance in all directions between k-means centroid point 1102D and surrounding point cloud points 1104A through 1104N. It is preferable to preserve the appearance of points 1104A through 1104N so that they do not exceed the average distance from k-means centroid point 1102D.

[0106] Finally, k-means centroid point 1102N also lies within the average density region of point cloud 1100. Neither upsampling nor downsampling is required. There may be longer distances in one direction and shorter or smaller distances in a second direction. Even so, point cloud points 1104A through 1104N can remain within both longer and shorter distances to k-means centroid point 1102N, ensuring that they do not exceed those distances.

[0107] Computing devices that can be used to implement the systems, techniques, machines, and / or apparatus described in this document can operate as clients and / or servers and can include one or more of a variety of suitable computing devices, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframes, mobile computing devices (e.g., PDAs, cellular phones, smartphones, and / or other similar computing devices), computer storage devices (e.g., Universal Serial Bus (USB) flash drives, RFID storage devices, solid-state drives, hard disk storage devices), and / or other similar computing devices. For example, a USB flash drive can store an operating system and other application programs and can include input / output components, such as a wireless transmitter and / or a USB connector that can be plugged into a USB port on another computing device.

[0108] Such computing devices may include one or more of the following components: a processor, memory (e.g., random access memory (RAM) and / or other forms of volatile memory), storage (e.g., a solid-state drive, a hard drive, and / or other forms of non-volatile memory), high-speed interfaces connecting various components to each other (e.g., connecting one or more processors to memory and / or high-speed expansion ports), and / or low-speed interfaces connecting various components to each other (e.g., connecting one or more processors to low-speed buses and / or storage). Such components may be interconnected using various buses and may be mounted across one or more motherboards that are communicatively connected to each other, or in other suitable ways. In some embodiments, a computing device may include multiple components listed above, including multiple processors, multiple memories, multiple types of memory, multiple storage devices, and / or multiple buses. Multiple computing devices may be connected to each other and may coordinate at least a portion of their computing resources to perform one or more operations, for example, to provide a multi-processor computer system, a computer server system, and / or a cloud-based computer system.

[0109] A processor may process instructions for execution within a computing device, including instructions stored in memory and / or storage. Such processing of instructions may cause various operations to be performed, including causing visual, auditory, and / or tactile information to be output by one or more input / output devices, such as a display configured to output graphical information, such as a graphical user interface (GUI). A processor may be implemented as a chipset comprising individual and / or multiple analog and digital processors. A processor may be implemented using any of several architectures, such as a CISC (Complex Instruction Set Computer) processor architecture, a RISC (Reduced Instruction Set Computer) processor architecture, and / or a MISC (Minimum Instruction Set Computer) processor architecture. A processor may provide, for example, coordination of other components of a computing device, such as a user interface, control of applications executed by the device, and wireless communications performed by the device.

[0110] The memory can store information within the computing device, including instructions to be executed by one or more processors. The memory can include one or more volatile memory units, such as synchronous RAM (e.g., double data rate synchronous dynamic random access memory (DDR SDRAM), DDR2 SDRAM, DDR3 SDRAM, DDR4 SDRAM), asynchronous RAM (e.g., fast page mode dynamic RAM (FPM DRAM), extended data output DRAM (EDO DRAM)), and graphics RAM (e.g., graphics DDR4 (GDDR4), GDDR5). In some embodiments, the memory can include one or more non-volatile storage units (e.g., flash memory). The memory can also be another form of computer-readable media, such as a magnetic disk and / or optical disk.

[0111] A storage device can provide mass storage for a computing device and can include computer-readable media such as a floppy disk drive, a hard disk drive, an optical disk drive, a microdrive or tape drive, a flash memory or other similar solid-state storage device, or an array of devices, including devices in a storage area network or other configuration. A computer program product can be tangibly embodied in an information carrier such as a memory, a storage device, a cache memory within a processor, and / or other suitable computer-readable media. A computer program product can also contain instructions that, when executed by one or more computing devices, perform one or more methods or techniques, such as those described above.

[0112] The high-speed controller can manage bandwidth-intensive operations of the computing device, while the low-speed controller can manage less bandwidth-intensive operations. This allocation of functions is exemplary only. In some embodiments, the high-speed controller is coupled to memory, a display (e.g., through a graphics processor or accelerator), and to high-speed expansion ports that can accept various expansion cards; and the low-speed controller is coupled to one or more storage devices and low-speed expansion ports, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), which can be coupled to one or more input / output devices such as a keyboard, a pointing device (e.g., a mouse, touchpad, trackball), a printer, a scanner, a copier, a digital camera, a microphone, a display, a tactile device, and / or a networking device such as a switch and / or router (e.g., through a network adapter).

[0113] The display may include any of a variety of suitable display devices, such as a TFT (thin-film transistor liquid crystal display) display, an OLED (organic light-emitting diode) display, a touch screen device, a presence sensing display device, and / or other suitable display technologies. The display may be coupled to appropriate circuitry for driving the display to output graphics and other information to a user.

[0114] Expansion memory may also be provided and connected to the computing device via one or more expansion interfaces, which may include, for example, a SIMM (Single In-Line Memory Module) card interface. Such expansion memory may provide additional storage space for the computing device and / or may store applications or other information accessible by the computing device. For example, the expansion memory may include instructions that implement and / or supplement the techniques described above, and / or may include security information (e.g., the expansion memory may include a security module and be programmed with instructions that permit secure use on the computing device).

[0115] The computing device can communicate wirelessly via one or more communication interfaces, which may include digital signal processing circuitry, where appropriate. The communication interfaces may provide for communication in various modes or protocols, such as GSM voice calls, messaging protocols (e.g., SMS, EMS, or MMS messages), CDMA, TDMA, PDC, WCDMA, CDMA2000, GPRS, 4G protocols (e.g., 4G LTE), and / or other appropriate protocols. Such communication may occur, for example, via one or more radio frequency transceivers. Additionally, short-range communication may occur, for example, using Bluetooth, Wi-Fi, or other such transceivers. Furthermore, a GPS (Global Positioning System) receiver module may provide the computing device with additional navigational and location-related wireless data, which may be used, as appropriate, by applications running on the computing device.

[0116] The computing device may also communicate audibly using one or more audio codecs that can receive spoken information from the user and convert it into usable digital information. Such audio codecs may also generate audible sounds for the user, for example, through one or more speakers that are part of or connected to the computing device. Such sounds may include sounds from voice phone calls, recorded sounds (e.g., voice messages, music files, etc.), and sounds generated by applications running on the computing device.

[0117] Various implementations of the systems, devices, and techniques described herein can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system that includes at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from a memory system and transmit data and instructions to a memory system, at least one input device, and at least one output device.

[0118] These computer programs (also referred to as programs, software, software applications, or code) may include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor.

[0119] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., an LCD display, an LED display) for displaying information to the user, a keyboard, and a pointing device (e.g., a mouse, a trackball, a touch screen) that the user can use to provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound, voice, or tactile input).

[0120] The systems and techniques described herein can be implemented in a computing system that includes a back-end component (e.g., as a data server); or includes a middleware component (e.g., an application server); or includes a front-end component (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any digital data communication form or medium, such as a communication network. Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), a peer-to-peer network (with self-organizing or static members), a grid computing infrastructure, and the Internet.

[0121] The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0122] The above description provides examples of some embodiments. Other embodiments not explicitly described above are also possible, such as embodiments based on modifications and / or variations of the features described above. For example, the techniques described above can be implemented in different orders, wherein the order includes one or more additional steps and / or excludes one or more identified steps. In addition, the steps and techniques performed by some computing devices and / or systems described above can be performed alternatively or in addition by other computing devices and / or systems described above or other computing devices and / or systems not explicitly described. Similarly, systems, devices and equipment may include one or more additional features, exclude one or more identified features, and / or include identified features combined in a manner different from that presented above. A feature described as single may be implemented as a plurality of such features. Similarly, a feature described as multiple may be implemented as a single example of such a feature. The figures are intended to be illustrative and may not accurately depict some embodiments. Changes in the size, position, shape, angle and / or positioning of features relative to each other are possible.

Claims

1. A method for determining facility information using point cloud data, the method comprising: receiving, by a computing system, image data of a facility, wherein the image data is captured by at least one imaging device located within the facility; generating, by the computing system, point cloud data from the image data, wherein the point cloud data comprises a plurality of points in three-dimensional (3D) space that approximate locations of physical surfaces within the facility; filtering, by the computing system, the plurality of points in the point cloud data to (i) retain points in the point cloud data that correspond to a specific type of outline of the physical surface within the facility, and (ii) remove points from the point cloud data that do not correspond to the specific type of outline of the physical surface within the facility, wherein the specific type of physical surface includes at least vertical poles and racks; generating, by the computing system, a map of the facility based at least in part on the points in the point cloud data corresponding to the contours of the particular type of the physical surface within the facility; determining, by the computing system, facility information based on the generated map, the facility information including at least identification of the vertical poles and the racks within the facility; as well as The facility information is returned by the computing system.

2. A method according to claim 1, wherein determining facility information by the computing system based on the generated map includes determining a size or occupancy rate of the facility, the size or occupancy rate of the facility indicating an amount of physical space within the facility that includes the physical surface.

3. The method of claim 1, wherein the physical surfaces include racks, vertical poles, doorways, vehicles, workers, and pallets.

4. The method of claim 1 , wherein removing points from the point cloud data that do not correspond to the contour of the particular type of the physical surface within the facility comprises removing points from the point cloud data that correspond to walls and ceilings in the facility.

5. The method of claim 1 , wherein retaining points in the point cloud data corresponding to the specific type of the contour of the physical surface within the facility comprises retaining points in the point cloud data corresponding to the racks and the vertical poles within the facility.

6. The method of claim 1, further comprising aligning, by the computing system, the filtered plurality of points in the point cloud data along x, y, and z dimensions corresponding to an orientation of the facility represented by the image data. 7 . The method of claim 1 , further comprising aligning, by the computing system, the point cloud data with a width of a wall in the facility to determine an orientation of the point cloud data corresponding to an orientation of the facility. 8 . The method of claim 1 , wherein the facility is a warehouse, and determining, by the computing system, facility information based on the generated map comprises determining a capacity of the warehouse based on remaining points in the point cloud data.

9. The method of claim 1, further comprising classifying, by the computing system, the filtered plurality of points in the point cloud data as corresponding to one or more of the particular types of the physical surfaces within the facility. 10 . The method of claim 9 , wherein classifying, by the computing system, the filtered plurality of points in the point cloud data comprises applying a spherical filter to the filtered plurality of points.

11. The method of claim 9, wherein classifying, by the computing system, the filtered plurality of points in the point cloud data comprises: determining a confidence score for each of the particular types of the physical surfaces identified within the facility, the confidence score indicating a likelihood that the identified physical surface is a vertical pole or a rack; as well as Based on the confidence score being less than a threshold confidence value, one or more points corresponding to the identified physical surface are removed from the point cloud data.

12. The method of claim 1 , wherein filtering, by the computing system, the plurality of points in the point cloud data comprises: generating at least one histogram based on the point cloud data; iteratively voxelizing the at least one histogram to a predetermined grid size; filtering the at least one voxelized histogram based on a sum of points in the at least one voxelized histogram; identifying a set of points among the plurality of points in the at least one filtered histogram, the set of points being nearest neighboring points to a center point in the set of points; as well as The set of points is removed from the plurality of points in the image data.

13. The method of claim 1, wherein the at least one imaging device is a stereo camera.

14. The method of claim 1, wherein the at least one imaging device is an unmanned aerial vehicle.

15. The method of claim 1, further comprising determining, by the computing system, a process flow of activities in the facility based on the generated map of the facility.

16. The method of claim 1, wherein determining, by the computing system, facility information comprises determining a layout of the facility based at least in part on the points in the point cloud data corresponding to the contours of the particular type of the physical surface within the facility.

17. A method according to claim 1, wherein the image data includes at least one 3D scan of a portion of a physical space within the facility, and wherein generating point cloud data from the image data by the computing system includes interpolating points corresponding to other portions of the physical space within the facility that were not captured in the at least one 3D scan.

18. The method of claim 1, further comprising aligning, by the computing system, the point cloud data with an orientation of the facility based on identifying reference points in the point cloud data corresponding to doorways, door frames, and windows within the facility.

19. A system for determining facility information using point cloud data, the system comprising: a storage facility comprising walls, a ceiling, vertical rods, and racks, the storage facility being configured to store pallets in one or more locations defined by at least the vertical rods and the racks; at least one imaging device positioned within the storage facility, wherein the at least one imaging device is configured to capture image data of a physical space within the storage facility; as well as a computing system in network communication with the at least one imaging device, the computing system comprising one or more processors and memory to map the storage facility and determine facility information based on the image data captured by the at least one imaging device, wherein the one or more processors are configured to perform operations comprising: receiving said image data of a facility from said at least one imaging device; generating point cloud data from the image data, wherein the point cloud data includes a plurality of points in three-dimensional (3D) space that approximate locations of physical surfaces within the storage facility; filtering the plurality of points in the point cloud data to (i) retain points in the point cloud data that correspond to a particular type of contour of the physical surface within the storage facility, and (ii) remove points from the point cloud data that do not correspond to the particular type of contour of the physical surface within the storage facility; generating a map of the storage facility based at least in part on the points in the point cloud data corresponding to the contours of the particular type of the physical surfaces within the storage facility; determining, by the computing system, facility information based on the generated map, the facility information including a size of the storage facility; as well as The facility information is returned by the computing system.

20. The system of claim 19, wherein the operations further comprise: classifying the filtered plurality of points in the point cloud data as corresponding to one or more of the particular types of the physical surfaces within the storage facility; determining a confidence score for each of the particular types of the physical surfaces classified within the storage facility, the confidence score indicating a likelihood that the identified physical surface is a vertical pole or a rack; as well as Based on the confidence score being less than a threshold confidence value, one or more points corresponding to the identified physical surface are removed from the point cloud data.

Images