Removal of dynamic points from three dimensional maps

Abstract

In some examples, a method comprises obtaining sensor data from a 3D LiDAR sensor, the sensor data comprising multiple LiDAR scan frames comprising multiple points and multiple poses associated with the multiple LiDAR scan frames, respectively, generating a map based on the obtained sensor data, combining at least some of the multiple LiDAR scan frames, whereby to obtain a submap, the submap comprising multiple planes, wherein each of the multiple planes is associated with a set of points of the multiple points, detecting the multiple planes in the submap and arranging the detected planes in order of size, and incorporating the set of points associated with at least one detected plane into the generated map based on the arrangement of the planes.

Description

[0001] REMOVAL OF DYNAMIC POINTS FROM THREE DIMENSIONAL MAPS

[0002] TECHNICAL FIELD

[0003] The present disclosure relates, in general, to removal of dynamic points from three dimensional (3D) maps. Aspects of the disclosure relate to generating 3D maps using LiDAR sensors.

[0004] BACKGROUND

[0005] SLAM (Simultaneous Localization and Mapping) algorithms are crucial in robotics and autonomous systems, allowing a device to map its environment while simultaneously determining its own position within that environment. LiDAR is a remote sensing method that uses laser pulses to measure distances to surrounding objects. By emitting laser beams and analysing the time it takes for the light to return after hitting an object, LiDAR can construct detailed, three-dimensional representations of the environment.

[0006] A SLAM algorithm accumulates 3D (three dimensional) point clouds from the 3D LiDAR sensor, along with the sensor's motion or pose, enabling the creation of a 3D map of the environment. Various commercial hand-held 3D LiDAR mapping devices using SLAM algorithms have been developed. Existing open-source SLAM algorithms that can work with any 3D LiDAR sensor to create same 3D maps with little adaptation are also available.

[0007] However, these maps may contain dynamic objects present in the environment during the mapping process. As the dynamic objects move across the environment during the mapping process, they leave residual dynamic object tracks in 3D maps. The resulting 3D maps with dynamic object tracks do not accurately represent the environment, thus making it difficult to develop downstream robot applications such as localisation, path planning, and navigation.

[0008] Moreover, these corrupted 3D maps with dynamic objects also have negative influence on multi-session map alignment, which essentially stitches multiple 3D maps with different overlapping areas into a much larger single 3D map. Having clean static 3D maps greatly enhances accurate registration and stitching of these multi-session 3D maps.

[0009] As such, clean static 3D maps are crucial for accurate 3D environment representation, robot localisation, path planning and navigation. Additionally, clean 3D maps play a crucial role in creating accurate and visually appealing representations for use in 3D digital world creation and digital twin applications.

[0010] SUMMARY

[0011] An objective of the present disclosure is to provide a mechanism for removing dynamic points from three dimensional (3D) maps.

[0012] The foregoing and other objectives are achieved by the features of the independent claims.

[0013] Further implementation forms are apparent from the dependent claims, the description and the Figures. A first aspect of the present disclosure provides a method comprising obtaining sensor data from a 3D LiDAR sensor, the sensor data comprising multiple LiDAR scan frames comprising multiple points and multiple poses associated with the multiple LiDAR scan frames, respectively, generating a map based on the obtained sensor data, combining at least some of the multiple LiDAR scan frames, whereby to obtain a submap, the submap comprising multiple planes, wherein each of the multiple planes is associated with a set of points of the multiple points, detecting the multiple planes in the submap and arranging the detected planes in order of size, and incorporating the set of points associated with at least one detected plane into the generated map based on the arrangement of the planes.

[0014] Accordingly, an effective method of dynamic point removal can be provided. The method is independent of the positioning or placement of the LiDAR sensor and therefore can be used with any hand-held, robot mounted or any custom LiDAR sensor device. As a result, more accurate and more dense 3D maps can be provided, thereby enhancing downstream applications such as navigation, localisation, and path planning.

[0015] Arranging the detected planes in order of size may comprise arranging the detected planes from the largest to the smallest, wherein incorporating the points associated with the at least one detected plane may comprise incorporating the points associated with the largest detected plane.

[0016] Incorporating the points associated with at least one detected plane into the generated map may comprise iteratively adding points associated with the detected planes to the generated map according to a ratio between a size of a detected plane to be incorporated and a size of the submap.

[0017] The method may further comprise specifying a number of detected planes to be incorporated into the generated map, and adding the points associated with the detected planes to the generated static map until the specified number is reached.

[0018] The method may further comprise, for each plane, comparing the set of points associated with a plane of the multiple planes of the submap to a threshold value, and incorporating the points which are above the threshold value into the generated map.

[0019] The threshold value may be selected based on at least one of the detected planes.

[0020] Generating the map based on the obtained sensor data may comprise applying an algorithm to the obtained sensor data, whereby to categorise each of the multiple points as static, dynamic, or unknown, and applying a nearest neighbour algorithm to categorise the unknown points as static or dynamic.

[0021] The method may further comprise, for each point categorised as a dynamic point, applying a radial filter to determine whether points neighbouring the dynamic point are static or dynamic, for each point categorised as a static point, applying the radial filter to determine whether points neighbouring the static point are static or dynamic, and, in response to determining that the points neighbouring the dynamic point are static points, reclassifying the dynamic point as a static point, and, in response to determining that the neighbouring points of the static point are dynamic points, re-classifying the static point as a dynamic point.

[0022] The method may further comprise classifying each LiDAR scan frame of the multiple scan frames based on the generated map, whereby to obtain a classified LiDAR scan comprising an indication of the static and dynamic points having the static and dynamic points denoted.

[0023] A second aspect of the present disclosure provides a computer program stored on a non-transitory medium and including code instructions, which, when executed on one or more processors, cause the one or more processors to execute the method described herein.

[0024] A third aspect of the present disclosure provides an apparatus comprising a processor; a memory coupled to the processor, the memory configured to store program code executable by the processor, the program code comprising one or more instructions, whereby to cause the apparatus to obtain sensor data from a 3D LiDAR sensor, the sensor data comprising multiple LiDAR scan frames comprising multiple points and multiple poses associated with the multiple LiDAR scan frames, respectively, generate a map based on the obtained sensor data, combine at least some of the multiple LiDAR scan frames, whereby to obtain a submap, the submap comprising multiple planes, wherein each of the multiple planes is associated with a set of points of the multiple points, detect the multiple planes in the submap and arrange the detected planes in order of size, and incorporate the set of points associated with at least one detected plane into the generated map based on the arrangement of the planes.

[0025] The program code comprising one or more instructions, whereby to cause the apparatus to arrange the detected planes in order of size may comprise program code comprising one or more instructions, whereby to cause apparatus to arrange the detected planes from the largest to the smallest, wherein the program code comprising one or more instructions, whereby to cause the apparatus to incorporate the set of points associated with the at least one detected plane may comprise program code comprising one or more instructions, whereby to cause apparatus to incorporate the points associated with the largest detected plane.

[0026] The program code comprising one or more instructions, whereby to cause the apparatus to incorporate the set of points associated with the at least one detected plane into the generated map may comprise program code comprising one or more instructions, whereby to cause apparatus to iteratively add points associated with the detected planes to the generated map according to a ratio between a size of a detected plane to be incorporated and a size of the submap.

[0027] The apparatus may further comprise the program code comprising one or more instructions, whereby to cause the apparatus to specify a number of detected planes to be incorporated into the generated map, and add the points associated with the detected planes to the generated static map until the specified number is reached. The apparatus may further comprise the program code comprising one or more instructions, whereby to cause the apparatus to, for each plane, compare the set of points associated with a plane of the multiple planes of the submap to a threshold value, and incorporate the points which are above the threshold value into the generated map.

[0028] These and other aspects of the disclosure will be apparent from the embodiment(s) described below.

[0029] BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order that the present disclosure may be more readily understood, embodiments of the disclosure will now be described, by way of example, with reference to the accompanying drawings, in which:

[0031] Fig. 1 is a flow chart of a method according to an example;

[0032] Fig. 2 is a flow chart of a method of dynamic object removal according to an example;

[0033] Fig. 3 is a flow chart of a method of dynamic object removal according to another example;

[0034] Fig. 4 is a flow chart of a method of dynamic object removal according to yet another example;

[0035] Fig. 5 is a flow chart of a method of plane detection according to an example; and

[0036] Fig. 6 is a schematic representation of an apparatus according to an example.

[0037] DETAILED DESCRIPTION

[0038] Example embodiments are described below in sufficient detail to enable those of ordinary skill in the art to embody and implement the systems and processes herein described. It is important to understand that embodiments can be provided in many alternate forms and should not be construed as limited to the examples set forth herein.

[0039] Accordingly, while embodiments can be modified in various ways and take on various alternative forms, specific embodiments thereof are shown in the drawings and described in detail below as examples. There is no intent to limit to the particular forms disclosed. On the contrary, all modifications, equivalents, and alternatives falling within the scope of the appended claims should be included. Elements of the example embodiments are consistently denoted by the same reference numerals throughout the drawings and detailed description where appropriate.

[0040] The terminology used herein to describe embodiments is not intended to limit the scope. The articles “a,” “an,” and “the” are singular in that they have a single referent, however the use of the singular form in the present document should not preclude the presence of more than one referent. In other words, elements referred to in the singular can number one or more, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises,” “comprising,” “includes,” and / or “including,” when used herein, specify the presence of stated features, items, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, items, steps, operations, elements, components, and / or groups thereof.

[0041] Unless otherwise defined, all terms (including technical and scientific terms) used herein are to be interpreted as is customary in the art. It will be further understood that terms in common usage should also be interpreted as is customary in the relevant art and not in an idealized or overly formal sense unless expressly so defined herein. Currently, there exist various approaches for dynamic point removal in point cloud data. One common method involves using LiDAR information exclusively to identify and eliminate dynamic objects from the data. However, this approach has limitations based on the placement and positioning of the LiDAR device. It is particularly constrained in scenarios where motion is restricted, such as in autonomous driving, where the vehicle or robot primarily moves on planar surfaces like roads. However, there is a specific class of algorithms based on OctoMap that operate without the need for these constraints.

[0042] The OctoMap algorithm is a 3D mapping framework used in robotics and autonomous systems for efficient and accurate representation of three-dimensional environments. OctoMap can incorporate sensor data, such as from LiDAR, to dynamically build and refine a probabilistic occupancy map. The OctoMap algorithm leverages an octree data structure, which recursively subdivides space into octants, to manage and store occupancy information about the environment. The OctoMap classifies voxels as occupied, empty, or unknown in terms of fused occupation probability based on data acquired from a sensor. The fused occupation probability is derived according to Bayes’ Theorem in probability theory. Dynamic points are classified as the points in the voxels where the occupation probability is lower than a threshold value.

[0043] However, the resultant static map and dynamic map generated using the OctoMap algorithm still suffer from issues affecting the accuracy of the map. Here, static points refer to parts of the environment that remain unchanged over time, such as buildings and roads, while dynamic points represent moving objects, like vehicles and pedestrians. The first issue is the presence of residual holes in planar regions, such as roads and building facades. The second issue is that the resulting static map is quite sparse compared to the original input scans, due to extensive filtering and some false positive detections by OctoMap.

[0044] In order to compensate for the holes in the road, which are quite evident in autonomous driving scenarios, ground segmentation has been exploited. In particular, an approach was proposed, based on a ground segmentation algorithm that uses height from LiDAR placement on the robot / vehicle to detect the static ground plane using a height-based projection algorithm. The detected static ground plane is added back as static points to the static map, such that the residual holes problem can be reduced. However, the major drawback of this approach is that some constraints based on LiDAR placement and motion to detect ground plane are still present. Furthermore, this approach does not fully solve the residual holes problem, as these holes can also exist on building facades and large planar regions, i.e., not just on the ground planes. Finally, the approach does not remedy in any way the sparseness of the static maps resultant from the underlying OctoMap algorithm.

[0045] According to an example, there is provided a mechanism to remove dynamic points from three dimensional (3D) maps. More specifically, aspects relate to generating 3D maps using LiDAR sensors, independently of the LiDAR sensor’s positioning. The mechanism provided advantageously enables the provision of more accurate and more dense 3D maps, thereby enhancing downstream applications such as navigation, localisation, and path planning. In comparison to currently available solutions for dynamic object removal, the residual holes corrupting the static maps can be minimised, and the sparse static maps can be densified. Examples in the present disclosure can be provided as methods, systems or machine-readable instructions, such as any combination of software, hardware, firmware or the like. Such machine-readable instructions may be included on a computer readable storage medium (including but not limited to disc storage, CD-ROM, optical storage, etc.) having computer readable program codes therein or thereon.

[0046] The present disclosure is described with reference to flow charts and / or block diagrams of the method, devices and systems according to examples of the present disclosure. Although the flow diagrams described above show a specific order of execution, the order of execution may differ from that which is depicted. Blocks described in relation to one flow chart may be combined with those of another flow chart. In some examples, some blocks of the flow diagrams may not be necessary and / or additional blocks may be added. It shall be understood that each flow and / or block in the flow charts and / or block diagrams, as well as combinations of the flows and / or diagrams in the flow charts and / or block diagrams can be realized by machine readable instructions.

[0047] The machine-readable instructions may, for example, be executed by a machine such as a general-purpose computer, user equipment such as a smart device, e.g., a smart phone, a special purpose computer, an embedded processor or processors of other programmable data processing devices to realize the functions described in the description and diagrams. In particular, a processor or processing apparatus may execute the machine-readable instructions. Thus, modules of apparatus (for example, a module implementing a comparator unit, or a firewall structure and so on) may be implemented by a processor executing machine readable instructions stored in a memory, or a processor operating in accordance with instructions embedded in logic circuitry. The term 'processor' is to be interpreted broadly to include a CPU, processing unit, ASIC, logic unit, or programmable gate set etc. The methods and modules may all be performed by a single processor or divided amongst several processors.

[0048] Such machine-readable instructions may also be stored in a computer readable storage that can guide the computer or other programmable data processing devices to operate in a specific mode. For example, the instructions may be provided on a non-transitory computer readable storage medium encoded with instructions, executable by a processor.

[0049] Fig. 1 is a flow chart of a method according to an example. The method comprises, in block 101, obtaining sensor data from a three-dimensional (3D) light detection and ranging (LiDAR) sensor. The LiDAR sensor may comprise a hand-held LiDAR device, a robot-mounted LiDAR device and / or a LiDAR device in any other custom configuration. In other words, any LiDAR device may be used, regardless of its positioning (i.e., the disclosure is not limited to a dynamic LiDAR device having motion constraint to a specific area / plane).

[0050] The sensor data obtained from the LiDAR sensor comprises multiple LiDAR scan frames. Each of the multiple LiDAR scan frames comprises multiple points associated therewith and a pose. Here, the term “multiple points” may refer to individual data points collected by the LiDAR sensor, with each point representing a specific location in the scanned environment. These points, also referred to as "point clouds," may collectively form a detailed three- dimensional representation of the surroundings. Each point in the cloud may be defined in terms of its coordinates (x, y, z), as well as any additional attributes, such as intensity or colour, depending on the LiDAR sensor’s capabilities. The pose, on the other hand, may represent the position and orientation of the LiDAR sensor at the time of data acquisition, typically represented as a combination of translation (location in space) and rotation (orientation angles). The LiDAR scan frames may be obtained directly from a LiDAR sensor, whereas the poses associated therewith may be estimated using a SLAM algorithm.

[0051] The method comprises, in block 102, generating a map based on the obtained sensor data. In other words, the multiple LiDAR scan frames may be combined to generate the map. Generating the map may comprise applying an algorithm to the obtained sensor data (i.e., the multiple points and the multiple poses), whereby to categorise each of the multiple points as static, dynamic, or unknown. The algorithm to be applied to the sensor data may comprise, for example, the OctoMap algorithm, but the disclosure is not limited thereto - that is, any algorithm capable of categorising the multiple points can be used. A nearest neighbour (NN) algorithm may be applied to the points categorised as “unknown”, so as to categorise the unknown points as static or dynamic, as will be described in more detail later in the description.

[0052] In block 103, the method comprises combining at least some of the multiple LiDAR scan frames, whereby to obtain a submap. Here, the term “submap” may refer to a small map created by accumulating a few of the LiDAR frames, along with LiDAR motion. Compared to a regular map, creating a submap may require fewer LiDAR scan frames. For instance, a submap may be generated using only 10-100 LiDAR scan frames, whereas hundreds or even thousands of LiDAR scan frames might be needed to create a regular map. In order to create a submap, motion estimates between the LiDAR scan frames may be required. The term “motion estimates” may refer to the calculated changes in position and orientation of the LiDAR sensor as it moves over time, used to enable proper alignment of consecutive LiDAR frames. By comparing the point clouds from successive frames, an algorithm can determine how the sensor has shifted and rotated.

[0053] The submap comprises multiple planes, with each of the multiple planes being associated with a set of points of the multiple points present in the LiDAR scan frames. The multiple planes may refer to static planes. A plane of the multiple planes may comprise a flat, two-dimensional surface that can be defined within the three-dimensional space captured by the LiDAR sensor. The plane may be represented by a group (set) of points that lie close to each other in space. The points may form a coherent, flat region within the point cloud data, which could correspond to real-world surfaces such as walls, floors, ceilings, or the ground.

[0054] In block 104, the method comprises detecting the multiple planes in the submap and arranging the detected planes in order of size. The plane detection will be described in more detail with reference to Fig. 5, which is a flow chart of a method for plane detection according to an example. In one example, arranging the detected planes in order of size may comprise arranging the detected planes from the largest to the smallest.

[0055] The method comprises, in block 105, incorporating the set of points associated with at least one detected plane into the generated map based on the arrangement of planes. In case the detected planes were arranged from the largest plane to the smallest plane, incorporating the set of points associated with the at least one detected plane into the generated map may comprise incorporating the set of points associated with the largest detected plane. By incorporating the set of points corresponding to the detected static plane into the generated map, the residual holes in the static map may be remedied, improving the density of the resulting map.

[0056] Incorporating the points associated with the at least one detected plane into the generated map in block 105 may further comprise iteratively adding points associated with the detected planes to the generated map according to a ratio between a size of a detected plane to be incorporated and a size of the submap, as will be described in more detail below.

[0057] As will be described in more detail in relation to Fig. 2, a height fdter may be employed in order to ensure that the correct planes are incorporated into the generated map. In such case, the method may comprise, for each plane of the multiple planes, comparing the set of points associated with a plane of the multiple planes of the submap to a threshold value, and incorporating the points which are above the threshold value into the generated map. The threshold value may be selected based on at least one of the points.

[0058] To aid understanding of the method of Fig. 1, reference will be made to Fig. 2, which is a flow chart of a method of dynamic object removal according to an example. Sensor data 201 (i.e., the LiDAR scan frames and poses associated therewith) may be passed to an algorithm 202 to obtain an initial classification of static, dynamic, and unknown points, whereby to generate a static map and a dynamic map. In parallel to this, at least some of the sensor data 201 (i.e., at least some of the LiDAR scan frames) may be combined in order to create a submap 203, the submap 203 comprising multiple planes.

[0059] A plane regression algorithm 204 may then be applied to the submap 203 in order to detect multiple planes. The plane detection and incorporation are described in more detail further in the specification, in relation to Fig. 5.

[0060] The plane regression algorithm 204 may comprise, for example, a RANSAC-based plane regression algorithm. The RANSAC-based plane regression algorithm may refer to an algorithm for fitting a plane to a set of 3D points, especially when the data contains a significant number of outliers. RANSAC, which stands for Random Sample Consensus, operates by iteratively selecting random subsets of the data points and fitting a plane to these subsets. In each iteration, the algorithm estimates the plane parameters and identifies how many points from the entire dataset are close to this estimated plane, known as inliers. The plane with the highest number of inliers is considered the best fit. This process is repeated for a predetermined number of iterations or until a satisfactory fit is achieved.

[0061] Following the detection of the multiple planes, in the particular example shown in Fig. 2, the largest plane 205 may be incorporated into the static map generated using the algorithm 202, but the disclosure is not limited thereto. That is, sets of points corresponding to planes other than the largest plane 205 may be combined with a raw output of the algorithm 202. By incorporating the at least one plane into the static map, the holes in the planar region can be alleviated, and the point density can be increased. This is because the direct detection of planes is more accurate than static points classification based on fused probability. The plane regression algorithm 204 may consider the remaining multiple planes 206 (not including the largest plane 205), assessing whether any of the multiple planes 206 are large enough relative to the submap 203, based on a ratio check 207. Based on the ratio check 207, the plane regression algorithm 204 may determine at least one additional plane 208 of the multiple planes 206 to be added to the generated map. In particular, the set(s) of points associated with the at least one additional plane 208 may be incorporated into the generated static map, in order to fill residual holes in planar areas. This addresses the density problem, as the residual holes typically lie within large planes. Alternatively or additionally, the user may specify the number of planes to be incorporated into the generated map. In such case, the sets of points associated with each of the planes to be included in the generated map may be added to the generated map until the specified number is reached.

[0062] In one example, the largest detected plane 205 may comprise a ground plane, and the subsequent large planes of the multiple planes 206 may comprise building facades. However, the disclosure is not limited thereto - regardless of what is represented by the plane, the mechanism described herein allows to add the points associated with the detected plane as static points.

[0063] Optionally, a height filter 209 may be applied. The detected largest plane 205 may comprise a ground plane, subject to additional consistency checks required to verify whether the largest plane 205 is, in fact, the ground plane. The height filter may be selected based on a height of the largest plane 205. A regressed plane equation may be used to calculate a point distance to the largest plane 205. Points whose distance from the largest plane 205 exceeds a predetermined threshold may be classified as static. This is due to the fact that, in urban environments, moving (i.e., dynamic) objects are typically located on the ground, e.g., vehicles or pedestrians.

[0064] After the above steps, the output of the algorithm 202 may be combined with the detected static planes (e.g., largest plane 205, and, optionally, any additional planes 208), in order to acquire 3D maps that are devoid of residual holes. However, at this stage, one remaining issue remains to be resolved - the sparsity of the static 3D maps with residual dynamic points present, caused by the inaccuracy of the algorithm 202 and / or incomplete sensor data 201.

[0065] In order to address the sparsity of the static map and to remove the remaining dynamic points, K-NN voting 210 may be applied. K-Nearest Neighbours (K-NN) voting is a method used for classifying data points based on their proximity to other points in a feature space, in this method, a data point is classified by assigning it to the most common category among its K closest neighbours, where K is a positive integer. The distance between data points is typically measured using metrics such as Euclidean distance. Each of the K nearest neighbours casts a vote for its respective class, and the data point is assigned to the class with the highest number of votes.

[0066] As described earlier in relation to block 102, the points in the sensor data 201 may be categorised as static, dynamic, or unknown. K-NN voting 210 may be applied to the unknown points in order to re-classify them as static or dynamic. Subsequently, in post-processing 211, statistical outlier removal may be performed on the resultant static maps. This process aids in the removal of noisy dynamic points that remain present in the static maps. Furthermore, also in the post-processing 211, radial filtering may be performed to reassign points close to static points in the dynamic map as static, reducing the number of misclassified dynamic points. Given spatial consistency, dynamic residual points which are located in the proximity of static points may be reclassified as static. This may be achieved by performing a radial filtering step, which uses a k-dimensional tree (or any other suitable data structure) for fast traversing. In general, a significant number of static points are incorrectly classified as dynamic, indicating that the dynamic point removal algorithm is excessively aggressive in filtering out static points. A radial filtering module may evaluate each dynamic point and, if a substantial number of its neighbouring points are identified as static, the dynamic point may be reclassified as static. This reduces the false positive detection of dynamic points and increases the density of the resulting static 3D map, making the maps more accurate. Advantageously, every single point in each one of the multiple LiDAR scan frames obtained from the LiDAR sensor may be classified as static or dynamic.

[0067] Optionally, a further K-NN based radius search may be performed on each point in every scan, whereby to classify each of the points as static or dynamic. Finally, a final 3D map may be generated by processing the classified input 3D scans. The generated 3D map may comprise a static map and a dynamic map.

[0068] Fig. 3 is a flow chart of a method of dynamic object removal according to another example. Briefly, the method of dynamic object removal is nearly identical to the method described in relation to Fig. 2. Same reference numerals have been used to denote the same elements in both figures, functioning likewise. In comparison to the method depicted in Fig. 2, the supplementation of the generated map using the multiple planes 206 may be skipped. In other words, although the largest plane 205 may still be incorporated into the static map, inclusion of any additional planes 208 based on the ratio check 207 may not be performed. The approach of Fig. 3 may be preferred for scenarios in which the sensor data 201 relates to a wild environment (i.e., a forest or similar), where artificial vertical planes may be scanty. By disabling the supplementation of additional planes 208 into the static map, wrong plane detection may be avoided. Additionally, although present in Fig. 3 , the height filter 209 may also be disabled. Disabling the height filter 209 may be preferred for use in drone-based 3D mapping applications, where the use of a height fdter may not be needed.

[0069] Fig. 4 is a flow chart of a method of dynamic object removal according to yet another example. Similarly to Fig. 3, the method of dynamic object removal of Fig. 4 builds upon the method described in relation to Fig. 2. Same reference numerals have been used to denote the same elements in both figures, the elements functioning likewise. In Fig. 4, in addition to all the elements present and described in relation to Fig. 2, classified scans 213 may be utilised. In particular, by utilising classified dynamic map and static map combined with the raw LiDAR scans (i.e., sensor data 201), the individual scan(s) may also be classified based on static and dynamic points.

[0070] Fig. 5 is a flow chart of a method of plane detection according to an example. By regressing the largest planes and supplementing the planes back to the static map, the residual holes in planar regions of the static maps may be filled. This ensures the generation of a dense and complete ground plane(s), thereby addressing the problem stemming from the use of the categorisation / generation algorithm 202 (for example, OctoMap). In order to ensure robust detection of planes, a series of LiDAR scans 501 (i.e., the set(s) of points and the pose(s) associated therewith) may be transformed into a global coordinate frame using poses, and combined together to form a submap 502. Here, the term “global coordinate frame” may refer to a reference coordinate system that is used to define the positions and orientations of objects in a consistent and absolute manner across a large-scale environment. The reference coordinate system may be fixed and defined by a set of orthogonal axes (labelled as, for example, x, y and z) with a specified origin point. As such, the global coordinate frame may allow data acquired from various LiDAR sensors to be shared and interpreted.

[0071] Following the generation of the submap 502, a regressive plane detection algorithm (e.g., RANSAC plane regression algorithm) may be invoked in order to acquire the current largest plane 504 (as well as the equation associated therewith, i.e., the equation defining the largest plane). The largest plane 504 may comprise a plane having the largest number of points associated therewith, compared to the remaining planes. The process may mn iteratively after removing the current largest plane, until the remaining points are below a certain proportion of the original points. In other words, a RANSAC plane regression algorithm may be applied to the input (i.e., the point cloud) in order to detect the largest plane. Following the detection, the largest place may be removed from the input point cloud, and a next largest plane can be detected. As such, the process may be performed iteratively, whereby to detect multiple planes in decreasing order of size.

[0072] A ratio check 505 may be performed for the extracted largest plane 504, such that the size of points in a plane should exceed a percentage threshold of previous planes. All the extracted planes satisfying the ratio check 505 may be stored in a list 506. The largest plane of the detected planes may be considered to be the ground plane, whereas the remaining detected large planes may be considered to be supplementary planes.

[0073] Fig. 6 is a schematic representation of an apparatus according to an example. The apparatus 600 comprises a processor 603, and a memory 605 coupled to the processor 603 and configured to store instructions or program code 607, executable by the processor 603. The apparatus 600 comprises the program code 607 arranged to cause the apparatus to perform the method described herein.

[0074] According to an example, machine-readable instructions can be loaded onto a computer or other programmable data processing devices, so that the computer or other programmable data processing devices perform a series of operations to produce computer-implemented processing, thus the instructions executed on the computer or other programmable devices provide an operation for realizing functions specified by flow(s) in the flow charts and / or block(s) in the block diagrams.

[0075] Further, the teachings herein may be implemented in the form of a computer or software product, such as a non- transitory machine-readable storage medium, the computer software or product being stored in a storage medium and comprising a plurality of instructions, e.g., machine readable instructions, for making a computer device implement the methods recited in the examples of the present disclosure. In some examples, some methods can be performed in a cloud-computing or network-based environment. Cloudcomputing environments may provide various services and applications via the Internet. These cloud-based services (e.g., software as a service, platform as a service, infrastructure as a service, etc.) may be accessible through a web browser or other remote interface of the user equipment for example. Various functions described herein may be provided through a remote desktop environment or any other cloud-based computing environment.

[0076] While various embodiments have been described and / or illustrated herein in the context of fully functional computing systems, one or more of these exemplary embodiments may be distributed as a program product in a variety of forms, regardless of the particular type of computer-readable-storage media used to actually carry out the distribution. The embodiments disclosed herein may also be implemented using software modules that perform certain tasks. These software modules may include script, batch, or other executable files that may be stored on a computer-readable storage medium or in a computing system. In some embodiments, these software modules may configure a computing system to perform one or more of the exemplary embodiments disclosed herein. In addition, one or more of the modules described herein may transform data, physical devices, and / or representations of physical devices from one form to another.

[0077] The preceding description has been provided to enable others skilled in the art to best utilize various aspects of the exemplary embodiments disclosed herein. This exemplary description is not intended to be exhaustive or to be limited to any precise form disclosed. Many modifications and variations are possible without departing from the spirit and scope of the instant disclosure. The embodiments disclosed herein should be considered in all respects illustrative and not restrictive. Reference should be made to the appended claims and their equivalents in determining the scope of the instant disclosure.

Claims

CLAIMS1. A method comprising: obtaining sensor data from a 3D LiDAR sensor, the sensor data comprising multiple LiDAR scan frames comprising multiple points and multiple poses associated with the multiple LiDAR scan frames, respectively (101); generating a map based on the obtained sensor data (102); combining at least some of the multiple LiDAR scan frames, whereby to obtain a submap (103), the submap comprising multiple planes, wherein each of the multiple planes is associated with a set of points of the multiple points; detecting the multiple planes in the submap and arranging the detected planes in order of size (104); and incorporating the set of points associated with at least one detected plane into the generated map based on the arrangement of the planes (105).

2. The method of claim 1, wherein arranging the detected planes in order of size (104) comprises arranging the detected planes from the largest to the smallest, wherein incorporating the points associated with the at least one detected plane comprises incorporating the points associated with the largest detected plane.

3. The method of claim 1 or 2, wherein incorporating the points associated with at least one detected plane into the generated map (105) comprises iteratively adding points associated with the detected planes to the generated map according to a ratio between a size of a detected plane to be incorporated and a size of the submap.

4. The method of claim 1 or 2, further comprising specifying a number of detected planes to be incorporated into the generated map, and adding the points associated with the detected planes to the generated static map until the specified number is reached.

5. The method of any one of claims 1 to 4, further comprising: for each plane, comparing the set of points associated with a plane of the multiple planes of the submap to a threshold value; and incorporating the points which are above the threshold value into the generated map.

6. The method of claim 5, wherein the threshold value is selected based on at least one of the detected planes.

7. The method of any one of claims 1 to 6, wherein generating the map based on the obtained sensor data (102) comprises: applying an algorithm to the obtained sensor data, whereby to categorise each of the multiple points as static, dynamic, or unknown; andapplying a nearest neighbour algorithm to categorise the unknown points as static or dynamic.

8. The method of claim 7, further comprising: for each point categorised as a dynamic point, applying a radial filter to determine whether points neighbouring the dynamic point are static or dynamic; for each point categorised as a static point, applying the radial fdter to determine whether points neighbouring the static point are static or dynamic; and in response to determining that the points neighbouring the dynamic point are static points, re-classifying the dynamic point as a static point, and, in response to determining that the neighbouring points of the static point are dynamic points, re-classifying the static point as a dynamic point.

9. The method of claim 7 or 8, further comprising classifying each LiDAR scan frame of the multiple scan frames based on the generated map, whereby to obtain a classified LiDAR scan comprising an indication of the static and dynamic points having the static and dynamic points denoted.

10. A computer program stored on a non-transitory medium and including code instructions, which, when executed on more or more processors, cause the one or more processors to execute the method of any of claims 1 to 9.

11. An apparatus (600) comprising: a processor (601); a memory (602) coupled to the processor (601), the memory (602) configured to store program code (603) executable by the processor (601), the program code (603) comprising one or more instructions, whereby to cause the apparatus (600) to: obtain sensor data from a 3D LiDAR sensor, the sensor data comprising multiple LiDAR scan frames comprising multiple points and multiple poses associated with the multiple LiDAR scan frames, respectively; generate a map based on the obtained sensor data; combine at least some of the multiple LiDAR scan frames, whereby to obtain a submap, the submap comprising multiple planes, wherein each of the multiple planes is associated with a set of points of the multiple points; detect the multiple planes in the submap and arrange the detected planes in order of size; and incorporate the set of points associated with at least one detected plane into the generated map based on the arrangement of the planes.

12. The apparatus of claim 11, wherein the program code (603) comprising one or more instructions, whereby to cause the apparatus (600) to arrange the detected planes in order of size comprisesprogram code (603) comprising one or more instructions, whereby to cause apparatus (600) to arrange the detected planes from the largest to the smallest, wherein the program code (603) comprising one or more instructions, whereby to cause the apparatus(600) to incorporate the set of points associated with the at least one detected plane comprises program code (603) comprising one or more instructions, whereby to cause apparatus (600) to incorporate the points associated with the largest detected plane.

13. The apparatus of claim 11 or 12, wherein the program code (603) comprising one or more instructions, whereby to cause the apparatus (600) to incorporate the set of points associated with the at least one detected plane into the generated map comprises program code (603) comprising one or more instructions, whereby to cause apparatus (600) to iteratively add points associated with the detected planes to the generated map according to a ratio between a size of a detected plane to be incorporated and a size of the submap.

14. The apparatus of claim 11 or 12, further comprising the program code (603) comprising one or more instructions, whereby to cause the apparatus (600) to: specify a number of detected planes to be incorporated into the generated map, and add the points associated with the detected planes to the generated static map until the specified number is reached.

15. The apparatus of any one of claims 11 to 14, further comprising the program code (603) comprising one or more instructions, whereby to cause the apparatus (600) to: for each plane, compare the set of points associated with a plane of the multiple planes of the submap to a threshold value; and incorporate the points which are above the threshold value into the generated map.

Images