Computing module, system and method of modelling an environment

Abstract

Disclosed herein is a system and method of modelling an environment. The system is configured to receive a LiDAR point cloud of the environment; determine an image point cloud based on a plurality of image frames of the environment, the plurality of image frames comprising at least one reference frame, wherein each of the at least one reference frame comprises a plurality of markers; scale the image point cloud to obtain a scaled point cloud based on a plurality of marker points in the image point cloud, the plurality of marker points corresponding to the plurality of markers; and register the scaled point cloud with the LiDAR point cloud to obtain a multi-modal point cloud of the environment.

Description

COMPUTING MODULE, SYSTEM AND METHOD OF MODELLING AN ENVIRONMENTCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of priority to the Singapore application no. 10202302109U filed July 25, 2023, the contents of which are hereby incorporated by reference in their entirety for all purposes.TECHNICAL FIELD

[0002] This application relates to 3D model building, and more particularly to a computing module for modelling an environment, a system for modelling an environment, and a method of modelling an environment.BACKGROUND

[0003] In the field of environmental modeling, such as Building Information Modelling (BIM), digitalization of an environment often entails construction and / or reconstruction of existing structures and facilities as digital 3D models from point clouds. During the modelling process, areas such as confined spaces / obstructions / occlusions are typically inaccessible to scanning devices, such as Light Detection and Ranging (LiDAR) devices, which present obstacles and physical limitations to the modelling process. However, such areas are often crucial in the various planning and construction phases, during operation and maintenance, as well as during demolition.SUMMARY

[0004] According to an aspect, disclosed herein a computing module for modelling an environment. The computing module comprises: a processing unit; and a non-transitory media readable by the processing unit, the media storing instructions that when executed by the processing unit, causes the processing unit to: receive a LiDAR point cloud of the environment; determine an image point cloud based on a plurality of image frames of the environment, the plurality of image frames comprising at least one reference frame, wherein each of the at least one reference frame comprises a plurality of markers; scale the image point cloud to obtain a scaled point cloud based on a plurality of marker points in the image point cloud, the plurality of marker points corresponding to the plurality of markers; and register the scaled point cloud with the LiDAR point cloud to obtain a multi-modal point cloud of the environment.

[0005] In another aspect, the present disclosure includes a system for modelling an environment. The system comprises an imaging device; and the computing module as described above in signal communication with the imaging device.

[0006] According to yet another aspect, the present disclosure includes a method of modelling an environment using a computing module. The method comprises: receiving a LiDAR point cloud of the environment; determining an image point cloud based on a plurality of image frames of the environment, the plurality of image frames comprising at least one reference frame, wherein each of the at least one reference frame comprises a plurality of markers; scaling the image point cloud to obtain a scaled point cloud based on a plurality of marker points in the image point cloud, the plurality of marker points corresponding to the plurality of markers; and registering the scaled point cloud with the LiDAR point cloud to obtain a multi-modal point cloud of the environment.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Various embodiments of the present disclosure are described below with reference to the following drawings:

[0008] FIG. 1 is a schematic diagram of a system for modelling an environment according to embodiments of the present disclosure;

[0009] FIG. 2 shows an exemplary target part of an environment which includes obstructing structures;

[0010] FIG. 3 is a process flow diagram of the system for modelling an environment of FIG. 1;

[0011] FIG. 4 shows a perspective view of obtaining a LiDAR point cloud according to various embodiments;

[0012] FIG. 5 shows a perspective view of obtaining a video according to various embodiments;

[0013] FIG. 6 A and 6B show respective perspective views of obtaining multiple still images according to various embodiments;

[0014] FIGs. 7A to 7C show multiple exemplar}' markers according to various embodiments;

[0015] FIG. 8 shows multiple exemplary ArUco markers according to various embodiments;

[0016] FIGs. 9A to 9C show target parts of three different environments with the obstructing structures according to various examples;

[0017] FIG. 10 is a flowchart illustrating processes of scaling an image point cloud according to various embodiments;

[0018] FIGs. 11 A and 1 IB show actual distances between markers and point cloud distance between markers according to various embodiments;

[0019] FIG. 12 shows inlier points and outlier points of markers points according to an exemplary implementation;

[0020] FIG. 13 shows the centre points computed with the averaging method and the plane segmentation method according to an exemplary implementation;

[0021] FIG. 14A shows a perspective view of sampling of multiple LiDAR sub-point clouds using a sampling boundary along a sampling path according to various embodiments;

[0022] FIG. 14B is a front view of FIG. 14A;

[0023] FIG. 15 is a photo showing a smartphone with a gimbal according to an exemplary implementation;

[0024] FIGs 16A and 16B are schematic diagrams showing an image frame capturing process of a marker set according to various embodiments;

[0025] FIG. 17 is a floor plan of an academic building with a targeted enclave highlighted by a rectangular box according to an exemplary environment;

[0026] FIG. 18 A shows a top view of a LiDAR scan of a lab with ceiling point clouds removed;

[0027] FIG. 18B shows an image frame of target pipes in the lab of FIG. 18 A;

[0028] FIGs. 19A to 19F shows the respective visualization corresponding to LiDAR scan, videogrammetry, and multi-modal point clouds of target pipes in the academic building of FIG. 17;

[0029] FIG. 20 shows the registration of scaled point cloud with LiDAR point cloud of the academic building of FIG. 17, with the box indicating the location of the scaled point cloud;

[0030] FIGs. 21 A to 2 ID shows the respective re-constructed pipe models based on LiDAR scan and multi-modal point clouds of target pipes in the academic building of FIG. 17;

[0031] FIGs. 22A to 22F shows the respective visualization corresponding to LiDAR scan, videogrammetry and multi-modal point clouds of the target pipes in the lab of FIG. 18 A;

[0032] FIGs. 23 A and 23B show the respective re-constructed pipe models based on LiDAR scan and multi-modal point clouds of the target pipes in the lab of FIG. 18 A;

[0033] FIG. 24A and 24B are views of pipe models re-constructed using LiDAR scan and multi-modal point cloud of FIG. 23A, with the rectangular boxes highlighting reconstruction improvements using the multi-modal point cloud;

[0034] FIG. 25 is a schematic diagram illustrating a method of modelling an environment according to various embodiments of the present disclosure; and

[0035] FIG. 26 shows a block diagram of a processing system for implementing embodiments of the present disclosure.DETAILED DESCRIPTION

[0036] The following detailed description is made with reference to the accompanying drawings, showing details and embodiments of the present disclosure for the purposes of illustration. Features that arc described in the context of an embodiment may correspondingly be applicable to the same or similar features in the other embodiments, even if not explicitly described in these other embodiments. Additions and / or combinations and / or alternatives as described for a feature in the context of an embodiment may correspondingly be applicable to the same or similar feature in the other embodiments.

[0037] In the context of various embodiments, the articles "a", “an” and “the” as used with regard to a feature or element include a reference to one or more of the features or elements.

[0038] In the context of various embodiments, the term “about” or “approximately” as applied to a numeric value encompasses the exact value and a reasonable variance as generally understood in the relevant technical field, e.g., within 10% of the specified value.

[0039] As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0040] The term “environment” may refer to a space / region / area of interest for 3D modelling or environmental modelling. The term “environment” may also refer to an indoor environment, an outdoor environment, a mixture of indoor and outdoor environments, etc. The term “environment” may also refer to a subset or a portion of a building or structure, such as a room, a basement, a single level of the building, etc. The environment may include obstructing objects or obstructions.

[0041] As used herein, the term “obstructing object” may be used to refer to any one or more of the terms “obstruction”, “occlusion”, “obstructive object”, “obstructing structure”, etc., as will be understood from the context. The term “obstructing object” may include building construction elements and / or post-construction elements. Building construction elements may include various components and elements that come together to create a structure like foundations, walls, floors, doors and windows, structural framework, heating, ventilation and air-conditioning (HVAC) systems, pipes and plumbing, electrical cables and systems, finishes, insulation, waterproofing features, etc. Post-construction elements may include tables, chairs, benches, speakers, screens, etc.

[0042] For the sake of brevity, the term "imaging device" may include a device for taking one or more image frames, such as still images / photos / videos of an environment. As used herein, the term “image frame” may be used to refer to any one or more of the terms “photo”, “portion of a photo”, “frame of a video”, “portion of a frame of a video”, etc., as will be understood from the context. The image frame may be in a variety of formats, such as, colour, monotone, infrared, ultraviolet, etc. The imaging device may include examples such as a camera, a video camera, a mobile phone with a camera / video camera, an aerial drone with a camera / video camera, a mobile vehicle with a camera / video camera, a camera or video camera with a gimbal and a combination of the above.

[0043] The term “LiDAR device” refers to a Light Detection and Ranging device for determining ranges by targeting an object or a surface with a laser and measuring the time for the reflected light to return to the receiver. The LiDAR device may be used to obtain a 3D point cloud of an environment.

[0044] As used herein, the term “point cloud” may be used to refer to any one or more of the terms “3D point cloud”, “spatial point cloud”, “3D data points”, “3d point cloud data”, etc., as will be understood from the context. The term “point cloud” may refer to a collection of spatial points that are represented in a coordinate system, such as a cartesian coordinate system with X-coordinates, Y-coordinates, and Z-coordinates. The point cloud data may be used to define a 3D surface or a 3D object that may be used for various purposes, including modelling features such as an interior or partial interior of a building, a terrain or natural landscaping, a mixture of man-made and natural structures, pathways, etc.

[0045] Obstructing objects / structures in an environment often bring about challenges in 3D modelling or 3D model building of the environment. For example, obstructing objects may block the line of sensing / sight during structural scanning, detrimentally affecting modelling accuracy. Tn some environments, building construction features such as wiring and pipes, which often have a complicated and layered structure, may act as obstructing objects during modelling. The building construction features are often congregated at a part of the environment, such as on the ceiling, which increases the difficulty in obtaining an accurate model for conventional scanning devices or scanning methods.

[0046] The present disclosure proposes a system and a method for modelling an environment. According to various embodiments, in addressing such challenges, the proposed system and method performs a point cloud reconstruction from image frames in complement to LiDAR scanning for modelling an environment. As an illustration, the proposed system may be used in the surveying or modelling of partially enclosed areas with obstructing objects. Theobstructing objects may cause obstruction or self-occlusion adversely affecting LiDAR scanning. As another example, the proposed system may be used for surveying a cluttered and confined space where multiple objects in the space obstruct each other. In the above examples, a conventional LiDAR scanner typically does not accurately reflect the environment during environmental modelling.

[0047] According to various embodiments of the present disclosure, the proposed system and method may generate a multi-modal point cloud of an environment based on multiple point clouds of different natures. The multi-modal point cloud may be used with Building Information Modelling (BIM) reconstruction methods to re-construct a 3D model of the environment. The multi-modal point cloud may include a combination, integration or registration of point clouds from multiple sources, overcoming obstructions and allowing higher accuracy and resolution in 3D environmental modelling. As an example, the multi-modal point cloud may include a fusion of LiDAR point clouds obtained from LiDAR scanners and image point cloud obtained from visual images. Motivations for the multi-modal point cloud include improvements to model accuracy, inclusion of missing / obstructed areas in the model, and a higher point cloud density for higher resolutions.

[0048] FIG. 1 illustrates a system 50 for modelling an environment according to various embodiments of the present disclosure. The system 50 may include a computing module 100 configured to generate a multi-modal point cloud of an environment. In comparison to a conventional LiDAR point cloud, the multi-modal point cloud may be a relatively more accurate representation of the environment, in particular, with respect to the target part(s) which include the obstructing objects. In various embodiments, the computing module 100 may include a point cloud reconstruction module 110; a scaling module 120; and a point clouds registration module 130. The computing module 100 may further include a processing unit 1800configured in signal communication with the point cloud reconstruction module 110, the scaling module 120 and the point clouds registration module 130.

[0049] In some embodiments, the system 50 may further be integrated with an imaging device 200, such as a portable device, in signal communication with the computing module 100. As an example, the portable device may be a mobile phone with a camera / video camera. The portable device may facilitate capturing different views of the target part 92 (as illustrated in FIG. 2) of the environment 90 (as illustrated in FIG. 2) as images or image frames. The portable device may also be light weight such that a user may move the portable device during image frame capturing without experiencing difficulties due to bulk or weight. In another example, the portable device may be a mobile drone or an aerial drone with a gimbal-assisted camera / video camera. The mobile / aerial drone may allow access to target part 92 which is deemed too dangerous for a human user, such as a high ceiling or a heated chamber.

[0050] In some embodiments, the system 50 may be in signal communication with a LiDAR device 300 in signal communication with the computing module 100. In this way, LiDAR point cloud captured by the LiDAR device 300 may be directly transmitted to the system 50. In various embodiments, the LiDAR device 300 may be in wired signal communication or wireless signal communication with the system 50.

[0051] According to various embodiments, the environment may include one or more obstructing structures which may detrimentally affect the accuracy of LiDAR scanning. In various embodiments, a target part or target portion of the environment may correspond to an area of the environment with the obstructing structure. In other words, the location of the obstructing structure may be considered the target part of the environment. Without being limiting, examples of the environment may include a partially open walkway of a building or an enclosure on an open terrain. In such examples, the target part of the environment may be a localized area in the environment.

[0052] FIG. 2 illustrates an exemplary target part 92 of an environment 90. The environment 90 may be a room 90 with pipes and wirings 94 (obstructing structures) fixed to a ceiling 92 of the room 90. In the example, the target part 92 may correspond to the ceiling 92 which has the pipes and wirings 94 fixed thereto. When viewed from one or multiple positions, portions of or whole of the pipes and wirings 94 may be obstructed or blocked by other structures. In addition, structural elements such as the ceiling area behind the pipes and wirings 94 may also be obstructed. Therefore, during LiDAR scanning or other form of 3D scanning, the obstructing structures act to occlude the line of sight of the scanning process.

[0053] In some embodiments, the obstructing structure 94 may be a deliberate structure such as a building construction structure 94 or mechanical, electrical, and plumbing (MEP) structures 94. The inclusion of the deliberate structure during model building facilitates the planning and execution during building construction and upgrading. In other embodiments, the obstructing structure may be a non-deliberate structure or an unintended structure, such as a broken pipe, detached wiring, or a collapsed roof. The inclusion of the non-dclibcratc structures during model building facilitates maintenance, repair and demolition.

[0054] FIG. 3 illustrates a process flow of the system 50 for modelling an environment. The process flow generally includes the stages of: i) the point cloud reconstruction module 110 to obtain an image point cloud 115 of a target area of the environment; ii) the scaling module 120 to scale the image point cloud 115 into a scaled point cloud 125 to achieve dimension accuracy with respect to a LiDAR point cloud 305, and iii) the point clouds registration module 130 to register the scaled point cloud 125 with the LiDAR point cloud 305 to form a multi-modal point cloud 135.

[0055] According to various embodiments, the process of modelling the environment 90 may begin with obtaining a LiDAR point cloud 305 of an environment 90 and obtaining a plurality of image frames 205 of the environment 90. The plurality of image frames 205 maybe captured based on a target part 92 or target portion of the environment 90. The LiDAR point cloud 305 and the plurality of image frames 205 may be obtained or captured at different time instances or at non-overlapping time points. In other words, the LiDAR point cloud 305 and the plurality of image frames 205 may be obtained separately in different sessions. This allows previously obtained LiDAR data and / or visual images of the environment 90 to be used or reused during the modelling process, advantageously reducing the cost relating to performing repeated LiDAR scans during environmental modelling.

[0056] In various embodiments, as shown in FIG. 4, a standalone LiDAR device 300 may be used to obtain the LiDAR point cloud 305 of the environment 90 at a first time instance (Tl). The LiDAR point cloud 305 may often be incomplete due to occlusion / obstructions from structures, thus limiting subsequent usage of the LiDAR point cloud / data such as scene parsing and 3D model reconstruction.

[0057] Further referring to FIG. 5, in addition to obtaining LiDAR point cloud 305, an imaging device 200 such as a video camera may be configured to capture a plurality of image frames 205 of a target part 92 of the environment 90 at a second time instance (T2). Therefore, the LiDAR point cloud and the plurality of image frames 205 may be obtained in respective non-overlapping time points. In various embodiments, as shown in FIG. 5, the plurality of image frames 205 may be in the form of a video. The video may be captured when the imaging device 200 moves from a first position 200a to a second position 200b. Therefore, the plurality of image frames 205 may be a series of continuous frames showing gradually different views of the target part 92 of the environment. In some examples, the first position 200a and the second position 200b may cover a 120-degrees angle of view of the target part 92.

[0058] In other embodiments, as shown in FIGs. 6A and 6B, the imaging device 200 may be a camera configured to capture a plurality of image frames 205 of a target part 92 of the environment 90 at a third time instance (T3). The plurality of image frames 205 may be in theform of still images captured by the camera 200 between a first position 200a and a second position 200b. It may be appreciated that the still images of the target part 92 may be taken from a variety of intermediate positions between the first position 200a and the second position 200b. Therefore, the plurality of image frames 205 may capture the target part 92 from multiple points of view or angles.

[0059] According to various embodiments, the LiDAR point cloud 305 and the plurality of image frames 205 may be received by or transmitted to the computing module 100 at different time instances. In some embodiments, to allow faster input acquisition, the image frames 205 may be a series of continuous frames of a video. This may also allow a simpler and more robust capturing process, requiring little operational knowledge from the user, such as photography knowledge. In some embodiments, the image frames 205 may include continuous frames from a video in addition to still images from a camera. The still images may be image frames of a higher resolution in comparison to the continuous frames from the video.

[0060] In some embodiments, the computing module 100 may trigger the imaging device 200 to capture the image frames 205 of the target part 92 of the environment 90 and receive the image frames 205 from the imaging device 200. In other words, the computing module 100 may assist in the image frames 205 capturing process of the target part 92. This allows the computing module 100 to capture image frames 205 of particular angle of view(s) which may improve the proposed building modelling process. In some embodiments, the computing module 100 may trigger the imaging device 200 to capture a first plurality of image frames based on a video and to capture a second plurality of image frames based on still images.

[0061] In some embodiments, referring to FIGs. 5, 6A and 6B, the target part 92 of the room 90 may include an obstructing structure 94, such as mechanical, electrical, and plumbing (MEP) structures 94. In addition, the target part 92 of the room 90 may include one or more physicalmarkers 80. The markers 80 may be attached to the target part 92 of the environment 90. In some embodiments, two or more markers 80 may collectively be known as a marker set 81.

[0062] In various embodiments, the plurality of image frames 205 captured by the imaging device 200 may include reference frames, which are image frames with a full maker set 81 or multiple markers 80 in the image frames. In addition, the plurality of image frames 205 may also include scene frames, which are image frames without markers 80 / marker set 81. Scene frames may also be frames w hich include only a single marker 80 or a part / portion of the marker set 81.

[0063] As an illustration, referring to FIG. 6A, in an example, the image frame 205 may be a reference frame 205a which includes the marker set 81 in the target part 92 of the environment 90. In other 'ords, the reference frame 205a includes a full image of the marker set 81. Referring to FIG. 6B, in another example, the image frame 205 may be a scene frame 205b which includes only a portion of the marker set 81 in the target part 92 of the environment 90. In other words, the scene frame 205b may include a partial image of the marker set 81 or docs not include the marker set 81. In some embodiments, the reference frames 205a may be captured before the scene frames 205b. Therefore, the image frames 205 include continuous frames of a video which include one or more reference frames 205a followed by one or more scene frames 205a in a sequential manner.

[0064] For each marker set 81, the relative distances or the actual distances between the plurality of markers 80 are known. Referring to FIGs. 7 A to 7C, each marker set 81 may include at least two markers 80a / 80b. In some embodiments, each marker 80a / 80b may include at least two dimensions. For example, as shown in FIG. 7A, each marker 80a / 80b may be a two- dimensional marker, with a width and a height. In other examples, as shown in FIG. 7B, each marker 80a / 80b may be a three-dimensional marker 80a / 80b, with a w'idth, a height and a depth. In yet another example, as shown in FIG. 7C, the markers 80a / 80b may extend along a singledirection with alternate shadings (black or white). In other embodiments, as shown in FIG. 8, the markers 80c / 80d / 80e / 80f or marker set 81 may be ArUco markers or ArUco marker sets. The markers 80 or ArUco markers 80c / 80d / 80e / 80f at the initial period or beginning of a video, may be used for scaling of frames. As it may be appreciated, the markers 80 may be configured or selected based on the application and / or implementation, such as the size and shape of the surface for attaching the markers 80.Point cloud reconstruction

[0065] In various embodiments, upon receiving or obtaining the plurality of image frames 205 comprising reference frames 205a and scene frames 205b, the computing module 100 may be configured to control the point cloud reconstruction module 110 to determine an image point cloud 1 15 based on the plurality of image frames 205.

[0066] In some embodiments, the point cloud reconstruction module 110 may generate a sparse point cloud based on the plurality of image frames 205. Thereafter, the point cloud reconstruction module 110 may generate a dense point cloud based on the sparse point cloud in a subsequent step. In some embodiments, the point cloud reconstruction module 110 may directly generate a dense point cloud based on the plurality of image frames 205. In some embodiments, the image point cloud 115 from the point cloud reconstruction module 110 may include a point cloud density higher than the LiDAR point cloud 305.

[0067] In some embodiments, the point cloud reconstruction module 110 may be a videogrammetry module and / or a photogrammetry module. In other words, the point cloud reconstruction module 1 10 may perform the digitalization technique of videogrammetry reconstruction and / or photogrammetry reconstruction, in which a collection of image frames 205 capturing an environment or an object from multiple viewpoints are used to form a 3D surface mesh or 3D point cloud.

[0068] In various embodiments, the videogrammetry reconstruction typically comprises the steps of: 1) image preparation and frame extraction, 2) Structure from Motion (SfM), and 3) Multi-View Stereo (MVS). Image preparation typically includes extracting frames from the video(s) to be used for videogrammetry reconstruction. In some embodiments, a uniform time sampling method may be adopted for frame extraction. In some embodiments, image preparation may also include augmenting the image frames 205 with metadata, wherein image frames 205 which lack metadata may be augmented to include metadata such as camera model, focal length, depth of field, aperture, etc.

[0069] In some embodiments, the point cloud reconstruction module 110 may sort the plurality of image frames 205 into reference frames and scene frames. For embodiments where ArUco markers are used, the reference frames may also be known as ArUco frames. In other words, the ArUco frames are frames with the markers 80c / 80d / 80e / 80f (FIG. 8) visible and detectable in the image frames, while the scene frames are the remaining frames. The sorting of image frames 205 may be performed in conjunction with the frame extraction and augmentation of metadata (if required).

[0070] In various embodiments, a 3D-based key frame selection algorithm may be implemented for the frame extraction. In various embodiments, when a frame is sorted and categorized as a reference frame (or an ArUco frame), a marker detection process may be performed. Multi-thread processing may be implemented to accelerate the process. In some embodiments, the frames to be extracted may be inserted into a queue with a corresponding sequence number.

[0071] The image frames as extracted may be used in the step of Structure from Motion (SfM) for the reconstruction of a sparse point cloud. Thereafter, a dense point cloud may be generated from the sparse point cloud in the Multi-View Stereo (MVS) step. It is worth noting that the point cloud reconstruction module 110 may determine the image point cloud 115 basedon both the reference frames (ArUco frames) and the scene frames. Therefore, avoiding the selective filtering process(es) which are often computationally expensive.

[0072] In an example implementation, OpenMVG and OpenMVS were used for the respective SfM step and MVS step of videogrammetry reconstruction. OpenMVG may be used for the SfM step. The SfM step may be a customizable step with multiple configurable parameters as provided in Table 1 .Table 1. Configurable parameters and corresponding options in OpenMVG.

[0073] As shown in FIGs. 9A to 9C, tests were conducted to determine the optimal option under each parameter using videos of respective target parts of three different environments with the obstructing structures including a table, a fire hose enclosure and pipes, respectively. Criteria for the option selection are camera orientation accuracy, reconstruction speed, and point cloud quality. Camera orientation accuracy is defined as the ability to orient all given images. Reconstruction speed is defined as the time required to complete reconstruction. Point cloudquality is the visual qualitative assessment of the generated image point cloud including the point coverage and accuracy of the target objects. Referring to Table 1, each of the parameters were tested separately by fixing the other parameters, with the options as selected underlined and in bold. Thereafter, the SIM output from OpenMVG was provided to OpenMVS for the MVS step, to re-construct a dense point cloud or the image point cloud 115 of the target part of the environment. It may be appreciated that the above implementation is merely exemplary, and other implementations of point cloud reconstruction may also be viable via the point cloud reconstruction module 110.Scaling of image point clouds

[0074] Referring to FIG. 10, in various embodiments, upon the point cloud reconstruction module 1 10 generating the image point cloud 1 15 from the plurality of image frames 205, the computing module 100 may control the scaling module 120 to scale the image point cloud 115 to obtain a scaled point cloud 125. In various embodiments, the scaled point cloud 125 may have a substantially similar scale in relative to the LiDAR point cloud 305. This allows improved point clouds registration or point clouds combination between the scaled point cloud 125 and the LiDAR point cloud 305.

[0075] In various embodiments, the scaling of the image point cloud 115 may include extracting data points from the image point cloud 115 and resolving an optimization problem using a cost function to determine a scale multiplier 124. In various embodiments, the scaling module 120 may scale the image point cloud 115 using the scale multiplier 214 to obtain the scaled point cloud 125.

[0076] In various embodiments, the scaling module 120 may determine a plurality of marker points 121 from the image point cloud 115. The marker points 121 may correspond to the position / location of the markers 80 or marker set 81 in the image point cloud 115. In some embodiments, the scaling module 120 may determine the marker points 121 directly from theimage point cloud 115. In some embodiments, the scaling module 120 may determine the marker points 121 from a sparse point cloud generated by the point cloud reconstruction module 110. In embodiments where a sparse point cloud is input into the scaling module 120, a dense point cloud may be generated upon completion of the scaling step. In other embodiments, the scaling module 120 may determine the marker points 116 based on a dense point cloud generated by the point cloud reconstruction module 1 10.

[0077] In various embodiments, the optimization problem may include minimizing an error between the respective pairs of actual distances 84 and re-constructed point cloud distances 123 of the markers 80. The actual distances 84 may refer to the distance between the markers 80 which were predetermined or known. Point cloud distance 123 may refer to the computed distance between the markers 80 based on the marker points 1 16 which correspond to the location / position of the markers 80. In other words, the actual distances 84 between the markers 80 (for example, along X-axis and Y-axis) are compared to point cloud distances 123 between the markers 80 obtained from the reconstruction process.

[0078] In some embodiments, the scaling module 120 may obtain a scale multiplier 124 based on an actual distance 84 (for example, AD1 / AD2 of FIG. 1 1 A) between the plurality of markers 80 and a point cloud distance 123 (for example, CD1 / CD2 of FIG. 1 IB) between the plurality of markers 80. hi some embodiments, the scale multiplier 124 may be obtained from a cost function, by minimizing an error between the actual distance 84 between the multiple markers 80 and the point cloud distance 123 between the multiple markers 80. Thereafter, the scaling module 120 may scale the image point cloud 1 15 based on the scale multiplier 124 to obtain a scaled point cloud 125.

[0079] In some embodiments as shown in path 120a of FIG. 10, to determine a point cloud distance 123, the scaling module 120 may first determine a respective centre point 82 for each of the marker 80 based on the marker points 121. The scaling module 120 may determine thecentre point 82 by computing an average based on the respective location of the marker points 121 of each marker 80. The scaling module 120 may then determine the point cloud distance123 based on the respective centre point 82 of the markers 80. Thereafter, the scale multiplier124 may be determined based on the point cloud distances 123. This method may be known as the averaging method in which an averaging of the marker points 121 is performed in determining the point cloud distance 123.

[0080] FIG. 8 illustrates an exemplary implementation of the averaging method. The markers 80 may include four ArUco markers 80e / 80d / 80e / 80f arranged in two rows by two columns array. The ArUco markers 80e / 80d / 80e / 80f may be configured from a selection of 7 by 7 squares. This advantageously provides more edges on the markers, and thus allowing more feature points (or marker points 1 16) to be extracted. In the implementation, the detection of ArUco markers may be implemented using OpenCV, which outputs the corner positions of the markers 80 and aids in identifying specific markers in the sparse point cloud from the vidcogrammctry process. Extraction of ArUco marker points may be performed leveraging on the intermediate output “feat” fde and “sfm data” file from OpenMVG. The “feat” file lists all feature points in each frame with XY coordinates and the “sfm data” file stores the association between points in the sparse point cloud and feature points in frames. Using the detected bounding box of each ArUco marker in each reference frame, feature points belonging to the ArUco marker may be identified. This enables the determination of feature points of the ArUco marker by tracing the relationships in the “sfm data” file. The respective location of each ArUco marker, i.e., the centre point of the marker, was then computed by averaging the respective feature points of each marker 80c / 80d / 80e / 80f. The scale multiplier may then be obtained by minimizing an error between the actual distances between markers and the point cloud distances in the sparse point cloud multiplied by the scale multiplier, as defined inEquation 1. FIG. 8 illustrates the respective marker distances defined as the horizontal offset(dhorizontal) <Hld vertical offsets (dvcrtical).

[0081] However, with reference to FIGs. 1 1 A and 1 1B, obtaining the point cloud distances 123 directly from the marker points by computing the marker locations through the averaging method may introduce errors in some instances. A reason for such error may be due to outlier points present in the marker points 116 and uneven distribution of the marker points 116, as shown in FIG. 11B.

[0082] Referring again to FIG. 10, in various embodiments as shown in path 120b, to determine the point cloud distance 123 between the multiple markers 80, the scaling module 120 may first perform a plane segmentation on the marker points 121 for each of the markers 80. The scaling module 120 may determine a segmentation plane 122 for each of the markers 80 based on the marker points 121. The scaling module 120 may determine the inlier points 121a from the marker points 121 corresponding to each of the markers 80, based on the respective segmentation plane 122. It may be appreciated that the inlier points 121a are a subset of the marker points 121. Similarly, outlier points 121b are also a subset of the marker points 121. In various embodiments, the scaling module 120 may determine a respective centre point 82 for each of the markers 80 based on the respective inlier points 121a. Therefore, outlier points 121b are not used in the determining of the respective centre points 82 of the markers 80.

[0083] In some embodiments, the centre point 82 may also include a collective centre point between all the markers 80. In other words, a centre point of the marker set 81. The centre points 82 of each of the markers 80 or market set 81 may be used to determine a point clouddistance 123 between the markers 80. This method may be known as the plane segmentation method in which outlier points 121b are filtered or removed prior to determining the centre point of the respective markers.

[0084] In an exemplary implementation of the plane segmentation method, Open3D was used to perform plane segmentation on marker points 121 of four ArUco markers 80c / 80d / 80e / 80f. FIG. 8 shows the four ArUco markers 80c / 80d / 80e / 80f and FIG. 12 shows the marker points 121 of the four ArUco markers 80c / 80d / 80e / 80f. As an initial step, the respective segmentation planes 122c / 122d / 122e / 122f may be determined for each of the markers 80c / 80d / 80e / 80f. Thereafter, inlier points 121a corresponding to each of the markers 80c / 80d / 80e / 80f may be determined based on the respective segmentation planes 122c / 122d / 122e / ! 22f. The marker points 121 within the segmentation planes 122c / 122d / 122e / 122f are inlier points 121a (in black) and the marker points outside the segmentation planes 122c / 122d / 122e / 122f are outlier points 121b (in white). The inlier points 121 may be used to determine edge points in each axis (X axis and Y axis) corresponding to the edges or extremes of each marker 80c / 80d / 80e / 80f. As the centre points correspond to a respective middle coordinate of each marker 80c / 80d / 80e / 80f, the centre points may be computed using a mid-point between the edge points for each marker 80c / 80d / 80e / 80f in both the X axis and Y axis. Thereafter, the point cloud distance 123 between the markers 80c / 80d / 80e / 80f (along X-axis and Y-axis) may be determined based on the computed centre points. This improves the accuracy of calculating the centre point of each marker as plane segmentation removes or filters the outlier points 121b and uneven distribution of marker points. Through the use of plane segmentation, a better scaling accuracy of the image point cloud 115 may be achieved.

[0085] FIG. 13 illustrates a comparison between the computed centre points 82c / 82d / 82e / 82f / 82g of the markers 80c / 80d / 80e / 80f (FIG. 8) using the averaging method(white with black surround) and the plane segmentation method (black with white surround). It may be seen that the centre points 82c / 82d / 82e / 82f / 82g of the plane segmentation method (black with white surround) are closer to the actual centre points of each marker 80c / 80d / 80e / 80f in comparison to the averaging method (white with black surround).Point clouds registration

[0086] According to various embodiments, referring again to FIG. 3, upon obtaining the scaled point cloud 125, the computing module 100 may control the point clouds registration module 130 to register the scaled point cloud 125 with the LiDAR point cloud 305 to obtain a multi-modal point cloud 135 of the environment 90. Registering the scaled point cloud 125 with the LiDAR point cloud 305 refers to aligning and combining the respective point clouds 125 / 305 to build a multi-modal point cloud 135.

[0087] In various embodiments, as the image frames 205 captured are targeted at one or more parts of the environment 90 instead of the entire environment 90, the scales and data sizes between the image point cloud 125 and the LiDAR point cloud 305 typically differ substantially. Further, different implementations may employ different imaging devices or parameters (such as different focal length, aperture, etc.), which also results in differing scales and sizes for the image frames 205 captured relative to the LiDAR point cloud 305. Therefore, in comparison to registering the image point cloud 115 directly with the LiDAR point cloud 305, registering the scaled point cloud 125 with the LiDAR point cloud 305 provides a more robust registration result due to the substantially similar scale between the scaled point cloud 125 and the LiDAR point cloud 305.

[0088] According to various embodiments, to register the scaled point cloud 125 with the LiDAR point cloud 305, the LiDAR point cloud 305 may be cropped or divided into smaller sections for point cloud registration. Referring to FIGs. 14A and 14B, the point cloudsregistration module 130 may be configured to sample multiple LiDAR sub-point clouds 307a / 307b / 307c from the LiDAR point cloud 305. The registration module 130 may sample the LiDAR sub-point clouds 307a / 307b / 307c by moving a sampling boundary 310 in the LiDAR point cloud 305. The sampling boundary 310 may include predetermined dimensions such as height (H), width (W), and length (L). In other embodiments, the predetermined dimension 310 may be determined based on the scaled point cloud 125. For example, the predetermined dimension 310 may be configured to be of similar size or view of field with the scaled point cloud 125.

[0089] In various embodiments, the sampling boundary 310 may move along a sampling path 312 to sample the LiDAR sub-point clouds 307a / 307b / 307c from the LiDAR point cloud 305. The sampling path 312 may be configured such that the sampling boundary 310 moves in each of the three axes (X-axis, Y-axis and Z-axis) sequentially or concurrently to sample the whole of the LiDAR point cloud 305. In some embodiments, the sampling path 312 may further include an inter-boundary distance which is a distance (along one or more of the three axes) between different positions of the sampling boundary 310 along the sampling path 312.

[0090] In various embodiments, the point clouds registration module 130 may determine whether each of the sampled LiDAR sub-point clouds has a point cloud count higher or equal to a predetermined point threshold. This ensures that each of the LiDAR sub-point clouds 307a / 307b / 307c may have sufficient cloud points density for an efficient point clouds registration process. Therefore, during the sampling process, samples which did not meet the point threshold may be discarded upon sampling. Discarding samples which did not meet the point threshold allows efficient use of resources in the computing module 100, reducing computational costs in the point cloud registration process.

[0091] In various embodiments, the point clouds registration module 130 may determine a matching sub-point cloud from the LiDAR sub-point clouds 307a / 307b / 307c. Based on thescaled point cloud 125, the point clouds registration module 130 may determine a matching sub-point cloud from the sampled and non-discarded LiDAR sub-point clouds 307a / 307b / 307c. In various embodiments, to determine the matching sub-point cloud, the point clouds registration module 130 may determine a respective similarity score between the scaled point cloud 125 and each of the LiDAR sub-point clouds 307a / 307b / 307c. Further, the point clouds registration module 130 may determine the matching sub-point cloud based on the LiDAR subpoint cloud with the highest similarity score with the scaled point cloud 125.

[0092] In some embodiments, the similarity score may be determined based on a non-linear regression function, a cost function, etc. formulated as an optimization problem, etc.

[0093] In some embodiments, to determine the similarity score, the point clouds registration module 130 may input the scaled point cloud 125 and each of the LiDAR sub-point clouds 307a / 307b / 307c into a machine learning model to obtain respective similarity score from the machine learning model. The machine learning model may be a convolutional neural network having multiple layers, such as pooling layers and dense layers. In some embodiments, batch calculation of the similarity scores between the LiDAR sub-point clouds and the scaled point cloud 125 may be performed.

[0094] In some exemplary implementations, the point clouds registration module 130 may compute the similarity score using the feature extraction descriptor Fully Convolutional Geometric Features (FCGF), as well as the correspondence confidence prediction module proposed by Deep Global Registration (DGR). FCGF embeds each input point with a feature vector of its local neighbourhood. An initial set of point correspondences between the two input point clouds may be generated naively by nearest neighbour search in the feature space. The point correspondences may then be evaluated using a 6D convolutional network which predicts the confidence that the correspondences are true. The total confidence of all correspondences may be regarded as the similarity score of the pair of input point clouds. Thereafter, the LiDARsub-point cloud with the highest similarity score may be registered or combined with the scaled point cloud.

[0095] In some embodiments, during the point cloud registration process, the point clouds registration module 130 may apply a coarse registration of the scaled point cloud with the matching sub-point cloud, this enabling a faster alignment of point cloud. Thereafter, the point clouds registration module 130 may apply a fine registration of the scaled point cloud with the matching sub-point cloud which allows fine positioning between the two point clouds. In some implementations, the coarse registration may be implemented using DGR followed by a fine registration using an iterative closest point (ICP) algorithm from the Open3D library. The scaled point cloud 125 may then be merged with the LiDAR point cloud 305 to generate the multimodal point cloud 135.Exemplary implementation of the proposed system

[0096] The following section discloses an exemplary implementation of the proposed system. Faro S70 was used as the LiDAR scanner for the scene scanning and a personal Samsung S22 smartphone was used as the imaging device for video recording using ultra-wide mode at ultra HD resolution, 60 FPS. The imaging device (Samsung S22) includes a known focal length for video-recording. A gimbal, DII OM 5, as shown in FIG. 15, was also used to facilitate the recording process by improving the stability and sharpness of the frames captured when the imaging device is in motion. DJI OM 5 was chosen for its unique feature of having a built-in extension rod, allowing the attachment and control of the smartphone via the handle spaced apart from the smart phone. The extension rod facilitates the control of the recording angle of the smart phone, which aids in recording obstructing objects, such as mechanical, electrical, and plumbing (MEP) structures, which are occluded during LiDAR scanning. Theextension rod also enables the MEP structures to be visible from angles inaccessible from the average user’s reach.

[0097] Upon identifying the target area or target part of the environment, the marker set 81 comprising two or more markers was attached to a wall near the obstructing object at the target part of the environment. The video-recording or image frame capturing was conducted with the capturing of the markers / marker set 81 as reference frames followed by the scene frames. Both the reference frames and the scene frames were collectively used for the point cloud reconstruction process (videogrammetry process). FIGs. 16A and 16B schematically illustrates an example of capturing the markers / marker set 81 as reference frames. Image frames (video) of the markers / marker set 81 were captured by moving the camera along an arc-shaped path 210 centered at the markers / marker set 81 . Tn addition, the image frames (video) were captured at three different heights to allow sufficient features for point cloud reconstruction of the markers / marker set 81. Upon the capturing of the markers / marker set 81 as reference frames, the session continued to the other part of the environment to be captured as scene frames.Experiments using the proposed implementation

[0098] The proposed implementation was tested on two separate environments. The first environment was on a single level of an academic building in Nanyang Technological University (NTU). The academic building was designed with a free -formed shape mimicking a hive. As shown in the floorplan in FIG. 17, each floor in the academic building comprises several rooms with enclaves formed therebetween. Each of the enclaves comprises MEP structures along the ceiling which were identified as potential target parts for the test. An incomplete enclave 92, as shown in FIG. 17, was identified as the targeted area or target part92 of the environment 90. FIG. 2 shows the target part 92 with obstructing pipes 94.

[0099] FIGs. 18A and 18B shows a research lab in NTU which was the second environment for the test. The ceiling and the pipes in the research lab were painted in black which is challenging for the extraction of image frames. An external light was used during the video recording to enhance the quality of the captured image frames. FIG. 18A illustrates the LiDAR scan performed with the target pail 92 ended. FIG. 18B shows the multiple piping systems in the lab arranged in close proximity near the ceiling and walls. The piping systems were chosen as the targeted obstructing objects in both the academic building and the lab environments as piping systems are often self-occluding which results in incomplete LiDAR scans.

[0100] Performance indicators and results of the point cloud reconstruction (videogrammetry) and point cloud registration are shown in Table 2.Table 2. Videogrammetry and point cloud registration results

[0101] 3D models of the piping system were re-constructed using a Building Information Modelling (BIM) reconstruction method. The targeted pipes in the LiDAR point cloud and the multi-modal point cloud were respectively re-constructed using the BIM reconstruction method. Table 3 shows the performance comparison between the BIM reconstruction based on the LiDAR scan (LiDAR point cloud) and the Multi-modal point cloud respectively.Table 3. BIM reconstruction performance on LiDAR and multi-modal point clouds

[0102] We first refer to the results obtained for the academic building environment. FIGs. 19A to 19F show the LiDAR scan, the scaled point cloud (videogrammetry point cloud), and the multi-modal point cloud of the target pipes. The ceiling and floor points were manually removed for a clearer visualization. Due to the occlusion and insufficient point clouds, the pipes in the LiDAR scan were either broken in the middle or have small surface coverage which resulted in an incomplete LiDAR scan.

[0103] In comparison to the LiDAR scan, the proposed implementation provided a portable and flexible solution for capturing different angles and positions of the target area. Further, the point cloud reconstruction method (videogrammetry method) generated a relatively complete point cloud of the pipes in the target area.

[0104] As shown in FIG. 17, challenges remain in the point cloud registration process due to the existence of eight other similar enclaves in the LiDAR point cloud. Parameters used in the point cloud registration are listed in Table 4. The input LiDAR scan was cropped into 2775 sections or sub-point clouds, with 2088 of the sub-point clouds filtered by or lower than the predetermined point threshold. As shown in FIG. 20, the proposed point cloud registration method was able to successfully identify the correct enclave for point clouds registration. In addition, the final root means square error (RMSE) between the two point clouds was at 0.0039 metres.Table 4. Automatic registration parameters

[0105] Accuracy of the scale rectification was evaluated by comparing the mean distances between two parallel planes extracted from the LiDAR point cloud and the scaled point cloud. The floors and ceilings of the single level of the academic building were chosen as the two parallel planes. Distances between points on the floor and points on the ceiling were used in both point clouds, and the difference of the mean distances was regarded as the error. For the test conducted in the single level of the academic building, the scale rectification error was 0.016%.

[0106] Further, as shown in FIGs. 21A to 21D, the targeted pipes were re-constructed using a Building Information Modelling (BIM) reconstruction method, based on the LiDAR point cloud and the multi-modal point cloud. The pipe segments re-constructed based on the LiDAR point cloud had three missing pipe segments and one incomplete pipe segments as indicated in the black boxes as shown in FIG. 2 IB. The missing pipe segments were due to incompleteness in the LiDAR point clouds. In contrast, as shown in FIG. 21 A, the pipe segments re-constructed based on the multi-modal point cloud were complete.

[0107] Similar experiments were conducted for the lab environment as shown in FIG. 18A. FIGs. 22A to 22F illustrate the LiDAR point cloud, the scaled point cloud (videogrammetry point cloud) and the multi-modal point cloud of the target pipes in the lab environment. As the lab environment is small in relative to the academic building (FIG. 17), the LiDAR point cloud was cropped into 23 sections or sub-point clouds, with 22 sub-point clouds satisfying the similarity check based on the point threshold. The proposed point cloud registration methodwas able to successfully register the scaled point cloud (videogrammetry point cloud) in 1.8 minutes with an inlier RMSE of 4 millimetres.

[0108] The Building Information Modelling (BIM) reconstruction method was also applied to the LiDAR point cloud and the multi-modal point cloud with the results shown in FIGs. 23 A to 23B. The BIM reconstruction from LiDAR point cloud had four missing pipes, one connection error, and three incomplete pipes as shown in FIG. 23A. Significant improvements were observed in the BIM reconstruction using the multi-modal point cloud, as highlighted by boxes in FIG. 23B. FIG. 24A shows the pipes corresponding to the location of the box in FIG. 23 A. Due to the high number of occlusions, reconstruction from the LiDAR point cloud resulted in three errors in the respective re-constructed BIM model, including one missing pipe, one connection error, and one incomplete pipe. Tn comparison, the same target part re-constructed from the multi-modal point cloud is shown in FIG. 24B. It is evident from FIG. 24B that the BIM reconstruction based on multi-modal point cloud does not include any error and has a higher accuracy in comparison to the BIM reconstruction based on LiDAR point cloud.

[0109] According to another aspect of the disclosure, FIG. 25 illustrates a method 700 of modelling an environment. The method 700 include in frame 710, receiving a LiDAR point cloud of the environment; in frame 720, determining an image point cloud based on a plurality of image frames of a target part of the environment, the plurality of image frames comprising at least one reference frame, wherein each of the at least one reference frame comprises a plurality of markers; in frame 730, scaling the image point cloud to obtain a scaled point cloud based on a plurality of marker points in the image point cloud, the plurality of marker points corresponding to the plurality of markers; and in frame 740, registering the scaled point cloud with the LiDAR point cloud to obtain a multi-modal point cloud of the environment.

[0110] The proposed system and method include a complete pipeline of image frame reconstruction from multiple image frames, scale rectification of an image point cloud, andautomatic registration of a scaled point clouds with a LiDAR point cloud to generated a multimodal point cloud. By using markers such as ArUco markers, the scale of the image point cloud may be rectified to near 99.8% accuracy in relative to actual marker dimensions. In addition to a reconstruction process of generating an image point cloud from multiple image frames, the present disclosure also includes a local-to-global automatic point cloud registration process which aids in registering two input point clouds, i.e. the scaled point cloud and the LiDAR point cloud, in view of the vast size or dimensional differences. Two experiments were performed showcasing the feasibility of the proposed solution in improving the point cloud-based Building Information Modelling (BIM) reconstruction.

[0111] In accordance with embodiments of the present disclosure, a block diagram representative of components of processing system 1800 that may be provided within controller 120 to carry out the digital signal processing functions or computations in accordance with embodiments of the disclosure, or any other modules or sub-modules of the system is illustrated in FIG. 26. One skilled in the art will recognize that the exact configuration of each processing system provided within these modules may be different and the exact configuration of processing system 1800 may vary and the arrangement illustrated in FIG. 26 is provided by way of example only.

[0112] In embodiments of the disclosure, processing system 1800 may comprise controller1801 and user interface 1802. User interface 1802 is arranged to enable manual interactions between a user and the computing module as required and for this purpose includes the input / output components required for the user to enter instructions to provide updates to each of these modules. A person skilled in the art will recognize that components of user interface1802 may vary from embodiment to embodiment but will typically include one or more of display 1840, keyboard 1835 and optical device 1836.[00113J Controller 1801 is in data communication with user interface 1802 via bus 1815 and includes memory 1820, processing unit, processing element or processor 1805 mounted on a circuit board that processes instructions and data for performing the method of this embodiment, an operating system 1806, an input / output (I / O) interface 1830 for communicating with user interface 1802 and a communications interface, in this embodiment in the form of a network card 1850. Network card 1850 may, for example, be utilized to send data from these modules via a wired or wireless network to other processing devices or to receive data via the wired or wireless network. Wireless networks that may be utilized by network card 1850 include, but are not limited to, Wireless-Fidelity (Wi-Fi), Bluetooth, Near Field Communication (NFC), cellular networks, satellite networks, telecommunication networks. Wide Area Networks (WAN) and etc.

[0114] Memory 1820 and operating system 1806 are in data communication with processor 1805 via bus 1810. The memory components include both volatile and non-volatile memory and more than one of each type of memory, including Random Access Memory (RAM) 1823, Read Only Memory (ROM) 1825 and a mass storage device 1845, the last comprising one or more solid-state drives (SSDs). One skilled in the art will recognize that the memory' components described above comprise non-transitory computer-readable media and shall be taken to comprise all computer-readable media except for a transitory, propagating signal. Typically, the instructions are stored as program code in the memory components but can also be hardwired. Memory 1820 may include a kernel and / or programming modules such as a software application that may be stored in either volatile or non-volatile memory.

[0115] Herein the term “processor” is used to refer generically to any device or component that can process such instructions and may include: a microprocessor, a processing unit, a plurality of processing elements, a microcontroller, a programmable logic device or any other type of computational device. That is, processor 1805 may be provided by any suitable logiccircuitry for receiving inputs, processing them in accordance with instructions stored in memory and generating outputs (for example to the memory components or on display 1840). In this embodiment, processor 1805 may be a single core or multi-core processor with memory addressable space. In one example, processor 1805 may be multi-core, comprising — for example — an 8 core CPU. In another example, it could be a cluster of CPU cores operating in parallel to accelerate computations.

[0116] All examples described herein, whether of methods, materials, or products, are presented for the purpose of illustration and to aid understanding and are not intended to be limiting or exhaustive. Modifications may be made by one of ordinary skill in the art without departing from the scope of the invention as claimed.

Claims

CLAIMS1. A computing module for modelling an environment, the computing module comprising: a processing unit; and a non-transitory media readable by the processing unit, the media storing instructions that when executed by the processing unit, causes the processing unit to: receive a LiDAR point cloud of the environment; determine an image point cloud based on a plurality of image frames of the environment, the plurality of image frames comprising at least one reference frame, wherein each of the at least one reference frame comprises a plurality of markers; scale the image point cloud to obtain a scaled point cloud based on a plurality of marker points in the image point cloud, the plurality of marker points corresponding to the plurality of markers; and register the scaled point cloud with the LiDAR point cloud to obtain a multi-modal point cloud of the environment.

2. The computing module as recited in claim 1, wherein each of the plurality of markers includes at least two dimensions.

3. The computing module as recited in any one of claims 1 to 2, wherein the image point cloud has a point cloud density higher than the LiDAR point cloud.

4. The computing module as recited in any one of claims 1 to 3, wherein the plurality of image frames comprises different views of a target part of the environment.

5. The computing module as recited in any one of claims 1 to 4, wherein the instructions to determine the image point cloud based on the plurality of image frames of the environment further comprises instructions for directing the processing unit to: sort the plurality of image frames into the at least one reference frames and at least one scene frames.

6. The computing module as recited in claim 5, wherein the instructions to determine the image point cloud based on the plurality of image frames of the environment further comprises instructions for directing the processing unit to: obtain the image point cloud based on the at least one reference frames and the at least one scene frames.

7. The computing module as recited in any one of claims 1 to 6, wherein the instructions to scale the image point cloud further comprises instructions for directing the processing unit to: determine a scale multiplier based on the plurality of marker points; and scale the image point cloud using the scale multiplier to obtain the scaled point cloud.

8. The computing module as recited in claim 7, wherein the instructions to determine the scale multiplier further comprises instructions for directing the processing unit to: obtain the scale multiplier by minimizing an error between an actual distance between the plurality of markers and a point cloud distance between the plurality of markers.

9. The computing module as recited in claim 8, wherein the instructions to determine the point cloud distance further comprises instructions for directing the processing unit to:determine a segmentation plane for each of the plurality of markers based on the plurality of marker points; determine a plurality of inlier points from the plurality of marker points for each of the plurality of markers based on the respective segmentation plane; determine a respective centre point for each of the plurality of markers based on the respective plurality of inlier points; and determine the point cloud distance between the plurality of markers based on the respective centre point of the plurality of markers.

10. The computing module as recited in any one of claims 1 to 9, wherein the non-transitory media further comprises instructions for directing the processing unit to: sample a plurality of LiDAR sub-point clouds from the LiDAR point cloud by moving a sampling boundary in the LiDAR point cloud; and based on the scaled point cloud, determine from the plurality of LiDAR subpoint clouds a matching sub-point cloud.

11. The computing module as recited in claim 10, wherein each of the plurality of LiDAR sub-point clouds has a point cloud count higher or equal to a predetermined point threshold.

12. The computing module as recited in any one of claims 10 to 1 1 , wherein the instructions for determining the matching sub-point cloud from the plurality of LiDAR sub-point clouds further comprises instructions for directing the processing unit to: determine a respective similarity score between the scaled point cloud and each of the plurality of LiDAR sub-point clouds; anddetermine the matching sub-point cloud as one of the plurality of LiDAR subpoint clouds with a highest similarity score.

13. The computing module as recited in claim 12, wherein instructions to determine the respective similarity score further comprises instructions for directing the processing unit to: input the scaled point cloud and respective ones of the plurality of LiDAR subpoint clouds into a machine learning model; and obtain the respective similarity score from the machine learning model.

14. The computing module as recited in any one of claims 12 or 13, wherein the non- transitory media further comprises instructions for directing the processing unit to: apply a coarse registration of the scaled point cloud with the matching sub-point cloud; and, apply a fine registration of the scaled point cloud with the matching sub-point cloud.

15. The computing module as recited in any one of claims 1 to 14, wherein the instructions for determining the image point cloud further comprises instructions for directing the processing unit to: trigger an imaging device to capture the plurality of image frames of the environment; and to send the plurahty of image frames to the computing module before the processing unit performs the step of determining the image point cloud.

16. The computing module as recited in any one of claims 1 to 15, wherein the LiDAR point cloud and the plurality of image frames are obtained in respective non-overlapping time points.

17. The computing module as recited in any one of claims 1 to 16, wherein the plurality of image frames is a series of continuous frames from a video.

18. A system for modelling an environment, comprising an imaging device; and the computing module as recited in any one of claims 1 to 17 in signal communication with the imaging device.

19. A method of modelling an environment using a computing module, the method comprising: receiving a LiDAR point cloud of the environment; determining an image point cloud based on a plurality of image frames of the environment, the plurality of image frames comprising at least one reference frame, wherein each of the at least one reference frame comprises a plurality of markers; scaling the image point cloud to obtain a scaled point cloud based on a plurality of marker points in the image point cloud, the plurality of marker points corresponding to the plurality of markers; and registering the scaled point cloud with the LiDAR point cloud to obtain a multimodal point cloud of the environment.

20. The method as recited in claim 19, further comprising obtaining the plurality of image frames using a portable device.

1. The method as recited in any one of claims 19 to 20, further comprising obtaining the LiDAR point cloud and the plurality of image frames in respective non-overlapping time points.

Images