Method and apparatus for accurate mobile data capture

Abstract

A system and method are provided for scanning an environment. The system includes a sensor that captures a plurality of images as the system traverses the environment. A processing system is provided having at least one processor that performs at least one photogrammetry function on the plurality of images to generate a sequence of data, and performs a two-pass sampling and interpolation on the sequence of data to generate a single structured point cloud.

Description

METHOD AND APPARATUS FOR ACCURATE MOBILE DATA CAPTURECROSS REFERENCE TO RELATED APPLICATIONS

[0001] The present application is related to the Provisional Patent Application Serial No. 63 / 542,922 entitled, “METHOD AND APPARATUS FOR ACCURATE MOBILE DATA CAPTURE SENSOR” filed October 6, 2023, the contents of which are incorporated by reference herein.BACKGROUND

[0002] The subject matter disclosed herein relates to trajectory estimation, and in particular, relates to estimating the trajectory of a three-dimensional mobile measurement device that is moving through and or relative to an environment.

[0003] Processing systems (e.g., smartphones, laptop computers, tablet computers, wearable computing devices, and / or the like including combinations and / or multiples thereof) can include a sensor for capturing images, such as of an object or environment. In some cases, the images are processed, analyzed, or otherwise used for some purpose, such as to measure environments or objects. For example, photogrammetry is a technique for measuring objects using images, such as photographic images acquired by a sensor or other suitable sensor of a processing system. Photogrammetry can make three- dimensional (3D) measurements from two- dimensional (2D) images or photographs. In some cases, photogrammetry involves determining 3D coordinates using triangulation based at least in part on common features or landmarks between two images.

[0004] Accordingly, while existing three-dimensional measurement systems are suitable for their intended purposes, a system having improved features is introduced herein.BRIEF DESCRIPTION

[0005] According to an aspect of the disclosure, a system is provided for scanning an environment. The system includes a sensor that captures a plurality of images as the system traverses an environment. A processing system is provided having at least one processor that performs at least one photogrammetry function producing a sequence of1MEI 50240747v.1data, and performs a two-pass sampling and interpolation on the sequence of data producing a single structured point cloud.

[0006] In additional embodiments, the method includes edge filtering on operation 1 data using a moving window that is four times the cell size, and edge filtering on operation 2 data using a moving window that is four times the cell size.

[0007] The above features and advantages, and other features and advantages, of the disclosure are readily apparent from the following detailed description when taken in connection with the accompanying drawings.BRIEF DESCRIPTION OF DRAWINGS

[0008] The foregoing and other features and advantages of one or more embodiments described herein are apparent from the following detailed description taken in conjunction with the accompanying drawings in which:

[0009] FIGS. 1A-1C depict a system for collecting and / or displaying images of an environment, according to one or more embodiments described herein;

[0010] FIG. 2 is a schematic illustration of a processing system for trajectory estimation using an image sequence, according to one or more embodiments described herein;

[0011] FIG. 3 is an example of a moving window where the grey grid lines represent the 24MP raster and the black box represents the window twice the size of the raster cells, according to one or more embodiments described herein;

[0012] FIG. 4 is a point pattern at the end of the resampling operations 1 & 2 below, according to one or more embodiments described herein;

[0013] FIG. 5 is a depiction of the moving windows in operations 1 & 2 below where certain grid lines represent the 24MP raster, other grid lines represents the operation 1 moving window, the remaining grid lines represent the operation 2 moving window and highlighted cells in the raster that will be populated by the corresponding operation, according to one or more embodiments described herein;

[0014] FIG. 6 is an example of a moving window centered over an empty raster cell that is being populated with an interpolated point distance, and further showing the2MEI 50240747v.1range of point distances that are able to be used to calculate the interpolated point distance, according to one or more embodiments described herein;

[0015] FIG. 7 is a depiction of interpolation using a moving window in the horizontal offset only, as in operation 3 below, according to one or more embodiments described herein;

[0016] FIG. 8 is a depiction of interpolation using a moving window in the vertical offset only, as in operation 4 below, according to one or more embodiments described herein;

[0017] FIGS. 9 and 10 show one type of mobile scanning device that makes use of the processes described herein, according to one or more embodiments; and

[0018] FIG. 11 is an output produced by a mobile scanner, according to one or more embodiments.

[0019] The following detailed description explains embodiments of the disclosure, together with advantages and features, by way of example with reference to the abovereferenced drawings.DETAILED DESCRIPTION

[0020] Embodiments described herein provide for trajectory estimation using a sequence of captured images from a mobile sensor apparatus that moves in and with respect to an environment of interest. Various embodiments provide advantages in reducing the computational requirements on one or more processors to reduce the time, data storage and bandwidth requirements therefor while determining a reliable and accurate trajectory of the sensors capturing the images.

[0021] Trajectory estimation is a process of accurately determining a path and motion of an object or agent over time, within or relative to a given environment. Trajectory estimation is useful in many fields, such as robotics, computer vision, aerospace engineering, autonomous systems, and / or the like including combinations and / or multiples thereof. Estimating trajectories is useful for understanding the behavior and dynamics of moving entities, which can provide precise predictions and control in dynamic scenarios.3MEI 50240747v.1

[0022] Trajectory estimation plays a role in a wide range of tasks. In robotics, for example, trajectory estimation is used for robot localization, mapping, and navigation. By accurately estimating a robot’s trajectory, the robot can determine its current position, create maps of its surroundings, and plan optimal paths to reach its destination or perform specific tasks. In computer vision, trajectory estimation is often used to track the movement of objects in video sequences, which enables applications like object tracking, surveillance, and activity recognition. In the field of aerospace engineering, trajectory estimation is used for spacecraft and aerial vehicle navigation and control. For example, trajectory estimation is used for maneuvering aircraft / spacecraft, determining orbits, and other tasks that can influence mission success.

[0023] Trajectory estimation can be challenging because of various sources of uncertainty and noise, such as sensor inaccuracies, occlusion, environmental changes, dynamic interactions with other objects (e.g., moving objects), and / or the like, including combinations and / or multiples thereof. To address these challenges, various techniques can be employed, such as sensor fusion, probabilistic filtering (e.g., Kalman filtering, particle filtering), machine learning, and optimization methods, among others.

[0024] To address various shortcomings in existing technologies, one or more embodiments described herein provide for trajectory estimation using image alignment for an image sequence of captured images. For example, when a sensor moves through an environment, the sensor captures images as it moves, resulting in an image sequence. The image sequence is used to estimate a trajectory of the movement of the sensor through the environment during capture, using one or more of the embodiments described herein.

[0025] Referring now to FIG. 1, an embodiment is shown of a system 100 for collecting and / or displaying images of an environment according to at least one embodiment described herein. Particularly, FIG. 1A depicts the system 100 for collecting and / or displaying images of an environment or object, the system 100 having a sensor 104 and a processing system 102.

[0026] The processing system 102 can be any suitable processing system, such as a built-in processing unit, a smartphone, tablet computer, laptop or notebook computer, node(s) of a cloud computing system, etc. Although not shown, the processing system 102 includes, in various embodiments, additional components, such as at least one processor4MEI 50240747v.1or microprocessors for executing processing instructions, a memory(such as a hard disk drive), random access memory, read-only memory, a memory chip and the like for storing the processing instructions and data in an electronically-stored, computer-readable format. The processing system 102, in various embodiments, further includes a display for displaying user interfaces, an input device for receiving inputs, an output device for generating outputs, a communications adapter for facilitating communications with other devices (e.g., the sensor 104), and / or the like including combinations and / or multiples thereof.

[0027] The sensor 104 captures one or more images, such as a panoramic image, of an environment. In examples, the sensor 104 is an ultra-wide angle sensor. As an example, the sensor 104 is an omnidirectional sensor, such as the RICO THETA or INSTA360 sensor. In an embodiment, the sensor 104 includes at least one sensor 110 (FIG. IB), which, in various embodiments, includes an array of photosensitive pixels. The sensor 110 is arranged to receive light from a lens 112. In the illustrated embodiment, the lens 112 is an ultra-wide angle lens that provides (in combination with the sensor 110) a field of view 0 between 100 and 270 degrees, for example. In an embodiment, the field of view 9 is greater than 180 degrees and less than 270 degrees. It should be appreciated that while embodiments herein describe the lens 112 as a single lens, this is for exemplary purposes and the lens 112 in other embodiments is comprised of a plurality of optical elements.

[0028] In some embodiments, the sensor 104 includes a pair of sensors 110A, HOB that are arranged to receive light from ultra-wide angle lenses 112A, 112B respectively (FIG. 1C). In this disclosure, the sensor 104 is sometimes referred to as a dual sensor because it has a pair of sensors 110A, HOB and corresponding lenses 112A andl 12B as shown. The sensor 110A and lens 112A are arranged to acquire images in a first direction, and the sensor 110B and lens 112B are arranged to acquire images in a second direction. In the illustrated embodiment, the second direction is opposite the first direction (e.g., 180 degrees apart). A sensor having opposingly arranged sensors and lenses with at least a 180 degree field of view is sometimes referred to as an omnidirectional sensor, a 360 degree sensor, or a panoramic sensor, since it acquires an image in a 360 degree volume about at least one axis of the sensor.

[0029] The sensor 104 can also be referred to as a “rig photogrammetry system” or simply a “rig,” as further described herein. In various embodiments, the sensor 104 (e.g.,5MEI 50240747v.1rig) is moved through an environment to capture a plurality of images about a surrounding environment. The plurality of images forms an “image sequence” or “sequence of images” as variously referenced herein. The processing system 102, or another suitable processing system or device, performs photogrammetry methods on the captured images that form the image sequence to generate a 3D representation, or model, of the environment in which the images were captured. In certain embodiments, it is desirable to estimate the trajectory of the sensor 104 as it moves throughout or relative to the environment to facilitate the determination of the 3D representation or model. Various methods for trajectory estimation are further described herein.

[0030] In various embodiments, more than two such images are captured at each point along a trajectory, and the image sequence captured along the entire trajectory includes hundreds, or thousands or tens of thousands of images from one or both of the sensors of the sensor 104.

[0031] Turning now to FIG. 2, a schematic illustration 200 of one embodiment of the processing system 102 for trajectory estimation using an image sequence is shown. Other useful embodiments are readily contemplated herein. The processing system 102 is any suitable computing device, such as a laptop computer, a desktop computer, a smartphone, a tablet computer, and / or the like, including combinations and / or multiples thereof. According to at least one embodiment described herein below, FIG. 5 depicts a processing system 500, which is another example of the processing system 102. Returning to FIG. 2, the processing system 102 includes a processing device 202, a system memory 204, a network adapter 206, a data store 208, a display 210, sensor(s) 211, a capture engine 212, and an alignment and trajectory engine 214.

[0032] In various embodiments, the components, modules, engines, etc. described regarding FIG. 2 (such as., the capture engine 212 and the alignment and trajectory engine 214) are implemented as computer processing instructions that are stored on a computer- readable storage medium, in hardware modules, as special-purpose hardware (e.g., application specific hardware, application specific integrated circuits (ASICs), application specific special processors (ASSPs), field programmable gate arrays (FPGAs), as embedded controllers, hardwired circuitry, etc.), or as some combination or combinations of these. According to aspects of the present disclosure, the engine(s) described herein is a combination of hardware and programming. The programming is embodied in6MEI 50240747v.1processor-executable instructions stored in a tangible memory, and the hardware can include the processing device 202 for executing those instructions. Thus, the system memory 204 can store program instructions that when executed by the processing device 202 implement the engines described herein. Other engines can also be utilized to include other features and functionality described in other examples herein. The network adapter 206 enables the processing system 102 to transmit data to and / or receive data from other sources, such as the sensor 104.

[0033] In various embodiments, the sensor 104 is any suitable device for collecting images of an object or an environment 222. For example, the processing system 102 receives data (e.g., an image or set of images of the environment 222) from the sensor 104 directly and / or via a wired or wireless network 207. The data (e.g., image(s)) from the sensor 104) is be stored in the data store 208 of the processing system 102 as data 209a (also referred to as “images 209a”). According to at least one embodiment described herein, the capture engine 212 causes the sensor 104 to capture the data 209a (e.g., images), such as by transmitting an activation signal to the sensor 104. According to at least one embodiment described herein, the sensor 104 captures the data 209a automatically and / or responsive to some other trigger event (e.g., a user command), and transmits the data 209a to the processing system 102 for processing and / or storage.

[0034] According to embodiments described herein, the capture engine 212 uses the sensor(s) 211 to capture additional data and / or images of the environment 222 as data 209b. According to embodiments described herein, the processing system 102 generates a representation of the data 209a and / or the data 209b as a point cloud, which is displayed on the display 210. The sensor(s) 211 includes at least one sensor for capturing image data and information about the environment 222. The sensor(s) 211 can include at least one of the following: a sensor, an omnidirectional sensor, a panoramic sensor, a 360-degree sensor, a LIDAR sensor, and / or the like, including combinations and / or multiples thereof.

[0035] The network 207 represents any of different types of suitable communications networks (including combinations thereof) such as, for example, cable networks, public networks (e.g., the Internet), private networks, wireless networks, cellular networks, and any other suitable private and / or public networks. Further, the network 207 can have any suitable communication range associated therewith and includes, for example, global networks (e.g., the Internet), metropolitan area networks (MANs), wide area networks7MEI 50240747v.1(WANs), local area networks (LANs), or personal area networks (PANs). In addition, the network 207 can include any type of medium over which network traffic carries including, but not limited to, coaxial cable, twisted- pair wire, optical fiber, a hybrid fiber coaxial (HFC) medium, microwave terrestrial transceivers, radio frequency communication mediums, satellite communication mediums, or any combination thereof.

[0036] During image capture, the sensor 104 is arranged on, in, and / or around the environment 222 to collect data / images of the environment 222. For example, the sensor 104 captures a sequence of images of the environment while moving along a path.

[0037] According to embodiments described herein, using images (e.g., the data 209a) received from the sensor 104 and / or the images (e.g., the data 209b) captured by the sensor(s) 211, the processing system 102 performs various photogrammetry functions. Photogrammetry is a technique for modeling objects using images, such as photographic images acquired by a digital sensor. Photogrammetry can make three-dimensional (3D) models from two-dimensional (2D) images or photographs. When more than one image is acquired at different positions that have an overlapping field of view, common points or features (sometimes referred to herein with respect to “tie points”) are identified in each image. By projecting a ray from the sensor location to the feature / tie point on the object, the 3D coordinate of the feature / point is determined using trigonometry or triangulation in various embodiments. In some examples, photogrammetry is based on markers / targets (e.g., lights or reflective stickers) placed on an object in the environment or based on distinctive natural features thereof. To perform photogrammetry, in various instances, images are captured, with a sensor having one or more sensors (such as a photosensitive array for example). By acquiring multiple images of an object, or a portion of the object, from different positions or orientations, 3D coordinates of points on the object are determined based on common features or points and information on the position and orientation (e.g., pose, or pitch, yaw, or roll angle) of the sensor when each image is acquired. In order to reliably obtain the desired information for determining 3D coordinates, the features are identified in two or more images. Since the images are acquired from different positions or orientations, the common features are located in overlapping areas of the field of view of the images. It should be appreciated that one method of performing photogrammetry techniques is described in commonly-owned U.S. Patent No. 10,477,180 entitled “PHOTOGRAMMETRY SYSTEM AND METHOD OF8MEI 50240747v.1OPERATION” filed in the name of Wolke et al., the contents of which are incorporated by reference herein. With photogrammetry, two or more images are captured and used to determine 3D coordinates of features.

[0038] Different configurations are possible, such as a linear configuration, a circular configuration, a back-to-back configuration, and the like including combinations and / or multiples thereof. The particular arrangement of sensors is selected to provide a desired overlap and coverage of the object or environment being captured. The sensors have the same or different focal lengths and / or resolutions depending on the specific aspects of the photogrammetric task being performed, in various instances.

[0039] ] Sensor calibration is performed for each individual sensor to determine its intrinsic parameters. Examples of intrinsic parameters can include focal length, principal point, lens distortion coefficients, skew, aspect ratio, and / or the like including combinations and / or multiples thereof. Sensor calibration is performed using on-the-fly calibration in which no additional data is obtained before or after the image capture. Common points produced through feature extraction and matching in images involving the same location in an environment or on an object, as described herein, can be used in this process. In various embodiments, each sensor is calibrated individually by using the portion of an image set selected for sensor calibration. A number of common features and root mean squares of errors (RMSE) of bundle adjustment, as described herein, is considered for image candidates. To obtain enough common points suitable for sensor calibration, and to have a sufficiently large base line among the sensor poses, the sensors can be arranged orthogonally to the direction of movement.

[0040] Once the intrinsic parameters of each sensor are computed, the relative orientations of the sensor(s) can be determined. This involves finding the translation and rotation between the sensors’ coordinate systems. The relative orientation calibration can be performed using a calibration object with known 3D points. This object is placed within the environment (e.g., the environment 222), and the 3D points of the object are captured in the images from each sensor. By matching the 3D points in the images, the relative pose between the sensors can be computed.

[0041] According to embodiments described herein, another approach to relative orientation calibration is based on an on-the-fly system calibration. This approach uses9MEI 50240747v.1common image points of the images of the image sequence and can be applied to applications like trajectory estimation, where scale information is not being used.

[0042] According to embodiments described herein, in some setup configurations, existing information can be used as a constraint to increase the reliability of system calibration. The existing information can be, for example, back- to-back sensor placement (when the distance between the two sensor positions is unknown) in sensor systems such as those shown and described herein (see, e.g., FIGS. 1A-1C) and / or the like including combinations and / or multiples thereof. The distance of the two sensors is unknown, for example, because an accurate distance is unknown in advance. System calibration, as described herein, is used to determine this unknown distance as well as certain other unknown parameters.

[0043] With continued reference to FIG. 2, according to embodiments described herein, using images (e.g., the data 209a or 209b) received from the sensor 104 and / or the images as captured by the sensor(s) 211, the processing system 102 can perform trajectory estimation using an image sequence using the alignment and trajectory engine 214.

[0044] According to embodiments described herein, the alignment and trajectory engine 214 performs image alignment, which is used for trajectory estimation. Image alignment is the process of determining the sensor location and sensor angular orientation with respect to a reference coordinate system. Image alignment is a pre-requisite for a desired level of accuracy in reconstruction of a scene in 3D space in various embodiments.

[0045] The alignment and trajectory engine 214 can perform at least one technique for image alignment in photogrammetry, such as feature-based matching, direct image alignment sequential alignment, bundle adjustment, global positioning systems (GPS) and control points, and / or the like, including combinations and / or multiples thereof.

[0046] Feature-based matching involves identifying and matching distinctive features in different images. These features could be key points, comers, or other salient points (i.e., tie points) that can be easily detected and described. Once the features are matched across the images, the alignment and trajectory engine 214 calculates the transformation (e.g., translation, rotation) to align the images.10MEI 50240747v.1

[0047] Sequential image alignment involves aligning images sequentially, where the images that are already oriented serve as a reference, and subsequent images are aligned to the reference.

[0048] Bundle adjustment is an optimization technique used to simultaneously refine the sensor parameters and 3D coordinates of points in the scene. In various implementations of such an approach, the alignment and trajectory engine 214 considers multiple images and features together, which provides for a consistent and accurate alignment by reducing or minimizing reprojection error, which is the difference between the observed image points and the projected 3D points.

[0049] GPS and control points, if available and depending on their accuracy, provide an initial image alignment. For example, the alignment and trajectory engine 214 use GPS coordinates and ground control points (GCPs) to perform an initial alignment following the georeferencing of the images and the entire photogrammetry network.

[0050] It should be appreciated that image alignment can be a computationally intensive process, especially when dealing with a large number of images and complex scenes. Therefore, in some embodiments, specialized software and / or libraries are used to implement efficient algorithms for image alignment.

[0051] Mobile data from a handheld or mobile scanner can be noisy (e.g. have erroneous data) and inaccurate when compared to data collected from a stationary scanner purely due to the nature of the data collection method. However, when creating a scan of a large environment the mobile data capture methodology is significantly faster than data capture from a series of stationary scanner locations. The processes and devices described herein combine the speed and efficiency of the mobile data capture methodology while maintaining an accuracy equivalent to that of stationary scanner data from an armmounted rig or the like to accomplish accurate data capture from large complex environments within a short capture time frame. Various types of useful scanners, and processes used therewith, in addition to those described herein are disclosed in PCT Publication WO2020 / 234575 entitled SENSOR SYNCHRONIZATION and U.S. Patent No. 11,176,353 entitled THREE-DIMENSIONAL DATASET AND TWO- DIMENSIONAL IMAGE LOCALIZATION, the entirety of both of which are hereby incorporated by reference.11MEI 50240747v.1

[0052] Some terms are used through this document, which will be defined in the following paragraph:

[0053] Spatial Flash: A form of FLASH that is based on the geometry of a point cloud rather than photogrammetry. Aspects of Spatial Flash is described in commonly-owned the following United States Patent Applications: United States Patent Application No. 18 / 449943 entitled “CALIBRATING SYSTEM FOR COLORIZING POINTCLOUDS”; United States Patent Application No. 17 / 451946 entitled “THREE DIMENSIONAL MEASUREMENT DEVICE HAVING A CAMERA WITH A FISHEYE LENS”; United States Patent Application No. 17 / 492801 entitled “DYNAMIC SELF-CALIBRATING OF AUXILIARY CAMERA OF LASER SCANNER”; United States Patent Application No. 17 / 379268 entitled “LASER SCANNER WITH ULTRA WIDE- ANGLE LENS CAMERA FOR REGISTRATION”; and United States Patent Application No. 17 / 678116 entitled “CALIBRATING SYSTEM FOR COLORIZING POINT-CLOUDS”, the contents of all of which are incorporated by reference in their entirety.

[0054] Mobile Data: Spatial data captured using handheld light detection and ranging (LIDAR) scanner designed to be used by walking around the environment that is being scanned, and that has then been run through SLAM.

[0055] Static Data: Spatial data captured from a single stationary position either through a stationary scanner or extracted from a single stationary time period in the mobile data.

[0056] SLAM: Simultaneous Localization and Mapping algorithms that build a point cloud incrementally using trajectory and distance data capture from a mobile Lidar Scanner.

[0057] Point Cloud: A collection of data points captured by a scanner.

[0058] Point: A single spatial datum derived from a single ray from a Lidar Scanner or the like which will contain primarily 3D coordinate data (i.e., X, Y, Z axis) that also contains the reflectivity of the surface that it represents (intensity), color (RGB values for red, blue, green) and the capture time from the scanner.12MEI 50240747v.1

[0059] Spatial: Work within the physical three dimensions of space.

[0060] Two-Pass: A resampling and interpolation methodology on a 2D grid to populate empty cells with data which involves two stages of resampling and two stages of interpolation populating different spatial regions of the grid for each pass.

[0061] Point Distance: The distance from the point to the scanner origin. This is the length of the laser ray to the physical real -world object with which it interacts.

[0062] Structured Point Cloud: A point cloud in which every Cartesian coordinate can be given a row and column value in a grid format.

[0063] FIG. 3 is an example of a moving window 300 where the grid lines 302 represent the 24 megapixel (MP) raster and the black box 304 represents the window twice the size of the raster cells according to embodiments described herein. One embodiment of the processes disclosed herein (sometimes referred to herein as Spatial Flash) creates structured and accurate data using point data gathered from a single stationary position in a mobile data capture. To achieve this, three-dimensional (3D) points gathered from a single stationary position are projected onto an equirectangular 2D raster with a resolution of 24MP. The bounds of each cell in this raster represents an azimuth and altitude angle, the values contained within each cell represent a distance from the scanner (point distance).

[0064] By the end of the algorithm each cell will contain a single point distance and by using the azimuth and altitude angular bounds of each cell the central angles can be derived and with the point distance a Cartesian coordinate can be reconstructed. This will produce a point cloud of up to about 24 million points regularly spaced in a row / column structured format, averaged out and smoothed to reduce outlier noise.

[0065] A two-pass re-sampling and interpolation methodology has been developed and incorporated. One example of the operations performed to achieve this are as follows:

[0066] Operation 1 : Resampling using a moving window that is two times the cell size.

[0067] Operation 2: Resampling using a moving window that is two times the cell size, offset by one cell in the horizontal and one cell in the vertical.13MEI 50240747v.1

[0068] Operation 3: Interpolation using the combined data from operation 1 and operation 2, using a moving window that is two times the cell size, offset by one cell in the horizontal direction only.

[0069] Operation 4: Interpolation using the combined data from operation 1 and operation 2, using a moving window that is two times the cell size, offset by one cell vertical direction only.

[0070] Operation 5: (Optional) Edge fdtering on the operation 1 data using a moving window that is four times the cell size.

[0071] Operation 6: (Optional) Edge fdtering on the operation 2 data using a moving window that is four times the cell size.

[0072] The results of operations 3, 4, 5 and 6 (or 1, 2, 3, 4 in the absence of edge fdtering) are combined to produce a single structured point cloud.

[0073] The moving window (see, Fig. 3) mentioned is, in various embodiments, a virtual box which covers a spatial dimension of two times the raster cell size and is centered over one of the cells. The window will use all point distance data within its bounds to derive a single point distance value which will be assigned as the distance value of the cell it is centered over.

[0074] FIG. 4 is a point pattern 400 at the end of the resampling operations 1 & 2 described above according to embodiments described herein. The resampling will extract a static point cloud from an approximately fifteen second stationary period in the mobile point cloud. This data will be projected onto the 24MP raster. Each cell contains multiple point distances at this stage and by using the moving window a single point distance value will be derived for each cell. In certain instances where there is a large depth range (i.e. greater than -0.05 meters) between the point distances in the cell, it is assumed an area of geometry similar to a corner has been found and so the smallest point distance (the point closest to the scanner) is selected to represent this cell. Otherwise, in certain instances, the median of the distances is used.

[0075] In various embodiments, the spatial area of these populated cells is then used as a template to retrieve all data from the full mobile point cloud which is within those14MEI 50240747v.1spatial areas and spatial areas that are nearby (e.g., within 2 cells of separation) and contain point distances close to the associated currently derived single cellular value. A median distance value for each cell is then recalculated, including data from outside of the ~15 second stationary time period, which aids with smoothing and gap fdling.

[0076] Operation 1 and operation 2 above will, in various embodiments, follow the same resampling process, where the only change between them is the offset nature of the moving window (see, e.g., FIG. 4 and FIG. 5).

[0077] FIG. 5 is a depiction of the moving windows 500 in operations 1 & 2 below where the lighter (grey) grid lines 502 represent the 24MP raster, the darker (black) grid lines 504 represents the operation 1 moving window, the dashed grid lines 506 represent the operation 2 moving window and the cells are those cells in the raster that will be populated by the appropriate operation, according to embodiments described herein.

[0078] FIG. 6 is an example of a moving window 600 centered over an empty cell that is being populated with an interpolated point distance, and showing the range of point distances that are used to calculate the interpolated point distance, according to embodiments described herein.

[0079] Interpolation uses the singular point distance values populated in each cell during operations 1 & 2 above rather than the actual point data. Interpolated point distances are created by centering a moving window over each cell where a value is desired. In various embodiments, a moving window is created on the combined data generated from operations 1 and 2. A new point distance is only created if the number of points within the moving window is >=3 and the range difference between point distances is <=0.05 meters (m). In various embodiments, the new point distance is the mean average range value for all point distances within the window (See, FIG. 6).

[0080] In various instances, edge fdtering uses the singular point distance values populated in each cell from operations 1 and 2 rather than the actual point data. The edge fdtering operations are optional operations that are designed to further remove anomalous points, which are created, for example, when a ray from the scanner skims an edge or surface of an object at an oblique angle and creates a scattering of points in the direction of the ray. In various embodiments, edge fdtering uses a larger moving window (4x4 raster cells) over every cell populated from operations 1 (operation 5 above) and 2 (operation 615MEI 50240747v.1above). If the range of the point distances gathered within the window is greater than -0.05 m, the cell will be cleared of data, and so, effectively, a point will be removed from the final output in such instances.

[0081] FIG. 7 is a depiction of interpolation 700 using a moving window in the horizontal offset only, as in operation 3 above, according to one or more embodiments described herein. Note that every other point in the longer rows displayed therein are the referenced points in various instances including highlighted points 702 and other points 704.

[0082] FIG. 8 is a depiction of interpolation 800 using a moving window in the vertical offset only, as in operation 4 above, according to embodiments described herein. Note that every other point in alternating rows as displayed therein are the referenced points in various instances, including highlighted points 802 and other points 804.

[0083] FIGS. 9 show one type of mobile scanning device 900 (sometimes referred to herein as ORBIS that is manufactured by FARO Technologies of Lake Mary, Florida, USA) that makes use of the processes described herein according to various embodiments. Such a hand-held scanner provides a fast and easy way to capture 3D point cloud data, by walking through the area of interest. In various embodiments, the scanner includes a 2D time-of-flight (TOF) laser range scanner rigidly coupled to an inertial measurement unit (IMU) mounted on a motor drive. The motion of the scanning head on the motor drive provides the third dimension used to generate 3D information. A 3D simultaneous localization and mapping (SLAM) algorithm (i.e., FARO’s 3D SLAM) combines the 2D laser scan data with the IMU data to generate accurate 3D point clouds.

[0084] The mobile scanner captures raw laser range measurement and inertial data in various embodiments. This data must be processed using a 3D SLAM algorithm to convert the raw data into a 3D point cloud. The data is processed using, for example, FARO HUB processing applications or the like.

[0085] FARO STREAM manufactured by FARO Technologies Inc., of Lake Mary, FL USA is one type of useful mobile app that connects hardware with cloud-based applications and services. By uniting hardware with cloud software, STREAM makes onsite capture workflows more efficient and brings captured data directly into the FARO ecosystem, all while providing live feedback and display of the captured scan (see, e.g.,16MEI 50240747v.1FIG. 11). One or more aspects of STREAM are described in commonly owned United States Patent No 11,340,058 entitled “REAL-TIME SCAN POINT HOMOGENIZATION FOR TERRESTRIAL LASER SCANNER”, the contents of which are incorporated by reference herein.

[0086] STREAM provides on-site efficiency for data capture with the scanner for scan operations in architecture, engineering, construction and facility management. Users can be confident in the successful and complete scan data they collect in real-time, and that no additional site visits will be required because of missing data, which expedites project finalization times.

[0087] A cloud application, such as FARO SPHERE, FARO Technologies provides a centralized, efficient, and collaborative environment across point cloud applications as well as customer support tools for faster 3D data capture, processing, and delivery, all through a secure, single point sign- on process in various embodiments. With STREAM and SPHERE, off-site colleagues can begin work on the data or share it with endcustomers via WEBSHARE Software or the like, including the collaborative point cloud project management solutions offered by FARO. One or more aspects of SPHERE is described in commonly-owned United States Patent Publ. Application No. 2023-0119214 entitled “FOUR-DIMENSIONAL DATA PLATFORM USING AUTOMATIC REGISTRATION FOR DIFFERENT DATA SOURCES”, the contents of which is incorporated by reference herein.

[0088] To download the raw scan data, the data logger of the scanner is powered on. Universal Serial Bus (USB) memory or the like is connected to the USB socket on the data logger front panel. An AUX LED light or similar will light, for example, green while the data is transferring to the flash drive. The USB flash drive must not be removed when the AUX LED is lit green. After a few seconds (dependent on the size of the data file to be transferred) the AUX LED will turn off. All data that has not previously been transferred will be transferred and the USB flash drive can be removed.

[0089] In various embodiments, the 3D SLAM algorithm used to process the raw laser scan data into a 3D point cloud relies on there being features in the scanned environment that are repeatedly scanned as the operator passes through the scanned environment. For a feature to be significant, the ratio of its size to its range is approximately l:10(e.g., at 517MEI 50240747v.1m range, for a feature to be significant it must be greater than 0.5 m in size). ‘Feature poor’ environments, include open spaces and smooth walled passageways. In smooth-walled hallways there are typically an insufficient number of identifiable features in the direction of travel for a 3D SLAM algorithm to determine forward motion.

[0090] In various instances, the 3D SLAM algorithm processes the raw scan data into a point cloud using a method analogous to the Traverse technique used in survey practice. With this technique, a previously known position of the scanner is used to determine the current position. This method can magnify small measuring errors, causing the calculated position to “drift”. It is good survey practice to “close the loop” by re-scanning a known position so that the compounded error can be spread around the loop.

[0091] At a minimum, one should start and end the survey in the same position to ensure at least one loop closure. It is recommended to close the loop as often as possible to reduce or minimize error and improve the accuracy of the resulting point cloud.

[0092] In general, it is desired to do circular loops instead of just retracing one’s operations. This applies to horizontal as well as vertical loops, so enter and exit through different doors, and move between floors via different stairwells.

[0093] In various instances, it is desired to scan the closed loop regions carefully to ensure the key features are scanned from a similar perspective. In an embodiment, the user turns around and returns to a region from a different direction. This is particularly desired in feature-poor environments, such as hallways and passageways.

[0094] In certain embodiments, data within a small range value is not processed (by default) in order to eliminate such data from the scanner operator being included in the final point cloud and preserve bandwidth and data storage sizes. Close proximity to walls and ceilings should be avoided in various instances.

[0095] The maximum range of the scanner is —100 meters in various embodiments. This range will be achieved in optimal conditions (indoors with good target reflectivity). The typical maximum range will be 60-80 m in most conditions. It is recommended that the range is kept to less than 50 m where possible to ensure good point density and to assist the 3D SLAM algorithm.18MEI 50240747v.1

[0096] In most cases the 3D SLAM algorithm can handle moving objects in the environment. In order to estimate the sensor trajectory, the algorithm assumes a large proportion of the environment is static. However, in some feature-poor environments where 3D structure is lacking in some dimensions, moving objects can have a greater impact on the solution. In particular, moving objects should be avoided in long tunnellike environments (e.g., corridors), relatively open spaces and when transitioning through doorways.

[0097] Referring to FIG. 10, in various embodiments a scanning kit 1000 is provided. In this embodiment, the scanning kit 1000 includes a hand-held scanner 1001 includes the following components as labelled therein: data logger 1002, battery 1003, data cable, battery charger and power supply unit 1005, reference base plate 1006, USB or other type of flash electronic memory drive 1007, backpack 1008 or similar transport case, and data logger 1009 shoulder strap for mobility. The scanning kit 1000 provides a convenient means for the use to transport the scanner 1001 with desired accessories to perform a scanning project.

[0098] FIG. 11 is an output 1100 produced by a mobile scanner according to one or more embodiments described herein. To view the processed dataset, when a VIEW button 1102 or the like next to the fde name in a dataset list is selected, a 3D view of the dataset opens in a new view window, one example of which is depicted in this FIG. 11. In various instances, the displayed output will be provided in color (not shown), not black and white as presented.

[0099] It will be appreciated that one or more embodiments described herein are embodied as a system, method, or computer program product and take the form of a hardware embodiment, a software embodiment (including firmware, resident software, micro-code, etc.), or a combination thereof. Furthermore, one or more embodiments described herein take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.

[0100] The term “about” is intended to include the degree of error associated with measurement of the particular quantity based upon the equipment available at the time of filing the application. For example, “about” can include a range of ± 8% or 5%, or 2% of a given value.19MEI 50240747v.1

[0101] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and / or “comprising,” when used in this specification, specify the presence of stated features, integers, operations, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, operations, operations, element components, and / or groups thereof.

[0102] While the disclosure is provided in detail in connection with only a limited number of embodiments, it should be readily understood that the disclosure is not limited to such disclosed embodiments. Rather, the disclosure can be modified to incorporate any number of variations, alterations, substitutions or equivalent arrangements not heretofore described, but which are commensurate with the spirit and scope of the disclosure. Additionally, while various embodiments of the disclosure have been described, it is to be understood that the embodiment(s) include only some of the described aspects.Accordingly, the disclosure is not to be seen as limited by the foregoing description.20MEI 50240747v.1

Claims

CLAIMSWhat is claimed:

1. A system that estimates trajectory moving through an environment comprising: a sensor that captures a plurality of images as it traverses the environment; and at least one processor that performs at least one photogrammetry function for producing a sequence of data from the plurality of images received from the sensor, and performs a two-pass sampling and interpolation on the sequence of data to generate a structured point cloud.

2. The system of claim 1, wherein the two-pass sampling and interpolation comprises:(i) a first resampling of the sequence of data using a moving window that is two times a cell size of a raster the sensor; and(ii) a second resampling of the sequence of data using a moving window that is two times the cell size and offset by one cell in a horizontal direction and one cell in a vertical direction.

3. The system of claim 2, wherein the two-pass sampling and interpolation further comprises a first interpolation using combined data from the first resampling and data from the second resampling using a moving window that is substantially two times the cell size and offset by one cell in the horizontal direction only.

4. The system of claim 3, wherein the two-pass sampling and interpolation further comprises a second interpolation using the combined data from the first resampling and the data from the second resampling using a moving window that is substantially two times the cell size and offset by one cell in the vertical direction only.

5. The system of claim 4, wherein the data from the first resampling is edge filtered.

6. The system of claim 4, wherein the data from the second resampling is edge filtered.21MEI 50240747v.

17. The system of claim 4, wherein the data from the first resampling, the second resampling, the first interpolation and the second interpolation is combined to form a single structure point cloud.

8. The system of claim 2, wherein the at least one processor extracts from data produced by the first resampling and the second resampling a static point cloud from a stationary period while traversing the environment.

9. The system of claim 8, where the static point cloud is projected onto a megapixel raster.

10. The system of claim 1, where the sensor is a camera.

11. The system of claim 1, where the sensor is a laser scanner.

12. The system of claim 1, where the sensor comprises two opposingly-arranged sensors, each having at least a 180-degree field of view.

13. A method that estimates a trajectory of a scanning system moving through an environment, the method comprising: capturing, with a sensor of the scanning system, a plurality of images as the sensor traverses the environment; performing at least one photogrammetry function on at least one of the plurality of images to produce a sequence of data; and performing two-pass sampling and interpolation on the sequence of data to generate a single structured point cloud.

14. The method of claim 13, wherein performing the two-pass sampling and interpolation further comprises:(i) resampling the sequence of data a first time using a first moving window that is substantially two times a cell size of a raster of the sensor; and(ii) resampling the sequence of data a second time using a second moving window that is two times the cell size offset by one cell in a horizontal direction and one cell in a vertical direction.22MEI 50240747v.

115. The method of claim 14, wherein performing the two-pass sampling and interpolation further comprises: performing a first interpolation using combined data from the resampling at the first time and from the resampling at the second time using a third moving window that is two times the cell size and offset by one cell in the horizontal direction only.

16. The method of claim 15, wherein performing the two-pass sampling and interpolation further comprises: performing a second interpolation using the combined data and a fourth moving window that is two times the cell size and offset by one cell in the vertical direction only.

17. The method of claim 14, wherein performing the two-pass sampling and interpolation further comprises edge filtering the sequence of data from the resampling at the first time.

18. The method of claim 14, wherein performing the two-pass sampling and interpolation further comprises edge filtering the sequence of data from the resampling at the second time.

19. The method of claim 16, further comprising: combining the sequence of data from the resampling at the first time, the resampling at the second time, the first interpolation and the second interpolation to form a single structure point cloud.

20. The method of claim 14, wherein a processor extracts from the sequence of data produced by the resampling at the first time and the resampling at the second time, a static point cloud from a stationary period.23MEI 50240747v.1

Images