Point cloud staining with occlusion detection

Description

BACKGROUND

[0001] The present invention relates generally to point cloud colorization, and more particularly to point cloud colorization with occlusion detection.

[0002] A point cloud is a set of data points associated with the 3D (three-dimensional) representation of objects or structures. A point cloud is generated using a 3D scanner system, such as, e.g., a mobile mapping system, a terrestrial laser scanner, or LiDAR (light detection and ranging) system. The 3D scanner system measures a set of data points corresponding to surfaces of an object and outputs the set of data points in a 3D coordinate system as the point cloud. However, a point cloud generated by such a 3D scanner does not include color information for the scanned objects.

[0003] From the publication "3D point cloud model colorization by dense registration of digital images" by Crombez et al., in The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Vol. XL-5 / W4, 2015, 3D Virtual Reconstruction and Visualization of Complex Architectures, 25-27 February 2015, Spain, pages 123-130, it is known a method for colorization of point cloud data points.

[0004] From the publication "Direct visibility of point sets" by Sagi Katz, in ACM Transactions on Graphics, Vol. 26, No. 3, Article 24, July 2007, it is known a method for determining the visible points in a point cloud.

[0005] Conventionally, point clouds are colorized using images acquired from cameras mounted on the 3D scanner system or from an external database of previously acquired images. The point cloud and the images are geo-referenced to a global coordinate system based on position information estimated from GNSS (global navigation satellite systems), IMUs (inertial measurement units), and / or wheel sensors. Each data point of the point cloud is colorized using the color value at a georeferenced coordinate of the data point in an image closest to that data point. However, where the data point is occluded in its closest image, the color value of the occluding object is incorrectly applied to colorize the data point. For example, where a white pole occludes a data point in an image closest to that data point, the data point is incorrectly colorized with the white color of the pole according to conventional approaches.

[0006] One approach for overcoming the problems associated with occlusions is to apply ray casting algorithms to search for another point in the image between the camera and the point of the point cloud currently being colored. However, such approach is computationally expensive and not feasible for mass data applications. Further, since the images being used are not spatially close to the point currently being colored, the accuracy of the mapping is low due to sensor error and position error of the system.BRIEF SUMMARY

[0007] The object of the invention is attained by a method for colorizing a set of point cloud data points associated with a particular object according to claim 1, an apparatus according to claim 7 and a non-transitory computer readable medium according to claim 8. Further developments of the invention are specified in dependent claim 2 to 6. These and other advantages of the invention will be apparent to those of ordinary skill in the art by reference to the following detailed description and the accompanying drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 exemplarily shows an environment at a time T1 in which a line of sight of an image acquisition device to an object is occluded; Figure 2 exemplarily shows an image generated by the image acquisition device at time T1; Figure 3 exemplarily shows an environment at a time T2 in which a line of sight of an image acquisition device to an object is not occluded; Figure 4 exemplarily shows an image generated by the image acquisition device at time T2; Figure 5A is a block diagram exemplarily illustrating the subject distance to a point for the image shown in Figure 2; Figure 5B is a block diagram exemplarily illustrating the subject distance to a point for the image shown in Figure 4; Figure 6 exemplarily shows a method for colorizing a data point of a point cloud; Figure 7 exemplarily shows a method for selecting an image, from a subset of the plurality of candidate images that include an un-occluded point of the particular object corresponding to the respective data point, that has a shortest subject distance to the un-occluded point of the particular object; Figure 8 exemplarily shows a mask image; Figure 9 exemplarily shows a method for generating a depth map; and Figure 10 exemplarily shows a graph illustrating the relationship between the pixel size and the subject distance in a depth map. DETAILED DESCRIPTION

[0009] Figure 1 illustrates aspects that aid in understanding the claimed invention. Figure 1 shows an environment 100 at a time T1. Environment 100 includes object 110 to be scanned by a surveying system 102 for generating a colorized point cloud of object 110. The colorized point cloud may be used for a number of different applications, such as, e.g., modeling of environment 100.

[0010] Object 110 may include any object of interest within environment 100, such as, e.g., a building or other structure. Surveying system 102 collects data on environment 100 as surveying system 102 travels along path 118 (e.g., a road) for generating a colorized point cloud of object 110. Surveying system 102 may be mounted on a vehicle for traveling along path 118.

[0011] Surveying system 102 includes one or more image acquisition devices 104, a scanner 106, and a processing system 108. Scanner 106 may be any device suitable for generating a point cloud of object 110. Scanner 106 may be a laser scanner, such as, e.g., a known LiDAR imaging scanner. Image acquisition device 104 may be any image acquisition device suitable for acquiring color images of object 110 such as, e.g., a 360 degree panoramic camera, a wide angle camera, and / or a telephoto camera. Processing system 108 includes memory 124 (e.g., random access memory) and storage 126 (e.g., persistent storage) operatively coupled to one or more processors 128. Storage 126 may store computer program instructions to perform, e.g., the steps of Figures 6, 7, and 9 for colorizing the point cloud generated by scanner 106 using images acquired by image acquisition device 104. Storage 126 may also store the point cloud generated by scanner 106 and the images acquired by image acquisition device 104. The computer program instructions may be loaded into memory 124 and executed by processor 128. While image acquisition device 104, scanner 106, and processing system 108 are shown as part of surveying system 102, it should be understood that image acquisition device 104, scanner 106, and processing system 108 may be discrete components. For example, processing system 108 may be a discrete component located remotely from image acquisition device 104 and scanner 106.

[0012] Scanner 106 scans environment 100 to generate the point cloud of object 110 as surveying system 102 travels over path 118. The point cloud may be stored in storage 126 and comprise a set of point cloud data points corresponding to a 3D representation of exterior surfaces of object 110. The set of data points are defined by a 3D coordinate system of the point cloud, such as, e.g., the Cartesian coordinate system representing the data points in X, Y, and Z coordinates. For example, scanner 106 may be a laser scanner that transmits one or more laser light pulses to object 110. A portion of the light is reflected back from object 110. Scanner 106 detects the reflected light and measures its round trip travel time to determine the distance to a point (e.g., point 112) on an exterior surface of object 110. This distance information is obtained for a plurality of points over one or more exterior surfaces of object 110, thereby resulting in the set of data points of the point cloud. Figure 1 shows an exemplary point 112 on an exterior surface of object 110. Point 112 on a surface of object 110 corresponds to, or is represented by, a point cloud data point.

[0013] The point cloud provides 3D spatial information of object 110 in environment 100 but does not include color information of object 110. The color information of object 110 may be determined using images of object 110. Image acquisition device 104 may acquire a plurality of images as surveying system 102 travels along path 118. The plurality of images may be stored in storage 126. Image acquisition device 104 may acquire at least some of the plurality of images at a same time or different time as scanner 106 scans object 110. Alternatively, separate image acquisition device may acquire the plurality of images at a same time or a different time as scanner 106 scans object 110 or as image acquisition device 104 acquires images. The plurality of images may include images previously acquired by image acquisition device 104 or another image acquisition device and stored in a database.

[0014] Figure 2 illustrates aspects that aid in understanding the claimed invention. Referring to Figure 2, with continued reference to Figure 1, it is assumed that image acquisition device 104 generates image 200 at position 120 and time T1 (in accordance with environment 100 in Figure 1). As shown in Figure 1, line of sight 116 of image acquisition device 104 to point 112 is occluded by occlusion 114 when surveying system 102 is at position 120 at time T1. Accordingly, point 112 on a surface of object 110 is not visible in image 200. Image point 130 is a location (e.g., a 2D coordinate) in image 200 that corresponds to the 3D coordinate of the data point representing point 112. Image point 130 is determined by mapping the 3D coordinate of the data point to the 2D coordinate of image 200, for example using Equation 1, as discussed below. Since occlusion 114 occludes point 112 on a surface of object 110 in image 200, image point 130 represents (e.g., depicts) a point on occlusion 114, instead of point 112 on a surface of object 110. Occlusion 114 may be any object occluding, at least in part, point 112 in image 200. Examples of occlusion 114 may include a vehicle, a tree, a pole, etc. It should be understood that image 200 may include any number of image points corresponding to different data points in the set of data points, where the different data points corresponding to points on a surface of object 110 that may or may not be visible in image 200.

[0015] Figure 3 illustrates aspects that aid in understanding the claimed invention. Figure 3 shows environment 100 in which surveying system 102 is assumed to be at position 122 on path 118 at a later time T2. While time T2 is described as being a time in the future with respect to time T1, it should be understood that time T2 may alternatively be a time in the past with respect to time T1.

[0016] Figure 4 illustrates aspects that aid in understanding the claimed invention. Referring to Figure 4, with continued reference to Figure 3, it is assumed that image acquisition device 104 generates image 400 at position 122 and time T2 (in accordance with environment 100 in Figure 3). As shown in Figure 3, line of sight 116 of image acquisition device 104 to point 112 is not occluded by occlusion 114 at position 122 and time T2. Accordingly, point 112 on object 110 is visible in image 400. Image point 132 is a location (e.g., a 2D coordinate) in image 400 that corresponds to the 3D coordinate of the data point representing point 112. Image point 132 is determined by mapping the 3D coordinate of the data point to a 2D coordinate of image 400, for example using Equation 1, as described below. Since point 112 on a surface of object 110 is not occluded by occlusion 114, image point 132 in image 400 is an un-occluded point of point 112 on object 110 corresponding to the respective data point (e.g., image point 132 in image 400 represents or depicts point 112). It should be understood that image 400 may include any number of image points corresponding to different data points in the set of data points, where the different data points corresponding to points on a surface of object 110 that may or may not be visible in image 400.

[0017] Scanner 106 may generate the point cloud of object 110 and image acquisition device 104 acquires the plurality of images during a pre-processing stage. During a post-processing stage, processing system 108 may colorize the point cloud generated by scanner 106 using the plurality of images acquired by image acquisition device 104. In particular, processing system 108 may colorize each data point associated with object 110 in the point cloud by selecting, from a subset of a plurality of candidate images that include an un-occluded point of object 110 corresponding to the respective data point, an image having a shortest subject distance to the un-occluded point of the particular object and colorizing each data point based on a color in the selected image of the un-occluded point of the particular object corresponding to that data point. As used herein, the subject distance to a point (e.g., a coordinate) in an image is defined as the distance between the image acquisition device and a location on a subject (e.g., object 110 or occlusion 114) corresponding to the point in the image at the time that the image was acquired by the image acquisition device. The subject distance may be measured with respect to any point on image acquisition device 104, such as, e.g., the front lens of image acquisition device 104.

[0018] Figure 5 illustrates aspects that aid in understanding the claimed invention. Referring to Figure 5A, with reference to Figures 1 and 2, a subject distance 134 to image point 130 in image 200 is illustratively depicted. Image 200 of Figure 2 shows image point 130, which corresponds to a location 502 on occlusion 114. Image point 130 on image 200 is mapped to location 502, e.g., using Equation 1 below. Subject distance 134 to image point 130 in image 200 is the distance between image acquisition device 104 and location 502 on occlusion 114 at a time T1 image acquisition device 104 acquired image 200.

[0019] Figure 5A shows a measured distance 136 as the distance between image acquisition device 104 and point 112 (e.g., as determined from the point cloud) at a time T1 that image acquisition device 104 acquired image 200. As point 112 is occluded by occlusion 114, subject distance 134 is smaller than measured distance 136.

[0020] Referring to Figure 5B, with reference to Figures 3 and 4, a subject distance 138 to image point 132 in image 400 is illustratively depicted. Image 400 of Figure 4 shows image point 132, which corresponds to a location 504 on object 110. Image point 132 on image 400 is mapped to location 504, e.g., using Equation 1 below. Subject distance 138 to point 132 in image 400 is the distance between image acquisition device 104 and location 504 on object 110 at a time T2 image acquisition device 104 acquired image 400.

[0021] Figure 5B shows a measured distance 140 as the distance between image acquisition device 104 and point 112 (e.g., determined from the point cloud) at a time T2 that image acquisition device 104 acquired image 400. As point 112 is not occluded, subject distance 138 is substantially the same as measured distance 140 (or within an error threshold).

[0022] An image having a shorter subject distance to a point provides more accurate color details of the point than an image with a longer subject distance to the point. Advantageously, processing system 108 of Figure 1 colorizes each data point of the point cloud for an object using an image having a shortest subject distance to an un-occluded point of the object that corresponds to that data point. Accordingly, the examples described herein provide for a point cloud colorized with a higher degree of accuracy and lower amount of artifacts compared to conventional techniques.

[0023] Figure 6 illustrates aspects that aid in understanding the claimed invention. Figure 6 shows a method 600 for colorizing a data point of a point cloud. Method 600 will be described with respect to environment 100 in Figures 1 and 3. Method 600 comprises steps 602 and 604 of pre-processing stage 600 and steps 612 and 614 of post-processing stage 610.

[0024] During pre-processing stage 600, at step 602, a point cloud is generated, e.g., by scanner 106 of Figures 1 and 3. The point cloud comprises data points for a scanned environment 100. The data points include a set of point cloud data points associated with an object 110. For example, set of data points may include one or more (e.g., all) data points associated with object 110. At step 604, a plurality of candidate color images associated with the set of data points is acquired, e.g., by image acquisition device 104. The set of candidate images may comprise images that are candidates for colorizing the set of data points. The set of candidate images may be selected from all images acquired by image acquisition device 104 that are spatially close to a point corresponding to a data point in the set of data points or that are temporally close to a time of the acquisition of a data point in the set of data points. For example, the set of candidate images may be selected as images that are acquired within a predetermined distance to a point corresponding to a data point in the set of data points or images acquired within a predetermined time to the acquisition of a data point in the set of data points. In another example, the set of candidate images may be selected as a predetermined number of images (e.g., 10 images) that are spatially closest to the position of a point corresponding to a data point in the set of data points or that are temporally closest to a time of the acquisition of a data point in the set of data points. In one example, the set of candidate images associated with the respective data point may include previously acquired images stored in a database. Image 200 of Figure 2 and image 400 of Figure 4 are examples of images that may be included in the set of candidate images. It should be understood that steps 602 and 604 may be performed at a same time or different times in any order. The set of data points of the point cloud generated at step 602 and the set of candidate images for each respective data point acquired at step 604 may be stored in storage 126 of processing system 108.

[0025] Post-processing stage 610 may be performed to colorize the point cloud generated at step 602 using the plurality of candidate images acquired at step 604. Steps 612 and 614 of post-processing stage 610 may be performed, e.g., by processing system 108 for each respective data point in the set of data points associated with object 110 to thereby colorize the point cloud of object 110.

[0026] At step 612, an image is selected, from a subset of the plurality of candidate images that include an un-occluded point of the particular object corresponding to the respective data point, that has a shortest subject distance to the un-occluded point of the particular object. The image may be selected from the plurality of candidate images acquired at step 604. According to the claimed invention, step 612 is performed by performing steps of the method 700 of Figure 7.

[0027] Figure 7 illustrates aspects that aid in understanding the claimed invention. Figure 7 shows a method 700 for selecting an image, from a subset of the plurality of candidate images that include an un-occluded point of the particular object corresponding to the respective data point, that has a shortest subject distance to the un-occluded point of the particular object.

[0028] According to the claimed invention, at step 702, for each candidate image in a specific set of candidate images, a particular point in that candidate image corresponding to the respective data point is determined. The specific set of candidate images may initially include the plurality of candidate images acquired at step 604. The particular point (e.g., a coordinate) in the candidate image that corresponds to the respective data point is determined by mapping the coordinate of the respective data point (in a 3D coordinate system of the point cloud) to a coordinate in a coordinate system of the candidate image. The coordinate of the respective data point may be mapped to a coordinate in a coordinate system of the candidate image according to the camera model equation of Equation 1: Coord image = Coord data point − Pos camera Rot camera − 1 Proj camera − 1 where Coordi mage is the coordinate of the particular point corresponding to the respective data point in a coordinate system of the candidate image, Coord datapoint is a coordinate of the respective data point in a coordinate system of the point cloud, Pos camera is the position of image acquisition device 104 at the time when the candidate image was acquired, Rot camera − 1 is the rotation or orientation of image acquisition device 104 at the time when the candidate image was acquired, and Proj camera − 1 is the matrix representing parameters for mapping a 3D coordinate of the point in the camera's coordinate frame into the an image coordinate. Coord datapoint is known from the point cloud, while Pos camera and Rot camera − 1 are measured by devices or sensors (e.g., global navigation satellite system, an inertial measurement unit, wheel sensors, and / or other suitable devices or sensors GPS, not shown) associated with image acquisition device 104. Proj camera − 1 may take different forms depending on the image projection (e.g., spherical projection, pinhole camera model, etc.).

[0029] According to the claimed invention, at step 704, for each candidate image in the set of candidate images, a subject distance to the particular point corresponding to the respective data point in that candidate image is determined.

[0030] In one exampole, the subject distance to the particular point in that candidate image is determined based on spatial information in the point cloud. In one example, Coord data point can be solved for in Equation 1 to determine a 3D coordinate (e.g., a data point) in the point cloud corresponding to the particular point in the candidate image, and the distance between the image acquisition device and the corresponding coordinate can be determined from the point cloud as the subject distance to the particular point in the candidate image. In one example, the distance may be calculated by x 2 + y 2 + z 2 , where x, y, and z are components of Coord data point , the 3D coordinate in the point cloud corresponding to the particular point in the candidate image. In another example, the point cloud is converted to the coordinate system of the candidate image according to Coord data point − Pos camera Rot camera − 1 . Since the coordinate of the particular point in the candidate image and the position of image acquisition device 104 in the coordinate system of the candidate image are known, the distance subject distance to the particular point in the candidate image may be calculated by x 2 + y 2 + z 2 , where x, y, and z are components the 3D coordinate the particular point in the candidate image.

[0031] In another example, the subject distance to the particular point in that candidate image is extracted from a depth map of that candidate image. A depth map of the candidate image is an image that comprises information relating to the subject distance to points of interest (e.g., points corresponding to each of the respective data points) in the particular image. In one example, a depth map for each candidate image is generated in pre-processing stage 600 by processing system 108 and stored in storage 126. The depth map for each candidate image may be generated using any suitable approach, such as, e.g., those known in the art. In one example, a depth map may be generated according to method 900 of Figure 9, discussed below.

[0032] In one example, the subject distance to the particular point in the candidate image may be stored as the coordinate of the particular point in the depth map. For example, the value of the subject distance to the particular point may be stored in each pixel. Since a depth map is generated for the candidate image, the coordinate system of a candidate image and the coordinate system of its corresponding depth map are the same. Accordingly, the coordinate of the point in the coordinate system of the candidate image is the same as the coordinate of the point in the coordinate system of its corresponding depth map.

[0033] According to the claimed invention, at step 706, an image is identified, from the specific set of candidate images, having a shortest subject distance to the particular point corresponding to the respective data point. For example, the subject distance to the particular point for each of the candidate images determined at step 704 may be compared with each other and a candidate image having a shortest subject distance to the particular point may be identified.

[0034] According to the claimed invention, at step 708, it is determined whether the particular point in the identified candidate image is the un-occluded point of the particular object. In one example, it is determined whether the particular point in the identified candidate image is the un-occluded point of the particular object by performing a comparison between 1) a measured distance (e.g., measured distances 136, 140 of Figures 5A and 5B) between the image acquisition device that acquired the identified candidate image (e.g., image acquisition device 104) and a location on the particular object corresponding to the respective data point at a time that the image acquisition device acquired the identified candidate image; and 2) the subject distance (e.g., subject distances 134, 138 of Figures 5A and 5B) to the particular point in the identified candidate image. In one example, the measured distance is determined from the point cloud. The subject distance to the particular point in the identified candidate image is determined at step 704.

[0035] In one example, comparing the measured distance with the subject distance may include calculating a difference between the measured distance and the subject distance. For example, the difference may be calculated as diff = measured distance - subject distance, where measured distance is the distance between the image acquisition device that acquired the identified candidate image and a location of the particular object corresponding to the respective data point at the time that the image acquisition device acquired the identified candidate image and subject distance is the subject distance to the particular point in the identified candidate image determined at step 704.

[0036] In one example, the difference may be compared with a predetermined error threshold to account for error. For example, the error threshold may be 0.5 meters. The particular point is determined to occlude the object 110 in the identified candidate image if the difference diff satisfies (e.g., is greater than) the error threshold.

[0037] In response to determining that the particular point in the identified candidate image is not the un-occluded point of the particular object (i.e., the location on the particular object is occluded), method 700 proceeds to step 710 and the specific set of candidate images is updated by removing the identified candidate image from the specific set of candidate images. Image 200 of Figure 2 is an example of an image having point 130 (of an occlusion 114) occluding object 110. Method 700 returns to step 706 to identify a candidate image from the updated specific set of candidate images.

[0038] In response to determining that the particular point in the identified candidate image is the un-occluded point of the particular object (i.e., the location on the particular object is not occluded), method 700 proceeds to (optional) step 712. Image 400 of Figure 4 is an example of an imaging having point 132 that does not occlude object 110 (i.e., point 132 is an un-occluded point of object 110 corresponding to the respective data point).

[0039] At step 712, it is determined whether the particular point in the identified candidate image should be masked. Certain points in the candidate images may be masked to prevent colorizing the respective data point with undesirable objects (e.g., surveying system 102, a vehicle that surveying system 102 is mounted on, etc.). In one example, a pixel intensity value in a mask image at a location (e.g., a coordinate) of the particular point may be determined. The mask image may be a predetermined mask image having masked portions that, e.g., identify undesirable objects present in all image of the specific set of candidate images. In one example, the mask image is generated during pre-processing stage 600 of Figure 6. Figure 8 shows an illustrative mask image 800. Masked portions 802 correspond to portions identifying undesirable objects in the set of candidate images. Masked portions 802 are represented in mask image 800 with a non-zero pixel intensity value while unmasked portions 804 are represented in mask image 800 with a zero pixel intensity value.

[0040] If the determined pixel intensity value is non-zero, the particular point should be masked. In response to determining that the particular point should be masked, method 700 proceeds to step 710 and the specific set of candidate images is updated to remove the identified candidate image from the specific set of candidate images. Method 700 returns to step 706 to select a candidate image from the updated specific set of candidate images.

[0041] If the determined pixel intensity value is zero, the particular point should not be masked. In response to determining that the particular point should not masked, at step 714, the identified candidate image is returned to step 612 of Figure 6 as the selected image

[0042] It should be understood that the steps of method 700 may be performed in any suitable order. For example, while method 700 is described by identifying a candidate image having a shortest subject distance to the particular point (step 706) before determining whether the particular point in the identified candidate image is the un-occluded point of the particular object (step 708), method 700 may also be performed by first identifying a subset of the plurality of candidate images that include an un-occluded point of the particular object corresponding to the respective data point, and then identifying a candidate image, from the subset, that has a shortest subject distance to its particular point.

[0043] According to claimed embodiments, at step 614, the respective data point is colorized based a color in the selected image of the un-occluded point of the particular object corresponding to the respective data point. In one example, the color in the selected image may be a color value (e.g., a red, blue, green (RGB) color value) extracted from the selected image at the un-occluded point of the particular object corresponding to the respective data point. The respective data point is colorized with the extracted color value by applying or assigning the extracted color value to the respective data point. For example, the respective data point may be assigned the extracted RGB color value to colorize the respective data point with the extracted RGB color value.

[0044] The examples described herein were experimentally applied to colorize a point cloud. The point cloud colorized using the examples described herein were compared with point clouds colorized using conventional approaches. It was found that the point cloud colorized using the examples described herein accurately colored data points that conventional approaches miscolored due to occlusions.

[0045] Figure 9 illustrates aspects that aid in understanding the claimed invention. Figure 9 shows a method 900 for generating a depth map for a particular image. Method 900 may be performed as part of pre-processing stage 600 of Figure 6. Method 900 may be repeated for each respective image in the set of candidate images to generate a depth map for each respective image. The generated depth maps may be used at step 704 of Figure 7 to determine a subject distance to the particular point in each particular image of the specific set of candidate images.

[0046] At step 902, for each specific coordinate of a plurality of coordinates of the respective image, a subject distance to the specific coordinate is determined. The plurality of coordinates may be any coordinates in the respective image, such as, e.g., coordinates corresponding to every pixel of the image. In one example, each coordinate in the plurality of coordinates corresponds to a data point of the set of data points of the point cloud associated with object 110. The plurality of coordinates may be mapped to the coordinate system of the respective image from coordinates of the data points in a coordinate system of the point cloud, e.g., using Equation 1 described above.

[0047] In one example, the subject distance to the specific coordinate in the respective image is determined based on spatial information in the point cloud. For example, the data point corresponding to the specific coordinate in the respective image is determined, e.g., using Equation 1, and the distance between the image acquisition device and the corresponding data point is determined from the point cloud as the subject distance to the specific coordinate in the respective image.

[0048] At step 904, each specific coordinate is represented based on its subject distance to that specific coordinate in the respective image to provide a depth map. The purpose of this step is to introduce a size to each measurement to fill in the "holes" of the generated depth map and prevent the depth map from being too sparse. In one example, each specific coordinate is represented having a pixel size determined based on its subject distance to that specific coordinate in the respective image. For example, the pixel size may be determined according to Equation 2. Pixel Size = 10 − min floor 2 × Subject Distance , 10 where pixel size is the size of the pixel determined for the specific coordinate and subject distance is the subject distance to the specific coordinate in the respective image. The relationship between the pixel size and the subject distance in Equation 2 is shown by graph 1000 of Figure 10. Other suitable approaches may be employed to represent coordinates in a depth map.

[0049] At step 906, the subject distance to each specific coordinate in the respective image is stored in the depth map. In one example, the value of the subject distance is stored in each pixel.

[0050] The depth map should retain the subject distance information for its closest points. For example, when a coordinate is represented with a larger pixel size, to ensure that surrounding pixels are not overwritten with information indicating longer subject distances, the subject distance determined at step 902 is compared with the current subject distance stored in the depth map. The current subject distance is overwritten if the determined subject distance is shorter than the current subject distance.

[0051] Systems, apparatuses, and methods described herein may be implemented using digital circuitry, or using one or more computers using well-known computer processors, memory units, storage devices, computer software, and other components. Typically, a computer includes a processor for executing instructions and one or more memories for storing instructions and data. A computer may also include, or be coupled to, one or more mass storage devices, such as one or more magnetic disks, internal hard disks and removable disks, magneto-optical disks, optical disks, etc.

[0052] Systems, apparatus, and methods described herein may be implemented using computers operating in a client-server relationship. Typically, in such a system, the client computers are located remotely from the server computer and interact via a network. The client-server relationship may be defined and controlled by computer programs running on the respective client and server computers.

[0053] Systems, apparatus, and methods described herein may be implemented within a network-based cloud computing system. In such a network-based cloud computing system, a server or another processor that is connected to a network communicates with one or more client computers via a network. A client computer may communicate with the server via a network browser application residing and operating on the client computer, for example. A client computer may store data on the server and access the data via the network. A client computer may transmit requests for data, or requests for online services, to the server via the network. The server may perform requested services and provide data to the client computer(s). The server may also transmit data adapted to cause a client computer to perform a specified function, e.g., to perform a calculation, to display specified data on a screen, etc. For example, the server may transmit a request adapted to cause a client computer to perform one or more of the method steps described herein, including one or more of the steps of Figures 6, 7, and 9. Certain steps of the methods described herein, including one or more of the steps of Figures 6, 7, and 9, may be performed by a server or by another processor in a network-based cloud-computing system. Certain steps of the methods described herein, including one or more of the steps of Figures 6, 7, and 9, may be performed by a client computer in a network-based cloud computing system. The steps of the methods described herein, including one or more of the steps of Figures 6, 7, and 9, may be performed by a server and / or by a client computer in a network-based cloud computing system, in any combination.

[0054] Systems, apparatus, and methods described herein may be implemented using a computer program product tangibly embodied in an information carrier, e.g., in a non-transitory machine-readable storage device, for execution by a programmable processor; and the method steps described herein, including one or more of the steps of Figures 6, 7, and 9, may be implemented using one or more computer programs that are executable by such a processor. A computer program is a set of computer program instructions that can be used, directly or indirectly, in a computer to perform a certain activity or bring about a certain result. A computer program can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0055] Processing system 108 of Figures 1 and 3 shows a high-level block diagram of an example computing device that may be used to implement systems, apparatus, and methods described herein. Processing system 108 includes a processor 128 operatively coupled to a data storage device 126 and a memory 124. Processor 128 controls the overall operation of processing system 108 by executing computer program instructions that define such operations. The computer program instructions may be stored in data storage device 126, or other computer readable medium, and loaded into memory 124 when execution of the computer program instructions is desired. Thus, the method steps of Figures 6, 7, and 9 can be defined by the computer program instructions stored in memory 124 and / or data storage device 126 and controlled by processor 128 executing the computer program instructions. For example, the computer program instructions can be implemented as computer executable code programmed by one skilled in the art to perform the method steps of Figures 6, 7, and 9. Accordingly, by executing the computer program instructions, the processor 128 executes the method steps of Figures 6, 7, and 9. Processing system 108 may also include one or more network interfaces (not shown) for communicating with other devices via a network. Processing system 108 may also include one or more input / output (I / O) devices (not shown) that enable user interaction with processing system 108 (e.g., display, keyboard, mouse, speakers, buttons, etc.).

[0056] Processor 128 may include both general and special purpose microprocessors, and may be the sole processor or one of multiple processors of processing system 108. Processor 128 may include one or more central processing units (CPUs), for example. Processor 128, data storage device 126, and / or memory 124 may include, be supplemented by, or incorporated in, one or more application-specific integrated circuits (ASICs) and / or one or more field programmable gate arrays (FPGAs).

[0057] Data storage device 126 and memory 124 each include a tangible non-transitory computer readable storage medium. Data storage device 126 and memory 124 may each include high-speed random access memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate synchronous dynamic random access memory (DDR RAM), or other random access solid state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices such as internal hard disks and removable disks, magneto-optical disk storage devices, optical disk storage devices, flash memory devices, semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM), digital versatile disc read-only memory (DVD-ROM) disks, or other non-volatile solid state storage devices.

[0058] Input / output devices may include peripherals, such as a printer, scanner, display screen, etc. For example, input / output devices may include a display device such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor for displaying information to the user, a keyboard, and a pointing device such as a mouse or a trackball by which the user can provide input to processing system 108.

[0059] Any or all of the systems and apparatus discussed herein, including image acquisition device 104 and scanner 106 of Figures 1 and 3, may also be implemented using one or more computing devices.

[0060] One skilled in the art will recognize that an implementation of an actual computing device may have other structures and may contain other components as well, and that processing system 108 of Figures 1 and 3 is a high level representation of some of the components of such a computing device for illustrative purposes.

[0061] The foregoing Detailed Description is to be understood as being in every respect illustrative and exemplary, but not restrictive, and the scope of the invention disclosed herein is not to be determined from the Detailed Description, but rather from the claims as interpreted according to the full breadth permitted by the patent laws. It is to be understood that the examples shown and described herein are only illustrative of the principles of the present invention and that various modifications may be implemented by those skilled in the art within the scope defined by the appended set of claims.

Claims

1. A method for colorizing a set of point cloud data points associated with an object (110) using a plurality of candidate images (200, 400) comprising images that are candidates for colorizing the set of point cloud data points, the method comprising: for each respective data point in the set of point cloud data points: (i) for each candidate image in a set of candidate images initially comprising the plurality of candidate images (200, 400), determining (702) a particular point in that candidate image corresponding to the respective data point by mapping the coordinates of the respective data point in the coordinate system of the point cloud to the coordinates in the coordinate system of that candidate image, (ii) for each candidate image in the set of candidate images, determining (704) the distance from an image acquisition device (104) that acquired that candidate image to the location (502, 504) corresponding to the particular point (130, 132) in that candidate image at the time that the image acquisition device (104) acquired that candidate image, (iii) identifying (706) the candidate image (200, 400), from the set of candidate images, having the shortest distance from the image acquisition device (104) that acquired the candidate image to the location (502, 504) corresponding to the particular point (130, 132) in the candidate image at the time that the image acquisition device (104) acquired the candidate image by comparing the determined distances with each other, (iv) determining (708) whether the particular point in the identified candidate image is an un-occluded point (132) of the object (110) by comparing the distance (136, 140) from the image acquisition device (104) that acquired the identified candidate image (200, 400) to the location (112) on the object (110) corresponding to the respective data point at the time that the image acquisition device (104) acquired the identified candidate image (200, 400) and the distance (134, 138) from the image acquisition device (104) that acquired the identified candidate image to the location (502, 504) corresponding to the particular point (130,132) in the identified candidate image (200, 400) at the time that the image acquisition device (104) acquired the identified candidate image (200, 400), (v) in response to determining that the particular point in the identified candidate image (400) is an un-occluded point (132) of the object (110), selecting (714) the identified candidate image (400), and (vi) colorizing (614) the respective data point based on the color in the selected image of the un-occluded point (132) of the object (110) corresponding to the respective data point.

2. The method of claim 1, wherein step (iv) comprises: determining whether the difference between the distance (136, 140) from the image acquisition device (104) that acquired the identified candidate image (200, 400) to the location (112) on the object (110) corresponding to the respective data point at the time that the image acquisition device (104) acquired the identified candidate image (200, 400) and the distance (134, 138) from the image acquisition device (104) that acquired the identified candidate image (200, 400) to the location (502, 504) corresponding to the particular point (130,132) in the identified candidate image (200, 400) at a time that the image acquisition device (104) acquired the identified candidate image (200, 400) is smaller than or equal to an error threshold.

3. The method of claim 1 or 2, wherein step (vi) comprises: extracting a color value from the un-occluded point (132) of the object (110) in the selected image (400); and assigning the extracted color value to the respective data point.

4. The method of any one of claims 1 to 3, further comprising: determining (712) that the un-occluded point (132) in the selected image (400) should not be masked based on a pixel intensity value at a coordinate in a mask image (800) corresponding to the particular point.

5. The method of claim 4, wherein the determining (712) that the un-occluded point (132) in the selected image (400) should not be masked based on a pixel intensity value at a coordinate in a mask image corresponding to the particular point comprises: determining that the un-occluded point (132) in the selected image (400) should not be masked when the pixel intensity value is zero.

6. The method of any one of claims 1 to 5, further comprising: in response to determining that the particular point in the identified candidate image (200) is not an un-occluded point (132) of the particular object (110), removing (710) the identified candidate image (200) from the set of candidate images (200, 400) and repeating steps (iii) and (iv) until it is determined that the particular point in the identified candidate image (400) is an un-occluded point (132) of the particular object (110).

7. An apparatus, comprising: a processor (128); and a memory (124) storing computer program instructions which, when executed on the processor (128), cause the processor (128) to perform the method according to any one of claims 1 to 6.

8. A non-transitory computer readable medium storing computer program instructions, which, when executed on a processor (128), cause the processor (128) to perform the method according to any one of claims 1 to 6.