Method and system for enhancing color in a set of images

Abstract

Methods and systems for enhancing colors in a set of images. The method includes: acquiring input data including 3D data and image data, the image data being a set of single images, each image having a portion overlapping other images and consisting of a plurality of pixels, each pixel providing radiation measurement information; performing a joint integration on the input data to create integrated data, the joint integration including a 3D registration of the 3D data and the image data, a spatial graph decomposition that generates the structured representation, and a multi-homography decomposition that estimates homography in the image data; performing multi-scale feature extraction and block-by-block feature embedding on the integrated data, thereby integrating features of the 3D data and the image data in a single feature vector and embedding both geometric information and radiation measurement information; based on block-by-block feature embedding, generating a block-by-block weight map for each homography in the overlapped part; a pixel-by-pixel correction is performed in the set of images based on the weight map to generate a set of corrected images with homogenized radiation measurement information.

Description

Technical Field

[0001] This invention relates to a method and system for enhancing color and other radiometric information in a set of images of an environment, for example, to generate panoramic images or colorize 3D point clouds or meshes. In particular, color enhancement includes improved normalization of color information from image data of multiple images of the environment using 3D data of the same environment. For example, this allows for more uniform colorization of 2D or 3D data of the environment, even when images are captured under different lighting conditions. Background Technology

[0002] Generating 3D point clouds is used to survey many different settings, such as construction sites, building facades, industrial facilities, building interiors, or any other applicable scenarios. The resulting survey can be used to obtain an accurate 3D model of the scene, which includes the point cloud. The points in this cloud are stored as coordinates in a coordinate system defined by the surveying instrument recording the point cloud. Typically, the origin of the coordinate system is formed by the instrument's center (particularly by the instrument's so-called nodal point). Points are typically surveyed by associating distances measured with a laser beam (using a time-of-flight method) with the alignment under which the distance is measured. The coordinate system is usually a spherical coordinate system, such that the points are characterized by their distance values, elevation angles, and azimuth angles relative to the origin of the reference coordinate system.

[0003] Common surveying instruments include units for emitting a scanning beam and receiving the reflected beam to measure the distance to a point pointed to by the beam. Typically, these instruments also include devices for rotatably changing the beam direction, usually a vertical rotation axis and a horizontal rotation axis, both sensed by angle sensors. Typically, rotation of the vertical axis is measured by azimuth, and rotation of the horizontal axis by elevation. If the surveying instrument is specifically implemented as a laser scanner, one of the axes can be a slow axis and the other a fast axis.

[0004] Distance can be calculated using time-of-flight (TOF) measurement by observing the time between signal transmission and reception. Alignment angles are achieved using angle sensors positioned along the vertical and horizontal axes.

[0005] In the field of surveying, providing colored 3D point clouds is a desirable feature, for example, for LiDAR-based surveying tools. Color features aid in understanding and navigating through scenes, as well as identifying elements of interest, thus providing a more "human-visual-friendly" product than uncolored point clouds. Furthermore, color features are widely used as input features for many state-of-the-art point cloud post-processing algorithms, such as segmentation, classification, and / or modeling algorithms. Additionally, the calibration and projection of 3D data into images is a desirable feature, as this creates a "metric" image where certain measurements can be performed directly within the image.

[0006] To provide better visualization, point clouds can be digitally colored. In various applications, to provide color information for colorizing point clouds, ground surveying is therefore supported by imaging data from at least one calibrated imaging sensor (e.g., a camera), which is combined with the surveying instrument by including the camera in the instrument or mounting it on the same platform as the instrument.

[0007] These imaging sensors are integrated or attached to a LiDAR measurement system with accurate intrinsic and extrinsic camera calibration, so that two features acquired by the imaging sensors can be projected / mapped onto a 3D LiDAR point cloud and vice versa.

[0008] Devices configured to generate a digital three-dimensional representation of an environment by simultaneously capturing 3D data along with panoramic images of the environment are also referred to as "reality capture devices." WO 2020 / 126123 A2 discloses such a reality capture device having a laser scanner and multiple RGB cameras. EP 4 095 561 A1 discloses a reality capture device that combines multiple time-of-flight cameras for capturing 3D data with multiple RGB cameras.

[0009] Some LiDAR measurement systems integrate simultaneous location and mapping (SLAM) and / or additional localization technologies that enable the dynamic use of these devices. Thus, to measure and reconstruct 3D scenes along dynamic trajectories, the LiDAR can be carried by an operator or mounted on a transport platform, such as an unmanned ground vehicle (UGV).

[0010] While LiDAR measurement technology is robust to changes in position and lighting conditions, the imaging sensors are highly sensitive to these changes, especially when capturing RGB data. Even when RGB data is collected simultaneously by imaging sensors mounted in the same device, minute shifts in the optical center due to construction constraints (called parallax) or differences in sensor orientation will most likely produce significant differences in color, brightness, and / or contrast, even when capturing the same scene. LiDAR is unaffected by lighting conditions such as multiple artificial light sources indoors, resulting in both highly illuminated and very dark areas; it is generally insensitive to reflections from sunlight and can handle overexposed areas.

[0011] Gaussian-like post-processing methods can be used to improve 3D visualization. However, these are optimized only by photometric loss and lack a learning mechanism to group image patches of the same color together.

[0012] EP 3 944 184 A1 discloses a method for enhancing images captured by a reality capture device. EP 4459 562 A1 discloses a method for applying color to regions of a 3D point cloud using image inpainting, where reliable RGB colors from an RGB image cannot be assigned to points due to point occlusion caused by parallax between the LiDAR and the camera. A neural network derives the missing colorization information from a joint evaluation of intensity values ​​from the LiDAR and color information from the camera. Summary of the Invention

[0013] The purpose of this invention is to provide an improved method and system for color matching and uniformization of a set of images, particularly a method for enhancing colors in panoramic images and colorized 3D data.

[0014] The specific objective is to provide a method that enables color matching and normalization across images acquired from different viewpoints and / or at different times.

[0015] Another objective is to provide a method that robustly matches and normalizes radiometric measurements (color, brightness, and contrast) across the entire image without introducing geometric aberrations in the process.

[0016] Another objective is to provide a method that allows the generation of panoramic images and colored 3D data, such as point clouds or meshes.

[0017] According to certain aspects of the invention, the fundamental principles of color matching and normalization, most likely applicable to image stitching and panorama generation use cases in existing literature, are enhanced with LiDAR guidance. This provides geometric reliability, eliminates any geometric aberrations often introduced by conventional methods during the "blending" process, and provides robustness to variations in position and lighting conditions. Furthermore, color matching and normalization are achieved along multiple different scan positions and scan times.

[0018] A first aspect of the invention relates to a computer-implemented method for homogenizing radiometric information in a set of images of an environment. The method includes:

[0019] - Acquire input data, which includes image data of the environment and 3D data including geometric information of the environment, wherein the image data is acquired as a set of single images, each image having a portion overlapping with other images in the set and consisting of multiple pixels, each pixel providing radiometric measurement information;

[0020] - Perform joint integration on the input data to create integrated data, the joint integration including 3D registration of the 3D data and the image data, spatial graph decomposition for generating structured representations, and multi-homography decomposition for estimating homography in the image data;

[0021] - Perform multi-scale feature extraction and block-by-block feature embedding on the integrated data to integrate the features of the 3D data and the image data into a single feature vector and embed geometric and radiometric measurement information;

[0022] - Based on the block-by-block feature embedding, generate a block-by-block weight map for each homography in the overlapping portion; and

[0023] - Based on the weight map, pixel-by-pixel correction is performed in the set of images to generate a set of corrected images with the homogenized radiometric measurement information.

[0024] According to some implementations, the method includes propagating a corrected image onto 3D data to generate a colored 3D model of the environment. For example, the colored 3D model may be a colored 3D point cloud or a colored 3D mesh.

[0025] According to some other implementations, the method includes stitching together corrected images to generate a panoramic image of the environment.

[0026] According to some implementations of this method, 3D data and image data are acquired by the same reality capture device.

[0027] In some implementations, input data is acquired at multiple times and / or from multiple locations of the real-world capture device, and 3D registration includes the registration of 3D data acquired at each time and / or multiple locations.

[0028] In some implementations, the method is performed by a computing unit of the reality capture device. For example, the computing unit is also configured to control the reality capture device's acquisition of input data.

[0029] According to some embodiments of the method, at least a subset of the images has one or more non-overlapping portions (i.e., portions that do not overlap with any other images). Each of these non-overlapping portions images a portion of the environment that is not imaged in any other image in that set of images. In this case, pixel-by-pixel correction can be performed in both the overlapping and non-overlapping portions of the images.

[0030] According to some implementations of this method, the 3D data is a point cloud acquired by a LiDAR unit or multiple ToF cameras.

[0031] According to some implementations of this method, the 3D data includes a depth map.

[0032] According to some implementations of this method, 3D registration includes image LiDAR radar intrinsic and extrinsic parameters.

[0033] According to some embodiments of the method, the radiation measurement information includes at least color. Optionally, the radiation measurement information also includes brightness and / or contrast.

[0034] According to some implementations of this method, the geometric information includes the 3D coordinates of multiple points.

[0035] According to some implementations of this method, multi-scale feature extraction and block-by-block feature embedding are performed by a neural network.

[0036] According to some implementations of this method, multiple individual images are acquired by multiple cameras with overlapping fields of view.

[0037] A second aspect of the invention relates to a reality capture device comprising: a plurality of sensors configured to acquire input data including image data and 3D data of the environment; and a computing unit configured to control the acquisition of the input data. The computing unit stores program code for performing a computer-implemented method according to the first aspect of the invention.

[0038] Multiple sensors may include, for example, multiple image sensors for capturing image data, and LiDAR units or multiple ToF cameras for capturing 3D data.

[0039] The third aspect of the invention relates to a computer program product comprising program code stored on a machine-readable medium or embodied by an electromagnetic wave including program code segments, and having computer-executable instructions for performing the method according to the first aspect of the invention, particularly when executed in the computing unit of a real-world capture device according to the second aspect of the invention. Attached Figure Description

[0040] In the following, the invention will be described in detail by way of exemplary embodiments with reference to the accompanying drawings, wherein:

[0041] Figures 1a to 1b Two exemplary implementations of the reality capture device are shown;

[0042] Figures 2a to 2b A simple method for generating panoramic images and colorized point clouds is shown;

[0043] Figure 3 A first exemplary embodiment of the method for generating panoramic images and colorized point clouds according to the present invention is shown;

[0044] Figures 4a to 4b This demonstrates how LiDAR data can be used to enhance matching and normalization; and

[0045] Figure 5 A second exemplary embodiment of the method according to the present invention is shown. Detailed Implementation

[0046] Figure 1a and Figure 1b Two exemplary embodiments of the reality capture devices 51 and 52 according to the present invention are shown. Both devices are configured to capture data about the environment, which includes at least image data and 3D data of the environment, particularly RGB data and 3D point clouds.

[0047] Figure 1a A first embodiment of the reality capture device 50 shown includes a plurality of cameras 55 configured to capture image data and a laser scanner (LiDAR) unit 53 configured to capture 3D point clouds. The LiDAR unit 53 and the cameras 55 can be configured to capture corresponding data simultaneously.

[0048] Although the term "camera" is used here, other image sensors can also be used to capture image data. Basically, any kind of sensor can be used to acquire radiometric measurement information in a structured manner (e.g., represented as a pixel map), regardless of the wavelength acquired (e.g., RGB, infrared, NVDI, or multispectral).

[0049] Device 50 includes a base 57 on which a main body 56 is mounted, allowing the main body 56 to rotate about an azimuth axis (or a vertical axis). A beam guiding unit of LiDAR unit 53 is mounted within the main body, allowing it to rotate about an elevation axis (or a horizontal axis) orthogonal to the azimuth axis. Camera 55 is attached to or integrated into the main body 56 of the device. LiDAR unit 53 includes a transmitting unit for providing (emitting) a transmitted beam and a detection unit for detecting (receiving) a received beam.

[0050] Figure 1b A second embodiment of the reality capture device 51 shown includes a plurality of RGB cameras 55 configured to capture image data and a plurality of ToF cameras 54 configured to capture 3D point clouds by capturing time-of-flight (ToF) data of a plurality of pixels. The ToF cameras 54 and the RGB cameras 55 are configured to capture data simultaneously in a 360° range. Instead of having a tripod and base, the second embodiment of the reality capture device 51 includes a handle 59 attached to the main body 56 to allow a user to carry the device 51 through the environment.

[0051] Components for controlling the reality capture devices 50, 51 and a computer (not shown here) for performing the method according to the invention can be integrated into the respective devices 50, 51, for example, within the main body 2. Alternatively, an external computer can be connected to the devices, for example, via a cable or wireless data connection.

[0052] Figure 2a and Figure 2b A simplified representation is shown for using from Figure 1a and Figure 1b One of the reality capture devices takes input to generate a data stream of panoramic images or colored point clouds. For example... Figure 2a As shown, data 1, including LiDAR data 11 (i.e., 3D point cloud) and image data 12 generated from a reality capture device, is used as input to an algorithm that generates a colorized panoramic image 5 and / or a colorized 3D point cloud (or mesh). Figure 2b The generated data 1 includes multiple sets of data 1', 1', 1'' generated at multiple different locations and / or at multiple different times (each set of data includes LiDAR data and image data).

[0053] In many known applications, image data is used to generate panoramic visualizations or representations of scenes, or to colorize 3D point clouds of the same scene, thereby enabling better visualization and understanding of 3D information. Furthermore, it can be used for several post-processing and downstream tasks. In all cases, consistency of radiometric measurements from the same and different locations on the measurement surface is critical.

[0054] In many use cases, it is important to preprocess image data to "homogenize" radiometric features, whether in images acquired simultaneously from the same scan location or between images acquired at different times and scan locations, in order to mitigate or eliminate certain artifacts or problems. Examples of these use cases include the following:

[0055] 1. Panoramic images generated from multiple partially overlapping images acquired simultaneously using image stitching strategies will suffer from multiple chromatic aberrations or radiometric aberrations without proper color matching and normalization. This will significantly reduce the end user's qualitative perception of the product and may even hinder the understanding of the depicted scene.

[0056] 2. The panoramic images generated as described in 1 will reduce and / or hinder any post-processing or downstream tasks. Post-processing techniques such as detection or segmentation may significantly degrade performance if unrealistic and unreliable radiometric measurements exist in the scene, especially if these differences create artificial or non-existent boundaries, edges, or even “soft” color gradients in the depicted scene.

[0057] 3. Any downstream task that combines images acquired from different viewpoints, i.e., for robust object detection and “tracking” (i.e., assigning unique identifiers to the same assets present in the scene even if acquired from different locations and viewpoints), may be compromised due to the described artifacts and subsequent differences in radiometric measurements (i.e., color, brightness, and / or contrast).

[0058] 4. “Colorized” point clouds (i.e., point clouds with filled features from any type of calibrated imaging sensor) will suffer from multiple chromatic aberrations or radiometric aberrations without proper color matching and normalization. Since “colorized” point clouds often consist of multiple “registration” scans—that is, the integration of LiDAR and image data acquired from different locations and times—the existence and impact of radiometric differences in the scene will be amplified. This will significantly reduce the end-user's qualitative perception of the product and may even hinder their understanding of the depicted scene.

[0059] 5. As described in section 4, “colorizing” point clouds (or other colorized 3D representations) will reduce and / or hinder any post-processing or downstream tasks. Post-processing techniques such as detection or segmentation may significantly degrade performance if unrealistic and unreliable radiometric measurements exist in the scene, especially if these differences create artificial or non-existent boundaries, edges, or even “soft” color gradients in the depicted scene.

[0060] Traditionally, image stitching tasks, which include basic color matching and normalization, can be accomplished for multiple overlapping images acquired at the same time and location, as shown below:

[0061] 1. Registration stage: The distortion matrix is ​​estimated, which is then used for image alignment.

[0062] 2. Fusion Stage: The aligned images are merged into a single fused image. Current research in this field can be broadly categorized into two main types: reconstruction-based methods and seam-based methods. Reconstruction-based methods typically employ encoder-decoder networks for pixel-by-pixel reconstruction of the fused image. Seam-based methods focus on identifying optimal seams to eliminate fusion ghosting.

[0063] 3. Rectification stage. Irregularly shaped fused images are transformed into standard rectangular formats.

[0064] Even when using cutting-edge AI-based technologies, this traditional approach has the following drawbacks:

[0065] 1. Generation of radiometric artifacts: Linearization of radiometric measurements around the “seam” region often reduces the shift or difference in color, brightness, or contrast between images. Even if a “smooth” transition can be achieved between stitched images, this problem fails to capture remote context and is most likely not to compensate for radiometric biases along the entire image.

[0066] 2. Generation of Geometric Artifacts: The wrapping matrix computed in the "registration stage" is tailored to "blend" the images involved in the process in a way that minimizes the generation of visual artifacts / discontinuities on the seam areas. There is no way to ensure that these "blending" are geometrically realistic. This problem becomes significant when propagating RGB features to a 3D point cloud.

[0067] 3. Color matching and normalization on images acquired from different viewpoints and potentially at different times cannot be solved using these techniques.

[0068] Figure 3 A data flow is shown in an exemplary embodiment of the method according to the invention, which allows for robust and reliable color matching and normalization.

[0069] By leveraging geometric features and relationships extracted from 3D point clouds, the described method robustly matches and normalizes radiometric measurements (color, brightness, and contrast) not only around the seams of individual images but also along the image. It ensures that no geometric aberrations are introduced during the process, and that the 2D geometric features present in the image are aligned with the geometric features of the assets in 3D measured by the LiDAR sensor. Furthermore, by utilizing 3D information provided by the LiDAR point cloud to identify “common” regions in the image data and spatial relationships in 2D (image array) and 3D (LiDAR point cloud), the system ensures that a longer context is captured, allowing for color matching and normalization on images acquired from different viewpoints and / or at different times.

[0070] Input data 1 is captured, including LiDAR scan data 11 and image data 12 from multiple cameras. Input data can be collected at a single location and at a single time. (Example) Figure 2b As shown, input data can also be captured at multiple locations and times.

[0071] Joint integration 2 is applied to input data 1. This includes analytical 3D registration 21, spatial graph decomposition 22, and multiple homography decomposition 23. Joint integration 2 of LiDAR and image data 10, 12 generates joint and uniform image patch segmentation. For example, it can be computed through AI-driven solutions, traditional CV-based solutions, or heuristic-based solutions (e.g., primitive geometry fitting).

[0072] The analytical 3D registration 21 includes image-LiDAR intrinsics and extrinsics. If multiple LiDAR scans are available, the 3D registration 21 also includes LiDAR-LiDAR transformation.

[0073] The concept of spatial graph decomposition is typically described in "Superpixel Image Classification with Graph Attention Networks" by Pedro HC Avelar et al. (arXiv:2002.05544v2, November 15, 2020) and "Scalable 3DPanoptic Segmentation As Superpoint Graph Clustering" by Damien Robert et al. (arXiv:2401.06704v2, February 7, 2024). In the context of the methods shown, spatial graph decomposition22 generates structured representations including nodes and edges, which, according to graph theory, not only identify and encode relevant segments in both LiDAR and image spaces, but also identify and encode the spatial relationships between them.

[0074] The concept of multi-homography decomposition is generally described in Simon Seibt et al.'s "Parallax-aware Image Stitching based on Homographic Decomposition". In the context of the method shown, multi-homography decomposition23 is only used to estimate homography (planar surfaces in space) and does not require computation of iterative dense feature matching and robust integration of LiDAR features as a source of 3D geometric ground reality.

[0075] Next, the neural network 3 performs multi-scale feature extraction in both LiDAR and image space 31. This includes performing block-by-block feature embedding, thereby integrating multi-scale LiDAR and image features into a single feature vector at the image block level, and embedding geometric and radiometric features.

[0076] The image data includes overlapping regions 15 and non-overlapping regions 16, where overlapping regions are portions of the image that are also imaged by other images, and non-overlapping regions are portions of the image that are imaged only by that image. A block-by-block multidimensional weight map 32 is computed for each estimated homography in the overlapping regions 15 and used for geometric blending and radiometric normalization. Then, for both the overlapping and non-overlapping regions 15, the block-by-block weight map 32 is used to fill in pixel-by-pixel corrections 33 and 34, thereby producing a corrected image 18.

[0077] Finally, the corrected image 18 is propagated or projected 4 onto the (colorized) panoramic image 5 or the registered colorized 3D point cloud 6. In the case of the panoramic image 5, this involves perspective transformation using calibrated camera intrinsic and extrinsic parameters, and blending using calculated pixel-by-pixel corrections 33, 34. In the case of the point cloud 6, this involves image correction (i.e., removing geometric distortion using inherent calibration) and propagation of direct radiometric features.

[0078] Figure 4a and Figure 4b The generation of the corrected image is shown. In Figure 4a The image shows three images 12a-12c of a first set of images of the environment captured simultaneously by a reality capture device and a 3D point cloud. The first image 12a has an overlapping region 15 with its neighboring images 12b and 12c, and a non-overlapping region 16 (i.e., an area where no other images were captured). Each image consists of multiple pixels; that is, each image constitutes a pixel map. Figure 4b In, it is shown Figure 4a The first image 12a-12c consists of three images 12a-12c and three additional images 12d-12f from a second set of images set in the same environment. Images 12d-12f from the second set of images were captured simultaneously and / or at different times with a second 3D point cloud from different locations. Due to the larger number of images, the first image 12a has a higher density than the first image 12c. Figure 4a The larger portion of the overlapping area is 15.

[0079] Due to varying lighting conditions, pixels in different images 12a-f imaging the same part of an environment can have different radiometric characteristics (color, brightness, contrast). This is disadvantageous for generating panoramic images or colorized point clouds, because, for example, when the same color in different parts of the environment is represented differently in the corresponding parts of the colorized 2D or 3D data, the result can be patchy, unpleasant to the eye, or even misleading.

[0080] Within the same location exist feature blocks that indicate different radiometric characteristics (such as the color of a location in an image). Without LiDAR information, users or algorithms must now rely solely on image information to find blocks such as identical objects, regions, or surfaces. This approach typically produces inconsistent results because it fails to identify true regions or surfaces. The assistance of LiDAR information eliminates all these uncertainties and allows for the identification of contiguous regions or surfaces, guiding colors to their true values.

[0081] Using point cloud data captured by the same device as image data (point cloud registration is applied if more than one point cloud is captured), certain points in the point clouds are selected as anchor points 11. Anchor points 11 represent points in the environment visible in more than one image, i.e., points imaged in the overlapping region 15. Due to point cloud registration and image stitching, points from all point clouds can be applied to images from all groups of images. Using anchor points 11, pixels from different images 12a-f can be registered with each other with high determinism. Therefore, these pixels are known to image the same parts of the environment and should therefore have the same radiometric characteristics (color, etc.). If the radiometric characteristics are not the same, correction is required.

[0082] Figure 5 This is a flowchart illustrating an exemplary embodiment of method 100 according to the present invention. The method begins by acquiring 110 input data, which includes 3D data providing geometric information of the environment and image data providing radiometric information of the environment. The image data is acquired as a plurality of individual images, each having a portion overlapping with other images and consisting of a plurality of pixels. 120 Joint integration is performed on the input data to create integrated data. The joint integration includes 3D registration of the 3D data and image data, spatial graph decomposition for generating a structured representation, and multihomography decomposition for estimating homography in the image data. 130 Multiscale feature extraction and block-by-block feature embedding are performed on the integrated data to integrate features of the 3D data and image data into a single feature vector and embed geometric and radiometric information. Based on the block-by-block feature embedding, 140 block-by-block weight maps are generated for each homography in the overlapping portion. Based on the weight maps, 150 pixel-by-pixel correction is performed in both the overlapping and non-overlapping portions to generate a corrected image.

[0083] If the task is to generate a colored 3D model of the environment, the calibrated images are propagated 160° to the 3D data. If the task is to generate a colored panoramic image of the environment, the calibrated images are stitched together 170°.

[0084] Although the invention has been described above with reference to some preferred embodiments, it should be understood that many modifications and combinations of different features of the embodiments can be made. All such modifications are within the scope of the appended claims.

Claims

1. A computer-implemented method (100) for homogenizing radiometric information in a set of images (12a-f) of an environment, the computer-implemented method comprising: Acquire (110) input data (1), the input data including 3D data (10), the 3D data including geometric information of the environment and image data (12) of the environment, wherein the image data (12) is acquired as a set of images (12a-f), each image having an overlap (15) with other images in the set of images and consisting of multiple pixels, each pixel providing radiometric measurement information; The input data (1) is subjected to joint integration (2) to create integrated data, the joint integration including 3D registration (21) of the 3D data (10) and the image data (12), spatial graph decomposition (22) for generating structured representations, and multi-homography decomposition (23) for estimating homography in the image data (12). (130) Multiscale feature extraction (31) and block-by-block feature embedding are performed on the integrated data to integrate the features of the 3D data and the image data into a single feature vector and embed geometric and radiometric measurement information; Based on the block-by-block feature embedding, a block-by-block weight map (32) is generated for each homography in the overlapping portion (15); and Based on the weight map (32), pixel-by-pixel correction (33) is performed (150) in the set of images (12a-f) to generate a set of corrected images (18) with homogenized radiometric measurement information.

2. The method (100) according to claim 1, wherein the method comprises propagating (160) the corrected image (18) to the 3D data (10) to generate a colored 3D model (6) of the environment, and particularly wherein, The colorized 3D model (6) is either a colorized 3D point cloud or a colorized 3D mesh.

3. The method (100) according to claim 1, wherein the method includes stitching (170) the corrected image (18) to generate a panoramic image (5) of the environment.

4. The method (100) according to any one of the preceding claims, wherein, The 3D data (10) and the image data (12) are acquired by the same reality capture device (50, 51) (110).

5. The method (100) according to claim 4, wherein The input data (1) is acquired (110) at multiple times and / or from multiple locations of the reality capture devices (50, 51); and The 3D registration (21) includes the registration of 3D data acquired at each of the times and / or multiple locations.

6. The method (100) according to claim 4 or claim 5, wherein, The method (100) is performed by the computing unit of the reality capture device (50, 51), and in particular, the computing unit is configured to control the acquisition of the input data (1) by the reality capture device (50, 51).

7. The method (100) according to any one of the preceding claims, wherein At least one subset of the images (12a-f) has non-overlapping portions (16), each non-overlapping portion imaging a portion of the environment that was not imaged in any other image (12a-f) of the set of images; and The pixel-by-pixel correction (33) is performed (150) in the overlapping portion (15) and the non-overlapping portion (16) of the image (12a-f).

8. The method (100) according to any one of the preceding claims, wherein, The 3D data (10) includes: Point clouds acquired by a LiDAR unit (53) or multiple ToF cameras (54); and / or Depth map.

9. The method (100) according to any one of the preceding claims, wherein, The 3D registration (21) includes image-LiDAR intrinsic and extrinsic parameters.

10. The method (100) according to any one of the preceding claims, wherein, The radiation measurement information includes at least color, and in particular, the radiation measurement information also includes brightness and / or contrast.

11. The method (100) according to any one of the preceding claims, wherein, The geometric information includes the 3D coordinates of multiple points.

12. The method (100) according to any one of the preceding claims, wherein, The multi-scale feature extraction (31) and the block-by-block feature embedding are performed by the neural network (3) (130).

13. The method (100) according to any one of the preceding claims, wherein, The multiple individual images (12a-f) are acquired by multiple cameras with overlapping fields of view.

14. A reality capture device (50, 51), said reality capture device comprising: Multiple sensors are configured to acquire input data (1) including image data (12) and 3D data (10) of the environment. and a computing unit configured to control the acquisition of the input data (1), wherein, in particular, the plurality of sensors include: Multiple image sensors are used to capture the image data (12); and A LiDAR unit (53) or multiple ToF cameras (54) for capturing 3D data (10). Its features The computing unit stores program code for performing the method (100) according to any one of the preceding claims.

15. A computer program product comprising program code stored on a machine-readable medium or embodied by an electromagnetic wave including program code segments, and having computer-executable instructions for performing the method (100) according to any one of claims 1 to 13, particularly when the computer-executable instructions are executed in a computing unit of a reality capture device (51, 52) according to claim 14.

Images