Dark image enhancement

Abstract

The present invention relates to dark image enhancement, and more particularly to a method for enhancing images for metrology applications. A sensor image represents a scene of interest with initial metrology information to determine the position and / or orientation of at least one object therein. The sensor image has sensor image geometric correctness. The sensor image is described by image parameters associated with a scoring function for evaluating the sensor image. An enhanced image generated from the sensor image is also provided. The enhanced image is described by image parameters with enhanced image parameter values. The enhanced image has a processed image geometric correctness lower than the sensor image geometric correctness. The method comprises: providing a geometrically corrected image having an image geometric correctness higher than the processed image geometric correctness and showing at least a portion of the scene of interest; and at least partially reducing the loss of initial metrology information in distorted metrology information by fusing the enhanced image with the geometrically corrected image.

Description

Technical Field

[0001] The present invention relates to a method, a computer program product and a system. Background Art

[0002] Images perceived by the human eye as dark are often acquired in dim or rapidly changing environments, where the rapid changes may come from fast camera movements. In the second case, a short exposure time of the camera capturing the image may be desired to avoid blurring the image; however, a short exposure time results in a human observer perceiving the image as dark. Such a dark image may additionally include a significant amount of noise. Therefore, it may be beneficial to improve the dark image before further processing, for example, to enable a human observer to identify objects in the image and to infer metrological information from the identified objects. For images with a low signal-to-noise ratio (SNR), the algorithm used to enhance the image is preferably non-linear, as the non-linearity may help to separate the signal from the noise and selectively amplify only the signal.

[0003] The image is usually presented to a human observer and / or a machine. The human observer and / or the machine extracts information from the image. EP15176238 provides an example of image-based metrology. The information extracted is usually spatial in nature and relates to, for example, distance information or relative positions of two objects to each other.

[0004] For example, for triangulation of an object, two images of the object may be used to infer the spatial position of the object. When the base distance between the cameras providing the two images is known, corresponding features in the two images (e.g., the features are corners of the object) can be detected and used to determine the 3D position of the portion of the object corresponding to the detected corresponding features. Triangulation using corresponding features requires accurate determination of the features in the two images. For example, if the detected features are offset far from their true positions in the two images, the object will not be correctly triangulated.

[0005] Feature detection accuracy usually depends on the SNR of the image being analyzed. Images acquired in dim or rapidly changing environments usually have features with low SNR. For such images, feature detection will produce many errors, and subsequent triangulation using partially erroneous features will provide incorrect 3D positions of the triangulated objects. Therefore, images with low SNR may need to be enhanced before further processing.

[0006] For example, in terrestrial laser scanning applications or mobile mapping applications, point cloud data acquired by a laser scanner can be colored using a color image provided by a color-resolving camera. If a color image is acquired in a dim or rapidly changing environment, the resulting color image is typically characterized by a low SNR. Since measurements in a colored point cloud typically depend on the color of the colored point cloud, it may be necessary to enhance the color image before coloring the point cloud to improve subsequent measurements in the colored point cloud. The user can also select individual points or areas in the colored point cloud based on color information, for example for further inspection. For example, it may also be necessary to improve color fidelity (corresponding to white balance) and / or make the colors more vivid before coloring the point cloud.

[0007] For example, the vividness of a color can be defined in the hue-saturation-lightness (HSL) color space. Vividness can be measured using the saturation component. For example, if you move out from the center of the HSL cylinder to the edge of the HSL cylinder, the color will become more vivid.

[0008] Ideally, properties such as the position of edges or corners of objects in an image should remain unchanged after image enhancement, i.e., ideally, the position of edges or corners of objects in an image should be the same before and after image enhancement. Since the mentioned properties may subsequently be used for triangulation, for example, the mentioned properties to be preserved are metrological properties. For metrological applications, ideally, object contours or similar details in an image should remain unchanged after image enhancement, since determining metrological information from an image usually requires information encoded by object contours or similar details in the image.

[0009] As an example of a metrology application, a mobile mapping platform (e.g., a car-based mapping platform) can be used to map an urban environment. The purpose of mapping an urban environment is to obtain dense 3D reconstructions of houses, trees, road assets, and other objects that are at least temporarily stationary in the city. For example, a car used to map an urban environment may be equipped with a lidar and a camera. The pixels of the image sensor of the camera typically correspond to a square area with a side length in the sub-centimeter range. Therefore, structures in the sub-centimeter range in the urban environment are typically mapped to one pixel. The car may drive through narrow streets lined with tall buildings, which block direct sunlight, resulting in a dim environment, especially before or after peak daylight conditions. Car-based mapping systems typically move at speeds of 10 meters per second to 30 meters per second. The dim urban environment and the required short exposure time - to avoid blurred images caused by car movement - combine to result in low SNR images, which are darker. Similar considerations about dim environments are generally applicable to indoor environments.

[0010] Static mapping of an environment as an example of a metrology application may also be affected by dim or rapidly changing environments. In the case of a terrestrial laser scanner equipped with a color resolving camera, the provided color image may be noisy and have a low SNR.

[0011] Brightness enhancement of dark images is a well-studied problem. Recently, two publications by Chen et al. (Chen, C., Chen, Q., Xu, J., & Koltun, V. (2018). Learning to see in the dark. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3291-3300) and Chen, C., Chen, Q., Do, MN, & Koltun, V. (2019). Seeing motion in the dark. In Proceedings of the IEEE International Conference on Computer Vision (pp. 3185-3194)) studied the problem of enhancing dark images. The first of the two articles proposed a technique for enhancing a single image, while the second of the two articles proposed a technique for enhancing a video. Both articles use a deep learning approach, i.e., using input / output training data containing dark images and corresponding bright ground truth images to learn the mapping from dark images to enhanced bright images. Due to the use of nonlinearity and prior knowledge encoded in the learned CNN, the SNR of the enhanced bright images is generally improved compared to the dark images provided to the CNN for enhancement. The techniques studied by Chen et al. in both articles do not take into account the metrological properties of the images, i.e., the algorithms are explicitly designed to enhance only the input images without any relation or consideration to metrology.

[0012] If applied to noisy images, deep learning methods can also be used to reduce noise, or increase the resolution of the image, or increase the color fidelity of the image, or transform the color of the image to more vivid colors. Neural networks can also be trained to provide a combination of the above functions, that is, for example, a neural network can jointly increase the resolution of an image and increase the brightness of an image. Summary of the invention

[0013] It is therefore an object of the present invention to provide a method of enhancing an image while maintaining metrological properties.

[0014] Another object of the present invention is to provide a method for enhancing images acquired by a mobile or static mapping platform while maintaining metrological properties.

[0015] Yet another object of the present invention is to provide a method for enhancing images obtained by a mobile or static mapping platform for mapping an urban environment while maintaining metrological properties.

[0016] The invention relates to a method for enhancing an image for metrology applications. A sensor image is provided that represents a scene of interest using initial metrology information to determine the position and / or orientation of at least one object within the scene of interest. The sensor image has sensor image geometric correctness. The sensor image is described by image parameters provided by a scoring function that evaluates the sensor image, the image parameters particularly relating to brightness and / or noise and / or resolution and / or color fidelity and / or color vividness of the sensor image, the image parameters having initial image parameter values. An enhanced image generated from the sensor image is also provided. The enhanced image is described by image parameters having enhanced image parameter values, the enhanced image parameter values ​​particularly relating to higher brightness and / or reduced noise and / or high resolution and / or high color fidelity and / or more vivid colors than the sensor image. The enhanced image has processed image geometric correctness, the processed image geometric correctness relating to distorted metrology information, the distorted metrology information representing the loss of initial metrology information caused by image processing using a neural network, the processed image geometric correctness being lower than the sensor image geometric correctness. The method further comprises the steps of: 1) providing a geometrically corrected image having a higher image geometric correctness than the processed image and showing at least a portion of a scene of interest, and 2) at least partially reducing a loss of initial metrological information in the distorted metrological information by fusing the enhanced image with the geometrically corrected image.

[0017] The scoring function may, for example, determine the SNR of the image to which the scoring function is applied, in which case the image parameter corresponds to the SNR. The scoring function may also, for example, determine the resolution and / or brightness and / or color fidelity and / or color vividness and / or brightness of the image to which the scoring function is applied. The image parameter may be at least a part of the function output of the scoring function.

[0018] The scene of interest may be located in an urban environment, which is typically characterized by narrow streets flanked by tall buildings that block direct sunlight. Such urban environments are typically dim, particularly before or after peak daylight conditions. For example, the scene of interest may be a street in a city, wherein the street is mapped. The view of the scene of interest in the urban environment may also be blocked from above, for example due to tree canopies or due to overhanging parts of buildings. The sensor image in the present invention preferably captures (images) the scene of interest from a viewing position at substantially the same height as the typical structures in the scene, such as buildings or trees, in the case where the scene of interest is located in an urban environment. Since the sensor image - particularly in an urban environment - captures the scene of interest from a viewing position close to the scene of interest, structures in the urban environment in the sub-centimeter range are typically mapped to one pixel of the image sensor providing the sensor image.

[0019] A scene of interest (e.g., a street in an urban environment) can be mapped using a mobile mapping platform, such as a car-based mapping platform equipped with an imaging device configured to capture the scene of interest. Since car-based mapping systems typically move at speeds of 10 meters per second to 30 meters per second, it may be necessary to capture sensor images (the image sensor providing the sensor image is, for example, mounted on a car-based mapping platform) with a shorter exposure time to avoid blurring the image due to the movement of the car. Coupled with the dim environment often encountered in urban environments, the sensor image may be dark, where dark refers to a qualitative statement made by a typical human observer of the sensor image. The sensor image may be perceived by a human observer as being dark. Such "dark" images also typically have a low SNR because noise sources can affect image quality, for example, thermal noise in the image sensor is more noticeable in low-light imaging conditions than, for example, imaging in broad daylight.

[0020] Terrestrial laser scanning systems are equipped with a digital camera that takes sensor images and may also take "dark" sensor images when working in a dimly lit room. Terrestrial laser scanning systems are usually stationary, i.e. they are usually fixed in a specific location.

[0021] The sensor image geometric correctness may correspond, for example, to the sensor image contrast or to a measure of the deviation of an edge in the sensor image from a true position of the edge, such as that which in principle could be obtained by perfect imaging without imaging errors. Therefore, the sensor image may have a low sensor image contrast. The image contrast may measure the difference between the highest intensity value and the lowest intensity value of the image. A low sensor image contrast may mean that few easily discernible structures are visible in the sensor image. For example, if the sensor image is a color image, the sensor image contrast may be calculated using, for example, the brightness calculated from individual pixels of the color image.

[0022] The geometrically corrected image may also have a higher radiometric correctness than the sensor image. The higher radiometric correctness may occur in case the sensor image is non-uniform (e.g. due to the recording settings of the camera that acquired the sensor image); the geometrically corrected image may be acquired in such a way that the non-uniformity in the geometrically corrected image is smaller than the non-uniformity in the sensor image, in particular if the geometrically corrected image is acquired by an active imaging system such as a LiDAR imaging system.

[0023] The image sensor providing the sensor image may be characterized at least in part by a mathematical camera model, such as a pinhole camera model, or an orthogonal projection, or a scaled orthogonal projection, or a quasi-perspective projection, or a perspective projection. Thus, the scene of interest is typically distorted by the image sensor according to at least a combination of the image sensor position and orientation, the spatial arrangement of the imaged scene of interest, and the mathematical camera model describing the image sensor. In the case of the sensor image, this distortion may also be referred to as a first projection distortion.

[0024] For example, using at least the sensor image, metrological information such as the location of a house or a car in an urban environment can be determined. The sensor image is described by image parameters, wherein the image parameters may particularly relate to the brightness and / or noise level and / or resolution and / or color fidelity and / or color vividness of the sensor image. The values ​​of the image parameters are determined by evaluating a scoring function on the sensor image, the evaluation providing initial image parameter values. The scoring function may, for example, be embodied as a function that calculates SNR and / or brightness, or any other measurable attribute using the image as input. The image parameters may also relate to any perceptible quality of the sensor image. The sensor image may be "dark" and / or noisy and / or have a low resolution and / or have a low color fidelity and / or have a low color vividness. It is beneficial to enhance the sensor image before further metrological processing. For example, for brightness enhancement, state of the art machine learning techniques, in particular neural networks, such as those of Chen et al. cited above, may be used for this purpose. Since such machine learning techniques contain knowledge about the task to be solved - obtained via a training process using input / output image pairs, the input image corresponding to a "dark" version of a "bright" ground truth output image, for example - the enhancement provided by such machine learning techniques can increase the SNR of the sensor image to which the neural network is applied, because the processing provided by the neural network is non-linear. The neural network usually only needs to be trained on input / output image pairs that include a subset of images, which can be recorded by a sensor that is substantially similar to the image sensor. The basic similarity is understood here in a statistical manner. The neural network can be trained using images characterized by a specific statistical distribution, where the specific statistical distribution is sufficiently close to the statistical distribution of the images acquired by the image sensor. As long as this condition is met, the network can be trained using images recorded by an image sensor different from the image sensor. For example, in the case of training a neural network to perform brightness enhancement, the enhanced image is brighter than the sensor image, where brighter is a specific sense, such as perceived by a typical human observer who perceives the brightness enhanced image and the sensor image. For example, in the case of training a neural network to increase the resolution of a sensor image, the enhanced image has a higher resolution than the sensor image, where the neural network performs interpolation.

[0025] For example, the vividness of a color can be defined in a hue-saturation-lightness (HSL) color space. Vividness can be measured using the saturation component. For example, if you move out from the center of the HSL cylinder to the edge of the HSL cylinder, the color becomes more vivid.

[0026] Enhancement of a sensor image provided by a machine learning technique specifically trained for enhancing sensor images results in distortion of the enhanced image. The processed image geometry correctness of the enhanced image is lower than the sensor image geometry correctness of the sensor image. For example, during brightness enhancement by a neural network, edges may shift, which means that edge information in the enhanced image may be less reliable compared to the sensor image. For example, where the enhanced image will be used for subsequent triangulation of an object in a captured scene of interest or for coloring a point cloud, ideally, properties such as the location of edges or corners of an object in the sensor image should remain unchanged after image enhancement, i.e., ideally, the location of edges or corners of an object in the sensor image should be the same before and after image enhancement. However, prior art machine learning techniques for image enhancement are only trained for enhancing images, such as increasing brightness, and not for maintaining metrological properties of the enhanced image.

[0027] According to the present invention, a geometrically corrected image can be used to provide additional geometric information to an enhanced image to improve further metrological processing of the enhanced image. For example, since metrological applications such as triangulation from images typically require corresponding feature detection in at least two images, and since the detected features are typically edges or corners of objects in the images, it is critical to retain such contour information for metrological processing of the enhanced image. The geometrically corrected image can provide additional contour information to the enhanced image. The geometrically corrected image according to the present invention has an image geometric correctness that is higher than the geometric correctness of the processed image. For example, the geometrically corrected image can better represent edge information than the enhanced image.

[0028] According to the present invention, a geometrically corrected image captures (includes) at least a portion of a scene of interest captured by a sensor image. The geometrically corrected image may be provided with a second projection distortion of at least a portion of the captured scene of interest. The second projection distortion may be similar to the first projection distortion. For example, in the case where the geometrically corrected image is acquired by a different image sensor, the different image sensor is different from the image sensor that can provide the sensor image, the sensor image and the geometrically corrected image may correspond to rectified images acquired by the image sensor and the different image sensor, wherein rectification means projection onto a common image plane. For such a rectified sensor image and geometrically corrected image, the first projection distortion and the second projection distortion will be the same.

[0029] The enhanced image and the geometrically corrected image are fused according to the present invention. For example, in the case where the first projection distortion and the second projection distortion are similar, and at least a portion of the scene of interest is captured jointly by the sensor image and the geometrically corrected image, the fusion allows information of higher image geometric correctness to be transferred from the geometrically corrected image to the enhanced image. For example, the enhanced image and the geometrically corrected image can be fused if they are sufficiently correlated, i.e., if they image the same scene with similar distortions of the imaged scene.

[0030] The method according to the invention can be used to solve assignment problems, where two (or more) data sources may need to be related to each other. An example of an assignment problem is coloring a 3D point cloud using a color image, where the color value of each pixel of the color image needs to be related to a 3D point of the 3D point cloud.

[0031] The fused image resulting from the fusion of the enhanced image and the geometrically corrected image may be used for metrology applications, such as for triangulation, wherein triangulation of an object captured in the fused image may be performed using at least the fused image.

[0032] In addition to geometric correctness, the present invention can also be used to enhance radiometric correctness. Radiometric correctness can be enhanced separately from geometric correctness or together with geometric correctness. A sensor image may have a sensor image radiometric correctness which - when processed by a neural network - may deteriorate, i.e. the processed image radiometric correctness may be lower than the sensor image radiometric correctness. The radiometrically corrected image may then be used - structurally equivalent to the use of the geometrically corrected image - to enhance the processed image by fusing the processed image with the radiometrically corrected image.

[0033] In one embodiment of the method, the sensor image is color-resolved, in particular acquired by a color-resolving camera with a Bayer pattern.

[0034] The sensor image may thus be an RGB image.

[0035] In the case where the camera uses a Bayer pattern, the data provided by the camera may need to be pre-processed before feeding them to the neural network for image enhancement. The pre-processing can provide three separate RGB images with a lower resolution than the original captured Bayer image. It is also possible to feed the original captured Bayer image directly to the neural network.

[0036] In another embodiment of the method, the geometrically corrected image is generated using the sensor image by a linear combination of color channels of the sensor image, in particular by applying a gamma expansion and / or a histogram equalization to the color channels before the linear combination.

[0037] A geometrically corrected image can be obtained from a sensor image by linearly combining the color channels of the sensor image, providing relative brightness. For example, for an RGB color model, the following linear combination of the color channels can be performed:

[0038] Y=0.2126R+0.7152G+0.0722B,

[0039] Wherein Y denotes the relative brightness, here the geometrically corrected image, and R, G and B denote the red, green and blue components of the sensor image. Other linear combinations are also possible. The sensor image may be gamma compressed by the image sensor. In this case, the sensor image may be gamma expanded before being linearly combined to derive the relative brightness. Linearly combining different color channels to obtain a relative brightness image may increase the image contrast of the linearly combined image compared to the image contrast present in the individual color channels. Metric information which depends, for example, on the detectability of object contours in the geometrically corrected image may therefore be more easily extracted from the geometrically corrected image obtained by linearly combining the color channels of the sensor image than from the sensor image itself.

[0040] In another embodiment of the method, the geometrically corrected image is provided by a monochrome camera.

[0041] Due to the lack of color filters, monochrome cameras generally have higher sensitivity than color resolving cameras, so images acquired by monochrome cameras generally have a high SNR compared to, for example, RGB images. Therefore, geometrically corrected images acquired by a monochrome camera can have higher geometric correctness than sensor images acquired by an RGB camera.

[0042] In another embodiment of the method, the geometrically corrected image is provided using a LiDAR imaging system, wherein the intensity of the received electromagnetic pulses is used to derive the geometrically corrected image.

[0043] For example, a lidar imaging system can be mounted on a car-based mapping platform to densely or sparsely map an urban environment. The lidar imaging system emits measurement pulses, in particular electromagnetic measurement pulses, wherein the emitted measurement pulses are reflected by objects in the scene of interest. The reflected measurement pulses are then received by the lidar imaging system. The intensity of the received measurement pulses can then be determined. Based on such intensity information, a geometrically corrected image can be constructed by scanning the scene of interest using the lidar imaging system. Depending on the scanning procedure performed by the lidar imaging system, the obtained raw intensity image may need to be re-gridded (possibly interpolated) to obtain a geometrically corrected image. Alternatively, using the arrival time of the received measurement pulses, 3D information about the scanned scene of interest can be derived. The 3D point cloud obtained in this manner can be centrally projected onto a lidar image plane, which corresponds in particular to a sensor image plane, which corresponds to a sensor image. After centrally projecting onto the lidar image plane, the projected 3D point cloud data can be re-gridded to conform to the pixel grid of the sensor image.

[0044] Since the LiDAR imaging system is an active imaging system, a high-contrast geometrically corrected image can be obtained even in a dim environment. The geometrically corrected image obtained by the LiDAR imaging system can have high geometric accuracy.

[0045] The fusion can also provide a colorized 3D point cloud. The color information from the enhanced sensor image can be transferred to the 3D point cloud obtained using the LiDAR imaging system.

[0046] The intensity values ​​provided by the LiDAR imaging system may also be used to define the brightness levels of the colors in the sensor image. In this way, non-uniformities in the sensor image caused by the position and orientation of the camera acquiring the sensor image and the surface of the object being photographed (the color information of the object being photographed may appear differently in sensor images taken from different positions and orientations and camera settings) may be mitigated. Setting the brightness levels of the colors in the sensor image using the LiDAR imaging system allows a more uniform sensor image to be obtained, which can then be used to colorize the 3D point cloud acquired using the LiDAR imaging system. Using a LiDAR imaging system in this manner is an independent invention and can be performed independently of the present invention.

[0047] In another embodiment of the method, the enhanced image is upsampled by a neural network, wherein a resolution of the upsampled enhanced image is higher than a resolution of the sensor image.

[0048] Higher resolution may relate to the ratio between the size of the field of view that may be captured by an image sensor and the number of pixels used in a representation of a sensor image that may be provided by the image sensor.

[0049] In another embodiment of the method, the fusion uses pan-sharpening.

[0050] Pan-sharpening can be used for the fusion. Image fusion based on principal component analysis (PCA) can also be used for the fusion, or fusion based on wavelet or curvelet, or other transform domain image fusion techniques can also be used for the fusion. Neural networks can also be used for the fusion.

[0051] In another embodiment of the method, the fusion includes the following steps: 1) forward transforming the enhanced image into a color model different from the color model used for the enhanced image, thereby providing a forward transformed enhanced image; 2) defining an inverse transform, wherein the inverse transform is defined relative to the forward transform; 3) using the forward transformed enhanced image and the geometrically corrected image to provide a geometrically enhanced forward transformed enhanced image; and 4) applying the inverse transform to the geometrically enhanced forward transformed enhanced image.

[0052] Fusion may be performed in different color spaces corresponding to different color models than the color model that provides the enhanced image. Forward transforming the enhanced image to a different color space may allow for better separation of geometric information, such as that encoded in illumination information, from other information encoded in the enhanced image. The forward transform has an inverse transform, referred to herein as the inverse transform. The forward transformed enhanced image and the geometrically corrected image may then be combined to obtain a geometrically enhanced forward transformed enhanced image. In order to return to the color model that provides the enhanced image, an inverse transform may be applied to the geometrically enhanced forward transformed enhanced image.

[0053] In another embodiment of the present invention, the different color model is a hue-saturation-value (HSV) color model.

[0054] In the HSV color model, the luminance component includes illumination information. Edge information, such as that required for triangulation of edge feature detectors, is primarily stored in the luminance component.

[0055] In another embodiment of the method, geometric enhancement is provided by calculating a point-by-point linear combination between the luminance components of the forward transformed enhanced image and the geometrically corrected image, wherein both the forward transformed enhanced image and the geometrically corrected image are provided in a grid representation and the point-by-point linear combination is calculated between corresponding grid elements.

[0056] The luminance component of the HSV color model may include geometric information required for subsequent metrology applications. The geometrically corrected image with higher geometric correctness may be combined with the luminance component of the forward transformed enhanced image by a point-by-point linear combination operation. In addition to the point-by-point linear combination, any other functional relationship between the luminance component of the geometrically enhanced forward transformed enhanced image and the luminance components of the forward transformed enhanced image and the geometrically corrected image is also feasible.

[0057] The fusion may be performed in a point-by-point manner, where each pixel of the forward transformed enhanced image may be fused with the corresponding pixel from the geometrically corrected image.

[0058] In another embodiment of the method, geometric enhancement is provided by replacing the luminance component of said forward transformed enhanced image with the luminance component of said geometrically corrected image.

[0059] In another embodiment of the method, the fusion corrects a geometric error present in the enhanced image, wherein the geometric error corresponds to a displacement error introduced into the sensor image by the enhancement provided by the neural network, wherein the displacement error refers to a displacement of corresponding points between the sensor image and the enhanced image.

[0060] The enhancement provided by the neural network may introduce geometric distortions to the image on which it operates. Using the geometrically corrected image, the fusion according to the present invention may advantageously at least partially compensate for this introduced geometric distortion. As a result, the resulting fused image may have a higher SNR (via the nonlinear enhancement provided by the trained neural network) and more reliable geometric information required for subsequent metrology applications performed on the fused image.

[0061] The invention also relates to a computer program product with a program code stored on a machine-readable medium or embodied as electromagnetic waves, said program code being configured to provide an image enhancement as in the method according to the invention.

[0062] The present invention also relates to an image processing system, which comprises a computing unit, a computer program product according to the present invention with program code stored on the computing unit, and at least one image sensor, which provides at least one image to the computing unit, wherein the image processing system is configured to perform an image enhancement method.

[0063] The invention further relates to an image processing system according to the invention, wherein the image processing system is specifically configured as part of a car-based mapping platform or a track-based mapping platform or a user-carried mapping platform or a static mapping platform.

[0064] A mobile mapping platform can map scenes of interest, such as streets in an urban environment. For example, a mobile mapping platform can be equipped with an RGB camera that acquires sensor images and a monochrome camera that acquires geometrically corrected images. Machine learning-based techniques can be used to enhance the sensor images and subsequently fuse them with the geometrically corrected images. In addition to operating on a single sensor image, a sequence of sensor images can also be enhanced jointly. The fusion of a sequence of geometrically corrected images with a corresponding sequence of enhanced images can be performed sequence element by sequence element.

[0065] A static mapping platform can be embodied, for example, by a terrestrial laser scanning system comprising a color resolution camera and a laser scanner. The color resolution camera can provide a sensor image, while the laser scanner can provide a geometrically corrected image. Examples of mobile mapping platforms are the Leica Pegasus 2 or Pegasus backpack. Examples of terrestrial laser scanners are the Leica RTC360 or the Leica BLK360. Laser trackers such as the Leica ATS600 can also be used. The Leica BLK2GO is an example of a mapping platform carried by a human user. Total stations (total station positioning systems, particularly those with scanning capabilities, such as the total station Leica Nova MS60), coordinate measuring machines, and articulated arms can also be used.

[0066] The image sensors may be mounted on a mobile or stationary mapping platform in such a way that the relative geometrical arrangement between the mounted image sensors is known. The mounted image sensors may also be calibrated.

[0067] Using the method according to the invention, metrology applications can be performed using images provided by cheaper image sensors than are typically used, for example, "dark" and low SNR images can be at least partially "corrected" by combining image brightness enhancement provided by a neural network and geometric enhancement provided by fusing the brightness enhanced image with a separate image. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The system of the invention is described in more detail below, by way of example only, with the aid of specific exemplary embodiments schematically illustrated in the accompanying drawings, other advantages of the invention are also being studied. In the drawings, the same elements are marked with the same reference numerals. In detail:

[0069] Figure 1 An embodiment of the present invention for mapping a scene of interest using a car-based mobile mapping platform is shown;

[0070] Figure 2 An embodiment of the method according to the present invention is shown;

[0071] Figure 3An embodiment of fusion according to the present invention is shown;

[0072] Figure 4 An embodiment of the method according to the present invention is shown;

[0073] Figure 5 An embodiment of the method according to the present invention is shown;

[0074] Figure 6 An embodiment of the method according to the present invention is shown;

[0075] Figure 7 An embodiment of the present invention for mapping a scene of interest using a laser scanner on a tripod is shown; and

[0076] Figure 8 An image processing system according to the invention is shown. DETAILED DESCRIPTION

[0077] Figure 1 An embodiment of the present invention for mapping a scene of interest 10 using a car-based mobile mapping platform 12 is shown. An RGB camera 1 and a LiDAR imaging system 11 are mounted on a car. While acquiring the environment, the car may continue to move. For example, a car-based mobile mapping platform 12 may typically move at a speed between 10 meters per second and 30 meters per second. Track-based systems can also be used for mapping. The mobile mapping system can also be carried by a person. Compared with car-based or track-based systems, such handheld mobile mapping systems may have lower movement speeds and greater dynamics such as rotation speeds. The LiDAR imaging system 11 and the RGB camera 1 may have known positions and orientations relative to each other, such as to allow image correction. The car-based mobile mapping platform 12 can travel through narrow streets in an urban environment, thereby mapping the environment it is traveling through. One metrological application may be to obtain a dense 3D reconstruction of an urban scene of interest 10. The dense 3D reconstruction should include the actual 3D reconstruction (i.e., the position information of the imaged surface in the 3D coordinate system) and the color information associated with the imaged surface. Using the method according to the invention, a dark and low SNR color image provided by the RGB camera 1 can be geometrically enhanced using, for example, a geometrically corrected image provided by a lidar imaging system 11, while the color information is provided by the enhanced image provided by a neural network.

[0078] Figure 2An embodiment of the method according to the present invention is shown. An RGB camera 1 and a monochrome camera 2 provide images. The RGB camera 1 provides a sensor image. The RGB camera 1 can provide a noisy three-channel RGB image 3 after preprocessing the raw RGB camera 1 output, wherein the noisy three-channel RGB image 3 can be taken with a shorter exposure time to avoid motion blur or the noisy three-channel RGB image 3 may be taken in a dim environment. Due to the higher sensitivity of the monochrome camera 2, the geometrically corrected image 4 provided by the monochrome camera 2 contains better geometric information. Due to the possible low signal-to-noise ratio of the noisy three-channel RGB image 3, the enhancement algorithm 5 operating on the noisy three-channel RGB image 3 is preferably nonlinear in its input. The enhancement algorithm may include a convolutional neural network (CNN), wherein the CNN can be trained for regression purposes. For example, the CNN can be trained to learn the mapping between a dark image and a bright image of the same scene, so that when a dark image is received as input, the CNN can transform the dark image into a brighter one. Therefore, the CNN can learn denoising, color conversion, demosaicing, and enhancement of dark and noisy input images. Since the CNN part is a black-box model, CNN may introduce geometric distortion in the three-channel RGB enhanced image8.

[0079] This geometric distortion is removed by using the additional information provided by the monochrome camera 2, wherein the required geometric information can be provided by the geometrically corrected image 4. The three-channel RGB enhanced image 8 and the geometrically corrected image 4 provided by the monochrome camera 2 are fused 6, for example by pan-sharpening. The overall three-channel RGB image 7 provided by the method is geometrically reliable and has a good color resolution. In its processed form 7, it is easier to undergo further metrological processing than the dark and noisy three-channel RGB image 3 provided directly by the RGB camera 1 after pre-processing.

[0080] Figure 3 An embodiment of the fusion according to the present invention is shown. The three-channel RGB enhanced image 8 is forward transformed into a different color model, in particular the Hue-Saturation-Value (HSV) color model. Figure 3 In an exemplary embodiment, the luminance component of the forward transformed three-channel RGB enhanced image 9 is replaced by a different luminance component 4a, in particular provided by the geometrically corrected image. After obtaining the geometrically enhanced forward transformed three-channel RGB enhanced image 9a, an inverse transform of the forward transform can be performed to obtain the desired three-channel RGB image 7. The luminance component 4a can provide accurate geometric information, while the three-channel RGB enhanced image 8 can provide color information that is not contained in the different luminance components 4a (in particular, the geometrically corrected image).

[0081] Figure 4An embodiment of the present invention for image enhancement is shown. An RGB camera 1 provides a noisy three-channel RGB image 3 of a scene of interest. The noisy three-channel RGB image 3 may have a large amount of noise because the image may be taken in a dim environment. Due to the possible movement of the RGB camera 1, a shorter exposure time may also be required. The dark and noisy three-channel RGB image 3 can be enhanced by a combination of an algorithm 5 including a convolutional neural network (CNN) and a pan-sharpening 6 that provides fusion, the CNN providing, for example, brightness enhancement. For fusion, a single-channel intensity image 4b ​​with lower noise can be derived from the noisy three-channel RGB image 3. The single-channel intensity image 4b ​​can be fused with the three-channel RGB enhanced image 8. The noisy three-channel RGB image 3 can be enhanced by a trained machine learning algorithm 5 including a CNN, wherein the CNN can be trained on an input / output pair of RGB images, wherein the input image can include a noisy and dark image of the scene of interest, and the output image can include, for example, a corresponding image that is brighter and less noisy. For example, the CNN can be trained in a way that learns a mapping from an input image to an output image. Due to the black-box nature of CNNs, geometric distortions may be present in the three-channel RGB enhanced image 8. The fusion 6 may aim to remove such distortions by using geometrically more reliable information 4b. The three-channel RGB image 7 provided by the fusion may be both enhanced and geometrically corrected, wherein possible geometric distortions of the machine learning algorithm 5 may be mitigated by a separate hard-coded fusion 6.

[0082] Figure 5 An embodiment of the method according to the invention is shown. An RGB camera 1 can provide a low-resolution three-channel RGB image 3a. An additional monochrome camera 2 with high sensitivity can provide a higher-resolution single-channel intensity image, i.e., a geometrically corrected image 4. In this embodiment, an algorithm 5 including a convolutional neural network (CNN) for enhancing the low-resolution three-channel RGB image 3a provided by the RGB camera 1 can include an upsampling function. For fusion 6, it may be beneficial if both the three-channel RGB enhanced image 8 and the geometrically corrected image 4 provided by the monochrome camera 2 have the same image resolution. After enhancement, in this case upsampling is performed by an algorithm 5 including a CNN, and the upsampled three-channel RGB enhanced image 8 and the monochrome image 4 are fused 6, for example by using pan-sharpening, ultimately providing a higher-resolution three-channel RGB image 7 as output. The neural network providing upsampling can also provide brightness enhancement and / or noise reduction and / or it can increase color fidelity and / or it can be combined with upsampling to make the colors more vivid.

[0083] Figure 6An embodiment of the method according to the invention is shown. A sensor image 3 is provided by an RGB camera 1. A lidar imaging system that images at least a portion of a scene of interest captured by the RGB camera 1 provides intensity values ​​13 of received electromagnetic pulses. The intensity values ​​13 may be re-gridded into a grid representation, which is a geometrically corrected image 14. In addition to using intensity values, alternatively, the 3D positions of points in a 3D point cloud recorded by the lidar imaging system may also be centrally projected onto the lidar image plane, the projected and possibly re-gridded image being a geometrically corrected image. The sensor image 3 is then enhanced by a neural network 5, and the enhanced image is then fused 6 with the geometrically corrected image 14. The fusion 6 provides an enhanced and geometrically reliable image 7.

[0084] Figure 7 An embodiment of the present invention for mapping a scene of interest 10 using a terrestrial laser scanner 15 on a tripod or a total station positioning system is shown. The terrestrial laser scanner 15 on a tripod is equipped with an RGB camera that acquires a sensor image. The sensor image is enhanced by a neural network, and the 3D point cloud provided by the laser scanner is transformed into a geometrically corrected image. The enhanced image and the geometrically corrected image are fused. For example, the scene of interest 10 can be a dimly lit urban environment or a dimly lit indoor room.

[0085] Figure 8 An image processing system according to the invention is shown. The image processing system comprises a computing unit 16 on which a computer program providing image enhancement according to the invention can be stored and executed. The computer program receives input from an RGB camera 1 providing a sensor image 3. Alternatively, additional inputs can be provided to the computer program, such as images provided by a monochrome camera or a LiDAR imaging system.

[0086] Image enhancement and fusion can be performed on-site at the measuring device or off-site after exporting the raw or pre-processed data from the measuring device. The raw measurement data can, for example, be transmitted to a cloud service where an enhanced and geometrically reliable image is computed.

[0087] It goes without saying that the illustrated figures are only schematic representations of possible exemplary embodiments.

[0088] Although the present invention has been illustrated above in part with reference to some preferred embodiments, it must be understood that various modifications and combinations of different features of the embodiments are possible. All these modifications are within the scope of the appended claims.

Claims

1. A method for enhancing an image for metrology applications, the method comprising the following steps: - providing a sensor image, the sensor image: representing a scene of interest using initial metrology information to determine a position and / or orientation of at least one object within the scene of interest, With sensor image geometric correctness, where The sensor image geometric correctness corresponds to a measure of the deviation of edges in the sensor image from their true positions, and is described by image parameters provided by a scoring function evaluated for the sensor image, the image parameters relating to brightness and / or noise and / or resolution and / or color fidelity and / or color vividness of the sensor image, the image parameters having initial image parameter values, - providing an enhanced image generated from the sensor image, wherein the enhanced image: is described by image parameters having enhanced image parameter values ​​relating to higher brightness and / or reduced noise and / or higher resolution and / or higher color fidelity and / or more vivid colors than the sensor image, and having a processed image geometric correctness, the processed image geometric correctness relating to distorted metrological information, the distorted metrological information representing a loss of initial metrological information due to image processing using a neural network, the processed image geometric correctness being lower than the sensor image geometric correctness, It is characterized in that - providing a geometrically corrected image, the geometrically corrected image: having an image geometric correctness that is higher than the processed image geometric correctness, showing at least a portion of the scene of interest, - at least partially reducing said loss of original metrology information in said distorted metrology information by fusing said enhanced image with said geometrically corrected image.

2. The method according to claim 1, It is characterized in that The sensor image is color resolved.

3. The method according to claim 2, It is characterized in that The geometrically corrected image is generated using the sensor image by a linear combination of its color channels.

4. The method according to claim 1 or 2, It is characterized in that The geometrically corrected image is provided by a monochrome camera.

5. The method according to claim 1 or 2, It is characterized in that The geometrically corrected image is provided using a LiDAR imaging system, wherein the intensity of received electromagnetic pulses is used to derive the geometrically corrected image.

6. The method according to claim 1, It is characterized in that The enhanced image is upsampled by the neural network, wherein a resolution of the upsampled image is higher than a resolution of the sensor image.

7. The method according to claim 1, It is characterized in that The fusion uses pan-sharpening.

8. The method according to claim 7, It is characterized in that The fusion comprises the following steps: - forward transforming the enhanced image into a color model different from the color model used for the enhanced image, thereby providing a forward transformed enhanced image, - defining an inverse transform, wherein the inverse transform is defined relative to the forward transform, - providing a geometrically enhanced forward transformed enhanced image using the forward transformed enhanced image and the geometrically corrected image, and - applying said inverse transform to said geometrically enhanced forward transformed enhanced image.

9. The method according to claim 8, It is characterized in that The different color model is a hue-saturation-lightness color model.

10. The method according to claim 9, It is characterized in that Geometric enhancement is provided by computing a point-by-point linear combination between the luminance components of the forward transformed enhanced image and the geometrically corrected image, wherein both the forward transformed enhanced image and the geometrically corrected image are provided in a grid representation and the point-by-point linear combination is computed between corresponding grid elements.

11. The method according to claim 9, It is characterized in that Geometric enhancement is provided by replacing the luminance component of the forward transformed enhanced image with the luminance component of the geometrically corrected image.

12. The method according to claim 1, It is characterized in that The fusion corrects geometric errors present in the enhanced image, wherein the geometric errors correspond to displacement errors introduced into the sensor image due to the enhancement provided by the neural network, wherein the displacement errors refer to the displacement of corresponding points between the sensor image and the enhanced image.

13. A computer program product having a program code stored on a machine-readable medium or embodied as an electromagnetic wave, the program code being configured to provide image enhancement as in the method according to any one of claims 1 to 12.

14. An image processing system comprising a computing unit, a computer program product according to claim 13 with program code stored on the computing unit, and at least one image sensor, the at least one image sensor providing at least one image to the computing unit, wherein: The image processing system is configured to perform an image enhancement method.

15. The image processing system according to claim 14, wherein: The image processing system is configured as part of a car-based mapping platform or a track-based mapping platform or a user-carried mapping platform or a static mapping platform.

Images