Low-light image enhancement method and device

Abstract

An artificial neural network (ANN) based on the Retinex theory estimates a reflection map and a noise map of an input image. The ANN is trained by unsupervised learning using multiple loss functions in order to facilitate extraction of feature points, such as in V-SLAM. The map is iteratively combined with the input low-light image. The processed image can be used in V-SLAM for a robot, especially an indoor robot and a construction robot in a low-light environment. And 65% RMSE reduction in the path trajectory is obtained.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 591,451, filed on October 19, 2023. The entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the technical field of machine learning, and in particular to image processing, and more specifically to image processing of low-light images. Background Art

[0004] In order to be used, the autonomous mobile operation of construction robots requires the collection of sufficient on-site data information. Currently, laser point clouds and red, green and blue (RGB) cameras are mainly used to provide data input to the positioning system. Visual data, such as image frames taken by cameras, potentially contains rich data information and provides a data foundation for construction site work.

[0005] One way to obtain information about the location is based on vision-based simultaneous localization and mapping (V-SLAM) algorithms and object detection methods, which can be used to build environmental maps and navigate. There are already a variety of methods available that rely on calculating the gray level changes between adjacent pixels of the input image of the environment and estimating the robot's pose by extracting and matching feature points between the images. Robot localization (V-SLAM) is designed to operate in environments with normal lighting or with auxiliary external light sources, rather than in low-light conditions.

[0006] In actual construction site environments, lighting conditions are often challenging due to factors such as uneven lighting intensity, insufficient exposure, or the robot is in a backlit location (e.g., indoor buildings, underground parking lots, outdoor construction sites at night, and backlit areas in mines). These adverse conditions can significantly degrade the image quality captured by the robot's camera, resulting in insufficient brightness and information loss in the captured image frames before any processing. Typical problems that can be observed include low brightness, low contrast, distorted colors, exposure-induced noise, and limited grayscale levels.

[0007] Therefore, the complex lighting conditions at construction sites pose a huge challenge to detection and localization based on visual information. How to obtain more complete visual information from the captured scene images under such low light and / or changing lighting conditions is the basis for the work of construction robots.

[0008] One option is to preprocess the images captured from the camera before inputting into V-SLAM. However, traditional low-light image enhancement schemes (such as histogram equalization or multi-scale retinal cortex (Retinex) with color restoration) cannot be directly applied to construction scenes because they require manual adjustment of parameters according to different lighting conditions. The lighting conditions on construction sites are variable, so it is impractical to adjust the parameters for each lighting condition encountered on the construction site.

[0009] Furthermore, known enhancement methods cannot address the problem of uneven illumination in a single image of a construction scene, for example, a scene comprising a dark corridor with no artificial lighting but with direct sunlight shining through window frames at regular intervals along the corridor walls.

[0010] Furthermore, existing enhancement methods lack control over low-light noise, which may result in amplifying noise during the image brightness adjustment process.

[0011] Another challenge is how to obtain suitable training data for machine learning used in this specific field. Generally, it is known to enhance image frames by computer. This can be achieved by deep learning based image enhancement techniques (e.g., trained artificial neural networks). Known artificial neural networks first undergo supervised learning, where paired image datasets (i.e., the same scene taken at different light levels that have been manually annotated) are used to train the artificial neural network. During training, existing techniques such as back propagation adjust the weights of the nodes in the neural network layers. However, due to the dynamic nature of construction sites, it is not feasible to obtain large-scale training datasets of low / normal brightness paired images on construction sites. In addition, deep learning methods sometimes introduce artifacts into the enhanced images due to the lack of separation of low-light image features. Summary of the invention

[0012] The disclosed method is based on extracting features from images after image enhancement processing.

[0013] In one aspect, the present disclosure provides a method for generating an artificial neural network for digital image processing of low-light images. An artificial neural network is established based on Retinex theory to estimate the reflectance map and noise map of an input image. The artificial neural network is trained by unsupervised learning using a loss function. The loss function is designed to facilitate the extraction of feature points from the input image after processing the reflectance map and noise map of the artificial neural network. The input image after the processing is a color-enhanced image. Color enhancement refers to an improvement in quality, which includes or is not limited to improvements in luminosity and / or other features (especially the results of the extraction and matching process of feature points in a subsequent V-SLAM). The extraction of feature points may refer to the extraction and / or matching of feature points in a subsequent V-SLAM process.

[0014] In some embodiments, at least multiple loss functions may be used, such as five or more.

[0015] In some embodiments, the loss function may include at least one of a feature loss, an exposure control loss, an exposure control loss, a distortion color loss, an illumination smoothness loss, and a noise suppression loss. The feature loss represents the difference between the features extracted in the processed image and the features extracted in the input image using a neural network image classification; the exposure control loss is used to adjust the pixel exposure brightness and uneven exposure by the difference between the good exposure level and the average intensity; the distortion color loss is used to align the color channels of the processed image; the illumination smoothness loss is used to maintain the smoothness of the illumination intensity in the color space of the processed image; and the noise suppression loss is used to suppress the noise map intensity.

[0016] In some embodiments, the feature loss is determined based on the feature extraction operator of the VGG-16 image extraction model, the input image, and the processed training image; the exposure control loss is determined according to the regions into which the image is divided, the exposure level of each region, and the number of non-overlapping local regions in the divided regions; the distortion color loss is determined according to the average intensity value of the RGB channels of the processed training image; the illumination smoothness loss is determined according to the horizontal gradient operator and the vertical gradient operator associated with the color space of the processed training image; and the noise suppression loss is determined according to multiple noise maps of the input image and the number of iterations in the iterative process.

[0017] In some embodiments, the training step includes: a new input image forming processing step, in which the input image is combined with the reflection map and the noise map is subtracted to form a new input image; then, the new input image is used as the input image, and the new input image forming processing step is iteratively performed multiple times to generate a processed training image; then, the artificial neural network is optimized based on the input image and the processed training image using a loss function.

[0018] The methods disclosed herein are computer-implemented.

[0019] In some embodiments, the loss function is applied to the training via a total loss function, which represents the loss functions used in the training, and weights are applied to each loss function so that each loss function used has a similar contribution to the total loss function.

[0020] In some embodiments, wherein the reflection map and the noise map comprise a set of paired reflection maps and noise maps, for example seven or more pairs of paired reflection maps and noise maps are generated for the input image or each input image, respectively, and starting from the input image, these pairs of reflection maps and noise maps are used through iterative processing to obtain a processed training image.

[0021] In some embodiments, the training comprises: passing a plurality of input images from a training dataset through the artificial neural network, the training dataset comprising at least a subset of a single image contrast enhanced (SICE) image dataset. In some embodiments, the artificial neural network comprises a plurality of (e.g., nine) layers that are convolutional layers, each layer having skip connections to maintain context between adjacent pixels.

[0022] According to an aspect of the present disclosure, a computer program product is provided, comprising a computer program code, which, when executed on a computer, will perform a method for generating an artificial neural network.

[0023] According to aspects of the present disclosure, a computer-implemented method for digital image processing of a low-light image using an artificial neural network generated by executing a method for generating an artificial neural network is provided. The method for digital image processing includes: inputting a scene image as an input image to the artificial neural network; in response to the input of the scene image, iteratively processing the scene image based on a plurality of reflection maps and a plurality of noise maps generated by the artificial neural network; and outputting a processed image.

[0024] In an embodiment of the method for digital image processing, said processed image has an increased brightness measure compared to said input image.

[0025] In an embodiment of the method for digital image processing, the method further comprises: adjusting the brightness of the processed image based on a comparison between the processed image output at time point t and other processed images output within its range; and outputting a brightness compensated image.

[0026] An aspect of the present disclosure provides a computer program product, comprising a computer program code, which when executed on a computer will perform a method for digital image processing of low-light images.

[0027] Aspects of the present disclosure provide a device for digital image processing of low-light images for a self-guided robot, wherein the device includes an image decomposition module and an image reconstruction module. The image decomposition module includes a trained artificial neural network, which estimates multiple reflection maps and multiple noise maps of an input image that can be obtained from a sensor of the self-guided robot based on Retinex theory, wherein the artificial neural network is trained by unsupervised learning using multiple loss functions. The image reconstruction module has an output for outputting a processed image, and the image reconstruction module is configured to iteratively process the input image by sequentially combining the reflection map and subtracting the noise map, and the reflection map and the noise map are generated for the input image respectively.

[0028] In some embodiments, the loss function facilitates extracting features from the image such that the root mean square localization error derived from the output of the visual localization and map building module is reduced or minimized.

[0029] Aspects of the present disclosure provide a computer-implemented method for positioning and mapping of an autonomous mobile machine, comprising: obtaining a processed image or a brightness-compensated image from a method for data image processing, and extracting features from the processed image or the brightness-compensated image by a visual positioning and mapping method for estimating the position of the autonomous mobile machine and / or generating a map of an environment corresponding to a scene image.

[0030] Aspects and embodiments disclosed herein relate to a deep learning model based on the Retinex theory. This is used to extract illumination and reflection information from low-light images and combine it with the image to enhance brightness. By exploiting the consistency of illumination and reflection according to the Retinex theory, the trained neural network adapts to different lighting conditions at construction sites, avoiding manual parameter settings.

[0031] These aspects and embodiments relate to the use of a combination of loss functions that are designed according to the expected use requirements of V-SLAM, particularly to facilitate the extraction of feature points and / or matching of feature points. The loss function trains an artificial neural network in an unsupervised manner, thereby reducing or eliminating the need for paired images for training.

[0032] In use, various combinations of loss functions enable more building information and feature points to be extracted from captured scene images or videos in V-SLAM. When integrated into the sensing system of a construction robot, this significantly improves the accuracy of the robot's map construction and mobility in low-light environments.

[0033] It should be understood that any changes, aspects and features described for devices and systems are also applicable to methods without departing from the teachings of the present disclosure, and vice versa. Any one or more of the above variations, aspects and features can also be combined without departing from the teachings of the present disclosure.

[0034] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, and is not intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings are used to better understand the technical solution and do not constitute a limitation of the present disclosure.

[0036] Figure 1 A flowchart of a method for generating an artificial neural network according to a first embodiment of the present disclosure is shown;

[0037] Figure 2 Shown based on Figure 1 A second embodiment of the method of an embodiment;

[0038] Figure 3 shows the architecture of the system, where an artificial neural network is deployed for training;

[0039] Figure 4 A flowchart of a digital image processing method for low-light images according to a third embodiment of the present disclosure is shown;

[0040] Figure 5 The architecture of the system is shown, wherein a trained artificial neural network is used to process low-light images;

[0041] Figure 6 is an illustration of the application of an embodiment of a sliding window of an inter-frame compensation module;

[0042] Figure 7 is a graph of the trajectory estimated according to V-SLAM relative to the actual trajectory for the input image sequence S1 (S1 is processed by the ANN trained using 3 loss functions and 5 loss functions);

[0043] Figure 8 is a graph of the trajectory estimated according to V-SLAM relative to the actual trajectory for the input image sequence S2 (S2 is processed by the ANN trained using 3 loss functions and 5 loss functions);

[0044] Fig. 9 is a graph of the trajectory estimated according to V-SLAM relative to the actual trajectory for the input image sequence S3 (S3 is processed by the ANN trained using 3 loss functions and 5 loss functions);

[0045] Fig.10 An example of a device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0046] Examples of systems and methods of embodiments of the present disclosure are described in detail below. Although several exemplary variations of systems and methods are described herein, other variations of systems and methods may include various aspects of the systems and methods described herein, combined in any suitable manner with a combination of all or some of the described aspects.

[0047] The details of various embodiments are described below in conjunction with the accompanying drawings for ease of understanding, and these descriptions are exemplary only. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. In addition, for the sake of clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0048] In the specification, it should be understood that the singular forms "a", "an" and "the" used in the following specification are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more related listed items. It should also be understood that the terms "include", "including ...", "comprising" and / or "comprising ..." when used herein refer to the presence of the features, wholes, steps, operations, elements, components and / or units, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, units and / or combinations thereof.

[0049] Retinex

[0050] Retinex theory simulates human color perception.

[0051] Based on the Retinex theory, the input image can be decomposed into source illumination and scene reflection, that is: I(x, y) = R(x, y)·L(x, y)

[0052] Where L(x,y) is the spatial distribution of the source illumination, and R(x,y) represents the distribution of the scene reflection. w,h represent the width and height of the image, and c represents the number of image channels. w,h represent the image (1080*720 image, w=1080h=720), and c=3 in the color image. Since images taken in low-light construction scenes often contain a lot of noise, the inventors modified the formula to include the noise component. The modified formula is as follows:

[0053] I(x,y)=R(x,y)·L(x,y)+N(x,y)

[0054] Where N(x,y) represents the noise component in the image.

[0055] Assuming that the illumination map L under normal light conditions is the identity matrix and N is a zero matrix, the reflection can be obtained for enhancement according to the Retinex formulation. However, the formulation of R(x,y) is a difficult problem; direct decomposition will lead to unnatural artifacts.

[0056] To solve this problem, the inventors use an iterative algorithm to progressively estimate the reflection and enhance the low-light input image as follows:

[0057]

[0058] Where n is the number of iterations. n is determined empirically.

[0059] Therefore, in the present disclosure, the reflectance maps R1 to Rn are first estimated from the input image and then iteratively combined with the low-light input image to enhance the low-light illumination component. The noise maps N1 to Nn are similarly estimated in the same process and can be subtracted from the input image after combination to reduce noise. During training, the neural network is trained to estimate the reflectance and noise components. During enhancement, this is used to guide the automatic enhancement of darker images.

[0060] First embodiment

[0061] In a first embodiment of general applicability, a method of generating an artificial neural network for digital image processing of low-light images is provided. Figure 1 The corresponding flow chart is described in . The artificial neural network is in an embodiment a deep learning model such as a convolutional neural network, which uses many layers.

[0062] In step 100, an artificial neural network 4 is established based on the Retinex theory. An input image 10 may be provided to the established artificial neural network. Establishing may refer to simply obtaining a properly set up Retinex-based neural network, and also refers to the act of setting up a neural network using the Retinex theory as disclosed herein.

[0063] In step 200, the reflection map 50 and the noise map 60 of the input image 10 (i.e., the image frame) formed in a dim light environment are estimated by the artificial neural network 4, and the artificial neural network is trained by unsupervised learning using a set of designed loss functions. For example, the artificial neural network is trained using the loss function so as to extract feature points from the input image after the reflection map and the noise map output by the artificial neural network are processed, wherein the input image after the processing is a brightness enhanced image.

[0064] It is necessary to design a loss function so as to facilitate the extraction of feature points from an input image after processing by a corresponding reflectance map and a noise map of an artificial neural network. At least four loss functions from a set of at least five loss functions are used to train the artificial neural network. The present disclosure allows a differentiable unsupervised learning method.

[0065] In low-light images, the extremely small differences between adjacent pixels pose a challenge to V-SLAM to effectively extract feature points by calculating grayscale changes. As discussed in the background, other factors can also interfere with the efficiency of V-SLAM in matching feature points.

[0066] The loss function can be regarded as a designed loss function, because the loss function is designed to enable the artificial neural network to learn how to effectively extract lighting features and noise features, that is, the parameters of the training neural network. The neural network trained using these loss functions can effectively enhance the lighting, noise and V-SLAM image features.

[0067] From the perspective of the results, the designed loss function can have the following effects in the processed image: increasing the brightness value and / or increasing the grayscale variation between adjacent pixels. These can improve the V-SLAM positioning results. For example, as disclosed herein, improving the image brightness can include increasing the MGV to substantially 100 or higher. For example, low lighting and / or low contrast below a grayscale value of 120 in an image may cause adverse interference to V-SLAM positioning. The designed loss function can increase the grayscale variation to substantially 120 or substantially higher.

[0068] The designed loss function can also achieve one or more other effects, as shown by the improvement in the error results produced by V-SLAM. For example, the effect can include restoring and highlighting feature information in the image, and / or reducing noise. Clear features can mean that the difference (contrast) of pixel values ​​is beneficial to V-SLAM positioning compared to low-light image enhancement.

[0069] In some embodiments, each loss function may produce an effect, each effect being different or synergistic.

[0070] By using differently designed loss functions in various combinations, one, some, or all of the effects can be achieved.

[0071] The purpose of training the artificial neural network 4 is to enable the trained neural network to produce a set of useful reflectance maps 50 and noise maps 60 for real images, which can then be used to produce a processed image 30 from an input image 10 .

[0072] In the processed image 30, one or more metrics of the actual input image have enhanced aspects compared to the actual input image 10. The metrics include characteristics of the input image, such as brightness and other features. Not all metrics can be observed by the human eye. However, the enhanced aspects can be verified by using the processed images in the V-SLAM technology, and the results show a reduction in positioning error due to improved feature extraction. Other techniques can be used to show improvements in other metrics, such as peak signal-to-noise ratio (PSNR), where higher PSNR scores indicate better image quality and more information.

[0073] To train an artificial neural network, multiple images in the form of image frames or image files are input. These come from a suitable dataset, which can include sequences of image frames of the same environment or the same scene. Figure 1 In the figure, for simplicity, only a single input image 10 is shown as data, and it should be understood that multiple images will be provided in sequence during training, which is sufficient to train the neural network.

[0074] In some training embodiments, the difference between the enhanced image obtained from the loss function calculation and the original input low-light image is used to update the parameters of the artificial neural network through back-propagation, enabling the ANN to better extract image lighting features and enhance brightness and contrast.

[0075] For example, during the training phase, the calculation of the loss function is performed on the output enhanced image (i.e., the processed training image). In other words, after all iterations are completed. Here, iteration refers to repeated image enhancement using multiple reflection maps and noise maps. The processed training image is obtained after enhancing its features. And the loss function is applied to the processed training image.

[0076] Second embodiment

[0077] Figure 2 A second embodiment is described, which is based on Figure 1 For simplicity, the establishment steps are omitted.

[0078] In the generating step 210, a plurality of reflection maps 50 and noise maps 60 are generated for the input image. Figure 2 In , multiple graphs are shown as two separate sets at the output. Each set is described as including two graphs, but this is only illustrative, and it should be understood that more graphs, such as 7 or more, can be used.

[0079] In the iterative processing step, the input image (ie, the original image) is iteratively processed according to the output reflectance map and the noise map of the input image 10 to obtain a processed image 30 .

[0080] In the optimization step, the artificial neural network is trained based on the input image and the processed training image 30 using, for example, four or five loss functions.

[0081] Figure 3 The architecture 1 of the system is shown, in which the artificial neural network 4 (ANN) of the first and / or second embodiment is deployed for training. The architecture 1 is described as comprising two connected stages: a decomposition stage 2 and a reconstruction stage 3.

[0082] An artificial neural network 4 is deployed in the decomposition stage 2. In use, it receives an input image at an input and outputs a corresponding reflectance map 50 and a noise map 60 for each input image 10, as shown in reference Figure 1 and Figure 2 A plurality of consecutive input images 10 are shown at the input, and Figure 3 , and label them as "Low-light frames".

[0083] The artificial neural network 4 generates a set of reflectance maps 50 and noise maps 60 in response to and based on the input image 10. These are shown in the reconstruction stage as terms N1, N2 .....Nn and R1, R2 ....Rn. The reflectance maps and noise maps are paired, that is, for each value of n, there is a corresponding R and N, for example, R1 and N1 are a pair, R5 and N5 are a pair, etc.

[0084] The reconstruction stage 3 is configured to receive as input the reflectance map and the noise map from the decomposition stage 2, and to receive as input the input image 10. In use, the reconstruction stage iteratively combines the input image with the reflectance map and the noise map generated for that particular input image. The reconstruction stage outputs a processed image, which is Figure 3 The "enhanced frames" are shown in FIG. 1 , each corresponding to a "low-light input frame". The "enhanced frames" are further used in navigation and / or mapping techniques (eg, V-SLAM).

[0085] The architecture is computer implementable on a processor, server, module, hardware, software, in a single device or distributed.

[0086] Next, the architecture and training scheme used in certain embodiments are disclosed in more detail.

[0087] ANN Architecture

[0088] The ANN 4 has a first layer 5 having an input configured to receive a training image as an input file.

[0089] In an embodiment, the artificial neural network 4 is a deep learning model, such as a convolutional neural network (CNN). As is common in the art, a deep learning model refers to a neural network with multiple node layers. Figure 2 The CNN shown in the embodiment of has 9 layers, each with skip connections to maintain the context between neighboring pixels.

[0090] The first four layers contain 32 convolution kernels of size 3*3 and stride 1, while the next four layers contain 64 convolution kernels of size 3*3 and stride 1. This can also be stated equivalently as the first four layers use 32 kernels, and each of the next four layers in turn combines one of the outputs of the first four layers with the output of the previous layer in the next four layers as input, each with 32 kernels. These connections are in Figure 3 The connection loop arrows from each of the first four layers to the next four layers are shown in FIG. 1 , i.e., the output of the first layer and the output of the fourth layer are the input of the fifth layer. The output of the second layer and the output of the fifth layer are the input of the sixth layer, and so on.

[0091] In an embodiment, the first eight layers contain 32 convolutional kernels of size 3×3 and stride 1, followed by an appropriate activation function, such as ReLU.

[0092] The ninth layer 6 consists of 8 convolutional kernels of size 3×3 and stride 1, optionally followed by a Tanh activation function.

[0093] A stride of 3 may also be used.

[0094] The ninth layer 6 generates the estimated reflectance map Ri and the noise map Ni, for example by using the SPLIT function.

[0095] Tables 1 and 2 further illustrate the architecture of an embodiment of a convolutional neural network, showing the input, dimensions, operators, and output of each layer, respectively. 256*256 is the size of the image of each layer; other image sizes can be used, but 256*256 is considered to be effective and suitable for robots with limited processing hardware. The image can be adjusted to these sizes before input. The last number in the size column refers to the image channel or kernel size. Therefore, Table 1 is particularly suitable for input grayscale images, while Table 2 is suitable for processing images with three color channels (e.g., RGB).

[0096] Table 1

[0097]

[0098] Table 2

[0099]

[0100] The role of the SPLIT function is to decompose the image features (matrices) extracted from the above 8 layers into 8 matrices. For example, 256*256*64 is decomposed into 8 matrices of 256*256*8. These 8 matrices are reflection maps R0 to R7, and the corresponding noise maps N0 to N7.

[0101] Loss Function

[0102] The input of the image file uses four or five loss functions from a set of five loss functions to train the ANN 4. In no particular order, the set of five loss functions includes: feature loss, exposure control loss, distortion color loss, lighting smoothness loss, and noise suppression loss.

[0103] For feature extraction, the above feature loss can use the VGG model. Alternatively, a model that can produce comparable or better results than the VGG model can be used.

[0104] With respect to the above-mentioned exposure control loss function, the light intensity ranges from 0 to 1, where 0 represents complete darkness and 1 represents the highest brightness level.

[0105] A good exposure level may be 0.65; the light intensity values ​​of the enhanced image may be compared to the good exposure level of 0.65.

[0106] Regarding the above-mentioned distortion color loss function, the color channel may refer to the RGB channel.

[0107] With respect to the above-mentioned illumination smoothness loss, color space can refer to RGB channels. Smoothness can refer to the absence of noticeable abnormal changes in the values ​​of adjacent pixels.

[0108] Feature loss

[0109] V-SLAM aims to extract high-quality feature points. Therefore, the VGG-16 model is used to extract features from the image, and then the feature loss information is evaluated, that is,

[0110]

[0111] VGG(I) is a feature extraction operator based on VGG-16, I0 is the input image, and In is the image after n iterations.

[0112] As used in another embodiment involving iterative processing, n is also 7. The feature loss exploits the concise structure of the VGG-16 network to efficiently compute the difference in feature information.

[0113] Exposure Control Loss

[0114] V-SLAM algorithms struggle to extract features from images that are too bright or too dark. The degradation of feature points is closely related to the pixel exposure brightness value. Typically, the pixel brightness values ​​of low-light images are concentrated in a lower brightness range. As a result, a large amount of feature information is compressed in a narrower range, making it challenging for V-SLAM algorithms to effectively utilize this information. At the same time, unevenly enhanced illumination also leads to failures in matching feature points.

[0115] The present disclosure uses an exposure control loss function to adjust pixel exposure brightness and uneven exposure. The image is divided into 16×16 regions, and the exposure level of each region is evaluated by calculating the average illumination value (expressed as Y). According to the exposure fusion theory, the good exposure level E is defined as the grayscale level in the RGB space. Therefore, the exposure control loss that measures the distance between the average intensity Y and the desired good exposure level E is calculated as follows:

[0116]

[0117] Where M represents the number of non-overlapping local regions (M=16), and E is empirically set to 0.65, which provides a good illumination level.

[0118] Distortion Color Loss

[0119] Color distortion may interfere with accurate feature extraction. When the value of a single color channel is too high or too low, it may cause color distortion and some information loss. Color constancy loss aligns the enhanced RGB channels at a similar level by connecting the three adjusted channels, i.e.

[0120]

[0121] Among them I p or I q Describes the average intensity value of the RGB channels of the enhanced image.

[0122] Lighting smoothness loss

[0123] The illumination component is updated using the illumination smoothness loss, which maintains the smoothness enhancement of the illumination intensity in the RGB space. During the iterative enhancement of the low-light image, the illumination smoothness loss helps to maintain the monotonicity of the surrounding environment at the pixel level and prevent the occurrence of ghosting artifacts. It is expressed as follows:

[0124]

[0125] Where n represents the number of iterations (in the embodiment, n=7), v is a first-order differential operator, and accordingly, v x and v y denote the horizontal and vertical gradient operations respectively, and E i It is the light component.

[0126] Noise suppression loss

[0127] In low-light construction sites, the hardware limitations of the camera (such as sensor size, camera sensitivity, and dynamic range) inevitably introduce noise into the image. In the process of low-light enhancement, image noise will be amplified and become more prominent. The inventors have found that high-frequency noise may mislead the V-SLAM algorithm during feature point extraction and matching, so it is useful to suppress the generation and amplification of noise during low-light enhancement. The noise suppression loss function limits the strength of the noise decomposition map Ni and is expressed as:

[0128]

[0129] Where Ni represents the noise map and n represents the number of iterations.

[0130] The total loss can be expressed as a linear weighted sum of the loss functions used. Therefore, in an embodiment using all five loss functions, the total loss can be expressed as:

[0131] L=wL fl +L ecl +L dcl +L isl +L nsl

[0132] where w is L fl weighted.

[0133] In the embodiment, w is empirically set to 200. This weight is responsible for ensuring that each individual loss function actually used operates with a similar magnitude in the total loss. If desired, a weight can be applied to each loss function.

[0134] train

[0135] In the training embodiment, an input image 10 from an appropriate training data set is first passed through the layers from the first layer 5 to the ninth layer 6, and then decomposed into R0 to R7 and N0 to N7 by the SPLIT function. These R and N are decomposed by the model at the first input of the image. The low-light image is then enhanced by processing it with these illumination feature maps (image matrix multiplied by the R matrix), and then the noise matrix is ​​subtracted. The process iterates from R1-R7, N1-N7, for a total of 7 times.

[0136] In this embodiment, n=7. The value of n=7 is selected based on experience, which can achieve a stable improvement in PSNR; a lower value can be used for lower PSNR values, but faster; a higher value can be used for substantially the same PSNR value, but longer. R0 and N0 are not used in this embodiment.

[0137] An example of pseudo code implementing the iteration to obtain the processed image is shown below:

[0138] pseudocode:

[0139] Input: Low-light input image (i.e. original / scene image)

[0140] Lighting noise decomposition stage:

[0141] Low-light image --> convolution layer --> reflection R and noise map N (matrix)

[0142] Image enhancement (i.e. reconstruction stage):

[0143] x=(low-light input image)*(r1+1)-n1

[0144] "x is the iteration image, r is the reflectance map, and n is the noise map; the image after each enhancement is used as the input for the next one."

[0145] x=(x)*(r2+1)-n2

[0146] x=(x)*(r3+1)-n3

[0147] x=(x)*(r4+1)-n4

[0148] x=(x)*(r5+1)-n5

[0149] x=(x)*(r6+1)-n6

[0150] x=(x)*(r7+1)-n7

[0151] Enhanced image = (x)*(r8+1)-n8

[0152] The same iterative process is used during the testing phase (when used), and after the neural network is trained, the same iterative process is also used for the actual input images.

[0153] Optimizer

[0154] In the training embodiment, the ADAM optimizer is used for optimization. The parameters are default, with a fixed learning rate of 0.001 and 200 iterations. Other optimizers can be used.

[0155] Training Dataset

[0156] Current deep learning methods require paired datasets and train the models through supervised learning methods.

[0157] In contrast, unsupervised learning is a machine learning method that learns from data that is not pre-labeled. No paired dataset is required.

[0158] Here, during the learning process, the algorithm iterates its own parameters through the loss function on a dataset of images with various lighting conditions to meet the requirements of the output processed image.

[0159] Any suitable dataset containing various lighting conditions can be used to train the ANN. The dataset should contain images from low light, normal light, and overexposed conditions. Those skilled in the art will know the meaning of these terms. Figure 4 The meanings provided in the embodiments are also applicable.

[0160] The images of the training image dataset are either grayscale images or advantageously in color, such as RGB.

[0161] In an embodiment, the training image dataset used is the entire Single Image Contrast Enhancement (SICE) dataset of images. This is a publicly available dataset of unpaired images in a range of lighting conditions.

[0162] In another embodiment, a subset of images is extracted from the SICE dataset of images and used as a training dataset, for example, the first 2000 images from SICE are used as the training dataset.

[0163] In yet another embodiment, a training dataset is created that includes a smaller subset of images from the SICE dataset and multiple images obtained from different construction sites with different lighting levels. For example, low-light indoor scenes, normal exposure scenes, alternating low-light and normal light scenes, and overexposed scenes. The training dataset includes the first 1000 images from SICE and 500 images from the construction site.

[0164] The inventors found that the best results were achieved with datasets containing at least 1,500 images, all with brightness between 40-200 MGV.

[0165] In V-SLAM, the localization errors of the trained ANN obtained using each of these data sets are equivalent. Using smaller data sets has the advantage of faster training time.

[0166] In another embodiment directed to using the artificial neural network thus trained, the same architecture is used to enhance scene files.

[0167] Processing low-light images

[0168] Figure 4 A flow chart of a third embodiment of a digital image processing method for low-light images is shown.

[0169] Here, the input image is a scene image 20 showing a scene from an environment, and it is desired to perform feature extraction on the scene image 20. As an example, the environment is an environment in which a construction robot is deployed, and the input image may be one of a sequence of image frames captured in real time by a camera of the construction robot.

[0170] The scene image is a grayscale image or a color image (eg, an RGB color space image).

[0171] In an embodiment, the low-light image refers to an image with a mean gray value less than 100. The mean gray value (MGV) indicates the brightness of the image, which is in the range of 0 to 255 (0 represents black, and 255 represents white).

[0172] Very low-light images (eg, MGV below 2) have very little information that can be recovered through processing.

[0173] An overexposed image is one with excessive brightness that results in loss of detail, a completely white canvas, and indistinguishable features. Typically, this occurs when the MGV exceeds 200. If the scene image consists primarily of overexposed areas that add up to more than 70% or 90% of the entire image, then very little information can be recovered through processing.

[0174] The sequence from which the input image frames are obtained may also have images under normal light conditions. In other words, the sequence may have low-light images and normal light images. The digital image processing method is able to maintain accurate positioning in an environment with alternating low-light conditions and normal light conditions.

[0175] In the input step, the scene image is input to the trained ANN obtained from one of the embodiments. With training, the weights of the ANN will be optimized, but the architecture is structurally the same. Therefore, the architecture and related information disclosed in the embodiments related to training are also applicable here.

[0176] In the generation step 201, the trained ANN estimates the reflection map 50 and the noise map 60 for each scene image 20 input, and outputs the reflection map 50 and the noise map 60. Here, four of each map are shown as an example. In an embodiment, each may have 7 or 8.

[0177] In an iterative processing step 220, the outputted reflectance map and noise map are used to process the scene image 20 in the same manner as disclosed in the training related embodiments, using the reflectance map and noise map generated for the particular scene image. The output result is a processed image 30.

[0178] In the processed image 30, the brightness level (MGV) has been increased relative to the corresponding scene image 20. Regarding brightness, the MGV of the scene image may be 40-80, but after processing it is 100-110 or higher. In an embodiment, the MGV of all processed images may exceed 110 MGV. Information should also be recovered.

[0179] Figure 4 An optional step 300 is shown, comprising:

[0180] adjusting the brightness of the processed image 30 based on a comparison between the processed image output (e.g., at time point t) and other processed image outputs within its range; and

[0181] A brightness compensated image 40 is output.

[0182] The range can be multiple frames or time.

[0183] The V-SLAM algorithm generates a pose graph using feature point matching, and the feature operator is sensitive to brightness changes. For example, significant changes in brightness within the same scene may lead to difficulties in feature matching.

[0184] To mitigate brightness differences between image frames within a range, for example for a certain number of consecutive frames, after processing scene images in a low-light architectural scene, the brightness adjustment step 300 stabilizes the transition between processed image frames.

[0185] The pseudo code of Algorithm 1 shown below includes the following steps, which may be involved in implementing the sampling process of adjusting the brightness step.

[0186] Algorithm 1 Inter-frame compensation system

[0187] Input: enhanced video frame D

[0188] Output: Adjusted video frame D*

[0189] 1: Step 1: Sliding window to extract frames

[0190] 2: for i in the range ((0,D),m) do

[0191] 3: Sliding window S~D[i:i+m]

[0192] 4: for j in range (S) do

[0193] 5: Step 2: Calculate the brightness in the sliding window: frame brightness B f , average sliding window brightness B a

[0194] 6: if B f -B a >20%Ba then

[0195] 7: Step 3: Linear Adjustment

[0196] 8: B f *{(B f -B a ) / B a +1)}

[0197] 9: end if

[0198] 10: end for

[0199] 11: Step 4: Reconstruct frame D*

[0200] 12: end for

[0201] In this example, the average brightness of the image is compared to the average brightness of all frames in the sliding window. If the brightness difference of a frame exceeds 20% of the average brightness of the sliding window, a linear enhancement or reduction is applied to the frame. The linear change is proportional to the difference. Other values ​​can be used, such as 18%, 22%, or more or fewer frames can be used.

[0202] In this way, the brightness of the processed image frame 30 will be adjusted towards the moving average.

[0203] The adjust brightness step 300 adjusts the brightness of some or all output processed frames. This is useful, for example, when the scene remains similar between consecutive frames but significant changes in brightness occur due to the presence of light sources. It solves the problem of local overexposure that occurs after low-light enhancement. It can reduce the impact of this problem on V-SLAM, but it cannot recover information from overexposed images by itself.

[0204] In the V-SLAM processing step 400, the processed image 30 or the brightness compensated image 40 is taken as input and image features are extracted. The V-SLAM processing step is optional. Figure 4 Potential applications of digital processing methods are shown.

[0205] Figure 5 The architecture 1 of an embodiment of the system is shown, in which the ANN 4 can be deployed after being trained. The structure of the decomposition phase 2 and the reconstruction phase 3 are similar to Figure 3 In addition, in an embodiment, the inter-frame compensation module 7 can receive the processed image 30. Figure 3 are marked as enhanced frames.

[0206] The inter-frame compensation module 7 is configured to perform the brightness adjustment step 300, such as Figure 4It is optional, as shown by the dotted line.

[0207] The processed image or the brightness compensated image can be input to the V-SLAM module 8 (shown here as a dotted line in an optional positioning stage). In an embodiment, the ORB-SLAM3 algorithm can implement the V-SLAM function in the V-SLAM module. Compared with other available V-SLAM algorithms (such as KAZE and AKAZE), the use of ORB-SLAM3 provides improved feature extraction of processed images of low-light scenes.

[0208] As known to those skilled in the art, V-SLAM estimates a pose graph and can create a map of the environment and trajectory.

[0209] Figure 6 is an illustration of an implementation of Algorithm 1 that may perform the step of adjusting brightness. Four consecutive processed image frames 30 are shown within a sliding window. In use, sliding changes the brightness (luminosity) of each or some of the image frames compared to the average brightness of the frames within the sliding window.

[0210] V-SLAM Results

[0211] 3 loss functions versus 5 loss functions

[0212] Figures 7 to 9 The robot trajectory versus the actual trajectory is shown. Figure 7 The results for image sequence S1 are shown; Figure 8 for the image sequence S2; and Fig. 9 For image sequence S3.

[0213] Sequences S1-S3 are characterized by the data in Table 3. The image sequences were collected from an indoor construction site in China; the scenes captured are rough rooms, i.e., basic walls built but without lighting. The sequences include low-light images, and scenes include backlit areas, dark staircases, and long corridors. The images were captured under natural lighting conditions without additional lighting using the automatic exposure setting on the Intel RealSense 435 camera.

[0214] Table 3

[0215]

[0216] At the same time, the actual trajectory is captured using markers. In the implementation, the AprilTags tagging system is used. The X coordinate corresponds to the ground direction in meters; the Z coordinate corresponds to the vertical ground direction, measured from the marker in meters.

[0217] Figures 7 to 9 3 plots each showing a trajectory:

[0218] i. Dashed line: Si-real

[0219] This depicts the true trajectory, ie it shows the real situation.

[0220] ii. Dashed line: Si-3LF

[0221] This depicts the trajectory computed using the methods generally disclosed herein. However, this particular ANN is trained with only three loss functions (3LFs) instead of the five loss functions disclosed herein. The three loss functions are: feature loss, exposure control loss, and lighting smoothness loss.

[0222] iii. Solid line: Si-5SF

[0223] This depicts the trajectories computed using the methods disclosed herein for an ANN trained using all five loss functions as disclosed herein.

[0224] Each figure shows that the trajectory of the ANN trained using the five loss functions is closer to the actual trajectory. In addition, the inventors have found that the ANN trained using the above three loss functions cannot obtain satisfactory enhancement effects and may even lead to direct failure in positioning.

[0225] Table 4 shows the localization results from an ANN trained with 3 loss functions (3LF), from an ANN trained with 5 loss functions (5LF), and from ORB SLAM3 on the same low-light images (but without any digital image processing).

[0226] Table 4

[0227]

[0228] The table shows the root mean square error (RMSE), maximum error, and average error as the metrics used to calculate the gap between the result and the true value of each V-SLAM. The RMSE is obtained as follows:

[0229]

[0230] where m represents the number of keyframes, h is the trajectory alignment method (it computes the distance between the coordinates of the estimated pose and the true coordinates of the same timestamp, and α and β are the generated trajectories of the keyframes and their associated true values.

[0231] Compared with the ANN trained with three loss functions, the (5LF) ANN trained with five loss functions leads to significant improvements in the metrics.

[0232] Compared to the unprocessed images, the 5LF ANN also has the following improvements: for S1, the average brightness after 5LF ANN processing is 112.3MGV; the RMSE percentage error is reduced by 65.44% compared to the unprocessed image; for S2, the corresponding numbers are 160.5MGV and 90.56%; for S3, the numbers are 160.1MGV and 84.52%. Therefore, the RMSE error is reduced by at least 65% for all sequences. Therefore, minimizing the RMSE means that the RMSE will be significantly reduced compared to the existing standard technology for V-SLAM.

[0233] 4 loss functions vs. 5 loss functions

[0234] The individual effect of each loss function is evaluated by excluding one of the five loss functions while keeping all other factors constant. The results of the localization experiments on the low-light construction dataset using different loss functions are shown in Table 5 as RMSE.

[0235] Table 5

[0236]

[0237] Therefore, in an embodiment, four loss function combinations from the set of five loss functions are as follows from most preferred to least preferred:

[0238] Exclude feature losses from the combination;

[0239] Exclude illumination gradient loss from the combination;

[0240] Excluding noise suppression losses from the combination;

[0241] Excludes distortion color loss from the mix;

[0242] Eliminate exposure control loss from the mix.

[0243] robot

[0244] The methods and apparatus disclosed herein are particularly useful in low-light environments, which are typically fully or partially indoors, such as tunnels, construction sites, warehouses, but also include outdoor environments in low-light conditions (eg, at night).

[0245] There are robots specially adapted and designed for this environment. They often have limited processor power and memory. Therefore, in an embodiment, the size of the trained ANN is about 0.4M and can be run in real time on a mobile computer platform such as NVIDIA JetsonOrin NX.

[0246] Various types of construction robots are available, and more are being developed. A construction robot can be a machine that operates on a construction site. Its function is to assist or replace tasks performed by humans on a construction site.

[0247] A construction site includes a location where construction or repair work is ongoing.

[0248] Mobile machines include construction robots, but are not limited to applications on construction sites. Thus, examples of mobile machines include: indoor robots for managing buildings, indoor robots that provide surveillance or inspection, construction robots, etc.

[0249] Construction robots and mobile machines can be defined by their ability to navigate their environment through visual simultaneous localization and mapping algorithms that generate a pose graph using input images from a camera typically mounted on the robot or machine.

[0250] The motion may be provided by a mobile mechanism, such as from drive wheels, tracks, legs, tracks, but not limited to these. In some cases, the robot may be stationary but have a component that can move relative to the robot in the environment to perform tasks, such as lifting objects, painting tools using V-SLAM.

[0251] The mobile machine may be teleoperated or autonomous.In the following description, deployment of the method and apparatus disclosed herein is advantageous because it lacks operator feedback for correcting trajectory errors.

[0252] Aspects and embodiments herein may be used with all or any of the above-described robots or machines.

[0253] Fig.10 A schematic diagram of an embodiment of the device is shown, wherein the CPU includes: an image decomposition module 22, an image reconstruction module 33 and an optional V-SLAM module 88, which communicates with the computer memory via a bus. The CPU has an input for inputting an image and an output for outputting a calculation result and / or a processed image from the V-SLAM module. The functions of the modules correspond to Figure 5 The similarly named stages shown in the embodiment of and using the Retinex-based ANN and loss function disclosed herein. The image decomposition module 22, the image reconstruction module 33 and the V-SLAM module 88 (optionally) can also use GPU accelerated calculations.

[0254] Another embodiment provides a low-light image enhancement method for positioning and detecting a construction robot, comprising the following steps:

[0255] a. Establish a deep learning model based on Retinex theory to estimate the reflection map and noise map from the input image;

[0256] b. Using five designed loss functions, deep neural network models were trained through unsupervised learning;

[0257] c. Obtaining the original (input) image under low light conditions through the sensor;

[0258] d. Iteratively, combine the estimated map with the low-light (input) image to enhance the low-light image;

[0259] e. Output enhanced images with improved brightness and information, enabling construction robots to further perceive and operate.

[0260] This embodiment uses the artificial neural network of the present disclosure as a deep learning model. This embodiment uses five loss functions of the present disclosure.

[0261] Certain aspects of the present disclosure include process steps and instructions described herein in algorithmic form. It should be noted that the process steps and instructions of the present disclosure can be implemented in software, firmware, or hardware, and when implemented in software, can be downloaded to reside on different platforms used by various operating systems and can be operated from different platforms used by various operating systems. Unless otherwise stated, it will be apparent from the following discussion that it should be understood that throughout the specification, discussions using terms such as "processing," "calculating," "computing," "determining," "displaying," "generating," etc. refer to the actions and processes of a computer system or similar electronic computing device that manipulates and converts data represented as physical (electronic) quantities within a computer system memory or register or other such information storage device, information transmission device, or information display device.

[0262] In some examples, the present disclosure also relates to a device for performing the operations herein. The device may be specially constructed for the desired purpose, or it may include a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory computer-readable storage medium, such as but not limited to any type of disk (including a floppy disk, a USB flash drive, an external hard drive, an optical disk, a CD-ROM, a magneto-optical disk, a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic card or an optical card), an application-specific integrated circuit (ASIC), or any type of medium suitable for storing electronic instructions, and each is connected to a computer system bus. In addition, the computer mentioned in this specification may include a single processor, or may be an architecture designed with multiple processors for increasing computing power. Suitable processors include a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), and an ASIC. The methods, devices, and systems described herein are not inherently related to any particular computer or other device. Various general-purpose systems may also be used with programs according to the teachings of this article, or it may prove convenient to construct a more specialized device to perform the required method steps. The structures required for various of these systems will be apparent from the following description. In addition, the present disclosure is described without reference to any particular programming language. It should be understood that various programming languages ​​can be used to implement the teachings of the present disclosure described herein.

Claims

1. A method for generating an artificial neural network, comprising: Establish artificial neural network based on Retinex theory; estimating a reflection map and a noise map of an input image formed in a low-light environment by using the artificial neural network; as well as The artificial neural network is trained using a loss function so as to extract feature points from an input image processed by the reflectance map and the noise map, wherein the processed input image is a brightness-enhanced image.

2. The method according to claim 1, wherein: The loss function includes four or five loss functions.

3. The method according to claim 1 or 2, wherein: The training includes: The new input image forming processing step includes: combining the input image with the reflection map and subtracting the noise map to form the new input image; Taking the new input image as the input image, iteratively executing the new input image forming processing step multiple times to generate a processed training image; and The artificial neural network is optimized based on the input image and the processed training image using the loss function.

4. The method according to claim 3, wherein: The loss function includes: Feature loss, representing the difference between features extracted in the processed training image and features extracted in the input image using the artificial neural network; Exposure control loss, used to adjust pixel exposure brightness and uneven exposure by the difference between the good exposure level and the average intensity; A distortion color loss for aligning color channels of the processed training image; An illumination smoothness loss for maintaining the smoothness of illumination intensity in the color space of the processed training image; and A noise suppression loss is used to suppress the intensity of the noise map.

5. The method according to claim 4, wherein: The feature loss is determined based on a feature extraction operator of a VGG-16 image extraction model, the input image, and the processed training image.

6. The method according to claim 4, wherein: The exposure control loss is determined according to the regions into which the image is divided, the exposure level of each region, and the number of non-overlapping local regions in the divided regions.

7. The method according to claim 4, wherein: The distortion color loss is determined according to the average intensity values ​​of the RGB channels of the processed training image.

8. The method according to claim 4, wherein: The illumination smoothness loss is determined based on a horizontal gradient operator and a vertical gradient operator associated with a color space of the processed training image.

9. The method according to claim 4, wherein: The noise suppression loss is determined based on a plurality of noise maps of the input image and a number of iterations in the iterative process.

10. The method according to claim 1, wherein: The loss functions are applied in the training via a total loss function, which represents the loss functions used in the training, wherein each loss function in the total loss function has a weight such that it has a similar contribution magnitude to the total loss function.

11. The method according to claim 1, wherein: The input image formed in the low-light environment includes images representing low-light environment, normal light and overexposure conditions.

12. The method according to claim 11, wherein: The input image formed in a low-light environment includes an image with a brightness ranging from 40 to 200 MGV and 1500 images.

13. The method according to claim 11 or 12, wherein: The input image formed in a low-light environment includes: at least a subset of a single image contrast enhanced (SICE) image data set.

14. The method according to claim 1 or 2, wherein: The artificial neural network includes multiple convolutional layers, each with skip connections to preserve context between adjacent pixels.

15. A computer program product comprising a computer program code, wherein when the computer program code is run on a computer, the method for generating an artificial neural network according to any one of claims 1 to 14 is executed.

16. A computer-implemented method for digital image processing of a low-light image using an artificial neural network generated according to any one of claims 1 to 14, the method comprising: Inputting a scene image as the input image into the artificial neural network; In response to an input of the scene image, iteratively processing the scene image based on a plurality of reflection maps and a plurality of noise maps generated by the artificial neural network; as well as Output the processed image.

17. The computer-implemented method of claim 16, wherein: The iterative process includes: Generate a set of paired reflection maps and noise maps, generate at least a plurality of pairs of reflection maps and noise maps for the scene image, and The processed image is obtained by combining the scene image with the reflectance map and subtracting the noise map.

18. The computer-implemented method of claim 17, wherein: The iterative processing includes: executing at least a plurality of processing operations in sequence, The first processing operation of the plurality of processing operations takes the scene image as input, Each subsequent processing operation after the first processing operation takes as input the image output by the previous processing operation, Each processing operation consists of combining the input with its corresponding reflectance map and subtracting its corresponding noise map. The final processing operation outputs the processed image, such that the scene image is iteratively enhanced by being processed once by the set of pairs of reflectance maps and noise maps.

19. The method according to any one of claims 16 to 18, wherein: The processed image has an increased brightness metric compared to the input image.

20. The method according to any one of claims 16 to 18, wherein: The method further comprises: adjusting the brightness of the processed image according to a comparison between the processed image output at time point t and other processed images output within the range thereof; and Output brightness compensated image.

21. A computer program product, comprising computer program codes, which, when executed on a computer, will perform the method for digital image processing of low-light images according to any one of claims 16 to 20.

22. A device for digital image processing of a self-guided robot, comprising: an image decomposition module, comprising a trained artificial neural network, the trained artificial neural network estimating a plurality of reflection maps and a plurality of noise maps of an input image obtainable from a sensor of the self-guided robot based on Retinex theory, wherein the artificial neural network is trained by unsupervised learning using a loss function; and An image reconstruction module is configured to iteratively process the input image by sequentially combining the reflectance map and subtracting the noise map.

23. The apparatus of claim 22, further comprising: The inter-frame compensation module is configured as follows: adjusting the brightness of the processed image based on a comparison between the output processed image and other processed images output within the range thereof; and Output brightness compensated image.

24. The device according to claim 22 or 23, further comprising a visual positioning and map building module, wherein the visual positioning and map building module comprises: an input module for receiving a processed image or a brightness compensated image as an input image, and A feature extraction module is configured to extract features from the image at the input module to estimate the position of the self-guided robot and / or generate a map of the environment.

25. The apparatus of claim 22 or 23, wherein: The loss function includes: feature loss, representing the difference between features extracted in the processed image using the neural network and features extracted in the input image; Exposure control loss, used to adjust pixel exposure brightness and uneven exposure by the difference between the good exposure level and the average intensity; a distortion color loss for aligning color channels of the processed image; A lighting smoothness loss for maintaining the smoothness of lighting intensity in the color space of the processed image; and A noise suppression loss is used to suppress the intensity of the noise map.

26. A computer-implemented method for positioning and mapping a mobile machine, comprising: Obtaining a processed image or a brightness compensated image from the method according to any one of claims 16 to 20, and Features are extracted from the processed image or the brightness compensated image by visual localization and mapping methods to estimate the position of the mobile machine and / or generate a map of the environment corresponding to the scene image.

27. An electronic device comprising: A memory storing a computer executable program; A processor executes the program to implement: The method according to any one of claims 1 to 14, or A method according to any one of claims 16 to 20; or The method according to claim 26.

Images