Confidence-dependent image brightening

In the image processing of the camera device, the output image is reconstructed using the confidence of various brightening methods, combined with traditional statistical methods and machine learning methods, the problem of poor brightening effects of dark scenes and high dynamic scenes is solved, and a high-quality and robust image brightening effect is achieved.

CN119968647APending Publication Date: 2025-05-09CONTINENTAL AUTONOMOUS MOBILITY GERMANY GMBH
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202380070542.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-10-14
Filing Date
2023-09-18
Publication Date
2025-05-09

AI Technical Summary

Technical Problem

The prior art uses poor image brightening effects when processing image of a camera device in dim scenes or high dynamic scenes, and traditional methods are difficult to implement in embedded systems. The performance of the machine learning method depends on the quality of the training data and the modeling ability of the model.

Method used

By providing the original data image and a variety of brightening images, the confidence of each brightening method is determined, and the output image is reconstructed based on the confidence, and combined with traditional statistical methods and machine learning-based methods, the robustness and high quality of image brightening are achieved.

Benefits of technology

Improves the robustness and quality of image brightening, reduces the occurrence of artifacts, adapts to different lighting conditions, and reduces the calculation time.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119968647A_ABST
    Figure CN119968647A_ABST
Patent Text Reader

Abstract

The invention relates to: a method, in particular a computer-implemented method, for brightening an image of a camera device (1); the application of the method in the field of computer vision; a data processing system comprising means for implementing the method according to the invention; a computer program; and a computer readable storage medium. The method comprises the following steps: providing at least one raw data image (R) of the camera device (1); providing at least one first highlighted image (I1) of the raw data image (R), the first highlighted image (I1) being highlighted by means of a first highlighting method; determining a first confidence (C1) for the brightening of the raw data image (R) of the camera device (1) by means of a first brightening method; and-reconstructing the output image (O) at least partially from the raw data image (R) and / or the first brightened image (I1) as a function of the first confidence (C1).
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a method for brightening an image of a camera device, and in particular to a computer-implemented method for brightening an image of a camera device. Background Art

[0002] Today's vehicles are generally equipped with cameras and driver assistance systems (English: Advanced driver assistance systems (ADAS)) with different functions, for example, for assisted and automated driving, or for providing the driver with displays and support when driving or parking, for example by displaying images of the surroundings. Thus, different methods known in connection with driver assistance systems include: methods for identifying objects or obstacles in the lane, methods for identifying lane boundaries and / or keeping the vehicle in a lane, methods for identifying rain on the windshield, or methods for supporting or implementing a parking process. These functions and others are usually based at least in part on images captured by means of a camera device fixed to the vehicle. The camera device can be, for example, a single camera device or a plurality of cameras for forward or rearward viewing, or a surround view camera system. In addition, these systems are also the basis for automated driving functions of automated driving systems (ADS).

[0003] In this context, the captured raw data images often have to be processed before further processing in order to optimize the visualization of the information contained in the image. The evaluation and image processing of the raw data images of the camera device not only plays a decisive role in driver assistance systems, but is also crucial in all other areas where an improvement in image quality over the raw data images is required. A core issue in image preprocessing is the handling of different illumination conditions. Images captured under daylight conditions or in good lighting conditions typically present no problems for further processing and can be implemented well both visually and mechanically. In contrast, dim scenes or highly dynamic scenes, for example scenes with very bright and very dark areas at the same time (such as in night photography), often present challenges for image brightening methods.

[0004] In principle, both conventional image processing methods and different approaches taking into account methods from the field of machine learning can be used for image brightening of camera images. In conventional filter-based image preprocessing, such as histogram-stretching methods, typically the entire bit area and dynamic range of the raw data image is fully utilized for different brightness adjustment matches. The corresponding methods use local or global statistics in the image and calculate the adjustment match for image brightening from this. As an alternative, another known method is to consider only a single brightness range of pixels and brighten them. In principle, this approach has the disadvantage that specific information in the raw data image is not taken into account, which is valuable for image brightening. In addition, some conventional filter-based methods are difficult to implement in embedded systems.

[0005] Among the methods for image brightening using methods from the field of machine learning, methods using neural networks, in particular convolutional neural networks (CNN) are particularly well known. Typically, the brightened image is reconstructed based on training data with different exposure times. At the same time, the image scene is modeled in the neural network and taken into account in each corresponding brightening process. This is achieved by learning textures and structures, which can provide the neural network with hints about specific types of brightening. As a result, local and global features in the original data image can be recognized and processed. In addition, the neural network can be trained with regard to image noise, which can at least partially suppress the image noise.

[0006] The performance of a neural network depends in principle on, among other things, the quality of the training data, the model used and the input data. The output quality of the neural network can also be affected by noise in the input data, insufficient modeling capabilities of the network and processing of traffic images that are not adequately described by the training data.

[0007] For example, DE 10 2019 220 168 A1 discloses a machine learning method for brightness conversion from input image data of a camera device to output image data by means of an artificial neural network, by means of which the input image data can be brightness converted.

[0008] Regardless of the method used, there are always cases where the image is not brightened enough or the quality of the brightening is poor. Poorly brightened images may be of very limited use in further processing. For example, in the field of driver assistance systems, it is extremely important to evaluate the quality of the image brightening, especially when the brightened images are further processed to implement safety-related functions. Problems in this area include, in particular, the handling of various image artifacts, such as color artifacts at edges. Image artifacts can also be caused by the image enhancement process, for example, well-lit areas in the raw data image are over-brightened. General methods typically do not adequately take into account the differences between well-lit and poorly lit areas in the raw data image. Summary of the invention

[0009] It is therefore an object of the present invention to provide a reliable method for evaluating image brightening of raw data images.

[0010] This object is achieved by a method according to claim 1 , by the use according to the invention of a brightened image according to the invention according to claim 12 , by a data processing system according to claim 13 , by a computer program according to claim 14 and by a computer-readable storage medium according to claim 15 .

[0011] In terms of the method, the object underlying the invention is achieved by a method, in particular a computer-implemented method, for brightening images of a camera device. The method comprises the following method steps:

[0012] - providing at least one raw data image of a camera device,

[0013] - providing at least one first brightened image of the raw data image, wherein the first brightened image is brightened by means of a first brightening method,

[0014] - determining a first confidence measure for brightening the camera raw data image by means of a first brightening method, and

[0015] - reconstructing an output image at least partially from the raw data image and / or the first brightened image according to the first confidence measure.

[0016] According to the invention, the reconstruction of the brightened output image depends on the confidence level. Here, the output image can be synthesized from the contribution of one or more different brightened images and the original data image. If the brightened output image needs to be further processed, and the further processing depends on the image information of the output image, then the consideration of the confidence level is particularly important.

[0017] Advantageously, the quality of the brightening methods can be determined during the implementation of the method, and the reconstruction of the brightened output image itself can be performed based on the performance of the different brightening methods. Furthermore, by first determining the brightened image and the confidence level, and at the same time only generating a further brightened image if the confidence level exceeds or falls below a predeterminable limit, the calculation time can be reduced.

[0018] Image regions can be synthesized from different brightened images and / or raw data images, i.e. different illuminations are appropriately used in different image regions. In this way, different image regions with different illuminations can be distinguished. Reconstructing the output image based on the confidence level also allows the image brightness to be adapted, in particular instantly adapted, to changing lighting conditions, such as, for example, switching on the exterior lights in a garage.

[0019] In principle, the confidence measure can be determined for the raw data image as a whole, or a plurality of confidence measure values ​​can be determined for individual image regions, or even a confidence measure can be determined for each individual pixel of the raw data image. In this case, the determination of the confidence measure can be suitably optimized with respect to the required computational effort. Furthermore, if the determined confidence measures have similar or almost identical values ​​in specific image regions or pixel regions, an average value of the confidence measure can be determined for these regions and used. Individual specific confidence values ​​can also be disregarded if they deviate significantly from the average value of the confidence measure in a specific region or in the entire brightened image.

[0020] In this respect, the confidence level can be understood as a measure of the quality or quality of the brightening. It can be an estimated value or a calculated value. For example, the confidence level can be given in the form of a probability. For example, some rules can be specified for reconstructing the brightened output image, such as which specific confidence values ​​or predefined confidence value intervals are respectively assigned to a specific brightened image or raw data image and / or a specific image area of ​​the brightened image or raw data image, and which images or image areas are used to reconstruct the output image. Interpolation between brightened images and / or raw data images can also be used.

[0021] Furthermore, in one embodiment, the method according to the present invention further comprises the following method steps:

[0022] - providing at least one second brightened image of the raw data image, wherein the second brightened image is brightened by means of a second brightening method different from the first brightening method, and

[0023] - reconstructing the output image at least partially from the original data image, the first brightened image and / or the second brightened image according to the first confidence level.

[0024] Therefore, the raw data image is brightened by means of two different brightening methods, and the confidence of at least one of the two brightened images is determined for reconstruction of the output image. Both brightened images and the raw data image can be used for reconstruction, or only one or two of the at least three images can be used for reconstruction. The selection also depends on the determined confidence.

[0025] In this context, it is advantageous to determine a second confidence measure for brightening the raw data image of the camera device by means of the second brightening method, wherein the output image is reconstructed at least partially from the raw data image, the first brightened image and / or the second brightened image according to the first confidence measure and / or the second confidence measure. In this case, the output image can also be reconstructed using two confidence measures, in particular confidence measures determined independently of one another. In this case, the two confidence measures can be considered separately from one another. However, it is also conceivable to determine an overall confidence measure from the two confidence measures and use it to reconstruct the output image.

[0026] Therefore, according to the present invention, one or more different brightening methods can be used to brighten the raw data image. The final output image is reconstructed according to the confidence level, thereby obtaining a high-quality optimal brightening result in which the artifacts are minimized.

[0027] The method according to the invention can also be designed iteratively. In this case, individual method steps are repeated, for example until a preset confidence limit is exceeded or undershot. Different methods or the same method with different parameters can be used for the individual image brightening runs, confidence determinations and reconstructions. In this way, for example, a minimum standard for image brightening can be predefined and always achieved.

[0028] In the method one design, at least one of the brightening methods is at least partially a statistical brightening method, preferably a filter-based method, in particular a histogram stretching method, a tone mapping method, a white balance method, or a combination of multiple of the above methods and / or a combination with other methods.

[0029] In another design scheme, at least one of the brightening methods is a method based on a machine learning method, preferably a method using one or more neural networks, especially convolutional neural networks, adversarial neural networks, recurrent neural networks, Transformer networks, neural graph networks or neural circuits, or a deep learning method, especially a deep learning method using a neural network with multiple hidden layers.

[0030] In this context, it is advantageous if the neural network is a trained neural network which is designed to determine, starting from an input in the form of at least one raw data image of the camera device, a brightened image which corresponds in particular to an image with a longer exposure time corresponding to the raw data image.

[0031] In a preferred embodiment of the method, the output image is reconstructed at least in partial regions by interpolation of the original data image, the first brightened image and / or the second brightened image. In the interpolation, the first confidence level and / or the second confidence level determined if necessary can be used as a weighted measure.

[0032] Therefore, according to another design solution, the determined first confidence level and / or second confidence level can also be used as a weighted measure or weighted coefficient. In this case, the output image is preferably reconstructed based on the superposition of the original data image, the first brightened image and / or the second brightened image, wherein the first confidence level and / or the second confidence level weights the contribution of the original data image and one or more brightened images for different regions or individual pixels.

[0033] It is more beneficial if the confidence is information about the quality of the image brightening of the raw data of the camera device, in particular a measure of the uncertainty of the brightening. Therefore, in the present invention, the confidence refers to a quality measure of the respective image brightening method used. In principle, it is a measure of the reliability of the image brightening, for example an estimate.

[0034] Here, the confidence level can be determined in different ways and methods, in particular depending on the brightening method used. If the confidence level is expressed in the form of a confidence measure, the output image will be interpreted probabilistically as the probability that the original data image is correctly brightened. However, the confidence level can also be expressed in the form of an uncertainty measure similar to the standard deviation. Here, the output image uncertainty is estimated as a difference describing the reconstruction. In this regard, for example, the uncertainty caused by the original data image itself and the uncertainty caused by the limited brightening accuracy of the respective methods can be taken into account. In addition, if a neural network is used for brightening, the scope and / or quality of the training data can also be taken into account. A combination of a confidence measure and an uncertainty measure can also be used to describe the confidence level. In addition, the uncertainty measure can be interpreted as a confidence measure by calibration.

[0035] In addition, many other possibilities for determining the confidence level are also conceivable, which also fall within the scope of the present invention. For example, a brightened image generated by means of a specific brightening method can be subjected to a predefined image transformation, in particular a mirror transformation, a filtering or similar transformation. Then, in a second step, the brightened image is compared with the brightened image that has been transformed, filtered or otherwise changed, wherein the confidence level of the brightening method is determined from the degree of consistency of the two images. The confidence level of the brightening method can also be determined based on a set of neural networks, especially in the case of methods based on machine learning methods. Here, multiple neural networks are trained to solve the same task. These networks may differ in their architecture, the training data used to train the networks, the weighting coefficients, the loss function or other parameters. Then, the confidence level can be determined based on the degree of consistency of the set.

[0036] Another possibility is to determine the confidence from the output distribution, i.e. the output of the neural network used for image brightening of the raw data image is modeled with a distribution function. The variance of the distribution function then gives the confidence. Finally, the confidence can also be determined based on the uncertainty of the knowledge. In this case, a distribution can be approximated by the weights of the neural network, for example by Monte Carlo Dropout or by Bayesian networks. The confidence can then be calculated from the uncertainty resulting from the influence of these weight distributions on the output of the neural network. For example, the output of a probabilistic neural network can be sampled multiple times and the mean and standard deviation or variance of these samples can be calculated.

[0037] In a design of the method according to the present invention, at least one limit value of the confidence is preset, wherein if the confidence value is lower than the preset limit value, the output image is at least partially reconstructed from the original data image, wherein if the confidence value exceeds the preset limit value, the output image is at least partially reconstructed from the first brightened image. As an alternative, if the confidence value exceeds the preset limit value, the reconstruction is performed from the original data image, and if the confidence value is lower than the limit value, the reconstruction is performed from the brightened image. In the case of using more than one brightening method, confidence intervals can also be preset, wherein if the confidence value is within each corresponding interval, a specific brightened image is used for reconstruction.

[0038] In an advantageous design scheme of the method according to the present invention, a confidence map is determined, which includes confidence values ​​of at least two sub-regions of the original data image, preferably includes confidence values ​​of each pixel of the original data image, wherein the output image is reconstructed based on the confidence map.

[0039] In this regard, it is useful to aggregate sub-regions of the raw data image whose confidence value differences do not exceed a predeterminable limit into a predeterminable confidence region. That is, regions with similar confidence values ​​are aggregated. For example, an average confidence value may be determined for a particular sub-region of the raw data image and indicated in a confidence map.

[0040] The output image reconstructed according to the invention is preferably used for applications in the field of computer vision, preferably for functions of a motor vehicle driver assistance system, in particular for a method for recognizing at least one traffic sign or traffic-related object. Traffic-related objects include, for example, different traffic participants, such as pedestrians, cyclists or motorcyclists, other vehicles (such as trucks, motor vehicles, buses or trains), lane markings. It is also conceivable to determine the intention of the traffic participant based on the output image. In addition, optical flows or depth maps can also be determined. However, the output image can also be used for any object recognition or directly for display purposes without any computer-assisted further processing or the like. For example, the output image can be displayed on a display to an observer who himself obtains information or takes action therefrom, for example in the case of supporting a vehicle parking process.

[0041] In addition, the task underlying the present invention is solved by a data processing system, which includes a device for implementing the method according to the present invention according to one of the described design solutions. The task underlying the present invention is solved by a computer program, which includes relevant instructions. When the computer implements the program, the instructions can enable the computer to implement the method according to the present invention according to one of the described design solutions. The task underlying the present invention is solved by a computer-readable storage medium, and the computer program according to the present invention is stored on the medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The present invention and its advantageous design solutions are explained in detail with reference to the following drawings, wherein:

[0043] FIG. 1 shows night scene images of two different scenes captured in the form of raw data images and images brightened by a brightening method;

[0044] FIG. 2 shows a diagrammatic illustration of a first preferred embodiment of the method according to the invention when a brightening method is used;

[0045] FIG. 3 shows a diagrammatic illustration of two further preferred designs of the method according to the present invention when two or more brightening methods are used; and

[0046] FIG. 4 shows a flow chart of a preferred design of the method according to the present invention.

[0047] Hereinafter, the same elements are provided with the same reference numerals. DETAILED DESCRIPTION

[0048] When photographing at night with insufficient lighting, it is often necessary to perform image brightening before further processing the captured image. As mentioned above, image brightening can be performed using various traditional methods or methods that at least partially use methods from the field of machine learning. However, as shown in Figure 1, these methods have different problems.

[0049] Figure 1a shows a raw data image taken at night at the tunnel exit. Figure 1b shows a brightened image of the same scene obtained using a correspondingly trained convolutional neural network (CNN). In the sky area, the raw data image (Figure 1a) contains very little image information, while a structure can be identified in the brightened image (Figure 1b), which is an error in image processing (white circle area).

[0050] Figure 1c shows an original data image of a county highway at night, while Figure 1d shows a brightened image of the same scene obtained using a correspondingly trained convolutional neural network (CNN). In this case, in the brightened image of Figure 1d, stripes can be seen in the night sky area, but not in the original data image (Figure 1c). When brightening the image, the sensor noise of the camera device is mistakenly processed as data information.

[0051] Similar problems also exist when using traditional methods, especially statistical methods, for image brightening.

[0052] The invention significantly improves image brightening of raw data images of a camera device. In particular, the robustness of the image brightening is improved by reconstructing the final output image from different brightened images and / or raw data images. The proposed reconstruction is based on the determination of the confidence level of at least one used brightening method.

[0053] For the situations shown in Fig. 1a and Fig. 1b, it is suitable to use the original data image, or the brightened image obtained by another method, or to establish the output image in the sky area based on the interpolation performed according to the brightened image shown in Fig. 1b and another brightened image and / or the original data image. In the case of the scenes shown in Fig. 1c and Fig. 1d, it is beneficial to use the original data image to establish the output area in the image area depicting the sky.

[0054] FIG2 shows an example of a first design of the method according to the invention. In a first method step 1, a raw data image 1 of a camera device is provided. In a further method step 2, a first brightened image I1 brightened by means of a first brightening method is created from the raw data image R, and in step 3 a first confidence C1 that the raw data image R was brightened by means of the first brightening method is determined. Finally, in step 4, an output image O is reconstructed from the raw data image R and / or the first brightened image I1 based on the first confidence C1. Here, depending on the value of the confidence C1, the output image O may originate at least partially or completely from the raw data image R, the brightened image I1 or an interpolation of both. In principle, the output images O of different image areas may be composed of different brightened images I and / or raw data images R.

[0055] FIG. 3 shows two further possible embodiments of the method according to the invention. In addition to the method steps shown in FIG. 2 , in the embodiment shown in FIG. 3 a , in addition to the first brightened image I1 generated according to method step 2 a , a second brightened image I2 is additionally generated by means of a second brightening method different from the first brightening method (step 2 b ) and is taken into account when reconstructing the output image O. In the embodiment shown in FIG. 3 a , no separate confidence level C is determined for the image brightening by means of the second brightening method, whereas in the embodiment shown in FIG. 3 b , a first confidence level C1 and a second confidence level C2 are respectively determined for both image brightenings (steps 2 a and 2 b ) (steps 3 a and 3 b ). In other embodiments, other brightening methods may be implemented and / or confidence levels C may be determined, which may be taken into account when reconstructing the output image O.

[0056] According to the present invention, image brightening can be performed by means of a conventional brightening method or by means of a method based on a machine learning method. Depending on the image brightening method used, the confidence level C can be determined in a suitable manner.

[0057] FIG4 shows another preferred design of the method. After the raw data image R is acquired from the camera device 1, it is first checked in a check step i whether the raw data image R is sufficiently illuminated. If so, the raw data image R is used as the output image O. If not, the image is brightened (step 2) and the confidence C of the image brightening is determined (step 3). A brightening method can be used as shown in FIG4 and similar to the design of FIG2. However, it is also possible to use a plurality of brightening methods, for example similar to the design according to FIG3.

[0058] If the determined confidence level C meets a predeterminable criterion (decision step ii), for example exceeds or falls below a predeterminable limit value, the brightened image I1 is used as output image O. Otherwise, according to step 4, the output image is reconstructed from the brightened image I1 and / or the raw data image R. During the reconstruction, the output image O can be composed of different image regions of the brightened image I1 or the raw data image O. As an alternative, the two images can also be interpolated, in which case the confidence level C can be used, for example, as a weighting measure.

[0059] For the design shown in FIG. 4 , it can be assumed, without limiting its generality, that the brightening method in step 2 is based on a machine learning method, for example a method using a convolutional neural network (CNN). Then, additional confidence estimates can be added to the neural network to determine the confidence of the image brightening. To determine the confidence C, an estimate of how reliably the network predicts its output can be determined. Thus, in general, the determination of the confidence is based on the calculation of an uncertainty measure. In this regard, a distinction can be made between epistemic uncertainty and stochastic uncertainty.

[0060] The determined confidence measure C can be calibrated in a further method step, for example, so that the value can be interpreted probabilistically. In this case, the calibration of the confidence measure C or the uncertainty measure can be used in particular for the comparability of the confidence measures C of different image regions and the raw data image R. For example, the calibrated confidence measure C can assume a value in the interval between [0; 1] and be interpreted as the probability that the neural network estimate will calculate a correct prediction.

[0061] Examples of confidence determination for neural networks can be found in particular in the article “What uncertainties do we need in bayesian deep learning for computer vision?” by A. Kendall et al., Advances in neural information processing systems, issue 30, 2017, “Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift?” by Y. Ovadia et al., Advances in neural information processing systems, issue 32, 2019, “A review of uncertainty quantification in deeplearning: Techniques, applications and challenges” by M. Abdar et al., Information Fusion, issue 76, 2021, pages 243, 297, or in the article by V. Kuleshov et al., International Conference on machine learning, 2018. Description from the PMLR (Proceedings of the International Conference on Machine Learning) article "Accurate uncertainties for deep learning using calibrated regression".

[0062] When determining the confidence level about random uncertainty, the network output is modeled as a distribution function. To this end, the neural network adds other outputs that map the parameters of the distribution function. In addition, the cost function can be adjusted and adapted so that the error between the overall expected distribution and the existing distribution of the existing data is minimized. The Kullback-Leibler Divergenz can also be used as a measure of the difference between the distributions. This is equivalent to the maximum likelihood estimation of the network parameters.

[0063] On the one hand, it is conceivable to determine a value for the confidence measure C for each raw data image R, or also to determine the values ​​of the confidence measure C for individual image regions and process them in the form of a confidence map. If a confidence measure C is determined individually for each pixel of the raw data image R, the resolution of the resulting confidence map is the same as the resolution of the raw data image R or the output image O.

[0064] In addition, optionally, the confidence map can also be filtered, for example, using a low-pass filter, etc., to reduce the influence of confidence C values ​​that deviate greatly from the confidence mean value. Image regions with similar confidence values ​​C can also be appropriately aggregated, which in turn can stabilize the reconstruction of the output image O. In addition, when the method is implemented in an embedded system, for example, the confidence map can also be optimized from the perspective of computational workload. This can be achieved, for example, by reducing the numerical representation of the confidence value to 8, 16 or 32 values ​​in the range of [0; 1]. Here, the resolution can be adjusted and adapted according to the integer data format, for example 3 bytes are used for 8 values. For this method, the confidence values ​​C are clustered, the cluster centers are calculated, and the individual values ​​are assigned to the cluster centers. In a possible expansion scheme, these cluster centers can be stored in a lookup table and can be arbitrary values.

[0065] It is also conceivable to combine the images, for example using the determined confidence maps, to calculate the semantic segmentation. For example, an average value of the confidence C can then be calculated for image regions with the same or similar semantics.

[0066] For the case where multiple brightening methods are to be considered when using methods from the field of machine learning, for example, different neural networks can be used, in particular networks with different architectures (such as convolutional neural networks, Transformer networks or in particular generative adversarial networks), different parameters can be used, different network weights learned due to different training (such as urban scenes and rural scenes), different layers can be used (such as different layers for final reconstruction), or at least partially different functions can be used (such as functions for object detection, depth calculation, optical flow calculation or semantic segmentation).

[0067] In principle, within the framework of the invention, for each raw data image R, each image region of a raw data image R or each pixel of a raw data image R, a brightened image I or raw data image R with the best, in particular the highest, confidence C is used for reconstruction. In this way, an optimal brightened output image O can be generated for a specific raw data image R from a plurality of possible brightening methods. Thus, the uncertainty in conventional image brightening can be compensated by a clever, at least partially confidence-controlled combination of different image brightening methods.

[0068] If the confidence C of another brightening method is too poor, especially too low, then the original data image R or the brightened image I for which the confidence C has not yet been determined can be used for reconstruction in this case, because the image brightening method with available confidence C cannot guarantee high-quality image brightening. In the case of high image brightening reliability, that is, good confidence C, especially high confidence, the brightened image I can be used. When selecting from different available images R, I, different selections can also be made for different image areas or for each pixel separately. In addition, an interpolation can be formed from the original image R and / or different brightened images I, for which the confidence C or confidence map is used as a weighted measure. Then, the output image is synthesized from the weighted input image and normalization / standardization. Standardization, for example, summarizes the weight of each pixel, image area or image, and normalizes the sum of the weights to 1. The original data image R and / or the brightened image I for which the confidence C has not yet been determined can be weighted using an inverted, available confidence C or confidence map.

Claims

1. A method for brightening an image of a camera device (1), in particular a computer-implemented method for brightening an image of a camera device, comprising the following steps: - providing at least one raw data image (R) of the camera device (1), - providing at least one first brightened image (I1) of the raw data image (R), wherein the first brightened image (I1) is brightened by means of a first brightening method, - determining a first confidence level (C1) for brightening a raw data image (R) of the camera device (1) by means of a first brightening method, and - reconstructing the output image (O) at least partially from the original data image (R) and / or the first brightened image (I1) according to the first confidence level (C1).

2. The method according to claim 1, in, The method further comprises the steps of: - providing at least one second brightened image (I2) of the raw data image (R), wherein the second brightened image (I2) is brightened by means of a second brightening method different from the first brightening method, and - reconstructing the output image (O) at least partially from the original data image (R), the first brightened image (I1) and / or the second brightened image (I2) according to the first confidence level (C1).

3. The method according to claim 2, in, In order to determine a second confidence level (C2) for brightening a raw data image (R) of a camera device (1) by means of a second brightening method, an output image (O) is reconstructed at least partially from the raw data image (R), a first brightened image (I1) and / or a second brightened image (I2) based on the first confidence level (C1) and / or the second confidence level (C2).

4. A method according to any one of the preceding claims, in, At least one of the brightening methods (I) is at least partially a statistical brightening method, preferably a filter-based method, in particular a histogram stretching method, a tone mapping method, a white balance method or a combination of multiple methods of the above methods.

5. A method according to any one of the preceding claims, in, At least one of the brightening methods (I) is a method based on a machine learning method, preferably a method using one or more neural networks, especially a method using a convolutional neural network, an adversarial neural network, a recurrent neural network, a Transformer network, a neural graph network or a neural circuit, or a deep learning method, especially a deep learning method using a neural network with multiple hidden layers.

6. The method according to claim 5, in, The neural network is a trained neural network which is configured to determine a brightened image (I) based on an input in the form of at least one raw data image (R) of a camera device (1), wherein the brightened image (I) corresponds in particular to an image having a longer exposure time and corresponding to the raw data image (R).

7. A method according to any one of the preceding claims, in, The output image (O) is reconstructed at least in a partial area by interpolation of the original data image (R), the first brightened image (I1) and / or the second brightened image (I2).

8. A method according to any one of the preceding claims, in, The confidence (C) is information about the quality of brightening the raw data image (R) of the camera (1), in particular, a measure of uncertainty in the brightening.

9. The method according to any one of the preceding claims, in, At least one predeterminable limit value of the confidence factor (C) is pre-set, and if the confidence factor (C) is below the predeterminable limit value, the output image is at least partially reconstructed from the original data image (R), and if the confidence factor (C) exceeds the predeterminable limit value, the output image is at least partially reconstructed from the first brightened image.

10. The method according to at least one of the preceding claims, in, A confidence map is determined, which contains confidence (C) values ​​for at least two sub-regions of the original data image (R), preferably for each pixel of the original data image (R), and the output image (O) is reconstructed according to the confidence map.

11. The method according to claim 10, in, Sub-regions of the raw data image (R) whose confidence (C) values ​​do not differ by more than a predefinable limit value are combined into a predefinable confidence region.

12. Use of the output image in the field of computer vision, preferably for a function of a driver assistance system of a motor vehicle, in particular for a method for recognizing at least one traffic sign or traffic-related object.

13. A data processing system comprising means for implementing the method according to any one of claims 1 to 11.

14. Computer program comprising instructions which, when executed by a computer, cause the computer to implement the method according to any one of claims 1 to 11.

15. A computer-readable storage medium storing the computer program according to claim 14.

Citation Information

Patent Citations

  • Brightness conversion of images from a camera

    DE102019220168A1