Irradiation adaptation method and picture recording device

Abstract

In one embodiment, a method for adapting illumination comprises: A) providing a picture recording device (1) comprising an image sensor (2) and a light source (3) configured to illuminate a scene (11) comprising a target (4) along different emission directions (D1 to DM), B) acquiring illumination information (II) of at least one of a background or a reference image (IR) of the scene (11), the illumination information (II) based on a situation when the light source (3) is turned off, c) generating an optimized weight vector (Lambda) on the basis of the illumination information (II), the optimized weight vector (Lambda) comprising at least one intensity value (Lambda) for each emission direction (D1 to DM), and D) capturing at least one target image (IT) of the target (4) by controlling the light emission of the light source (3) in the emission directions (D1 to DM) according to the optimized weight vector (Lambda).

Description

[0001] A method for adapting illumination and a picture recording device are provided.

[0002] Document JP 2022-003 372A relates to a rotating flash unit.

[0003] The problem to be solved is to provide a picture recording device and a corresponding method for improving image quality.

[0004] This object is achieved in particular by a method and a picture recording device as defined in the independent claims. Further exemplary developments form the subject matter of the dependent claims.

[0005] With the method and picture recording device described herein, for example, indirect illumination of the object to be imaged is used and its direction is adjusted by emitting a defined light pattern next to the object by controlling an adjustable photo flash, in particular realized by a multi-LED light source.

[0006] According to at least one embodiment, the method is used to adapt the illumination. For example, by means of the method, a photographic flash for capturing an image is provided. The at least one image to be captured may be a single picture, or may also be a series of pictures, such as an animated image or a video.

[0007] According to at least one embodiment, the method includes the step of providing a picture recording device. The picture recording device includes one or more image sensors, such as a CCD sensor. Further, the picture recording device includes one or more light sources, such as an LED light source. At least one light source is configured to illuminate a scene including a target to be photographed along different emission directions. In other words, the at least one light source is configured to provide a plurality of illumination areas around the target, for example. The emission directions are different from each other in pairs, so that there are no emission directions that are parallel or consistent with each other.

[0008] The term "light source" may refer to visible light, such as white light or red, green and / or blue light, but may also include infrared radiation, for example, near infrared radiation with a spectral range from 750 nanometers to 1.2 micrometers. That is, along each emission direction, visible light and / or infrared radiation may be emitted. For example, there are M emission directions, where M is a natural number. For example, 2≤M≤40 or 6≤M≤30 or 10≤M≤20.

[0009] According to at least one embodiment, the method comprises the step of acquiring illumination information of at least one of the scene background or the reference image, the illumination information being based on a situation when the light source is switched off. Thus, in order to obtain the illumination information, the picture recording device does not emit visible flashes or the like. The illumination information may be partially or completely submitted to the picture recording device by an external source or may be partially or completely calculated in the picture recording device. For example, the illumination information is a background light pattern or an illumination pattern of a scene to be captured later, or a mood of a reference image, which mood is transferred to an image to be captured by the picture recording device.

[0010] According to at least one embodiment, the method comprises the step of generating an optimized weight vector based on the illumination information, the optimized weight vector comprising at least one intensity value for each emission direction.

[0011] It is possible that the weight vector is a row vector or a column vector, depending on its purpose. In particular, the dimension of the vector is M or p*M, where p is a natural number, in particular, p∈{1; 2; 3; 4}. For example, in order to optimize the weight vector based on the illumination information, an objective function can be used, which can be a loss function.

[0012] According to at least one embodiment, the method comprises the step of capturing one or more target images of the target by controlling the light emission of the light source along the emission direction according to the optimized weight vector. In other words, the light intensity of each emission direction, or the light intensity of the light emission unit in the light source corresponding to the respective emission direction, is encoded by the intensity value assigned in the optimized weight vector.

[0013] In at least one embodiment, the method is for adapting illumination and comprises the following steps, for example in the order described:

[0014] A) providing a picture recording device comprising an image sensor and a light source, wherein the light source is configured to illuminate a scene containing a target along different emission directions,

[0015] B) obtaining illumination information of at least one of the scene background or the reference image, the illumination information being based on a situation when the light source is turned off,

[0016] C) generating an optimized weight vector based on the illumination information, the optimized weight vector comprising at least one intensity value for each emission direction, and

[0017] D) capturing at least one target image of the target by controlling the light emission of the light source along the emission direction according to the optimized weight vector.

[0018] In other words, for example, a method is provided for controlling a set of light emitting units of a light source to match a target light distribution when illuminating a scene without capturing any previous image of the actual scene, and in particular without emitting visible light from a picture recording device before taking the target image.

[0019] Cameras in mobile devices, such as smartphones, are very small and cannot receive a lot of light, and therefore do not perform well in low-light environments, producing images with a lot of noise. In order to get a good image exposure, attempts are usually made to add artificial lighting to the scene by turning on some artificial light source during image capture. The nature of this additional light can have a huge impact on the quality of the final picture, and the method described in this article provides a solution to improve the way that flash LEDs can introduce light into low-light scenes. The method focuses on improving the quality of artificial flash in indoor environments. One possibility to solve the problem of low-light photography is to take several images at a time and merge them together using motion adaptation techniques in order to reduce the amount of motion blur. However, this solution is limited by the fact that it acquires images over a few seconds and tries to merge them together, and therefore it is prone to motion blur when photographing moving objects.

[0020] Another possibility relies on very high ISO capability. Other image sensors are generally optimized for low ISO. Very high ISOs generate a lot of noise, but noise reduction can be used. While this solution is promising, the heavy denoising algorithms necessary to compensate for this very high ISO tend to wash out details in the image.

[0021] The more standard and historical approach to low light problems is to use a flash to illuminate the scene. There are two main use cases for flash in photography:

[0022] - In some cases, external light sources can be used that can be configured exactly as needed. Those light sources often provide indirect lighting, i.e. the light bounces off reflective surfaces and then runs to the object to be photographed, or passes through a diffuser to avoid sharp shadows. This use case mainly concerns professional photographers who shoot in a studio as a controlled environment.

[0023] - Sometimes, the environment cannot be controlled and the light source must be kept close to the camera, for example, in a smartphone or compact camera. Therefore, the flash sends direct light to the scene, i.e., there is a straight line between the light source and the subject, which can cause many problems, such as strong reflections, bad shading, strong direct light blinding the subject, overexposure of close subjects and / or sharp shadows.

[0024] In the latter specific use case, there are some possibilities to mitigate the above issues:

[0025] - You can use depth and RGB cameras to analyze the scene using the first RGBD capture, and then use a video projector to flash spatially distributed light to provide better illumination for the background and avoid overexposure of foreground objects.

[0026] - Several different colored LEDs covering the entire spectrum can be used. By analyzing the spectral distribution with a first picture without flash, and then controlling the LED flash to match or compensate for the initial distribution of light, ambient tones can be preserved or active white balance correction can be provided.

[0027] - A standard flash unit can be mounted on a mobile structure that is attached to a digital single-lens reflex (DSLR) camera. By applying an algorithm that uses an additional depth sensor and a fisheye camera to analyze the scene, the optimal direction of the mobile flash can be derived.

[0028] In the method described herein, a picture recording device is used that includes a set (e.g., M) of independently controlled light emitting units, all of which are near the camera, but each pointing in a different direction. A process or algorithm is used to optimize the intensity applied to each light emitting unit during the flash. The weight applied to each light emitting unit can be optimized according to different criteria.

[0029] The light emitting unit should be oriented so that the amount of light directly entering the field of view of the camera (i.e. the image sensor) is as low as possible. Thus, the direct light of a standard flash is replaced by indirect light, which bounces off nearby surfaces around the object to be photographed. For example, a useful emission direction is oriented at an angle of about 60° to the main optical axis and has a beam angle of about 25°.

[0030] The method optimizes the intensity of each of, for example, M light sources by finding an optimal vector Λ of p M weights (also called λ) and applying it to the light emitting units, where for each λ, it applies: λ∈[0;1]. A weight of 0 means that the corresponding light emitting unit is turned off, while a weight of 1 means that the corresponding light emitting unit is at full power.

[0031] The following steps can be used to find the best intensity to use, for example:

[0032] - Capture information about the environment of the object to be photographed (i.e. the scene). This can be performed in various ways, such as

[0033] --Take low-light pictures without turning on the light source.

[0034] --Estimate a 3D representation of the scene in the image sensor's field of view using a 3D sensor or by estimating 3D from a monocular image.

[0035] --As the user moves the mobile device in front of the target, a 3D representation of the scene including areas outside the field of view of the image sensor is estimated via the image sensor of the phone and / or via the 3D sensor of the mobile device. This approach can also be used, for example, for augmented reality applications.

[0036] -- Estimate the spatial distribution of light in the scene, e.g., intensity and color, using a segmented ambient light sensor pointed at the scene.

[0037] - Based on different types of applications, estimate the set of weights to apply to each light emitting unit in order to capture the best picture. This estimation can be performed in various ways, for example:

[0038] --Running inference on a neural network that is trained to directly predict the optimal weights to be applied to the individual light emitting units of a light source in order to capture the best picture based on different types of applications. All such applications share the fact that task-specific datasets need to be collected and ground truth target illumination needs to be defined.

[0039] --Estimating, based on the captured 3D representation of the ambient surfaces, the ambient surfaces that can advantageously reflect the light from the light emitting unit onto the target. This can be done by geometric calculations or ray tracing. Based on the position of the surface, at least one surface to be illuminated by the light emitting unit is selected to utilize the surface for the flash to bounce. In this method, the color and orientation of the surface may be important parameters to consider.

[0040] -- Use computer graphics methods to render pictures of the scene. For example, each picture simulates illumination of the scene by a single emission direction. The rendering may use the geometry of the scene as estimated in the previous step. It may also use an estimate of the geometric distribution of light.

[0041] - Take the optimal weights predicted in the previous step and apply those weights to the corresponding light emitting units to capture the final target image.

[0042] Neural networks or weight optimization methods can be trained for different tasks based on, for example, the results achieved in previous scenarios.

[0043] One possibility for training such a neural network is to use a corresponding image recording device to produce target images of various scenes in different application scenarios. The training may include:

[0044] -Turn on each emission direction in turn at maximum power and capture the corresponding calibration picture with the light source serving only one emission direction. Then, train the neural network by linearly combining all the individual calibration images and the best weights, and compute, for example, the L2 loss or any other regression loss between the synthetic image from the linear combination and the application-specific ground truth. In this way, the weights predicted by the neural network can be associated with some specific vision task.

[0045] In the described algorithm, the weight λ can have any value between, for example, 0.0 (i.e., the light is off) and 1.0 (i.e., the light is on at maximum intensity). This range is continuous, and each weight can take on an almost infinite number of values. This is especially true for a weight vector Λ that contains many weight values ​​λ. The number of combinations is almost infinite, and the algorithm to optimize the vector can therefore be quite complex.

[0046] We can also imagine a system where the intensity of each light emitting unit can only be selected from a limited set of values, such as {0.0; 0.5; 1.0}. In this case, the number of combinations is finite. For example, for ten light emitting units with only three possible intensities, the total number of possibilities is 3. 10 = 59049. One could then imagine a different type of algorithm, e.g. a brute force algorithm that tests all possibilities, to optimize the weight λ.

[0047] Depending on the application, one can choose an objective function to be optimized during training, such as a loss function.

[0048] Some possible examples of application scenarios for the approach described in this article are:

[0049] -Peripheral Light Preservation:

[0050] It tries to illuminate the scene while preserving the ambient light and visual tone from low-light environments. Most of the time, even without artificial light, the scene will still be dimly illuminated. The human eye is very good at adapting to low brightness, and it is desirable to capture a picture that reproduces the world as the human eye sees it, i.e. with the same light distribution but with a good exposure.

[0051] In this case, the ground truth is the weight vector corresponding to the picture that best matches the "long exposure" image (i.e., an image with exactly the same content and shot at exactly the same angle as the low-light image, but with a longer exposure time). The loss function can be calculated considering only the luminance channel of the target and predicted images, which represents only the luminance intensity component of the image.

[0052] -Style transfer:

[0053] In this case, an attempt is made to transfer the style of an arbitrarily chosen image to the final picture to be taken by actively optimizing the flash. In this use case, it is considered that the color of the light emitted by each light emitting unit can preferably also be controlled independently. Since in this case the light is preferably color controlled, the size of the weight vector Λ to be optimized is three times larger. It contains intensity values ​​for each color channel, namely red, green and blue, RGB for short, instead of a universal intensity value.

[0054] In this case, the reference truth is a weight vector corresponding to a picture with spherical harmonics that best matches those of the reference image. This vector represents the harmonic coefficients of the reference light in LAB color space. This spherical harmonic representation can also be obtained by using a target style reference image that is trained to perform such a decomposition with an additional neural network. In this case, the target style reference image can be added to the neural network as an additional input.

[0055] - Scene Relighting:

[0056] The idea here is to use the picture recording device to relight the scene to match any specific light conditions chosen by the user, not just ambient light. One application might be in the field of background customization for video conferencing, where it is desirable to illuminate a person's face in a way that matches the illumination given by a selected background image.

[0057] In this case, the neural network can be trained with synthetic images, allowing for example a high dynamic range image (HDRI) background to be varied effortlessly; or with separate calibration images to obtain additional information such as foreground masks, normal maps and / or albedo, allowing for the creation of a reference realistic re-lit image according to the target HDRI lighting environment. Also in this case, the target HDR map can be added in the vector provided as input to the neural network.

[0058] In order to better understand the advantages of the method described in this article, it is necessary to point out again the main problems of flash photography discussed previously, which are: strong reflections, bad light-dark transitions, overexposure, target glare, sharp shadows. Considering that the most popular low-light mobile photography currently does not use flash at all, but enhances the pictures with night mode algorithms, the main disadvantages associated are: motion blur, artifacts.

[0059] In the method described herein, for example, two innovations are employed: the first is to use a set of light emitting units pointing in different emission directions outside the field of view, and the second is to control the intensity of those light emitting units to match the reference illumination.

[0060] Using bounced light solves many of the problems that come with direct flash. When light bounces off a surface, it is equivalent to using a much larger light source placed on the corresponding surface; the size of this virtual light is equal to the footprint of the flash on the surface. Using such a light essentially eliminates strong reflections and sharp shadows.

[0061] The light optimization algorithm used is specifically designed to detect undesirable shading and overexposure caused by certain light sources, and reduce the intensity of these light sources to eliminate the problem. The fact that the final picture is taken with the weight vector applied to the light emitting cells means that there will be no artifacts such as those caused by heavy denoising, and motion blur is reduced due to the shorter exposure time.

[0062] In summary, the use of bounced light eliminates strong reflections and sharp shadows, the optimization of independent light emission units provides better light and dark changes and reduces overexposure, and retaking pictures without heavy denoising does not cause any artifacts and reduces motion blur. Further, there is no need to wait for any pre-captured images, making it faster and more user-friendly to capture the target image.

[0063] For example, the method can be used in the following embodiments and / or applications:

[0064] -Used in smartphones for low light photography

[0065] The main embodiment of the method described herein may involve mobile photography. If a sufficiently powerful LED with the required light distribution for bounced light could be miniaturized and placed on the back of a smartphone, it would be possible to take indoor flash pictures without all the drawbacks of direct artificial light.

[0066] -Colored light

[0067] For example, for style transfer applications, another possible embodiment is to have a colored light source. In this case, not only the light intensity can be distributed spatially, but also its spectrum. The control of the light source color can be of different types.

[0068] For example, in the case of RGB, the exact color of each light emitting unit can be controlled over a wide range of values ​​covering the entire spectrum or color gamut. In this case, the intensity is controlled by three parameters, for example, one parameter for each channel, such as red, blue and green. The algorithm used works exactly the same as indicated above, except that it optimizes a weight vector of three parameters for each light emitting unit, rather than a weight vector of one parameter in the case of a monochromatic light source.

[0069] For example, in the case of correlated color temperature (CCT for short), many light sources, including LEDs, can emit light in a decreasing spectrum from "warm" to "cold." The parameter that defines the color of light on this scale is called "temperature." Recent mobile phones have even come up with a "dual-tone" flash that has one cool white emitting LED and one warm white emitting LED, and automatically picks a mix of the two to emit light at the CCT that best suits the scene.

[0070] Such a "two-tone" flash setting can be used for each individual light emitting unit of the light source. In this case, the light emitted in each emission direction is controlled by two parameters: intensity and temperature. The above algorithm works exactly the same in this scenario, except that it optimizes the weight vector of two parameters for each light emitting unit instead of just one parameter.

[0071] - Infrared light

[0072] The light source can also emit light in the infrared (IR for short) spectral range. In such a system, the camera will also have IR capabilities in order to see the light emitted by one or more IR sources. In this case, the intensity of the light emitting unit is optimized, for example, by having an IR calibration picture of all emission directions in order to obtain information about the emission directions served by the visible flash, and / or by obtaining information about suitable surfaces next to the target for indirect lighting. In the first case, each emission direction can be equipped with an additional IR light emitting unit for indirect lighting, and in the latter case, direct IR lighting can be provided. IR light can also be used to avoid dazzling the target; the IR flash picture can then be used to denoise the enhanced low-light picture without losing fine grain details.

[0073] The main advantage of this approach using an IR source is that the flash does not emit any visible light until the target image is captured, thus greatly reducing distractions to people in the room and providing a better user experience.

[0074] - Dynamic visual effects

[0075] Another way to control emitted light is to use dynamic weights and permanent illumination instead of flashes. To create a specific tone, for example, for video content creation, light emitting units can be dynamically controlled to create visual effects such as standing next to a campfire or underwater. In this use case, the weights are constantly re-evaluated to adapt to the target animation.

[0076] Further, additional sensors can be used. That is, the number of input parameters of the optimization algorithm can be increased using information from, for example, a depth sensor and / or a wide-angle camera. Information from those sensors will provide additional information for a better performing weight optimizer.

[0077] In a modification of the method, different input images are used for the neural network. Instead of using low-light images as input to the neural network, it is also conceivable to use images captured by the picture recording device under any different light conditions, for example, using images captured with all light sources switched on.

[0078] Even imagine using a set of these images as input to the network, rather than just one, since the information they contain is often complementary. A low-light image contains information about the distribution of ambient light, but is still very dark and shows no detail. A full-flash picture, where all light-emitting units are on at full power, eliminates the ambient light somewhat, but contains more light and more information about the composition of the scene.

[0079] In a modification of this method, there is a pre-shoot. In this case, the weights are optimized by using pre-shot pictures (i.e. calibration pictures). The following algorithm can optimize the intensity of each light emission direction by performing, for example, the following steps:

[0080] - One picture is taken for each emission direction, in which only the corresponding light emission unit is turned on, while the other light emission units are turned off.

[0081] - Choose an arbitrary distribution for the weights, for example, all weights are equal to 1, which means that all light emitting units are at full power. The set of weights is called a weight vector.

[0082] - Optimize the weight vector using the gradient descent algorithm. Gradient descent is an iterative optimization algorithm that will improve the weight vector by running, for example, the following optimization sequence a certain number of times:

[0083] --Slightly change the weight vector.

[0084] --Numerically combine images according to their weights.

[0085] --Run the objective function (such as the loss function), which returns a number telling whether the resulting image is good.

[0086] --Backpropagate the loss gradients to evaluate the next change to apply to the weight vector, specifically if the loss improved due to the last change, then continue changing the weight vector in that direction, otherwise try a different direction.

[0087] -Once the best weight vector is found, the weights are applied to the emission direction and the target image is taken.

[0088] The objective function or loss function may vary depending on the desired outcome.As indicated above, the preceding method may be used to train a neural network.

[0089] Another way to adjust the weights is to first apply the weights to the light emitting units and view the results directly through the camera lens to evaluate the quality of the light and dark changes. The weights can be adjusted step by step based on what is visible in the camera preview for iterative optimization, for example, by using gradient descent. This is similar to using pre-shoots, but does not require any pre-captured images as input parameters. This approach can be called "flash autofocus" because it behaves in the same way as the camera's "autofocus", i.e., automatically adjusting the focus of the lens via a so-called "through the lens" (TTL) algorithm. This alternative is a TTL adjustment algorithm for multiple bounce flashes.

[0090] According to at least one embodiment, the image sensor and the light source and preferably the target are in the same position throughout method steps B) and / or D).Therefore, the picture recording device may not be intentionally moved during and between steps B) and D).

[0091] According to at least one embodiment, in step D), the target is illuminated in an indirect manner, so that all or some or most of the emission directions are directed beside the target. In other words, all or some or most of the emission directions are not directed towards the target.

[0092] According to at least one embodiment, the emission directions of the light sources are fixed relative to the orientation of the image sensor. That is, the emission directions do not change their orientation relative to each other and relative to the image sensor.

[0093] According to at least one embodiment, the diameter of the light source is at most 0.3 m or at most 0.2 m or at most 8 cm or at most 4 cm in a top view of the image sensor. Thus, for example, the lateral dimensions of the light source are smaller than the lateral dimensions of a mobile phone.

[0094] According to at least one embodiment, step B) comprises: B1) capturing a low-light image of the scene with the light source turned off. Thus, the low-light image is captured without emitting visible light from the picture recording device to illuminate the scene.

[0095] According to at least one embodiment, step B) comprises: B2) estimating a three-dimensional representation of the scene within the field of view of the image sensor. Thus, surfaces suitable for reflecting light in the emitted direction can be found, thereby providing indirect illumination. In this case, these surfaces can be located near the target.

[0096] According to at least one embodiment, step B) comprises: B3) estimating a three-dimensional representation of the scene beside the field of view of the image sensor.

[0097] Thus, the illumination information may include information about reflective surfaces beside the target.

[0098] According to at least one embodiment, step B) comprises: B4) analyzing illumination conditions of the reference image, wherein the illumination information comprises information about at least one illumination direction used to capture the reference image. Thus, no pictures may be captured from the scene and no analysis of the scene geometry may be performed before capturing the target image. This is possible because all illumination information may come from the reference image without the reference image having any actual connection to the scene containing the target.

[0099] Method steps B1), B2), B3) and B4) can be combined with one another.

[0100] According to at least one embodiment, step C) comprises: C1) feeding the illumination information to a neural network and running it to generate an optimized weight vector. Thus, the optimization can be done based on a previously trained neural network.

[0101] According to at least one embodiment, step C) comprises: C2) estimating the reflective surface to be illuminated by the light source based on the illumination information and calculating the optimized weight vector accordingly. Thus, the illumination information can be based on 3D information of the scene captured by the picture recording device before taking the target image.

[0102] According to at least one embodiment, step C) comprises: C3) simulating the illumination of the scene for each emission direction based on the illumination information. For example, each simulation assumes that the light source emits radiation only along a subset of the emission directions (e.g., exactly along one emission direction). In other words, based on 3D information of the scene that can be previously captured by a picture recording device, the emission of light along the emission directions is simulated so that virtual calibration images can be generated. These virtual calibration images can be used in combination with the objective function, and the weight vectors cause a linear combination of the virtual calibration images, thereby generating a combined image to be compared with the desired result of the target image.

[0103] Method steps C1), C2) and C3) can be combined with one another.

[0104] According to at least one embodiment, the reference image is an image taken independently of the method described herein. Thus, no spatial and / or temporal connection is required between the location and time at which the reference image is generated and the location and time at which the method is performed. For example, the reference image is an image downloaded from the Internet, an image shared by another user, a picture taken from a movie, or a graphic generated by a computer or another user. Thus, in principle, the reference image can be chosen arbitrarily.

[0105] According to at least one embodiment, step C) comprises: calculating a spherical harmonic representation of a reference ambient light distribution of the reference image. In other words, the illumination conditions present in the reference image are analyzed.

[0106] According to at least one embodiment, step C) comprises: calculating the same spherical harmonic representation of a linear combination of at least some of the calibration pictures, the objective function comprising a measure between two spherical harmonic representations. In other words, the illumination conditions of the synthetic image can be analyzed in the same way as in the case of the reference image. The weight vector is optimized to resemble as well as possible the illumination conditions of the reference image under the light source. In this case, the light along the emission direction can be colored light, in particular RGB light, so that three color channels can be considered for each emission direction for the optimization.

[0107] In accordance with at least one embodiment, for example, in the case of a scene relighting application, a foreground mask and / or a background mask is calculated.

[0108] According to at least one embodiment, the emission angle between the optical axis of the image sensor and all or most or some of the emission directions is at least 30° or at least 45° or at least 55°. Alternatively or additionally, the angle is at most 75° or at most 70° or at most 65°. The angle may refer to the direction of maximum intensity of the respective emission direction.

[0109] According to at least one embodiment, for all or most or some emission directions, the emission angle width of each emission direction is at least 15° or at least 25°. Alternatively or additionally, the angle is at most 45° or at most 35°. The angle may be referred to as full width at half maximum, or FWHM for short.

[0110] The same transmission parameters may be applied to all transmission directions, or the transmission parameters may differ between different transmission directions.

[0111] According to at least one embodiment, the radiation emitted in the emission direction is emitted from outside the field of view of the image sensor, that is, the radiation does not provide direct illumination of the object to be photographed.

[0112] According to at least one embodiment, there are at least six or at least 10 or at least 12 transmission directions. Alternatively or additionally, there are at most 30 or at most 20 or at most 18 transmission directions. For example, the number of transmission directions is between 12 (inclusive) and 16 (inclusive).

[0113] According to at least one embodiment, the light source comprises a light emitting unit for each emission direction. The light emitting unit may be an emitter with a fixed emission characteristic, or may be an emitter with adjustable emission characteristics, such as, for example, an RGB emitter. All light emitting units may have the same construction, i.e., the same emission characteristics, or there may be light emitting units that intentionally have different emission characteristics.

[0114] According to at least one embodiment, the positions of the light emitting units relative to each other are fixed. That is, in the intended use of the picture recording device, the light emitting units cannot move relative to each other. Further, in the intended use of the picture recording device, the light emitting units preferably cannot move relative to the image sensor.

[0115] According to at least one embodiment, the light emitting units are arranged in a circular manner from a top view of the image sensor. For example, the image sensor can be arranged within the circle in which the light emitting units are arranged. The emission direction can be directed inwards.

[0116] According to at least one embodiment, the light source comprises an additional light emitting unit, which is configured for direct illumination of the target. The additional light emitting unit can be used in other situations and / or applications than the light emitting unit for indirect illumination. Thus, with the picture recording device, both direct illumination and indirect illumination can be solved.

[0117] According to at least one embodiment, the method is performed indoors. Therefore, the intended use case is in a room rather than in an open environment, especially not in natural daylight. According to at least one embodiment, in step D), the light source emits a photographic flash. Optionally, the light source can also be configured for short-term or continuous illumination.

[0118] According to at least one embodiment, the distance between the image recording device and the target is at least 0.3 meters or at least 1 meter. Alternatively or additionally, the distance is at most 10 meters or at most 6 meters or at most 3 meters. In other words, the image recording device and the target are intentionally relatively close to each other.

[0119] According to at least one embodiment, the light source is configured to independently emit a plurality of light beams having different colors along all or some or most emission directions. Thus, RGB light can be provided.

[0120] According to at least one embodiment, the light source is configured to emit only a single light beam along at least some emission directions. Thus, the light source may have a single fixed color to be emitted. In this case, "color" may refer to specific coordinates in the CIE color table.

[0121] According to at least one embodiment, the light source comprises one or more emitters for invisible radiation, such as near infrared radiation. There may be only one common emitter for invisible radiation, or there may be one emitter for invisible radiation per emission direction.

[0122] According to at least one embodiment, the picture recording device comprises a 3D sensor. With the aid of the 3D sensor, the picture recording device can, for example, acquire three-dimensional information of the scene before step C). The 3D sensor can, for example, be based on a stereo camera setup, a time-of-flight setup or a reference pattern analysis setup.

[0123] According to at least one embodiment, the picture recording apparatus is a single device, such as a single mobile device, comprising an image sensor and a light source and optionally at least one additional light emitting unit, at least one emitter for invisible radiation and / or at least one 3D sensor.

[0124] According to at least one embodiment, the picture recording device is a mobile phone, such as a smartphone.

[0125] Additionally, a picture recording device is provided. The picture recording device is controlled by means of the method indicated in combination with at least one of the above embodiments. Therefore, the features of the picture recording device are also disclosed for the method, and vice versa.

[0126] In at least one embodiment, the picture recording device is a mobile device and includes an image sensor, a light source, and a processing unit, wherein

[0127] - the processing unit is configured to obtain illumination information of at least one of a background or a reference image of a scene including the object, the illumination information being based on a situation when the light source is turned off,

[0128] - the processing unit is further configured to generate an optimized weight vector based on the illumination information, the optimized weight vector comprising at least one intensity value for each emission direction, and

[0129] The image sensor is configured to capture at least one object image of the object by controlling the light emission of the light source along the emission direction according to the optimized weight vector by the processing unit.

[0130] The method and picture recording device described herein are explained in more detail below by way of exemplary embodiments with reference to the accompanying drawings. In each figure, the same reference numerals are used to indicate the same elements. However, the relationship between the elements is not shown to scale, but the individual elements may be exaggerated to help understanding.

[0131] In each figure:

[0132] Figure 1 is a schematic side view of an exemplary embodiment of a method of using a picture recording device as described herein,

[0133] Figure 2 yes Figure 1 A schematic elevation view of the method,

[0134] Figure 3 is a schematic block diagram of an exemplary embodiment of the method described herein,

[0135] Figure 4 and Figure 5 is a schematic representation of method steps of an exemplary embodiment of the method described herein,

[0136] Figure 6 is a schematic representation of the emission characteristics of a light emission unit of an exemplary embodiment of a picture recording device described herein,

[0137] Figure 7 and Figure 8 is a schematic top view of an exemplary embodiment of a picture recording device described herein, and

[0138] Fig. 9 and Fig.10 is a schematic cross-sectional view of a light emitting unit of an exemplary embodiment of a picture recording device described herein.

[0139] Figure 1 and Figure 2 The figure shows an exemplary embodiment of a method using a picture recording device 1. The picture recording device 1 is a mobile device 10 and comprises an image sensor 2 configured to take pictures and / or videos. Further, the picture recording device 1 comprises a light source 3. Figure 1 and 2 A user of the picture recording device 1 is not shown.

[0140] In the intended use, the picture recording device 1 is used to take an object image IT of, for example, an object 4 in a scene 11 indoors. For example, the object 4 is a person to be photographed. For example, the distance L between the object 4 and the picture recording device 1 is between 1 meter and 3 meters. The size H of the object 4 may be about 1 meter to 2 meters. The object 4 may be located in front of a wall 12 or any other object, for example, the object in front of the object 4 provides a rebound surface on the side of the object 4, so that indirect lighting can be provided. The object 4 may be directly at the wall, or may be kept at a certain distance from the wall 12.

[0141] The light source 3 is configured to emit radiation R, such as visible light and / or infrared radiation, along a plurality of emission directions D1 to DM. Thus, there are M emission directions. For example, M is between 10 (inclusive) and 20 (inclusive). By means of the light source 3, for example, for each emission direction D1 to DM, there is an illumination area 13 next to the object 4 outside the field of view of the image sensor 2. Thus, the light source 3 provides indirect illumination. The radiation emission along the emission directions D1 to DM can be adjusted by means of a processing unit of the picture recording device 1.

[0142] For example, in the room where the picture recording device 1 and the object 4 are located, there is a luminaire 8 providing weak lighting. The picture recording device 1 will reproduce this tone provided by the luminaire 8. In order to do this and achieve a high picture quality, the light source 3 for example specifically addresses an illumination area 13 which is oriented approximately the same relative to the object 4 as the luminaire 8. Figure 2 In the example, this would be the illuminated area 13 of the upper left area next to the lamp 8. In this simple example, when taking a picture by letting the light source 3 act as an adapted photographic flash, the tone can be maintained while good illumination conditions can be present.

[0143] Combination Figure 3 An example of a method of achieving this is schematically illustrated.

[0144] In method step SA, a picture recording device 1 is provided which comprises an image sensor 2 and a light source 3 which is configured to illuminate a scene 11 comprising a target 4 along different emission directions D1 to DM.

[0145] In step SB, illumination information II of at least one of the background or reference images IR of the scene 11 is acquired, the illumination information II being based on the situation when the light source 3 is switched off. Therefore, in order to obtain the illumination information II, the picture recording device 1 does not emit visible light.

[0146] In method step SC, an optimized weighting vector Λ is generated based on the illumination information II, which includes at least one intensity value λ for each emission direction D1 to DM.

[0147] In method step SD, at least one object image IT of object 4 is recorded by controlling the light emission of light source 3 in emission directions D1 to DM according to the optimized weight vector Λ. In other words, for example, by emitting a photographic flash in previously calculated emission directions D1 to DM.

[0148] exist Figure 4 A more detailed map shows Figure 3 Some optional aspects of the method. Steps SB and SD are performed by processing unit 7.

[0149] As input, for example, the processing unit 7 receives a low-light image IL of a scene 11 including a target 4. Illumination information is obtained from the low-light image IL. For example, there are some light and dark variations due to the illumination conditions, represented by some hatching. The processing unit 7 analyzes the illumination conditions. Therefore, the weight vector Λ is optimized to best resemble the illumination conditions of the target image IT. Therefore, the method may include a step SB1 of taking a low-light image IL of the scene 11 with the light source 3 turned off.

[0150] Further, in Figure 4, it is shown that the input may be a three-dimensional representation I3D of a scene 11 around a target 4. Thus, a reflective surface 14 such as a wall 12 next to the target 4 may be found. Therefore, by addressing emission directions D1 to DM suitable for the respective reflective surfaces 14, the weight vector L is optimized to provide the desired illumination. The reflective surface 14 may be a Lambertian reflective surface. The three-dimensional representation I3D of the scene 11 may be within or outside the field of view 22 of the image sensor 2. Accordingly, the method may include steps SB2 and / or SB3 of analyzing the 3D situation of the scene 11 when the light source 3 is turned off.

[0151] The processing unit 7 may simulate a composite image IC based on, for example, a three-dimensional representation I3D. The composite image IC may be a linear combination of the emission directions D1 to DM provided with radiation. For example, the target image IT may be captured with a weight vector Λ that causes a minimum difference between the composite image IC and the desired tone or ambient light conditions of the low-light image IL.

[0152] Further, the illumination conditions of the reference image IR may be analyzed, wherein the illumination information II then comprises information about at least one illumination direction used to capture the reference image IR. Accordingly, the method may comprise a step SB4 of analyzing the reference image IR, wherein when this step is performed, the light source 3 may be irrelevant. The reference image IR may be used as a virtual background for the object 4.

[0153] exist Figure 5 In FIG. 1 , a reference image IR is also provided. For example, the illumination condition is analyzed and extracted from the reference image IR by a neural network. Figure 5 In the image, the illumination conditions are represented by means of the light and dark changes indicated in the reference image IR.

[0154] The weight vector Λ is then optimized to resemble these illumination conditions as closely as possible. This is indicated by light and dark variations in the (possibly simulated) optional composite image IC. Accordingly, the tone of the reference image IR can be transferred to the target image IT.

[0155] Preferably, in the case of a key shift, each emission direction D1 to DM has RGB channels.

[0156] exist Figure 6 , exemplary parameters of the emission directions D1 to DM are shown. For example, the angle 23 between the optical axis 20 of the image sensor 2 and the emission directions D1 to DM is approximately 60°. In each case, the emission angle width 5 of the emission directions D1 to DM may be approximately 30°. Thus, the light source 3 emits no or almost no radiation R into the field of view 23 of the image sensor 2.

[0157] exist Figure 7 and Figure 8 , exemplary embodiments of a picture recording device 1 are shown. In both cases, the picture recording device 1 is a mobile device 10, such as a smartphone.

[0158] The light source 3 includes a plurality of light emitting units 31 to 3M. The light emitting units 31 to 3M may be light emitting diodes, or LEDs for short. The light emitting units 31 to 3M may be arranged in a circular manner, i.e., on a circle. Since the distance between the light emitting units 31 to 3M is very small compared to the distance between the irradiation areas 13 (see Figure 2 ), therefore, the arrangement order of the light emitting units 31 to 3M does not necessarily correspond to the arrangement order of the irradiation areas 13. Therefore, alternatively, the light emitting units 31 to 3M may also be arranged in a matrix, for example.

[0159] If the light emitting units 31 to 3M are arranged on a circle, the respective emission directions D1 to DM associated with the light emitting units 31 to 3M may point inwards, ie may pass through the center of the circle.

[0160] In addition, the picture recording device 1 includes at least one image sensor 2. Optionally, the picture recording device 1 may include at least one of an additional light emitting unit 61, an emitter 62 for invisible radiation, or a 3D sensor 63. Further, the picture recording device 1 includes a processing unit 7 configured to perform the above method. The processing unit 7 may be a main board or an auxiliary board of the picture recording device 1.

[0161] according to Figure 7 , the light source 3 is integrated in the housing of the picture recording device 1. The light emitting units 31 to 3M are arranged around the image sensor 2. Optionally, from a top view of the image sensor 2, at least one of the additional light emitting unit 61, the emitter 62 for invisible radiation or the 3D sensor 63 can also be located within the arrangement of the light emitting units 31 to 3M.

[0162] Apart from Figure 7 In addition to the arrangement shown, an additional light emitting unit 61, at least one of an emitter 62 for invisible radiation or a 3D sensor 63 and the image sensor 2 may be located outside the arrangement of the light emitting units 31 to 3M, as shown. Figure 8 shown.

[0163] In addition, Figure 8 , the light emitting units 31 to 3M are shown to be arranged in a spider-like manner. In this case, the light emitting units 31 to 3M can be arranged to protrude from the housing, but can also be completely inside the housing (as seen from a top view of the image sensor 2, rather than Figure 8 shown).

[0164] Thus, the light source 3 may be an external unit mounted (eg clamped or glued) on the housing. The electrical connection between the housing and the light source 3 may be accomplished by, for example, a USB Type C connection.

[0165] In addition, Figures 1 to 6 The same situation can also be applied to Figure 7 and Figure 8 , and vice versa.

[0166] exist Fig. 9 , an exemplary light emitting unit 31 of the light source 3 is shown. In this case, the light emitting unit 31 has only one channel, ie, for example, is configured to emit in a fixed color along a specified emission direction D1. For example, the color is white light.

[0167] On the contrary, according to Fig.10 , the light emission unit 31 comprises three color channels, for example, for red light, green light and blue light. Thus, three light beams D1R, D1G, D1B are emitted along a specified emission direction D1 to form a radiation R. The three color channels are preferably electrically addressable independently of one another, so that the emission color of the light emission unit 31 can be tuned. For example, each color channel is implemented by its own LED chip as a corresponding light emitter.

[0168] Fig. 9 and Fig.10 The light emitting unit 31 can be used in all embodiments of the picture recording device 1 and can also be used in combination with each other.

[0169] besides, Figures 1 to 8 The same situation can also be applied to Fig. 9 and Fig.10 , and vice versa.

[0170] The invention described here is not limited by the description based on the exemplary embodiments. On the contrary, the invention covers any novel feature and any combination of features, which in particular includes any combination of features in the patent claims, even if this feature or this combination itself is not explicitly specified in the patent claims or exemplary embodiments.

[0171] This patent application claims the priority of German patent application 10 2022 114 077.6, the disclosure content of which is incorporated herein by reference.

[0172] Reference Mark List

[0173] 1 picture recording device; 10 mobile device; 11 scene; 12 wall; 13 illuminated area; 14 reflective surface; 2 image sensor; 20 optical axis; 22 field of view; 23 emission angle; 3 light source; 3.. light emitting unit; 4 target; 5 emission angle width; 61 additional light emitting unit; 62 emitter for invisible radiation; 63 3D sensor; 7 processing unit; 8 luminaire; D.. emission direction; H size; I three-dimensional representation of 3D scene; IC synthetic image; II illumination information; IL low-light image; IR reference image; IT target image; L distance; S.. method steps; R radiation; λ intensity value; Λ weight vector.

Claims

1. A method for adaptive irradiation, comprising: A) A picture recording device (1) is provided, comprising an image sensor (2) and a light source (3), wherein the light source (3) is configured to illuminate a scene (11) including a target (4) along different emission directions (D1 to DM), B) acquiring illumination information (II) of at least one of a background or a reference image (IR) of the scene (11), the illumination information (II) being based on a situation when the light source (3) is turned off, C) generating an optimized weight vector (Λ) based on the illumination information (II), the optimized weight vector (Λ) comprising at least one intensity value (λ) for each of the emission directions (D1 to DM), and D) capturing at least one target image (IT) of the target (4) by controlling the light emission of the light source (3) along the emission directions (D1 to DM) according to the optimized weight vector (Λ), Therein, in step D), the target (4) is illuminated in an indirect manner, so that at least some of the emission directions (D1 to DM) are directed beside the target (4) rather than onto the target (4).

2. The method according to the preceding claim, wherein in step D), the target (4) is illuminated by the light source (3) only indirectly, wherein the emission direction (D1 to DM) of the light source (3) is fixed relative to the orientation of the image sensor (2), From a top view of the image sensor (2), the diameter of the light source (3) is at most 0.3 meters.

3. The method according to any one of the preceding claims, Wherein obtaining the irradiation information (II) in step B) includes one or more of the following: B1) capturing a low-light image (IL) of the scene (11) with the light source (3) turned off, B2) estimating a three-dimensional representation (I3D) of the scene (11) within the field of view (22) of the image sensor (2), and / or B3) estimating a three-dimensional representation (I3D) of the scene (11) next to the field of view (22) of the image sensor (2), The illumination information (II) comprises information about reflective surfaces (12, 14) next to the target (4).

4. The method according to claim 1 or claim 2, Wherein step B) comprises: B4) analyzing the illumination conditions of the reference image (IR), The illumination information (II) comprises information about at least one illumination direction used to capture the reference image (IR).

5. The method according to any one of the preceding claims, Wherein step C) comprises one or more of the following: C1) feeding the illumination information (II) to a neural network and running it to generate the optimized weight vector (Λ), C2) estimating the reflective surface (12) to be illuminated by the light source (3) based on the illumination information (II) and calculating the optimized weight vector (Λ) accordingly.

6. The method according to any one of the preceding claims, Wherein step C) comprises: C3) based on the illumination information (II), simulating the illumination of the scene (11) for each of the emission directions (D1 to DM), wherein each simulation assumes that the light source (3) emits radiation (R) only along a subset of the emission directions (D1 to DM).

7. The method according to any one of the preceding claims, wherein an emission angle (23) between an optical axis (20) of the image sensor (2) and at least some of the emission directions (D1 to DM) is between 30° (inclusive) and 75° (inclusive), wherein for at least some of the emission directions (D1 to DM), the emission angle width (5) of each emission direction (D1 to DM) is between 15° (inclusive) and 45° (inclusive), The radiation (R) emitted in the emission direction (D1 to DM) is emitted from outside the field of view (22) of the image sensor (2).

8. The method according to any one of the preceding claims, There are at least six and at most 30 transmission directions (D1 to DM).

9. The method according to any one of the preceding claims, wherein the light source (3) comprises a light emitting unit (31 to 3M) for each of the emission directions (D1 to DM), the positions of the light emitting units (31 to 3M) relative to each other being fixed, From a top view of the image sensor (2), the light emitting units (31 to 3M) are arranged in a circular manner.

10. The method according to any one of the preceding claims, The light source (3) comprises an additional light emitting unit (61), and the additional light emitting unit (61) is configured to directly illuminate the target (4).

11. The method according to any one of the preceding claims, Indoors, wherein in step D), the light source (3) emits a photographic flash, The distance between the image recording device (1) and the target (4) is between 1 meter (inclusive) and 6 meters (inclusive).

12. The method according to any one of the preceding claims, The light source (3) is configured to independently emit a plurality of light beams having different colors along at least some of the emission directions (D1 to DM).

13. The method according to any one of the preceding claims, The light source (3) is configured to emit only a single beam of light along at least some of the emission directions (D1 to DM).

14. The method according to any one of the preceding claims, The light source (3) comprises an emitter (62) for invisible radiation.

15. The method according to any one of the preceding claims, The image recording device (1) comprises a 3D sensor (63), By means of the 3D sensor (63), the image recording device (1) acquires three-dimensional information of the scene before step C).

16. The method according to any one of the preceding claims, The picture recording device (1) is a single mobile device (10), and the mobile device (10) includes the image sensor (2) and the light source (3).

17. The method according to the preceding claim, The picture recording device (1) is a smart phone.

18. A picture recording device (1), which is a mobile device (10) and comprises an image sensor (2), a light source (3) and a processing unit (7), wherein - the processing unit (7) is configured to obtain illumination information (II) of at least one of a background or a reference image (IR) of a scene (11) containing the object (4), the illumination information (II) being based on a situation when the light source (3) is turned off, - the processing unit (7) is further configured to generate an optimized weight vector (Λ) based on the illumination information (II), the optimized weight vector (Λ) comprising at least one intensity value (λ) for each emission direction (D1 to DM); and -The image sensor (2) is configured to capture at least one target image (IT) of the target (4) by controlling the light emission of the light source (3) along the emission direction (D1 to DM) by the processing unit (7) according to the optimized weight vector (Λ).

Images