Multi-frame depth-based multi-camera illumination adjustment of images

Abstract

A method includes capturing multiple first images using a first image sensor and capturing multiple second images using a second image sensor. The method also includes generating a short single view, a short depth map, a long single view, and a long depth map based on the first image and the second image. The method also includes generating a multi-frame depth map using the short single view, the short depth map, the long single view, and the long depth map. The method also includes generating a gain map using the multi-frame depth map and the lighting conditions, wherein the lighting conditions introduce a lighting adjustment effect into the gain map. The method also includes generating a lighting-adjusted short single view and a lighting-adjusted long single view by applying the gain map to the short single view and the long single view. In addition, the method includes generating a lighting-adjusted image by fusing the lighting-adjusted short view and the lighting-adjusted long view.

Description

Technical Field

[0001] The present disclosure relates generally to image capture systems and more particularly to multi-frame depth-based multi-camera relighting of images. Background Art

[0002] The lighting within an image is often a key factor affecting the aesthetic quality of an image. The ability to perform lighting adjustments on an image (i.e., change the lighting within an image after it has been captured) is a powerful tool. Additionally, this allows consumers to tap into their inner artist to create the desired effect. Summary of the invention

[0003] Technical issues

[0004] However, most image editing tools operate on a single processed image file, such as a Joint Photographic Experts Group (JPEG) file. Unfortunately, when lighting adjustment tools operate on processed image files, a large amount of original image data (such as color and illumination data) is lost.

[0005] Technical Solutions

[0006] The present disclosure relates to multi-frame depth-based multi-camera illumination adjustment of images.

[0007] In a first embodiment, a method includes capturing a plurality of first images using a first image sensor, and capturing a plurality of second images using a second image sensor. The method also includes generating a short single view, a short depth map, a long single view, and a long depth map based on the plurality of first images and the plurality of second images. The method also includes generating a multi-frame depth map using the short single view, the short depth map, the long single view, and the long depth map. The method also includes generating a gain map using the multi-frame depth map and the illumination condition, wherein the illumination condition introduces an illumination adjustment effect into the gain map. The method also includes generating an illumination-adjusted short single view and an illumination-adjusted long single view by applying the gain map to the short single view and the long single view. In addition, the method includes generating an illumination-adjusted image by fusing the illumination-adjusted short single view and the illumination-adjusted long single view.

[0008] The method may also include aligning multiple first images; fusing images of the multiple first images having a common exposure value into a first short frame and a first long frame; aligning multiple second images; fusing images of the multiple second images having a common exposure value into a second short frame and a second long frame; generating a short single view based on the first short frame and the second short frame, and generating a long single view based on the first long frame and the second long frame; and calculating a short depth map using the first short frame and the second short frame, and calculating a long depth map using the first long frame and the second long frame.

[0009] The method may also include generating a motion map using a deghosting operation on the short single view and the long single view; generating a saturation map using exposure analysis on the short single view and the long single view; generating a hybrid map by combining the motion map and the saturation map; and generating a multi-frame depth map by applying the hybrid map to each of the short depth map and the long depth map.

[0010] The method may also include generating a normal map using the multi-frame depth map and lighting conditions, wherein the normal map contains, for each position in the scene, a value of a surface normal at the position; generating a light vector map based on the multi-frame depth map and lighting conditions, wherein the light vector map indicates, for each position in the scene, a light vector from a light source to the position; and combining the normal map and the light vector map using a shading model to generate a gain map.

[0011] The method may also include generating a saturation map using exposure analysis for the illumination-adjusted short single view and the illumination-adjusted long single view; generating a blend map by combining the saturation map and the motion map; and generating an illumination-adjusted image by applying the blend map to the illumination-adjusted short single view and the illumination-adjusted long single view using blend and hue map operations.

[0012] The lighting conditions may include point light source lighting adjustments, and the lighting adjusted image may have light properties applied from a single light source.

[0013] The lighting conditions may include parallel light source effects, and the lighting adjusted image may have light properties applied from multiple parallel light sources.

[0014] Directional light sources can be applied to create foreground brightening effects.

[0015] Directional light sources can be applied to create background brightening effects.

[0016] Beneficial Effects

[0017] In a second embodiment, an electronic device includes a first image sensor configured to capture a plurality of first images, a second image sensor configured to capture a plurality of second images, and at least one processing device operably coupled to the first image sensor and the second image sensor. The at least one processing device is configured to generate a short single view, a short depth map, a long single view, and a long depth map based on the plurality of first images and the plurality of second images. The at least one processing device is also configured to generate a multi-frame depth map using the short single view, the short depth map, the long single view, and the long depth map. The at least one processing device is also configured to generate a gain map using the multi-frame depth map and the illumination condition, wherein the illumination condition introduces an illumination adjustment effect into the gain map. The at least one processing device is also configured to apply the gain map to the short single view and the long single view to generate an illumination-adjusted short single view and an illumination-adjusted long single view. In addition, the at least one processing device is configured to fuse the illumination-adjusted short single view and the illumination-adjusted long single view to generate an illumination-adjusted image.

[0018] The at least one processing device may also be configured to: align multiple first images; fuse images of the multiple first images having a common exposure value into a first short frame and a first long frame; align multiple second images; fuse images of the multiple second images having a common exposure value into a second short frame and a second long frame; generate a short single view based on the first short frame and the second short frame, and generate a long single view based on the first long frame and the second long frame; and calculate a short depth map using the first short frame and the second short frame, and calculate a long depth map using the first long frame and the second long frame.

[0019] The at least one processing device may also be configured to: generate a motion map using a deghosting operation for the short single view and the long single view; generate a saturation map using exposure analysis for the short single view and the long single view; combine the motion map and the saturation map to generate a hybrid map; and apply the hybrid map to each of the short depth map and the long depth map to generate a multi-frame depth map.

[0020] The at least one processing device may also be configured to: generate a normal map using multi-frame depth maps and lighting conditions, wherein the normal map contains, for each position in the scene, a value of a surface normal at the position; generate a light vector map based on the multi-frame depth maps and lighting conditions, wherein the light vector map indicates, for each position in the scene, a light vector from a light source to the position; and combine the normal map and the light vector map using a shading model to generate a gain map.

[0021] The at least one processing device may also be configured to: generate a saturation map using exposure analysis for the illumination-adjusted short single view and the illumination-adjusted long single view; combine the saturation map and the motion map to generate a blended map; and apply the blended map to the illumination-adjusted short single view and the illumination-adjusted long single view using blending and tone map operations to generate a illumination-adjusted image.

[0022] In a third embodiment, a non-transitory machine-readable medium includes instructions that, when executed, cause at least one processor of an electronic device to capture multiple first images using a first image sensor and to capture multiple second images using a second image sensor. The medium also includes instructions that, when executed, cause at least one processor to generate a short single view, a short depth map, a long single view, and a long depth map based on the multiple first images and the multiple second images. The medium also includes instructions that, when executed, cause at least one processor to generate a multi-frame depth map using a short single view, a short depth map, a long single view, and a long depth map. The medium also includes instructions that, when executed, cause at least one processor to generate a gain map using a multi-frame depth map and an illumination condition, wherein the illumination condition introduces an illumination adjustment effect into the gain map. The medium also includes instructions that, when executed, cause at least one processor to apply the gain map to a short single view and a long single view to generate an illumination-adjusted short single view and an illumination-adjusted long single view. In addition, the medium includes instructions that, when executed, cause at least one processor to fuse an illumination-adjusted short single view and an illumination-adjusted long single view to generate an illumination-adjusted image.

[0023] Other technical features will be clear to those skilled in the art from the following drawings, description and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 An example network configuration including electronic devices according to the present disclosure is shown.

[0025] Figure 2A Example multi-frame images without lighting adjustment are shown.

[0026] Figure 2B Example multiple frames of lighting-adjusted images are shown.

[0027] Figure 3A An example no-depth relighted image is shown.

[0028] Figure 3B An example depth-aware single-frame lighting-adjusted image is shown.

[0029] Figure 3CExample depth-aware multi-frame lighting-adjusted images are shown.

[0030] Figure 4 An example method for multi-frame depth-based multi-camera illumination adjustment of images according to the present disclosure is shown.

[0031] Figure 5 An example method of multi-exposure depth generation for image lighting adjustment according to the present disclosure is shown.

[0032] Figure 6 An example method of multi-frame depth fusion for image illumination adjustment according to the present disclosure is shown.

[0033] Fig. 7A An example method of motion map generation for image lighting adjustment according to the present disclosure is shown.

[0034] Figure 7B An example implementation of reference frame block operations is shown.

[0035] Figure 7C An example implementation of the main block operation is shown.

[0036] Fig.7D An example implementation of sub-master block operations is shown.

[0037] Figure 8 An example method for exposure analysis according to the present disclosure is shown.

[0038] Fig. 9 An example method of blend map generation according to the present disclosure is shown.

[0039] Fig.10 An example method for multi-frame depth map generation according to the present disclosure is shown.

[0040] Fig.11A An example method for multi-frame lighting adjustment according to the present disclosure is shown.

[0041] Fig. 11B An example of a depth-based lighting adjustment operation is shown.

[0042] Fig. 11C An example of a gain map generation operation is shown.

[0043] Fig.11D An example of a normal map is shown.

[0044] Fig. 12A Shows an example of the lighting adjustment effect of a point light.

[0045] Fig. 12B An example of the effect of a directional light source is shown.

[0046] Fig. 12C An example of a graph related to foreground brightening lighting effects is shown.

[0047] Fig.12D An example of a graph related to background brightening lighting effects is shown.

[0048] Fig.13A Example images obtained using conventional methods are shown.

[0049] Fig. 13B An example image obtained using the method described in this patent document is shown.

[0050] Fig.14 An example method for multi-frame depth-based multi-camera illumination adjustment of images according to the present disclosure is shown. DETAILED DESCRIPTION

[0051] The following discussion is described with reference to the accompanying drawings. Figures 1 to 14 As well as various embodiments of the present disclosure. However, it should be understood that the present disclosure is not limited to these embodiments, and all variations and / or equivalents or replacements thereof also belong to the scope of the present disclosure. Throughout the specification and the accompanying drawings, the same or similar reference numerals may be used to refer to the same or similar elements.

[0052] As mentioned above, the lighting in an image is often a key factor affecting the aesthetic quality of the image. The ability to perform lighting adjustments on an image (i.e., to change the lighting within an image after it has been captured) is a powerful tool. Among other things, this allows consumers to tap into their inner artist to create the desired effect. However, most image editing tools operate on a single processed image file, such as a Joint Photographic Experts Group (JPEG) file. Unfortunately, when a lighting adjustment tool operates on a processed image file, a large amount of the original image data (such as color and illumination data) is lost.

[0053] The present disclosure provides various techniques for performing image illumination adjustment in a multi-frame imaging system before image fusion occurs. As described in more detail below, a series of images can be captured at different exposure levels, and each image can be illuminated differently depending on its exposure level. In addition, a depth map can be used to create an illumination-adjusted "gain" map that is applied to the image. The illumination-adjusted images can then be combined together to form a final image of the scene, which ideally has a more aesthetically pleasing illumination adjustment effect.

[0054] Figure 1 An example network configuration 100 including electronic devices according to the present disclosure is shown. Figure 1 The embodiment of the network configuration 100 shown is for illustration only. Other embodiments of the network configuration 100 may be used without departing from the scope of the present disclosure.

[0055] According to an embodiment of the present disclosure, an electronic device 101 is included in a network configuration 100. The electronic device 101 may include at least one of a bus 110, a processor 120, a memory 130, an input / output (I / O) interface 150, a display 160, a communication interface 170, or a sensor 180. In some embodiments, the electronic device 101 may exclude at least one of these components, or may add at least one other component. The bus 110 includes circuits for connecting components 120-180 to each other and for transmitting communications (such as control messages and / or data) between components.

[0056] The processor 120 includes one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). The processor 120 is capable of controlling at least one of the other components of the electronic device 101 and / or performing communication-related operations or data processing. In some embodiments, the processor 120 may be a graphics processor unit (GPU). As described in more detail below, the processor 120 may receive image data captured by a plurality of cameras or other image sensors, and process the image data to generate an image of a scene, and the processor 120 may perform multi-frame depth-based illumination adjustment of the image.

[0057] The memory 130 may include a volatile memory and / or a non-volatile memory. For example, the memory 130 may store commands or data related to at least one other component of the electronic device 101. According to an embodiment of the present disclosure, the memory 130 may store software and / or a program 140. The program 140 includes, for example, a kernel 141, middleware 143, an application programming interface (API) 145, and / or an application program (or "application") 147. At least a portion of the kernel 141, the middleware 143, or the API 145 may be represented as an operating system (OS).

[0058] The kernel 141 can control or manage system resources (such as bus 110, processor 120 or memory 130) for executing operations or functions implemented in other programs (such as middleware 143, API 145 or application 147). The kernel 141 provides an interface that allows the middleware 143, API 145 or application 147 to access the various components of the electronic device 101 to control or manage system resources. The application 147 includes one or more applications for image capture and image processing, as described below. These functions can be performed by a single application or multiple applications, each of which performs one or more of these functions. For example, the middleware 143 can act as a relay that allows the API 145 or application 147 to communicate data with the kernel 141. Multiple applications 147 can be provided. The middleware 143 can control the work request received from the application 147, such as by assigning a priority of using the system resources (such as bus 110, processor 120 or memory 130) of the electronic device 101 to at least one of the multiple applications 147. The API 145 is an interface that allows the application 147 to control functions provided by the kernel 141 or the middleware 143. For example, the API 145 includes at least one interface or function (such as a command) for archive control, window control, image processing, or text control.

[0059] The I / O interface 150 serves as an interface that can transmit commands or data input from a user or other external devices to other components of the electronic device 101, for example. The I / O interface 150 can also output commands or data received from other components of the electronic device 101 to the user or other external devices.

[0060] The display 160 includes, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a quantum dot light emitting diode (QLED) display, a micro-electromechanical system (MEMS) display, or an electronic paper display. The display 160 may also be a depth perception display, such as a multi-focal display. The display 160 is capable of displaying, for example, various contents (such as text, images, videos, icons, or symbols) to the user. The display 160 may include a touch screen and may receive, for example, a touch, gesture, proximity, or hover input using an electronic pen or a user's body part.

[0061] The communication interface 170 can, for example, establish communication between the electronic device 101 and an external electronic device (such as the first electronic device 102, the second electronic device 104, or the server 106). For example, the communication interface 170 can be connected to the network 162 or 164 through wireless or wired communication to communicate with the external electronic device. The communication interface 170 can be a wired or wireless transceiver or any other component for sending and receiving signals (such as images).

[0062] Wireless communication can use, for example, long term evolution (LTE), long term evolution-advanced (LTE-A), fifth generation wireless system (5G), millimeter wave or 60 GHz wireless communication, wireless USB, code division multiple access (CDMA), wideband code division multiple access (WCDMA), universal mobile telecommunications system (UMTS), wireless broadband (WiBro) or global system for mobile communications (GSM) as at least one of the cellular communication protocol. Wired connection can include, for example, at least one of universal serial bus (USB), high-definition multimedia interface (HDMI), recommended standard 232 (RS-232) or plain old telephone service (POTS). Network 162 or 164 includes at least one communication network, such as a computer network (such as a local area network (LAN) or a wide area network (WAN)), the Internet or a telephone network.

[0063] The electronic device 101 also includes one or more sensors 180 that can measure physical quantities or detect the activation state of the electronic device 101 and convert the measured or detected information into electrical signals. For example, one or more sensors 180 include multiple cameras or other imaging sensors for capturing images of scenes. The sensor 180 may also include one or more buttons for touch input, a posture sensor, a gyroscope or gyro sensor, a pressure sensor, a magnetic sensor or a magnetometer, an acceleration sensor or an accelerometer, a grip sensor, a proximity sensor, a color sensor (such as a red, green, and blue (RGB) sensor), a biophysical sensor, a temperature sensor, a humidity sensor, an illumination sensor, an ultraviolet (UV) sensor, an electromyogram (EMG) sensor, an electroencephalogram (EEG) sensor, an electrocardiogram (ECG) sensor, an infrared (IR) sensor, an ultrasonic sensor, an iris sensor, or a fingerprint sensor. The sensor 180 may also include an inertial measurement unit, which may include one or more accelerometers, gyroscopes, and other components. In addition, the sensor 180 may include a control circuit for controlling at least one of the sensors included here. Any of these sensors 180 may be located within the electronic device 101.

[0064] The first external electronic device 102 or the second external electronic device 104 may be a wearable device or a wearable device (such as an HMD) that can be mounted on an electronic device. When the electronic device 101 is mounted in the electronic device 102 (such as an HMD), the electronic device 101 may communicate with the electronic device 102 through the communication interface 170. The electronic device 101 may be directly connected to the electronic device 102 to communicate with the electronic device 102 without passing through a separate network. The electronic device 101 may also be an augmented reality wearable device including one or more cameras, such as glasses.

[0065] Each of the first external electronic device 102 and the second external electronic device 104 and the server 106 can be a device of the same or different type as the electronic device 101. According to certain embodiments of the present disclosure, the server 106 includes a group of one or more servers. In addition, according to certain embodiments of the present disclosure, all or some operations performed on the electronic device 101 can be performed on another or multiple other electronic devices (such as electronic devices 102 and 104 or server 106). In addition, according to certain embodiments of the present disclosure, when the electronic device 101 should automatically or upon request perform some functions or services, instead of or in addition to performing the functions or services itself, the electronic device 101 can request another device (such as electronic devices 102 and 104 or server 106) to perform at least some functions associated with it. Other electronic devices (such as electronic devices 102 and 104 or server 106) are capable of performing the requested functions or additional functions and transmitting the execution results to the electronic device 101. The electronic device 101 can provide the requested function or service by processing the received results as is or in addition. To this end, for example, cloud computing, distributed computing or client-server computing technology can be used. Although Figure 1 The electronic device 101 is shown to include a communication interface 170 for communicating with an external electronic device 104 or a server 106 via a network 162 or 164 , but according to some embodiments of the present disclosure, the electronic device 101 may operate independently without a separate communication function.

[0066] The server 106 may include components that are the same or similar to the components 110-180 of the electronic device 101 (or a suitable subset thereof). The server 106 may support driving the electronic device 101 by performing at least one of the operations (or functions) implemented on the electronic device 101. For example, the server 106 may include a processing module or processor that may support the processor 120 implemented in the electronic device 101. In some embodiments, the server 106 processes the image data to perform multi-frame depth-based multi-camera illumination adjustment of the image.

[0067] although Figure 1 An example of a network configuration 100 including an electronic device 101 is shown, but the Figure 1 Various changes may be made. For example, network configuration 100 may include any number of each component in any suitable arrangement. In general, computing and communication systems have a variety of configurations, and Figure 1 The scope of the present disclosure is not limited to any particular configuration. Figure 1 One operating environment is shown in which the various features disclosed in this patent document may be used, but these features may be used in any other suitable system.

[0068] Figure 2A and Figure 2B An example multi-frame image 200-201 according to the present disclosure is shown. Specifically, Figure 2A An example multi-frame image 200 without illumination adjustment is shown, while Figure 2B An example multi-frame illumination-adjusted image 201 is shown. Here, a "multi-frame" image refers to an image produced by fusing or otherwise combining multiple input images or image frames. For ease of explanation, images 200 and 201 may be described as being generated using Figure 1 The images 200 and 201 may be captured by electronic device 101. However, images 200 and 201 may be captured by any suitable electronic device and in any suitable system.

[0069] In this example, the multi-frame image 200 represents an image captured with a standard exposure time. This particular image 200 is an image of a scene outside an office building. In this scene, there are plants in a rock garden next to a sidewalk. The sidewalk extends from the foreground of the image 200 to the background, and a shadow falls on the sidewalk in the foreground of the image.

[0070] For short exposure times, dark portions 205 of the scene may experience significant loss of detail due to underexposure. As exposure time increases, the amount of light captured by the image sensor increases, allowing for more detail in dark portions 205 of the scene. For long exposure times, bright portions 210 of the scene may experience significant loss of detail due to overexposure. As exposure time decreases, the amount of light captured by the image sensor decreases, allowing for more detail in bright portions 210 of the scene. Averaging the exposure times may result in the loss of small details in both bright and dark areas of the scene.

[0071] Some methods for lighting adjustment may include adjusting specific colors to enhance the shot. For example, lighting adjustment may transition the foreground of a scene from a blue hue on the left to a red hue on the right. However, this effect may result in the plants in the front of image 200-201 having a purple hue, while the rocks in the foreground of image 200-201 have a blue hue on the left and a red hue on the right.

[0072] although Figure 2A and Figure 2B An example of multiple frame images 200-201 is shown, but the Figure 2A and Figure 2B For example, images of any number of scenes may be captured, and Figure 2A and Figure 2B The images 200-201 in FIG. 2 are merely examples of images that may be captured.

[0073] Figure 3A , Figure 3B and Figure 3C Example illumination-adjusted images 300-302 are shown according to the present disclosure. Specifically, Figure 3A An example non-depth, light-adjusted image 300 is shown. Figure 3B An example depth-aware single-frame illumination-adjusted image 301 is shown, and Figure 3C An example depth-aware multi-frame illumination-adjusted image 302 is shown. For ease of explanation, the images 300-302 may be described as being captured using Figure 1 The images 300-302 may be captured by electronic device 101. However, the images 300-302 may be captured by any suitable electronic device and in any suitable system.

[0074] Here, images 300-302 capture the same scene as images 200-201. The depth-free, light-adjusted image 300 represents an image without a concept of depth, meaning that additional lighting has been added throughout image 300. With this indiscriminate lighting, brighter areas of the scene may lose detail. For example, in image 300, detail is lost around the top of the building in the upper left of image 300, where the roof line is barely visible. Similar detail is lost with respect to the seats on the right side of image 300, where at least one seat is barely visible due to the added lighting in image 300.

[0075] Depth-aware single-frame lighting-adjusted image 301 takes depth into account in the lighting adjustment, but only to the extent that additional light is provided to the foreground of image 301. Considering depth in the scene when performing lighting adjustment operations allows details in the background to be preserved relative to the depth-free lighting-adjusted image 300. Thus, for example, the roofline of the building at the top right of image 301 is now visible, as are the seats on the right side of image 301. However, lighting adjustment in the foreground still has the problem of losing detail, such as in bright areas of the scene. For example, the sidewalk is over-brightened in a way that makes the shadowed portions of the sidewalk in the foreground of the image not visible.

[0076] The depth-aware multi-frame light-adjusted image 302 uses the depth within the scene and multiple image frames captured using different exposure times. In this example, the addition of light is only added to the foreground while maintaining an overall good exposure of the scene. This approach allows bright areas of the foreground to avoid saturation. In this particular example, this allows the shadows on the sidewalk in the foreground to remain visible. Various techniques for performing multi-frame depth-based multi-camera light adjustment of an image are described below.

[0077] although Figure 3A , Figure 3B and Figure 3C Examples of light-adjusted images 300-302 are shown, but may be Figure 3A , Figure 3B and Figure 3C For example, you can capture images of any number of scenes, and Figure 3A , Figure 3B and Figure 3C Images 300-302 in FIG. 3 are merely examples of images that may be captured.

[0078] Figure 4 An example method 400 for multi-camera illumination adjustment based on multi-frame depth of an image according to the present disclosure is shown. For ease of explanation, Figure 4 The method 400 shown can be described as being Figure 1 However, the method 400 may be used in any suitable electronic device and in any suitable system.

[0079] like Figure 4 As shown, the electronic device 101 or other device uses multiple cameras or other image sensors to capture an image of the scene 402. To simplify the discussion, it is assumed that the electronic device 101 includes a first image sensor 404 and a second image sensor 406. In the following discussion, it is assumed that the image sensors 404 and 406 are side by side, with the image sensor 404 located on the left and the image sensor 406 located on the right. However, the image sensors 404 and 406 can have any suitable arrangement on the electronic device 101, such as side by side, up and down, or offset. In addition, the electronic device 101 can include any suitable number of image sensors in any suitable arrangement.

[0080] The first image sensor 404 captures a plurality of first images 408 of the scene 402 and the second image sensor 406 captures a plurality of second images 410 of the scene 402. Depending on the implementation, the first images 408 and the second images 410 may be captured with different exposure times or similar exposure times (typically in rapid succession). In some embodiments, the exposure times may vary or alternate with the first image sensor 404 and the second image sensor 406. Furthermore, the sequence of image capture may be consistent between the first image sensor 404 and the second image sensor 406, meaning that the image sensors 404 and 406 may capture the same number of images using the same sequence of exposure times.

[0081] The first image 408 and the second image 410 are processed using a depth-based multi-frame multi-camera illumination adjustment operation 412. The depth-based multi-frame multi-camera illumination adjustment operation 412 generally operates to process the images 408 and 410 to generate at least one illumination-adjusted image 414 that represents an image of the scene 402 generated by applying the illumination adjustment operation to at least some of the images 408 and 410 and by combining at least some of the images 408 and 410. Multi-frame depth considerations provide an improved depth map that is robust to poor exposure conditions. Multi-frame illumination adjustment also provides pre-fusion illumination adjustment of the images (meaning that illumination adjustment occurs before the images 408 and 410 are combined), which results in a well-exposed illumination-adjusted image 414. An example implementation of the depth-based multi-frame multi-camera illumination adjustment operation 412 is described below, and examples of illumination-adjusted images 414 that can be generated using the depth-based multi-frame multi-camera illumination adjustment operation 412 are described below.

[0082] although Figure 4 An example of a method 400 for multi-camera illumination adjustment based on multi-frame depth for an image is shown, but may be used for Figure 4 Various changes are made. For example, method 400 may include using more than two image sensors.

[0083] Figure 5 An example method 500 for generating multiple exposure depths for image illumination adjustment according to the present disclosure is shown. For ease of explanation, Figure 5 The method 500 shown can be described as being Figure 1 However, the method 500 may be used in any suitable electronic device and in any suitable system.

[0084] like Figure 5 As shown, the electronic device 101 or other device uses multiple cameras or other image sensors to capture images of the scene 506. To simplify the discussion, assume that the electronic device 101 includes a first image sensor 508 and a second image sensor 510. In the following discussion, it is assumed that the image sensors 508 and 510 are side by side, with the image sensor 508 on the left and the image sensor 510 on the right. However, the image sensors 508 and 510 can have any suitable arrangement on the electronic device 101, such as side by side, up and down, or offset. In addition, the electronic device 101 can include any suitable number of image sensors in any suitable arrangement. The first image sensor 508 captures multiple first images 514 of the scene 506, and the second image sensor 510 captures multiple second images 516 of the scene 506. Note here that these elements can be connected with each other. Figure 4 The corresponding elements shown in and described above are the same or similar.

[0085] Image 514 is processed using a first registration and single exposure fusion operation 518, and image 516 is processed using a second registration and single exposure fusion operation 520. Note that the first and second registration and single exposure fusion operations 518 and 520 may actually represent the same registration and single exposure fusion operations simply repeated for different sets of images 514 and 516. Because multiple images 514 or 516 are captured over time, it is possible that the images 514 or 516 are misaligned due to, for example, motion of the electronic device, which is common with handheld devices. Each registration and single exposure fusion operation 518 and 520 operates to align the images 514 or 516 to produce an aligned version of the images 514 or 516. Furthermore, after alignment, each registration and single exposure fusion operation 518 and 520 operates to fuse or combine the aligned versions of the images 514 or 516 captured using a common exposure value. For example, the first registration and single exposure fusion operation 518 can fuse the image 514 captured using the shorter exposure to produce the short exposure first frame 522, and can fuse the image 514 captured using the longer exposure to produce the long exposure first frame 524. The second registration and single exposure fusion operation 520 can similarly fuse the image 516 captured using the shorter exposure to produce the short exposure second frame 526, and can fuse the image 516 captured using the longer exposure to produce the long exposure second frame 528. Note that "short" and "long" as used in this patent document refer to relative exposure, meaning that a "short" exposure involves a shorter exposure level than a "long" exposure. These terms do not assign any specific numerical values ​​to the exposure levels.

[0086] The first depth estimator 530 receives frames 522 and 526, and the second depth estimator 532 receives frames 524 and 528. Note that the first depth estimator 530 and the second depth estimator 532 may actually represent the same depth estimator that is simply reused for different sets of frames 522 and 526 or 524 and 528. Each depth estimator 530 and 532 operates to calculate a depth map of the scene 506 using the same exposure view from the image sensors 508 and 510. For example, each depth estimator 530 and 532 may determine the depth of each pixel based on its input fused image to create a depth map based on the determined depth. Each depth estimator 530 and 532 also operates to produce a fused image and an associated depth map of the fused image. For example, the first depth estimator 530 may generate and output a short single view 534 and a short depth map 536 based on the short exposure first frame 522 and the short exposure second frame 526. The short single view 534 is a unified image from the short exposure first frame 522 and the short exposure second frame 526 corresponding to the short depth map 536. Similarly, the second depth estimator 532 can generate and output a long single view 538 and a long depth map 540 according to the long exposure first frame 524 and the long exposure second frame 528. The long single view 538 is a unified image from the long exposure first frame 524 and the long exposure second frame 528 corresponding to the long depth map 540.

[0087] although Figure 5 An example of a method 500 for multi-exposure depth generation for image illumination adjustment is shown, but may be used for Figure 5 Various changes may be made. For example, method 500 may include using more than two image sensors. Furthermore, method 500 may include using more than two registration and single exposure fusion operations and more than two depth estimators.

[0088] Figure 6 An example method 600 for multi-frame depth fusion for image illumination adjustment according to the present disclosure is shown. For ease of explanation, Figure 6 The method 600 shown can be described as being Figure 1 However, the method 600 may be used in any suitable electronic device and in any suitable system.

[0089] Figure 6 The method 600 shown continues Figure 5 The process shown in FIG. 1 is to receive and process a short single view 534, a long single view 538, a short depth map 536, and a long depth map 540. Figure 6As shown, the short single view 534 and the long single view 538 are provided to the deghosting operation 602 and the exposure analysis 604. The deghosting operation 602 operates to identify motion regions in the input images 534 and 538 so that blending in these regions can be suppressed. This is useful because blending in motion regions of the image can result in the generation of ghosting artifacts. The deghosting operation 602 can be performed in any suitable manner.

[0090] In some embodiments of the deghosting operation 602, one view 534 or 538 is selected as a reference image frame, while the other view 538 or 534 is designated as a non-reference image frame. For images of the same exposure (such as all long images or all short images), without loss of generality, any image can be selected as a reference image frame. The output of the deghosting operation 602 is a motion map 606, which (in some embodiments) can have values ​​in the range [0, 1]. For example, a value of "0" can indicate motion, while a value of "1" can indicate no motion. Reference can be made below Figure 7A-7D Details of an example implementation of the deghosting operation 602 are found.

[0091] The exposure analysis 604 operates to generate a saturation map 608 using the short single view 534 and the long single view 538. The saturation map 608 indicates which portions of the short single view 534 and the long single view 538 are oversaturated or too bright. The output of the exposure analysis 604 is the saturation map 608, which (in some embodiments) may have values ​​in the range [0, 1]. For example, a value of "1" may indicate saturation, while a value of "0" may indicate unsaturation. The exposure analysis 604 may be performed using a variety of techniques. Reference may be made to the following. Figure 8 Details of an example implementation of exposure analysis 604 are found.

[0092] The blending map generation operation 610 receives the motion map 606 and the saturation map 608. The blending map generation operation 610 uses the motion map 606 and the saturation map 608 to generate a blending map 612. The blending map 612 represents an initial estimate of which portions of the input images 534 and 538 can be fused or blended and which portions of the input images 534 and 538 cannot be fused. Note that in the blending map generation, there may be differences in the noise levels in the images 534 and 538, such as blending a long image typically has lower noise than blending a short image. Therefore, it may be desirable to minimize the blending between the two images 534 and 538. In addition, the blending operation may take into account the motion map 606 and the saturation map 608, so the blending map 612 may be based on both maps 606 and 608. Reference may be made to the following Fig. 9 Details of an example implementation of the hybrid graph generation operation 610 are found.

[0093] The multi-frame (MF) depth map generation operation 614 receives the mixed map 612, the short depth map 536, and the long depth map 540. The MF depth map generation operation 614 operates to generate a multi-frame depth map 616 based on the mixed map 612, the short depth map 536, and the long depth map 540. For example, the depth map 616 may represent a combination of the short depth map 536 and the long depth map 540, where the combination is based on or controlled by the mixed map 612. Fig.10 Details of an example implementation of the depth map generation operation 614 are found.

[0094] although Figure 6 An example of a method 600 for multi-frame depth fusion for illumination adjustment of an image is shown, but the Figure 6 Various changes are made. For example, method 600 may involve using more than two input images and more than two depth maps.

[0095] Fig. 7A , Figure 7B , Figure 7C and Fig.7D An example method 700 for motion map generation for illumination adjustment of an image according to the present disclosure is shown. In some embodiments, the method 700 shown here is used by the deghosting operation 602 to generate a motion map 606. For ease of explanation, the method 700 shown in FIG. 7 can be described as being composed of Figure 1 However, the method 700 may be used in any suitable electronic device and in any suitable system. Figure 7A-7D In the discussion of , the YUV color system is used, where "Y" represents luminance information (brightness) and "U" and "V" represent chrominance information (color). However, any other suitable imaging data may be processed herein.

[0096] like Fig. 7A As shown, the method 700 for motion map generation herein includes a reference frame block operation 702 and a master block operation 704. The inputs of the reference frame block operation 702 include a Y non-reference frame 708 (meaning the illumination data of the non-reference image frame) and a Y reference frame 710 (meaning the illumination data of the reference image frame). The reference frame block operation 702 generates motion statistics 714 based on these inputs. Figure 7B An example implementation of a reference frame block operation 702 is shown. The inputs to the master block operation 704 include motion statistics 714, YUV frames 716, and tuning parameters 718. The YUV frames 716 represent complete image data for reference image frames and non-reference image frames, and the tuning parameters 718 represent parameters that can be set as needed or desired in any particular implementation. The master block operation 704 uses these inputs to generate a fusion / motion map 720, which can represent the motion map 606. Figure 7C One example implementation of main block operation 704 is shown.

[0097] like Figure 7B As shown, the reference frame block operation 702 includes a downscale operation 722, a downscale operation 726, a photometric correction operation 730, a difference operation 734, and a motion percentage estimation operation 740. The downscale operation 722 produces a Y non-reference downscaled frame 724, which represents a downscaled version of the Y non-reference frame 708. The downscale operation 726 produces a Y reference downscaled frame 728, which represents a downscaled version of the Y reference frame 710. The downscale operations 722 and 726 may use any suitable technique for downscaling image data, such as a nearest neighbor method.

[0098] The photometric correction operation 730 produces a Y non-reference downscaled correction frame 732, which represents a corrected version of the Y non-reference downscaled frame 724. For example, the photometric correction operation 730 may adjust the overall brightness level of the Y non-reference downscaled frame 724 to more closely match the overall brightness level of the Y reference downscaled frame 728. The difference operation 734 produces a Y difference downscaled frame 736, which represents the difference in illumination data between the Y reference downscaled frame 728 and the Y non-reference downscaled correction frame 732. The motion percentage estimation operation 740 generates the motion statistics 714 based on the Y difference downscaled frame 736 and the importance estimate 738 (which is based on the ISO level).

[0099] like Figure 7C As shown, the main block operation 704 includes an edge intensity filter 742, a YUV conversion operation 746, and a sub-main block operation 756. The edge intensity filter 742 operates to identify edges in the Y reference frame 710 and generates a normal map 744. The YUV conversion operation 746 generates a YUV reference frame 754 and a YUV non-reference frame 752 based on the Y reference frame 710, the UV reference frame 750 (which represents the chrominance data of the reference image frame), the Y non-reference frame 708, and the UV non-reference frame 748 (which represents the chrominance data of the non-reference image frame). The sub-main block operation 756 generates a fusion map 758 based on the motion statistics 714, the importance estimate 738, the normal map 744, the YUV reference frame 754, and the YUV non-reference frame 752. Fig.7D One example implementation of sub-master block operation 756 is shown.

[0100] like Fig.7DAs shown, the sub-primary block operation 756 includes a difference operation 760, a difference operation 766, an average / downscaling operation 774, a summation / capping operation 778, and a low pass filter (LPF) / division operation 782. The difference operation 760 generates a Y difference frame 764 by comparing each Y reference value 761 from the Y reference frame 710 with a corresponding Y value 762 from the Y non-reference frame 708. The difference operation 766 generates a U difference frame 772 by comparing each U reference value 767 from the UV reference frame 750 with a corresponding U value 768 from the UV non-reference frame 748. The difference operation 766 also generates a V difference frame 770 by comparing each V reference value 767 from the UV reference frame 750 with a corresponding V value 768 from the UV non-reference frame 748.

[0101] An averaging / downscaling operation 774 generates a Y difference downscaled frame 776 from the Y difference downscaled frame 764, such as by averaging the values ​​in the Y difference frame 764 or otherwise downscaling the Y difference frame 764. A summing / capping operation 778 generates a difference map 780 from the Y difference downscaled frame 776, the U difference frame 772, and the V difference frame 770. An LPF / division operation 782 generates a fusion map 758 using a scalar reference weight 784, a scalar multi-frame weight 786, the difference map 780, and the normal map 744. The scalar reference weight 784 and the scalar multi-frame weight 786 may be included in the tuning parameters 718.

[0102] Note that additional details regarding an example implementation of the deghosting operation can be found in U.S. Patent Application No. 16 / 278,543, filed on February 18, 2019, which is incorporated herein by reference in its entirety. However, any other suitable implementation may be used for the deghosting operation 602.

[0103] although Fig. 7A , Figure 7B , Figure 7C and Fig.7D An example of a method 700 for motion map generation for illumination adjustment of an image is shown, but may be used for Fig. 7A , Figure 7B , Figure 7C and Fig.7D Various changes may be made. For example, more than two image frames may undergo deghosting at the same time. In addition, the motion map may be generated in any other suitable manner.

[0104] Figure 8 An example method for exposure analysis 604 according to the present disclosure is shown. For ease of explanation, Figure 8 The exposure analysis 604 shown can be described as consisting of Figure 1The exposure analysis 604 may be used with any suitable electronic device and in any suitable system. Figure 8 The exposure analysis 604 shown details Figure 6 The exposure analysis 604 shown receives and processes the short single view 534 and the long single view 538 and outputs a saturation map 608 .

[0105] like Figure 8 As shown, one or more input images 802 (which may represent a short single view 534 and a long single view 538) are received. Each pixel in the one or more input images 802 is processed using a "max over RGB" function 804 to determine the maximum saturation between the R, G, and B values ​​of the pixel. In other words, the differences between the R value and the G value, the R value and the B value, and the G value and the B value of each pixel are calculated, and the maximum difference among the calculated differences is used as the saturation value of the pixel. The saturation level of each pixel is compared to a threshold value by a threshold function 806 to determine the exact saturation. In some cases, the threshold value may be determined based on the maximum saturation or the maximum change in adjacent pixels. The output of the threshold function 806 forms a saturation map 608, which indicates the corresponding saturation level of the pixel in each input image 802. For example, the saturation map 608 may have values ​​between 0 and 1, where a value of 1 indicates no saturation and a value of 0 indicates full saturation. The saturation map 608 here may represent Figure 6 Saturation map 608 in FIG.

[0106] although Figure 8 An example of exposure analysis 604 is shown for exposure analysis, but may be used for Figure 8 Various changes may be made. For example, the saturation map may be generated in any other suitable manner.

[0107] Fig. 9 An example method for hybrid graph generation 610 according to the present disclosure is shown. For ease of explanation, Fig. 9 The hybrid graph generation 610 shown can be described as being Figure 1 However, the hybrid map generator 610 may be used in any suitable electronic device and in any suitable system. Fig. 9 The hybrid graph generation 610 shown in FIG. Figure 6 The blend map generation 610 is shown receiving and processing the motion map 606 and the saturation map 608 and outputting a blend map 612 .

[0108] like Fig. 9As shown, the hybrid map generation method 610 here receives the motion map 606 (which can also be represented as the motion map 720) and the saturation map 608. In some embodiments, the motion map 606 and the saturation map 608 both have values ​​between 0 and 1. The multiplication operation 906 generates a value by multiplying the corresponding values ​​of each pixel from the motion map 606 and the saturation map 608. The inversion operation 908 calculates the inverse of the value generated by the multiplication operation 906. In this example, the inversion operation 908 includes subtracting each value generated by the multiplication operation 906 from the value 1. The resulting values ​​form the hybrid map 612. Here, the values ​​in the hybrid map 612 may include a first value (such as a value of 1) to indicate the position where the pixel value from the long depth map should be used, and a second value (such as a value of 0) to indicate the position where the pixel value from the short depth map should be used.

[0109] although Fig. 9 An example of a hybrid graph generation method 610 is shown, but may be used for Fig. 9 Various changes may be made. For example, the hybrid graph may be generated in any other suitable manner.

[0110] Fig.10 An example multi-frame depth map generation 614 for multi-frame depth map generation according to the present disclosure is shown. For ease of explanation, Fig.10 The multi-frame depth map generation 614 shown can be described as being Figure 1 However, the multi-frame depth map generation 614 may be used in any suitable electronic device and in any suitable system. Fig.10 The multi-frame depth map generation 614 shown in FIG. Figure 6 The multi-frame depth map generation 614 shown receives and processes the long depth map 540 , the short depth map 536 , and the mixed map 612 and outputs a MF depth map 616 .

[0111] like Fig.10 As shown, the multi-frame depth map generation 614 for multi-frame depth map generation receives the long depth map 540, the mixed map 612, and the short depth map 536. In some embodiments, the mixed map 612 contains values ​​between 0 and 1, and each of these values ​​controls whether a pixel from the long depth map 540 or a pixel from the short depth map 536 should be used. The multiplication operation 1008 multiplies the values ​​in the mixed map 612 with the corresponding values ​​in the long depth map 540. The inversion operation 1010 calculates the inverse of the values ​​in the mixed map 612, and in this example, the inversion operation 1010 includes subtracting each value in the mixed map 612 from the value 1. The multiplication operation 1012 multiplies the inverse value from the mixed map 612 with the corresponding value in the short depth map 536. The summation operation 1014 combines the values ​​output from the multiplication operations 1008 and 1012 to produce the values ​​in the multi-frame depth map 616.

[0112] although Fig.10 An example of a method for multi-frame depth map generation 614 is shown, but the Fig.10 Various changes may be made. For example, the multi-frame depth map may be generated in any other suitable manner.

[0113] Fig.11A , Fig. 11B , Fig. 11C and Fig.11D An example method 1100 for multi-frame illumination adjustment according to the present disclosure is shown. For ease of explanation, Fig.11A , Fig. 11B , Fig. 11C and Fig.11D The method shown can be described as consisting of Figure 1 However, the method can be used in any suitable electronic device and in any suitable system. Figure 6 or Fig.10 The process shown is receiving and processing the short single view 534, the long single view 538, the motion map 606 and the MF depth map 616.

[0114] like Fig.11A As shown, the method 1100 for multi-frame illumination adjustment receives the short single view 534, the long single view 538, and the multi-frame depth map 616 (which may be generated as described above). A depth-based illumination adjustment operation 1102 applies illumination adjustment to the short single view 534 using the multi-frame depth map 616, and a depth-based illumination adjustment operation 1106 applies illumination adjustment to the long single view 538 using the multi-frame depth map 616. Fig. 11B The operations of the depth-based lighting adjustment operations 1102, 1106 are discussed in more detail. The depth-based lighting adjustment operation 1102 generates a lighting-adjusted short single view 1108, and the depth-based lighting adjustment operation 1106 generates a lighting-adjusted long single view 1110.

[0115] The exposure analysis 1112 generates a post saturation map 1114 from the lighting adjusted short single view 1108 and the lighting adjusted long single view 1110. This may occur in the same or similar manner as discussed above with reference to the exposure analysis 604. A blend map generation operation 1116 generates a blend map 1118 by combining the post saturation map 1114 and the motion map 606. This may occur in the same or similar manner as discussed above with reference to the blend map generation operation 610. A blending and tone mapping operation 1120 generates a final lighting adjusted image 1122 using the lighting adjusted short single view 1108, the lighting adjusted long single view 1110, and the blend map 1118.

[0116] like Fig. 11B As shown, depth-based illumination adjustment operations 1102 and 1106 each include a gain map generation operation 1126 and a multiplication gain adjustment operation 1132. Gain map generation operation 1126 uses depth map 616 and illumination condition 1124 to generate gain map 1128. Illumination condition 1124 represents one or more simulated light sources to be used during the illumination adjustment operation. Fig. 11C The gain map generation is described in more detail. The lighting condition 1124 is a condition for changing an image using different lighting effects. The lighting condition 1124 may include, for example, the position of the simulated light source, the lighting direction of the simulated light source, and the light intensity of each color of the simulated light source. The multiplication gain adjustment operation 1132 generates a lighting-adjusted single view 1134 based on the single view 1130 (which may be represented as a short single view 534 or a long single view 538), which corresponds to the lighting-adjusted short single view 1108 based on the short depth lighting adjustment 1102 and the lighting-adjusted long single view 1110 based on the long depth lighting adjustment 1106.

[0117] like Fig. 11C As shown, the gain map generation operation 1126 includes a normal map generation operation 1136, a light vector map generation operation 1138, and a shading model operation 1144. The normal map generation operation 1136 generates a normal map 1140 based on the depth map 616 and the lighting conditions 1124. In some embodiments, for each (x, y) position in the image, the normal map 1140 includes the value of the surface unit normal vector 1146 at the position (nx, ny, nz), such as Fig.11D shown.

[0118] Light vector map generation operation 1138 generates a light vector map based on depth map 616 and lighting conditions 1124. The light source position from lighting conditions 1124 may be used to determine the light vector. In some embodiments, unit incident light vector 1148(l) may be determined as follows:

[0119] l(x,y,z)=(x,y,z)-light source position (1)

[0120] Shading model operation 1144 uses normal map 1140 and light vector map 1142 to generate gain map 1128. In some embodiments, the gain factor G for each point in gain map 1128 may be determined as follows:

[0121]

[0122] G=α+β r,g,b ·S (3)

[0123] Where S represents the shading coefficient, d represents the distance from the simulated light source to the specified point, represents the surface unit normal vector 1146 at the specified point, represents the unit incident light vector 1148 at the specified point, α represents a positive coefficient, and β represents the brightness of the simulated light source. The gain factor can be applied to the original image to generate the light-adjusted image 1122. In some embodiments, the light-adjusted image 1122 (I) can be determined as follows:

[0124]

[0125] although Fig.11A , Fig. 11B , Fig. 11C and Fig.11D An example of a method 1100 for multi-frame illumination adjustment is shown, but may be used for Fig.11A , Fig. 11B , Fig. 11C and Fig.11D Various changes may be made. For example, method 1100 may include using more than two input images.

[0126] Fig. 12A , Fig. 12B , Fig. 12C and Fig.12D 1126. For ease of explanation, Fig. 12A , Fig. 12B , Fig. 12C and Fig.12D The example technique shown can be described as consisting of Figure 1 The example techniques may be used with any suitable electronic device and in any suitable system.

[0127] like Fig. 12A , a point light source lighting adjustment effect 1202 is shown. Here, the point light source lighting adjustment effect 1202 uses a point light source 1214 to perform lighting adjustment on the original image I=[Ir, Ig, Ib]. The point light source lighting adjustment effect 1202 can also adjust light properties (such as color) based on the light intensity of each color channel. In some cases, this can be expressed as β=[βr, βg, βb], where β can be adjusted to add a color lighting adjustment effect. Here, the point light source lighting adjustment effect 1202 can provide a lighting effect on the image based on the position and color profile of the point light source 1214.

[0128] like Fig. 12BAs shown, a parallel light source effect 1204 is shown. Here, the parallel light source effect 1204 uses a simulated parallel light source 1216 to perform lighting adjustments on the original image I. The simulated parallel light source 1216 here can be used with constant light properties or variable light properties. For example, the color profile of each light source in the parallel light source 1216 can be different to provide a variable light effect.

[0129] exist Fig. 12C , a foreground incremental lighting effect 1206 is shown, which is a special case of the parallel light source effect 1204. For this lighting effect, the reference curve 1218 represents an example curve for a soft (γ=2) front lighting effect, while the foreground lighting curve 1220 represents a strong (γ=5) lighting effect. When the simulated light source is applied to the image, the gain used to form the lighting-adjusted image Inew can be determined as follows:

[0130] z = w(x,y) (5)

[0131] S = F(z) (6)

[0132]

[0133] G=α+ β·S (8)

[0134] I new =G·I (9)

[0135] Where the position of each pixel is defined by (x, y), z represents the depth at position (x, y), and F(z) represents a monotonically decreasing polynomial function as the depth increases.

[0136] exist Fig.12D , a background incremental lighting effect 1208 is shown, which is a special case of the parallel light source effect 1204. For this lighting effect, the reference curve 1222 represents an example curve for a soft (γ=2) back lighting effect, while the background lighting curve 1224 represents a strong (γ=5) lighting effect. When the simulated light source is applied to the image, the gain used to form the lighting-adjusted image Inew can be determined as follows:

[0137] z = w(x,y) (10)

[0138] S = F(z) (11)

[0139]

[0140] G=α+ β·S (13)

[0141] I new =G·I (14)

[0142] Where the position of each pixel is defined by (x, y), z represents the depth at the position (x, y), and F(z) represents a monotonically increasing polynomial function with increasing depth.

[0143] although Fig. 12A , Fig. 12B , Fig. 12C and Fig.12D A specific example of a technique for gain map generation is shown, but may be used for Fig. 12A , Fig. 12B , Fig. 12C and Fig.12D Various changes may be made. For example, the gain map may be generated for any other suitable simulated light source and any other suitable lighting adjustment operation.

[0144] Fig.13A and Fig. 13B 1 shows an example result of a light adjustment operation according to the present disclosure. Specifically, Fig.13A An example image 1302 obtained using conventional methods is shown. Fig. 13B An example image 1304 obtained using the method described in this patent document is shown.

[0145] from Fig.13A As can be seen in the image 1302, image 1302 is an indoor scene with two apples on a table in the foreground and structures visible through a window in the background. In image 1302, the foreground of the scene is darker, reducing the amount of detail visible in the image. Lighting adjustment increases the brightness of the foreground in the indoor light-adjusted image 1304. The details of the apples become more apparent, and the light shining on the table is clearer after the lighting adjustment effect is added.

[0146] although Fig.13A and Fig. 13B An example of the result of a lighting adjustment operation is shown, but Fig.13A and Fig. 13B Various changes may be made. For example, images of multiple scenes may be captured under different lighting conditions, and these figures do not limit the scope of the present disclosure. These figures are only intended to illustrate example benefits that may be obtained using the above-described techniques.

[0147] Fig.14 An example method 1400 for multi-camera illumination adjustment based on multi-frame depth of an image according to the present disclosure is shown. For ease of explanation, Fig.14 The method 1400 is described as using Figure 1 The electronic device 101 shown is used for execution. However, Fig.14 The illustrated method 1400 may be used with any other suitable electronic device and in any suitable system.

[0148] In operation 1405 , the electronic device 101 captures a plurality of first images and a plurality of second images. In some cases, the plurality of first images are captured by a first camera or other image sensor 180 , and the plurality of second images are captured by a second camera or other image sensor 180 .

[0149] In operation 1410, the electronic device 101 generates a short single view, a short depth map, a long single view, and a long depth map. The short single view, the short depth map, the long single view, and the long depth map are generated from the first image and the second image, such as by using the above-mentioned technology. As a specific example, the first images can be registered with each other, and the first images with common exposure can be fused to produce a first short frame and a first long frame. In addition, the second images can be registered with each other, and the second images with common exposure can be fused to produce a second short frame and a second long frame. In addition, the short depth map can be calculated using the first short frame and the second short frame, and another short depth map can be calculated using the first long frame and the second long frame.

[0150] In operation 1415, the electronic device 101 generates a multi-frame depth map 616, such as by using the above-described techniques. The multi-frame depth map 616 may be generated using the short single view 534, the short depth map 536, the long single view 538, and the long depth map 540. As a specific example, the motion map 606 may be generated, such as by using the deghosting operation 602 for the short single view 534 and the long single view 538. In addition, the saturation map 608 may be generated, such as by using the exposure analysis 1112 for the short single view 534 and the long single view 538. In addition, the hybrid map 612 may be generated, such as by combining the motion map 606 and the saturation map 608. Then, the multi-frame depth map 616 may be generated by applying the hybrid map 612 to each of the short depth map 536 and the long depth map 540.

[0151] In operation 1420, the electronic device 101 generates a gain map 1128, such as by using the above-described techniques. The gain map 1128 may be generated using the multi-frame depth map 616 and the lighting conditions 1124, which introduce lighting adjustment effects into the gain map 1128. The lighting conditions 1124 may include point light sources 1214 or parallel light sources 1216. The parallel light sources 1216 may be applied in the front in the image to produce a foreground brightening effect 1206, or in the back in the image to produce a background brightening effect 1208. As a specific example, the multi-frame depth map 616 and the lighting conditions 1124 may be used to generate a normal map 1140, wherein the normal map 1140 contains, for each location in the scene, the value of the surface normal 1146 at that location. In addition, a light vector map 1142 may be generated based on the multi-frame depth map 616 and the lighting conditions 1124, wherein the light vector map 1142 indicates, for each position in the scene, a light vector 1148 from the light source to the position. A shading model 1144 may be used to combine the normal map 1140 and the light vector map 1142 to generate a gain map 1128.

[0152] In operation 1425, the electronic device 101 generates a short single view 1108 adjusted for illumination and a long single view 1110 adjusted for illumination, such as by using the above-described techniques. For example, the short single view 1108 adjusted for illumination and the long single view 1110 adjusted for illumination can be generated by applying the gain map 1128 to the short single view 534 and the long single view 538.

[0153] In operation 1430, the electronic device 101 fuses the illumination-adjusted short single view 1108 and the illumination-adjusted long single view 1110 to form an illumination-adjusted image 1122. For example, the illumination-adjusted image 1122 may be generated by fusing the illumination-adjusted short single view 1108 and the illumination-adjusted long single view 1110 based on the additional blending map 1118. As a specific example, the post-saturation map 1114 may be generated using exposure analysis 1112 for the illumination-adjusted short single view 1108 and the illumination-adjusted long single view 1110. In addition, the additional blending map 1118 may be generated by combining the post-saturation map 1114 and the motion map 606. Then, the blending map 1118 may be applied to the illumination-adjusted short single view 1108 and the illumination-adjusted long single view 1110 using a blending and tone mapping operation 1120 to generate the illumination-adjusted image 1122. At this point, any final post-processing operations can occur.

[0154] although Fig.14 An example of a method 1400 for multi-camera illumination adjustment based on multi-frame depth for an image is shown, but may be used for Fig.14 For example, although shown as a series of steps, Fig.14 The steps in the process may overlap, occur in parallel, occur in a different order, or occur any number of times.

[0155] It should be noted that although the above describes the use of one or more devices to perform various operations, these operations can be implemented in any suitable manner. For example, each function in the electronic device 101 can be implemented or supported using one or more software applications or other software instructions executed by at least one processor 110 of the electronic device 101. In other embodiments, at least some of the functions in the electronic device 101 can be implemented or supported using dedicated hardware components. In general, the operations of the device can be performed using any suitable hardware or any suitable combination of hardware and software / firmware instructions.

[0156] Although the present disclosure has been described with reference to various exemplary embodiments, various changes and modifications may occur to those skilled in the art. The present disclosure is intended to encompass such changes and modifications as fall within the scope of the appended claims.

Claims

1. A method comprising: capturing a plurality of first images using different exposure values ​​of the first image sensor, and capturing a plurality of second images using different exposure values ​​of the second image sensor; generating a short single view, a short depth map, a long single view, and a long depth map based on the plurality of first images and the plurality of second images; Generate a multi-frame depth map using a short single view, a short depth map, a long single view, and a long depth map; Generating a gain map using the multi-frame depth map and the lighting condition, wherein the lighting condition introduces a lighting adjustment effect into the gain map; generating a lighting-adjusted short single view and a lighting-adjusted long single view by applying the gain map to the short single view and the long single view; and Generate an illumination-adjusted image by fusing the illumination-adjusted short single view and the illumination-adjusted long single view, wherein the short single view and the short depth map are generated by using short exposure frames among the plurality of first images and the plurality of second images, and The long single view and the long depth map are generated by using long exposure frames among the plurality of first images and the plurality of second images.

2. The method according to claim 1, further comprising: registering a plurality of first images; fusing the images of the plurality of first images having a common exposure value into a first short frame and a first long frame; registering a plurality of second images; fusing the plurality of second images having a common exposure value into a second short frame and a second long frame; generating a short single view according to the first short frame and the second short frame, and generating a long single view according to the first long frame and the second long frame; as well as A short depth map is calculated using the first short frame and the second short frame, and a long depth map is calculated using the first long frame and the second long frame.

3. The method according to claim 1, further comprising: Using a deghosting operation on the short single view and the long single view to generate a motion map; Exposure analysis is used for short and long single views to generate saturation maps; generating a hybrid map by combining the motion map and the saturation map; as well as A multi-frame depth map is generated by applying the hybrid map to each of the short depth map and the long depth map.

4. The method according to claim 1, further comprising: Generate a normal map using the multi-frame depth map and the lighting conditions, wherein the normal map contains, for each position in the scene, a value of a surface normal at the position; Generate a light vector map according to the multi-frame depth map and the lighting conditions, wherein the light vector map indicates, for each position in the scene, a light vector from a light source to the position; and Use a shading model to combine the normal map and the light vector map to generate a gain map.

5. The method according to claim 1, further comprising: generating a saturation map using exposure analysis for the illumination-adjusted short single view and the illumination-adjusted long single view; generating a hybrid map by combining the saturation map and the motion map; as well as Using blending and tone mapping operations, a light-adjusted image is generated by applying the blending map to the light-adjusted short single view and the light-adjusted long single view.

6. The method according to claim 1, wherein: The lighting conditions include point light source lighting adjustment, and The lighting adjusted image has light attributes applied from a single light source.

7. The method according to claim 1, wherein: The lighting conditions include parallel light source effects, and The lighting adjusted image has light attributes applied from a plurality of parallel light sources.

8. The method according to claim 7, wherein: The parallel light source is applied to produce a foreground brightening effect.

9. The method according to claim 7, wherein: The parallel light source is applied to produce a background brightening effect.

10. An electronic device comprising: a first image sensor configured to capture a plurality of first images using different exposure values ​​of the first image sensor; a second image sensor configured to capture a plurality of second images using different exposure values ​​of the second image sensor; as well as at least one processing device operably coupled to the first image sensor and the second image sensor, the at least one processing device configured to: generating a short single view, a short depth map, a long single view, and a long depth map based on the plurality of first images and the plurality of second images; Generate a multi-frame depth map using a short single view, a short depth map, a long single view, and a long depth map; Generating a gain map using the multi-frame depth map and the lighting condition, wherein the lighting condition introduces a lighting adjustment effect into the gain map; applying the gain map to the short single view and the long single view to generate a lighting-adjusted short single view and a lighting-adjusted long single view; and fusing the illumination-adjusted short single view and the illumination-adjusted long single view to generate an illumination-adjusted image, wherein the short single view and the short depth map are generated by using short exposure frames among the plurality of first images and the plurality of second images, and The long single view and the long depth map are generated by using long exposure frames among the plurality of first images and the plurality of second images.

11. The electronic device according to claim 10, wherein: The at least one processing device is further configured to: registering a plurality of first images; fusing the images of the plurality of first images having a common exposure value into a first short frame and a first long frame; registering the plurality of second images; fusing the plurality of second images having a common exposure value into a second short frame and a second long frame; generating a short single view according to the first short frame and the second short frame, and generating a long single view according to the first long frame and the second long frame; as well as A short depth map is calculated using the first short frame and the second short frame, and a long depth map is calculated using the first long frame and the second long frame.

12. The electronic device according to claim 10, wherein: The at least one processing device is further configured to: Using a deghosting operation on the short single view and the long single view to generate a motion map; Exposure analysis is used for short and long single views to generate saturation maps; combining the motion map and the saturation map to generate a hybrid map; as well as The hybrid map is applied to each of the short depth map and the long depth map to generate a multi-frame depth map.

13. The electronic device according to claim 10, wherein: The at least one processing device is further configured to: Generate a normal map using the multi-frame depth map and the lighting conditions, wherein the normal map contains, for each position in the scene, a value of a surface normal at the position; Generate a light vector map according to the multi-frame depth map and the lighting conditions, wherein the light vector map indicates, for each position in the scene, a light vector from a light source to the position; and Use a shading model to combine the normal map and the light vector map to generate a gain map.

14. The electronic device according to claim 10, wherein: The at least one processing device is further configured to: generating a saturation map using exposure analysis for the illumination-adjusted short single view and the illumination-adjusted long single view; combining the saturation map and the motion map to generate a hybrid map; as well as The blending map is applied to the illumination-adjusted short single view and the illumination-adjusted long single view using blending and tone mapping operations to generate a illumination-adjusted image.

15. The electronic device according to claim 10, wherein: The lighting conditions include point light source lighting adjustment, and The lighting adjusted image has light attributes applied from a single light source.

Images