Image processing method and apparatus

Abstract

Embodiments of the present application provide an image processing method and device, and relate to the technical field of terminals. The method comprises: acquiring a first image; determining a luminance information map corresponding to the first image; determining a luminance gain map according to the first image and the luminance information map; and performing dynamic range expansion on the first image by using the luminance gain map to obtain a target image. In this way, the electronic device can deduce the luminance value of each pixel point in the actual shooting scene according to the first image and the first parameter, obtain the luminance information map, and then perform dynamic range expansion on the first image based on the luminance gain between the luminance information map in the real scene and the first image, so as to obtain the target image that can reflect the real shooting scene.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to an image processing method and apparatus. Background Technology

[0002] With the popularization and development of the Internet, people's functional needs for electronic devices have become increasingly diversified. In some embodiments, many electronic devices can support taking pictures to obtain high dynamic range (HDR) images. HDR images can present rich color details and tonal gradations, which can better match the human eye's perception of real-world scenes.

[0003] Typically, electronic devices can cover the brightness range of the entire scene by exposing the same scene multiple times with different exposure levels. These images with different exposure levels are then combined into an HDR image, which the electronic device can then display on a screen.

[0004] However, given the limited dynamic range supported by electronic devices, HDR images displayed by these devices differ from the actual shooting environment. Summary of the Invention

[0005] This application provides an image processing method applied in the field of terminal technology to improve the image quality when displaying HDR images.

[0006] In a first aspect, embodiments of this application provide an image processing method applied to an electronic device. The method includes: upon receiving an operation to launch a camera application, displaying a first interface, the first interface including a shutter button; upon receiving an operation on the shutter button, acquiring a first image; determining a brightness information map corresponding to the first image, the brightness information map being composed of brightness values ​​corresponding to each pixel, the brightness information map being determined based on a first parameter and a first exposure map corresponding to the first image, the first parameter being used to characterize the average exposure when shooting at a preset brightness; determining a brightness gain map based on the first image and the brightness information map; and using the brightness gain map to dynamically expand the first image to obtain a target image.

[0007] In this way, the electronic device can deduce the brightness value of each pixel in the actual shooting scene based on the first image and the first parameters, obtain a brightness information map, and then dynamically expand the first image based on the brightness gain between the brightness information map in the real scene and the first image, so as to obtain a target image that can reflect the real shooting scene.

[0008] In one possible implementation, the first parameter is positively correlated with the preset brightness, the first parameter is positively correlated with the first exposure time of the camera in the test device, and the first parameter is positively correlated with the first sensitivity of the camera in the test device.

[0009] The preset brightness can be the calibrated brightness described in the embodiments of this application.

[0010] Understandably, a first parameter related to the preset brightness, first exposure time, and first sensitivity can be pre-calculated during the test, so that the electronic device can deduce the brightness value of the first image in the real shooting scene through the first parameter.

[0011] In one possible implementation, the first parameter is obtained based on the following formula:

[0012]

[0013] Where C is the first parameter, N is the pixel value of any point in the second exposure image, iso1 is the first sensitivity, expo1 is the first exposure time, nit1 is the preset brightness, and the second exposure image is obtained by taking a picture of a uniform lightbox calibrated to the preset brightness using the camera of the test equipment.

[0014] It can be understood that iso1*expo1*nit1 can be interpreted as the total exposure detected by the testing device when the brightness is nit1, and C can be interpreted as the average exposure. This allows the testing device to determine the primary parameter that reflects the average exposure of a pixel through brightness calibration.

[0015] In one possible implementation, determining the brightness information map corresponding to the first image includes: using a first parameter, the pixel value of each pixel in the first exposure image, the second exposure time of the camera in the electronic device, and the second sensitivity of the camera in the electronic device to determine the brightness information map.

[0016] In one possible implementation, the luminance value NImg(x,y) at the (x,y) coordinate position in the luminance information map is obtained based on the following formula:

[0017]

[0018] Where N(x,y) is the pixel value at position (x,y) in the first exposure image, iso2 is the second sensitivity, expo2 is the second exposure time, and C is the first parameter.

[0019] In this way, the electronic device can calculate the brightness value of any pixel in the real shooting scene based on the numbers C, N(x,y), expo2, and iso2, and then obtain a brightness information map with the brightness values ​​of all pixels.

[0020] In one possible implementation, determining the luminance gain map based on the first image and the luminance information map includes: linearizing the first image to obtain a linear map; and calculating the luminance gain between the linear map and the luminance information map to obtain the luminance gain map.

[0021] Since the first image is an RGB image, in order to facilitate the calculation of the brightness gain map, the first image can be linearized to obtain a linear map, and then the brightness gain between the images can be determined based on the fusion of the linear map and the brightness information map.

[0022] In one possible implementation, the method further includes: after receiving an operation to launch the gallery application, displaying a second interface, the second interface including: a first thumbnail; and performing dynamic range expansion on the first image using a luminance gain map to obtain a target image, including: after receiving an operation on the first thumbnail, performing dynamic range expansion on the first image using a luminance gain map to obtain the target image, and displaying the target image.

[0023] The second interface can be Figure 1B The interface shown can have the first thumbnail as... Figure 1B Thumbnail 105.

[0024] It is understood that the electronic device can perform the target image generation process while displaying the image. The embodiments of this application do not specifically limit the timing of acquiring the target image. For example, the electronic device can also generate the target image before displaying the target image.

[0025] Secondly, embodiments of this application provide an image processing apparatus, which may be an electronic device, a chip, or a chip system within an electronic device. The image processing apparatus may include a display unit and a processing unit. When the image processing apparatus is an electronic device, the display unit may be a display screen. The display unit is used to perform display steps to cause the electronic device to implement an image processing method described in the first aspect or any possible implementation of the first aspect. When the image processing apparatus is an electronic device, the processing unit may be a processor. The image processing apparatus may further include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit to cause the electronic device to implement an image processing method described in the first aspect or any possible implementation of the first aspect. When the image processing apparatus is a chip or a chip system within an electronic device, the processing unit may be a processor. The processing unit executes the instructions stored in the storage unit to cause the electronic device to implement an image processing method described in the first aspect or any possible implementation of the first aspect. The storage unit can be a storage unit within the chip (e.g., a register, cache, etc.) or a storage unit located outside the chip within the electronic device (e.g., a read-only memory, random access memory, etc.).

[0026] Specifically, upon receiving an operation to launch the camera application, the display unit displays a first interface, which includes a shutter button. Upon receiving an operation on the shutter button, the processing unit acquires a first image. The processing unit further determines a brightness information map corresponding to the first image. The brightness information map is composed of the brightness values ​​corresponding to each pixel. The brightness information map is determined based on a first parameter and a first exposure map corresponding to the first image. The first parameter represents the average exposure when shooting at a preset brightness. The processing unit further determines a brightness gain map based on the first image and the brightness information map. The processing unit further performs dynamic range expansion on the first image using the brightness gain map to obtain a target image.

[0027] Thirdly, embodiments of this application provide an electronic device, which includes: one or more processors and a memory; the memory is coupled to one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the electronic device to perform the methods described in the first aspect or any possible implementation of the first aspect.

[0028] Fourthly, embodiments of this application provide a computer-readable storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the methods described in the first aspect or any possible implementation thereof.

[0029] Fifthly, embodiments of this application provide a computer program product including a computer program. When the computer program product includes computer program code, when the computer program code is run on an electronic device, it causes the electronic device to perform the method described in the first aspect or any possible implementation of the first aspect.

[0030] Sixthly, this application provides a chip system applied to an electronic device. The chip system includes one or more processors, which are used to invoke computer instructions to cause the electronic device to perform the methods described in the first aspect or any possible implementation of the first aspect.

[0031] In one possible implementation, the chip system described above in this application further includes at least one memory storing instructions. The memory can be an internal storage unit of the chip system, such as a register or cache, or it can be a storage unit of the chip system itself (e.g., read-only memory, random access memory, etc.).

[0032] It should be understood that the second to sixth aspects of this application correspond to the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description

[0033] Figures 1A-1C A scenario diagram provided for an embodiment of this application;

[0034] Figure 2 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0035] Figure 3 A schematic diagram of the software structure of an electronic device provided in an embodiment of this application;

[0036] Figure 4 This is a schematic diagram illustrating the acquisition of calibration parameters and brightness information map as provided in an embodiment of this application.

[0037] Figure 5 This is a schematic diagram of an image processing method provided in an embodiment of this application;

[0038] Figure 6 A schematic diagram of a tone mapping process provided in an embodiment of this application;

[0039] Figure 7 A schematic diagram illustrating an image effect provided in an embodiment of this application;

[0040] Figure 8 A schematic diagram of module interaction for an image processing method provided in an embodiment of this application;

[0041] Figure 9 A flowchart illustrating another image processing method provided in an embodiment of this application;

[0042] Figure 10 This is a schematic diagram of the hardware structure of another electronic device provided in an embodiment of this application. Detailed Implementation

[0043] To facilitate a clear description of the technical solutions in the embodiments of this application, some terms and technologies involved in the embodiments of this application will be briefly introduced below:

[0044] 1. Exposure

[0045] Exposure refers to the amount of light received by a light-sensitive element (such as a camera's sensor or film) during shooting. Exposure is determined by three main factors: aperture, shutter speed (or exposure time), and ISO. These three factors together determine the final exposure.

[0046] For example, without changing the aperture size and ISO, the longer the exposure time, the greater the exposure; without changing the exposure time and ISO, the larger the aperture, the greater the exposure; without changing the aperture size and exposure time, the greater the ISO, the greater the exposure.

[0047] 2. ISO sensitivity

[0048] ISO describes the sensitivity of the camera's sensor to light, and determines the camera's exposure capability under different lighting conditions.

[0049] 3. Pixel value

[0050] A pixel value is the light intensity value (or simply light intensity value) recorded by a sensor under specific exposure conditions. The pixel value reflects the amount of light received by each pixel during shooting and ultimately determines the brightness and detail of the image. Pixel values ​​are typically integers, with the range depending on the sensor's bit depth. For example, for an 8-bit sensor, the pixel value ranges from 0 to 255; for a 12-bit sensor, the range is 0 to 4095.

[0051] During the shooting process, the amount of light received by the sensor undergoes an analog-to-digital converter (ADC) to convert the light signal into a digital signal, i.e., pixel value. These pixel values ​​constitute the final digital image.

[0052] The light intensity value is directly proportional to the brightness value (or brightness information), that is, the greater the light intensity, the greater the brightness.

[0053] 4. Linearization

[0054] Linearization typically refers to the process of converting an image from a gamma-coded non-linear color space back to a linear color space.

[0055] 5. Standard Dynamic Range (SDR) Images

[0056] SDR images refer to images with a traditional dynamic range, as opposed to high dynamic range (HDR) images. Compared to HDR images, SDR images have a narrower dynamic range and color gamut, making them suitable for most traditional display devices and content. Furthermore, SDR images can be generated by tone mapping HDR images.

[0057] 6. High dynamic range (HDR) images

[0058] HDR refers to images that can display more brightness levels and a wider color gamut than SDR images. HDR images typically range in brightness from very dark blacks to very bright whites, more realistically reflecting lighting conditions in nature.

[0059] HDR images can display higher brightness and deeper blacks, thus providing richer details and a more realistic visual experience.

[0060] From a color space perspective, HDR images typically use a wider color space, such as Rec.2020 (BT.2020) or DCI-P3, while traditional SDR images typically use the Rec.709 color space. A wider color space means that HDR images can display more colors and richer color details.

[0061] 7. Tone mapping

[0062] Tone mapping can be understood as the process of mapping colors from one dynamic range to another.

[0063] Understandably, since the color gamut of a monitor is smaller than that of the real world, tone mapping is needed to adjust the brightness and color of an image to ensure that the image is presented on a standard display device in a way that matches the original scene.

[0064] Tone mapping involves adjusting the dynamic range of an image, and includes two methods: global tone mapping and local tone mapping. Global tone mapping uses a constant scaling parameter, while local tone mapping allows the scaling parameter to vary with spatial location to better simulate the adaptive characteristics of the human visual system.

[0065] 8. Brightness gain map

[0066] A luminance gain map, also known as a gain map or gain information, is used to describe the brightness adjustment information of different regions in an image. For example, each pixel value in the luminance gain map represents the brightness gain of that region. The brightness gain, also called gain or gain value, is a coefficient or scaling factor for adjusting the brightness of an image.

[0067] Gain information is typically stored in floating-point format, representing the gain value in linear space.

[0068] 9. Normal frame image

[0069] Also known as a normal exposure image, N-frame image, N-frame, medium frame, or medium frame image, this is an image captured by a camera with an exposure of 0 EV. In other words, the exposure of a normal exposure image is 0 EV. Here, 0 EV is a relative value, not an absolute zero exposure. For example, exposure = exposure time * ISO. Assuming a normal exposure image is captured at ISO 200 and an exposure time of 50 milliseconds, the actual exposure corresponding to 0 EV is the product of 200 and 50 milliseconds.

[0070] 10. Short frame images

[0071] Also known as short frame, S-frame, or S-frame image, it is an image captured by a camera when the exposure is less than 0 EV.

[0072] 11. Long frame images

[0073] Also known as long frame, L-frame, or L-frame image, it is an image captured by a camera when the exposure is greater than 0 EV.

[0074] Normal frame images, short frame images, and long frame images can be understood as images with different levels of exposure. It is understandable that an electronic device can obtain the original image by image fusion of at least two of these three images.

[0075] 12. Other terms

[0076] In the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with substantially the same function and purpose. For example, "first chip" and "second chip" are used only to distinguish different chips and do not limit their order of execution. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.

[0077] It should be noted that, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0078] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, and c can be single or multiple.

[0079] 13. Electronic equipment

[0080] The electronic devices in this application embodiment may include handheld devices with image recognition capabilities, vehicle-mounted devices, etc. For example, some electronic devices are: mobile phones, tablet computers, PDAs, laptops, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving cars, wireless terminals in remote medical surgery, etc., and this application embodiment is not limited to these.

[0081] The electronic devices in the embodiments of this application may also be referred to as: terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent, or user device, etc.

[0082] Figures 1A-1C This is a schematic diagram of a scenario provided in an embodiment of this application. Figures 1A-1C The corresponding embodiments use a mobile phone as an example for illustration, but this example does not constitute a limitation on the embodiments of this application.

[0083] In response to a user launching the camera app, the electronic device can display something like... Figure 1A The interface shown. Figure 1A This can be referred to as a camera interface, which may display one or more of the following: a preview window 101, a camera button 102, a camera mode option bar 103, and a button 104 for viewing the captured image, etc.

[0084] Preview window 101 can be used to display a preview image. The preview image can be an image displayed after the electronic device has performed simple processing on the image captured in real time by the camera.

[0085] Figure 1A The preview image may include multiple subjects, such as building 20 and light source 10, which may be provided by at least one luminous object (such as a lantern).

[0086] The camera button 102 can be used to receive the user's shooting operation.

[0087] The camera mode option bar 103 can display at least one shooting mode button. For example, from left to right, the camera mode option bar can display: aperture mode button, night scene mode button, portrait mode button, photo mode button, video mode button, short video mode, and a button for viewing more modes, etc.

[0088] A selection indicator can be displayed below the camera mode button, indicating that the camera mode button is selected. Figure 1A It can also be called the shooting interface corresponding to the photo mode.

[0089] Button 104 allows you to view the last captured image obtained from the camera application.

[0090] In response to a user's click on the camera button 102, the electronic device can acquire at least two images with different exposure levels using the camera, and then fuse these two images to obtain a captured image. The difference in exposure level can manifest as a difference in exposure duration (e.g., different exposure times).

[0091] In response to the user opening the gallery app, the electronic device displays as follows: Figure 1B The interface shown is Figure 1B The screen can display thumbnail 105. In response to a user's click on thumbnail 105, the electronic device displays as follows: Figure 1C The interface shown is Figure 1C The image that can be displayed is the captured image 106 corresponding to the thumbnail 105, and the captured image 106 can be an HDR image.

[0092] It is understandable that the image fusion process described above produces an HDR image. However, electronic devices are limited by their dynamic range display capabilities when displaying HDR images, which may result in poor image quality that differs from the actual shooting environment. For example, when an electronic device shoots in a dark environment... Figure 1A When viewing the building shown, and opening an HDR image in the gallery app, the light sources in the HDR image are darker and do not match the actual brightness of the light sources.

[0093] In view of this, embodiments of this application provide an image processing method that enables an electronic device to acquire an original image after receiving a user's click on the camera button, calculate the brightness value of each pixel in the original image under the real shooting scene using preset calibration parameters, obtain a brightness information map, and then dynamically expand the original image by using the brightness gain between the brightness information and the original image to obtain the target image under the real shooting scene, thereby improving image quality.

[0094] The image processing method provided in this application can be used not only in the field of photography, but also in the fields of video recording, live streaming, and video, etc. This application does not limit it.

[0095] To better understand the embodiments of this application, the structure of the electronic device according to the embodiments of this application is described below. For example, Figure 2 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application.

[0096] The electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, an indicator 192, a camera 193, and a display screen 194, etc.

[0097] Optionally, the sensor module 180 may also include one or more of the following: proximity sensor, ambient light sensor, touch sensor, pressure sensor, gyroscope sensor, barometric pressure sensor, magnetic sensor, accelerometer, distance sensor, fingerprint sensor, temperature sensor, or bone conduction sensor, etc. Figure 2 (not shown in the text), and this application does not specifically limit this aspect in the embodiments.

[0098] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0099] The processor 110 may include one or more processing units, and the processor 110 may implement the processing steps in the image processing method provided in the embodiments of this application.

[0100] For example, the processor can start the camera 193 to capture the original image. It then calculates the luminance information map corresponding to the original image, and determines the luminance gain map based on the ratio between the luminance information map and the luminance map. This luminance gain map is then used to dynamically expand the original image to obtain the target image (or enhanced image).

[0101] USB interface 130 is an interface that conforms to the USB standard specification, specifically it can be a Mini USB interface, Micro USB interface, USB Type C interface, etc.

[0102] The charging management module 140 is used to receive charging input from the charger. The power management module 141 is used to connect the charging management module 140 and the processor 110.

[0103] The wireless communication function of electronic devices can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0104] Electronic devices utilize GPUs, displays (194), and application processors to achieve display functions. The GPU is a microprocessor for image processing, connecting the displays (194) and the application processor.

[0105] The display screen 194 is used to display images and videos, for example, the display screen 194 can be used to display a target image with enhanced brightness.

[0106] Electronic devices can achieve shooting functions through image signal processors (ISPs), cameras 193, video codecs, GPUs, displays 194, and application processors.

[0107] Camera 193 is used to capture still images or videos. Camera 193 may include a front-facing camera and / or a rear-facing camera. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, converting it into an image visible to the naked eye. In some embodiments, the ISP may be located in camera 193. Exemplarily, the electrical signal transmitted from the camera's photosensitive element to the ISP may be an image sequence, and the ISP may preprocess the image sequence (such as image preprocessing), etc. The image sequence after ISP processing may be a RAW format image.

[0108] The external storage interface 120 can be used to connect an external storage card.

[0109] Internal memory 121 can be used to store computer executable program code, which includes instructions.

[0110] Electronic devices can implement audio functions such as music playback and recording through audio modules 170, speakers 170A, receivers 170B, microphones 170C, headphone jacks 170D, and application processors.

[0111] Button 190 includes the power button, volume buttons, etc.

[0112] The software systems of electronic devices can adopt layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture, etc., which will not be elaborated here.

[0113] For example, Figure 3 This is a schematic diagram of the software structure of an electronic device provided in an embodiment of this application.

[0114] A layered architecture divides the system into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the system is divided into five layers, from top to bottom: application layer, application framework layer, hardware abstraction layer (HAL), driver layer, and hardware layer.

[0115] The application layer may include a series of application packages. In this embodiment, the application packages may include a camera application and a gallery application.

[0116] Camera apps are used to capture images using a webcam.

[0117] Gallery apps can be used to view photos taken with a camera; they may also be called camera apps.

[0118] The application framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer includes some predefined functions.

[0119] The application framework layer may include: camera access interface.

[0120] The camera access interface is used to provide camera applications with application programming interfaces and programming frameworks related to cameras. The camera access interface can include camera management and camera devices.

[0121] The hardware abstraction layer is an interface layer located between the application framework layer and the driver layer, providing a virtual hardware platform for the operating system. In this embodiment, the hardware abstraction layer may include: a camera hardware abstraction layer, a camera algorithm library, and an audio hardware abstraction layer.

[0122] The camera hardware abstraction layer can provide virtual hardware for camera device 1, camera device 2, or more camera devices. Camera devices may include the cameras described in the hardware layer.

[0123] The camera algorithm library may include runtime code and data for implementing the image processing methods provided in the embodiments of this application. For example, the camera algorithm library may include a brightness information map calculation module and a brightness gain map calculation module.

[0124] The brightness information graph calculation module is used to utilize the calibration system and exposure. Figure 2 The actual brightness information of each pixel in the original image is calculated to obtain a brightness information map. The brightness information map calculation module can execute steps S501 or S804.

[0125] The luminance gain map calculation module is used to calculate the luminance gain map using the luminance information map and the original map. For example, the luminance gain map calculation module can execute the steps in S502 or S805.

[0126] The driver layer is the layer between hardware and software. It includes drivers for various hardware components, such as camera drivers, digital signal processor drivers, and image processor drivers.

[0127] The camera driver is used to drive the camera to acquire images and to drive the image signal processor to preprocess the images. The digital signal processor driver is used to drive the data signal processor to process digital signals. The image processor driver is used to drive the image processor to process images.

[0128] The hardware layer may include: camera, image signal processor, digital signal processor and image processor, etc.

[0129] The following is in conjunction with the above. Figure 3 The software architecture described herein provides a detailed description of the image processing methods in the embodiments of this application:

[0130] In response to a user's action of launching the camera application, such as clicking the camera application icon, the camera application calls the camera access interface in the application framework layer to launch the camera application, and then sends a command to launch the camera by calling the camera device in the camera hardware abstraction layer. The camera hardware abstraction layer sends the command to launch the camera to the camera device driver in the kernel layer. Upon receiving the command, the camera device driver can launch the corresponding camera and acquire image light signals through the camera. One camera device in the camera hardware abstraction layer corresponds to one camera sensor in the hardware layer. Then, the camera sensor can transmit the acquired image light signals to the image signal processor for preprocessing (or image preprocessing) to obtain the image electrical signals (or raw image), and transmit the raw image to the camera hardware abstraction layer through the camera device driver.

[0131] The camera algorithm library stores program code that implements the image processing method provided in the embodiments of this application.

[0132] The camera algorithm library can also send images captured by the camera to the camera hardware abstraction layer. The camera hardware abstraction layer can then display these images.

[0133] It is understood that the embodiments of this application do not specifically limit the software layers involved in the software architecture, the modules contained in the software layers, and the functions of the modules.

[0134] Combination Figure 3The description of the software architecture in this application is followed by a detailed explanation of the technical solution and how it solves the aforementioned technical problems, using specific embodiments. These specific embodiments can be implemented independently or in combination. Similar or identical concepts or processes may not be described again in some embodiments.

[0135] It is understood that the image processing method provided in this application embodiment can be applied to the following three scenarios.

[0136] In Scenario 1, after receiving a user's click on the camera button, the electronic device can generate an original image (which can be an SDR or HDR image) and a corresponding brightness gain map, and then save both the original image and the brightness gain map. When the user views an image from a gallery application, the electronic device can use the brightness gain map to dynamically expand the original image to obtain a target image (i.e., an HDR image), and then display the target image. The original image can be obtained by fusing at least two frames from a long frame, a short frame, or a normal frame.

[0137] In scenario two, after receiving a user's click on the camera button, the electronic device can generate the original image and a corresponding brightness gain map. It then uses the brightness gain map to dynamically expand the original image to obtain the target image, which is then stored. The electronic device can display the target image when the user views an image from the gallery application.

[0138] In scenario three, the electronic device can generate the original image after receiving a user's click on the camera button. When the user views an image from the gallery app, the electronic device can generate a brightness gain map corresponding to the original image, use the brightness gain map to dynamically expand the original image to obtain the target image, and then display the target image.

[0139] In the descriptions of scenarios one through three, the method for obtaining the brightness gain map can be found in the description of S502 below. Furthermore, the timing of obtaining the brightness gain map and the target image is not limited in this embodiment.

[0140] The following is combined Figures 4-6 The corresponding embodiments provide a detailed description of the process of acquiring the brightness gain map and the target image.

[0141] Before executing the image processing method, the test equipment can pre-calculate calibration parameters, which can be stored in memory as an XML file.

[0142] like Figure 4As shown, the calibration parameters can be obtained as follows: The camera of the test equipment is used to photograph a uniform lightbox with a calibrated brightness of nit1, resulting in exposure image 1. When photographing a uniformly bright lightbox, the pixel values ​​at each point in exposure image 1 are the same and can be represented by N.

[0143] Given that the camera's exposure time is expo1 (or first exposure time) and sensitivity is iso1 (or first sensitivity), determine the calibration parameter C.

[0144] The calibration parameter C can be calculated using the following formula:

[0145]

[0146] Here, iso1*expo1*nit1 can be understood as the total exposure detected by the testing device when the brightness is nit1, and C can be understood as the average exposure.

[0147] This is understandable, since the aperture size of the camera generally remains constant during the shooting process, the effect of the aperture can be ignored when calculating the total exposure.

[0148] Next, the original image is acquired using the camera of an electronic device, and the exposure corresponding to the original image is determined. Figure 2 And the camera's exposure time expo2 and ISO sensitivity 2, based on exposure Figure 2 The pixel values, expo2, and iso2 at each pixel point are used to calculate the brightness information map associated with the original image. The specific process of calculating the brightness information map can be found in the description in S501.

[0149] It is understood that, in determining the calibration parameter C, the embodiments of this application do not limit the value of the calibration brightness. Furthermore, the testing device and the electronic device described in the embodiments of this application can be the same device or different devices.

[0150] like Figure 5 As shown, the process of obtaining the luminance information map and the luminance gain map can be as follows:

[0151] S501, Electronic equipment utilizes the exposure corresponding to the original image. Figure 2 And calibrate parameter C to determine the brightness information map.

[0152] The brightness information map can include the brightness information of each pixel (or brightness value, in nits).

[0153] For example, electronic devices can acquire exposure. Figure 2The brightness value of each pixel is calculated by taking the pixel value of each pixel, the camera's exposure time expo2 (or second exposure time) and sensitivity iso2 (or second sensitivity), and then obtaining the brightness information map.

[0154] The luminance information map NImg(x,y) can be obtained by the following formula:

[0155]

[0156] Here, N(x,y) can be understood as the pixel value of the pixel at the (x,y) coordinate position, and NImg(x,y) can be understood as the brightness information of the pixel at the (x,y) coordinate position in the brightness information map.

[0157] expo2 and iso2 can be the lens parameters of the camera in the electronic device, while expo1 and iso1 can be the lens parameters of the camera in the test device. It can be understood that when the electronic device and the test device are different devices, expo1 and expo2 can be different, and / or iso1 and iso2 can be different.

[0158] In this way, the electronic device can calculate the brightness value of any pixel in the real shooting scene based on the numbers C, N(x,y), expo2, and iso2, and then obtain a brightness information map with the brightness values ​​of all pixels.

[0159] S502, The electronic device fuses the original image and the brightness information image to obtain a brightness gain image.

[0160] For example, an electronic device can convert the original image into a linear image (i.e., perform linearization on the original image), perform fusion calculations on the linear image and the brightness information image to obtain an initial brightness gain image, and then perform log2 encoding, normalization, and image format conversion on the initial brightness gain image to obtain the brightness gain image.

[0161] The initial luminance gain map GN can be obtained using the following formula:

[0162]

[0163] Here, LN(x,y) can be understood as the pixel value at position (x,y) in the linear graph, and offset can be a preset positive number. Setting the offset can prevent the denominator from being 0 during the calculation of GN.

[0164] The process of performing log2 encoding on GN to obtain GN' can be represented by the following formula:

[0165] GN' = log2(GN)

[0166] The process of normalizing GN' to 0 to 1 can be represented by the following formula:

[0167]

[0168] Among them, GN ′ .min() can be understood as the minimum value in GN, where GN is the minimum value. ′ `.max()` can be understood as the maximum value in GN. The process of converting GN' to an 8-bit luminance gain map can be represented by the following formula:

[0169] GMap = GN' * 255

[0170] Here, GMap can be understood as a luminance gain map. In this case, the value of any pixel in the luminance gain map can be any value between 0 and 255.

[0171] It is understandable that electronic devices can determine the brightness gain of each pixel by comparing the ratio between the brightness information map and the linear graph, thus obtaining a brightness gain map.

[0172] S503. The electronic device uses the brightness gain map to dynamically expand the original image to obtain the target image.

[0173] Since the original image is a three-channel RGB image, and the values ​​of each pixel in the brightness gain image are within the range of 0-255, the electronic device can map the brightness gain image to a fixed range, such as [1, m], using the tone mapping function F(x). Then, the electronic device can use the mapped range [1, m] to expand the dynamic range of each of the three channels in the original image. Here, [1, m] can be understood as the range of brightening factors, and m can take values ​​such as 2 or 5.

[0174] like Figure 6 As shown, F(x) can map the value range [0, 255] to [1, m]. For example, if the three channel values ​​at position (i, j) in the original image are red (R), green (G), and blue (B), and the gain value at position (i, j) in the brightness gain map is GMap[i][j], then through the mapping of F(x), it can be determined that the three channel values ​​at position (i, j) in the target image are R*F(GMap[i][j]), G*F(GMap[i][j]), and B*F(GMap[i][j]).

[0175] For example, for an 8-bit image, let n = 2, F(x) = 1 + 1 / 255 * x. Then the value of each pixel in the target image can be calculated using the following formula:

[0176] R[i][j]new=R[i][j]ori*(1+GMap[i][j] / 255)

[0177] G[i][j]new=G[i][j]oir*(1+GMap[i][j] / 255)

[0178] B[i][j]new=B[i][j]ori*(1+GMap[i][j] / 255)

[0179] Here, R[i][j]new can be understood as the value of the R channel at position (i, j) in the target image, G[i][j]new can be understood as the value of the G channel at position (i, j) in the target image, and B[i][j]new can be understood as the value of the B channel at position (i, j) in the target image. R[i][j]ori can be understood as the value of the R channel at position (i, j) in the original image, G[i][j]ori can be understood as the value of the G channel at position (i, j) in the original image, and B[i][j]ori can be understood as the value of the B channel at position (i, j) in the original image.

[0180] In combination with the above Figures 4-6 The description in the text is as follows, combined with the following: Figure 7 The image effects during the execution of the image processing method are illustrated. Figure 7 A can be a schematic representation of a primitive diagram. Figure 7 B can be a representation of a linear graph. Figure 7 C can be a schematic diagram of a brightness gain map. Figure 7 D can be a representation of a target image. That is, the electronic device can acquire... Figure 7 Process A is Figure 7 B, yes Figure 7 B is obtained by fusing it with the luminance information map. Figure 7 C, to Figure 7 C and Figure 7 A is fused to obtain Figure 7 D.

[0181] Figure 7 A and Figure 7 D can be an RGB image. Compared to Figure 7 The darker areas in section A (such as lanterns and shop signs) Figure 7 The bright areas in D are enhanced, the brightness changes are more obvious, and the enhancement effect is similar to that of the actual shooting scene.

[0182] The following example, using scenario one, illustrates the interaction process between modules in the image processing method. This example does not constitute a limitation on the embodiments of this application.

[0183] Figure 8 This is a schematic diagram illustrating the module interaction of an image processing method provided in an embodiment of this application. Figure 8 In a corresponding embodiment, the electronic device may include: a gallery application, a camera application, a camera HAL, a camera algorithm library, and a camera device driver. The function of each module can be found in [reference needed]. Figure 3 The description in the text will not be repeated here.

[0184] like Figure 8 As shown, the image processing method may include the following steps:

[0185] S801. After receiving a user's click on the shutter button, the camera application sends a command to the camera device driver to start the camera via the camera HAL.

[0186] For example, the specific process by which a camera application sends a command to the camera device driver to start the camera through modules such as the camera HAL can be found in [reference needed]. Figure 3 The description in the text will not be repeated here.

[0187] S802, the camera device driver acquires the original image through the camera.

[0188] The S803 camera algorithm library obtains the original image from the camera device driver.

[0189] Optionally, the camera driver can also drive the image signal processor to perform image preprocessing on the original image, obtaining a preprocessed original image, and then pass the preprocessed original image to the camera algorithm library to execute subsequent image processing. This image preprocessing improves the image quality of multiple frames. Image preprocessing can include one or more of the following: dead pixel correction, RAW domain noise reduction, black level correction, optical shadow correction / automatic white balance, or gamma correction, etc.

[0190] S804, the camera algorithm utilizes calibration parameters and the exposure corresponding to the original image. Figure 2 Calculate the brightness information map corresponding to the original image.

[0191] The camera algorithm library can obtain calibration parameters by reading and parsing XML files. The process of calculating the brightness information map by the camera algorithm library can be found in the description in S501.

[0192] The S805 camera algorithm library converts the original image into a linear image, and performs image fusion on the linear image and the brightness information image to obtain a brightness gain image.

[0193] The process of calculating the brightness gain map using the camera algorithm library can be found in the description in S502.

[0194] Optionally, the S806 and the gallery application can obtain the original image and the brightness gain image from the camera algorithm library.

[0195] In S806, the camera algorithm library can save the original image and brightness gain map as image data (or image files), such as storing them in the image decoding module. The image decoding module can be set up in the gallery application for its use; alternatively, it can be located elsewhere, allowing the gallery application to decode the original image and brightness gain map from the image data by calling the image decoding module.

[0196] S807. In response to a user's operation of viewing an image in the gallery application, the gallery application uses a brightness gain map to dynamically expand the original image to obtain the target image.

[0197] The process of calculating the target image for the image library application can be found in the description in S503.

[0198] S808, the gallery application displays the target image on the monitor.

[0199] Based on this, embodiments of this application can determine the display brightness of each pixel in an image in a real scene using a luminance information map, and then expand the dynamic range of the original image using a luminance gain map related to the luminance information map. By applying different gains to different brightness areas, details in dark areas can be made clearer while preserving details in bright areas. Through local gain adjustment, the overall brightness of the original image can be enhanced while retaining details and textures, achieving fine-grained control over the dynamic range. This allows the target image displayed on the electronic device to present an effect similar to an HDR image.

[0200] In the above Figures 4-8 Based on the corresponding embodiments, Figure 9 This is a schematic flowchart illustrating an image processing method provided in an embodiment of this application. Figure 9 As shown, the image processing method may include the following steps:

[0201] S901. Upon receiving an operation to launch the camera application, the electronic device displays the first interface.

[0202] The first screen includes a camera button. The first screen can be used for... Figure 1A The interface shown.

[0203] S902. After receiving an operation on the camera button, the electronic device acquires the first image.

[0204] The first image can be the original image described in the embodiments of this application.

[0205] S903, The electronic device determines the brightness information map corresponding to the first image.

[0206] The brightness information map is composed of the brightness value corresponding to each pixel. The brightness information map is determined based on the first parameter and the first exposure map corresponding to the first image. The first parameter is used to characterize the average exposure when shooting under the preset brightness.

[0207] The first parameter can be the calibration parameter described in the embodiments of this application. The process of determining the calibration parameter can be found in [reference needed]. Figure 4 The process for determining the luminance infographic is described in section S501.

[0208] S904. The electronic device determines the brightness gain map based on the first image and the brightness information map.

[0209] The process of determining the luminance gain map based on the first image and the luminance information map can be found in the description in S502.

[0210] S905. The electronic device uses a brightness gain map to dynamically expand the first image to obtain the target image.

[0211] The process of using the brightness gain map to extend the dynamic range of the first image can be found in the description in S503.

[0212] Based on this, the electronic device can deduce the brightness value of each pixel in the actual shooting scene from the first image and the first parameters, obtain a brightness information map, and then dynamically expand the first image based on the brightness gain between the brightness information map in the real scene and the first image, thus obtaining a target image that can reflect the real shooting scene.

[0213] It should be noted that the interface described in the embodiments of this application is only an example and does not constitute a limitation on the embodiments of this application.

[0214] It should be noted that the module names involved in the embodiments of this application can all be defined as other names, as long as they can achieve the function of each module, and no specific restrictions are placed on the module names.

[0215] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0216] The image processing method of the present application embodiments has been described above. The apparatus for performing the above method provided in the present application embodiments is described below. Those skilled in the art will understand that the methods and apparatus can be combined with and referenced by each other, and the related apparatus provided in the present application embodiments can perform the steps in the above list sorting method.

[0217] Figure 10 This is a schematic diagram of the hardware structure of another electronic device provided in an embodiment of this application.

[0218] The electronic device includes a processor 1001, a communication line 1004, and at least one communication interface. Figure 10 (The example described uses communication interface 1003 as an example).

[0219] The processor 1001 may be a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application.

[0220] The communication line 1004 may include circuitry for transmitting information between the aforementioned components.

[0221] Communication interface 1003 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, wireless local area networks (WLAN), etc.

[0222] Possibly, the electronic device may also include a memory 1002.

[0223] The memory 1002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited to these. The memory may exist independently and be connected to the processor via communication line 1004. The memory may also be integrated with the processor.

[0224] The memory 1002 stores computer execution instructions for implementing the scheme of this application, and the processor 1001 controls the execution. The processor 1001 executes the computer execution instructions stored in the memory 1002 to implement the method provided in the embodiments of this application.

[0225] It is possible that the computer execution instructions in the embodiments of this application may also be referred to as application code, and the embodiments of this application do not specifically limit this.

[0226] In a specific implementation, as one example, the processor 1001 may include one or more CPUs, for example... Figure 10 CPU0 and CPU1 in the CPU.

[0227] In a specific implementation, as one example, an electronic device may include multiple processors, for example... Figure 10 Processors 1001 and 1005 are mentioned. Each of these processors can be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor here can refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0228] The image processing method provided in this application can be applied to electronic devices with communication functions. The electronic devices include terminal devices, and the specific device form of the terminal devices can be referred to the above-described related descriptions, which will not be repeated here.

[0229] This application provides a terminal device, which includes a processor and a memory; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, causing the terminal device to perform the above-described method.

[0230] This application provides a chip. The chip includes a processor, which is used to call a computer program in memory to execute the technical solutions in the above embodiments. Its implementation principle and technical effects are similar to those in the related embodiments described above, and will not be repeated here.

[0231] This application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, it implements the methods described above. The methods described in the above embodiments can be implemented wholly or partially by software, hardware, firmware, or any combination thereof. If implemented in software, the functionality can be stored as one or more instructions or code on or transmitted over the computer-readable medium. The computer-readable medium can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium accessible by a computer.

[0232] In one possible implementation, a computer-readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage or other magnetic storage devices, or any other medium targeted to carry or to store the required program code in the form of instructions or data structures, and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used herein, disks and optical discs include optical discs, laser discs, optical discs, Digital Versatile Discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs optically reproduce data using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0233] This application provides a computer program product, which includes a computer program that, when run, causes the computer to perform the above-described method.

[0234] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing device, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.

[0235] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that, Applied to electronic devices, the method includes: Upon receiving an instruction to launch the camera application, a first interface is displayed, which includes a shutter button. Upon receiving an operation on the camera button, acquire the first image; A brightness information map corresponding to the first image is determined. The brightness information map is composed of the brightness value corresponding to each pixel. The brightness information map is determined based on a first parameter and a first exposure map corresponding to the first image. The first parameter is used to characterize the average exposure when shooting under a preset brightness. A brightness gain map is determined based on the first image and the brightness information map; The target image is obtained by dynamically expanding the first image using the brightness gain map; The first parameter is positively correlated with the preset brightness, the first parameter is positively correlated with the first exposure time of the camera in the test device, and the first sensitivity of the camera in the test device.

2. The method according to claim 1, characterized in that, The first parameter is obtained based on the following formula: Where C is the first parameter, and N is the pixel value of any point in the second exposure image. This is the first photosensitivity. The first exposure time is nit1, the preset brightness is nit1, and the second exposure image is obtained by taking a picture of a uniform lightbox calibrated to the preset brightness using the camera of the test equipment.

3. The method according to claim 1 or 2, characterized in that, Determining the brightness information map corresponding to the first image includes: The brightness information map is determined using the first parameter, the pixel value of each pixel in the first exposure image, the second exposure time of the camera in the electronic device, and the second sensitivity of the camera in the electronic device.

4. The method according to claim 3, characterized in that, The brightness value at the (x,y) coordinate position in the brightness information map It is obtained based on the following formula: in, Let be the pixel value at position (x, y) in the first exposed image. This is the second sensitivity. C is the second exposure time, and C is the first parameter.

5. The method according to claim 1, characterized in that, Determining the brightness gain map based on the first image and the brightness information map includes: The first image is linearized to obtain a linear graph; The luminance gain between the linear graph and the luminance information graph is calculated to obtain the luminance gain graph.

6. The method according to claim 1, characterized in that, The method further includes: after receiving the operation to launch the gallery application, displaying a second interface, the second interface including: a first thumbnail; The step of dynamically expanding the first image using the brightness gain map to obtain the target image includes: after receiving an operation on the first thumbnail, dynamically expanding the first image using the brightness gain map to obtain the target image, and displaying the target image.

7. An electronic device, characterized in that, The electronic device includes: a camera, a display, one or more processors, and a memory; The camera is used to acquire image data, the display is used to perform the interface display step, the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 6.

8. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes computer program code that, when run on an electronic device, causes the electronic device to perform the method as described in any one of claims 1 to 6.

Images