Image processing method and device

Abstract

The embodiment of the invention provides an image processing method and device, and relates to the technical field of terminals. The method comprises the steps that after an operation for a photographing button is received, a first HDR image is acquired, and the first HDR image corresponds to a first scene; processing the first HDR image into a first SDR image, and determining a first gain graph based on a dynamic range difference between the first HDR image and the first SDR image; and after an operation for the first thumbnail in the second interface is received, a second HDR image is displayed, and the second HDR image is an image obtained by performing dynamic range expansion on the first SDR image based on the first gain graph and a gain value corresponding to the first scene. Therefore, the electronic equipment can determine the first scene through scene recognition, and performs dynamic range expansion on the first SDR image in the current shooting scene in a targeted manner based on the gain value related to the first scene, so that the picture quality of the generated second HDR image is matched with the shooting scene.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to an image processing method and apparatus. Background Technology

[0002] With the popularization and development of the Internet, people's functional needs for electronic devices have become increasingly diversified. In some embodiments, many electronic devices can support taking pictures to obtain high dynamic range (HDR) images. HDR images can present rich color details and tonal gradations, which can better match the human eye's perception of real-world scenes.

[0003] Typically, electronic devices can cover the entire brightness range of a scene by taking multiple exposures of the same scene with different exposure levels, and then combine these images with different exposure levels into a single HDR image. However, when displaying HDR images, electronic devices may encounter problems such as the image being too dark or too bright. Summary of the Invention

[0004] This application provides an image processing method, applied in the field of terminal technology, for improving the image quality when displaying HDR images.

[0005] In a first aspect, embodiments of this application provide an image processing method applied to an electronic device. The method includes: upon receiving an operation to launch a camera application, displaying a first interface, the first interface including a shutter button; upon receiving an operation on the shutter button, acquiring a first HDR image, the first HDR image corresponding to a first scene, the first scene being used to characterize the category to which the shooting scene belongs; processing the first HDR image into a first SDR image, and determining a first gain map based on the dynamic range difference between the first HDR image and the first SDR; upon receiving an operation to launch a gallery application, displaying a second interface, the second interface including a first thumbnail; upon receiving an operation on the first thumbnail, displaying a second HDR image, the second HDR image being an image obtained by dynamically expanding the first SDR image based on the first gain map and the gain value corresponding to the first scene.

[0006] Electronic devices can identify a first scene through scene recognition, and based on the gain value associated with the first scene, perform dynamic range expansion on the first SDR image in the current shooting scene in a targeted manner, so that the image quality of the generated second HDR image matches the shooting scene and meets the user's shooting needs.

[0007] In one possible implementation, the first HDR image includes at least one photographed object, and the method further includes: determining a first region and a second region, wherein the first region is determined by the first photographed object among the at least one photographed object, and the second region is a region in the first HDR image other than the first region; processing the first HDR image into a first SDR image, including: performing different tone mapping processes on the first region and the second region to obtain a tone-mapped first region and a tone-mapped second region; and fusing the tone-mapped first region and the tone-mapped second region to obtain the first SDR image.

[0008] The first region can be the key region described in the embodiments of this application, and the second region can be other regions described in the embodiments of this application. The process of performing different tone mapping processes on the first region and the second region can be found in the description in S407.

[0009] Compared to applying the same tone mapping to the first HDR image, electronic devices can apply different degrees of tone mapping to different regions of the first HDR image, allowing the dynamic range compression process to vary with different regions, thus achieving refined tone mapping processing.

[0010] In one possible implementation, different tone mapping processes are performed on the first region and the second region, including: performing tone mapping processing on the first region using a first scaling parameter, and performing tone mapping processing on the second region using a second scaling parameter, wherein the first scaling parameter is greater than the second scaling parameter.

[0011] By applying tone mapping to different regions using varying scaling parameters, the dynamic range and color gamut of the main subject region (i.e., the first region) can be preserved as much as possible. Tone mapping with larger scaling parameters can enhance the difference in dynamic range between the first HDR image and the first SDR image, allowing subsequent electronic devices to obtain a dynamic range more relevant to the subject during gainmap calculations.

[0012] In one possible implementation, determining the first region includes: when the first subject meets preset conditions, determining a first region containing the first subject, wherein the preset conditions include one or more of the following: the first subject is located at the focus position of the first HDR image, the first subject occupies the largest area among all subjects in the first HDR image, or the first subject is located in the foreground of the first HDR image.

[0013] Electronic devices can identify a first subject among at least one subjects, use the first subject as the first region in the image, and perform tone mapping on the first region and the second region to ensure that the processed image better matches the user's viewing experience of the main subject and other regions in the image.

[0014] In one possible implementation, at least one subject includes one or more of the following: people, skin color, blue sky, vegetation, water, beach, buildings, roads, vehicles, pets, sky, food, or flowers.

[0015] Electronic devices can recognize various shooting scenarios and process images from different scenarios accordingly.

[0016] In one possible implementation, processing the first HDR image into a first SDR image includes: processing the first HDR image into a first SDR image using a preset model, wherein the preset model is used to perform tone mapping processing on the first HDR image based on brightness information and / or color temperature information.

[0017] The preset module can be the display environment model described in the embodiments of this application, the brightness information can be the Display_Luminance described in the embodiments of this application, and the color temperature information can be the Display_Color_Temperature described in the embodiments of this application.

[0018] By using real-world environment models, electronic devices can utilize information such as brightness and color temperature to perform more refined tone mapping processing, thereby obtaining SDR images that better match the brightness and color temperature of the display environment.

[0019] In one possible implementation, the gain value corresponding to the first scene may include one or more of the following: a first gain value related to the illuminance information of the first scene, a second gain value related to the contrast information of the first scene, or a third gain value related to the color information of the first scene.

[0020] The illuminance information can be AE_Lux_Index as described in the embodiments of this application, the contrast information can be Scene_Contrast as described in the embodiments of this application, and the color information can be Color_Saturation_Info as described in the embodiments of this application.

[0021] The first gain value can be gain 1 as described in the embodiments of this application, the second gain value can be gain 2 as described in the embodiments of this application, and the third gain value can be gain 3 as described in the embodiments of this application.

[0022] The embodiments of this application can take into account different situations in the scenario during the gainmap calculation process, so as to improve the precision of the gainmap calculation.

[0023] In one possible implementation, the method further includes: adjusting the first gain map using at least one gain value among the gain values ​​corresponding to the first scene to obtain a second gain map; the second HDR image is an image obtained by dynamically expanding the first SDR image based on the first gain map and the gain values ​​corresponding to the first scene, including: the second HDR image is an image obtained by dynamically expanding the first SDR image based on the second gain map.

[0024] This application embodiment can perform more refined gain map calculations by using the gain value corresponding to the first scene. Different gain map calculation methods are also used for highlight areas, low-light areas, and different scene semantic content within the same scene. This enables more refined image adjustment, such as image processing based on scene awareness.

[0025] In one possible implementation, the first scene includes one or more of the following: portrait, night scene, flowers, food, landscape, pet, sports, sunset or sunrise, blue sky, snow scene, beach or fireworks.

[0026] Electronic devices can identify a variety of subjects and perform different levels of image processing on images containing different subjects.

[0027] In one possible implementation, after receiving the operation to launch the camera application, the method further includes: acquiring a first image sequence; performing scene recognition on the preview images in the first image sequence to obtain scene recognition results, wherein the scene recognition results include: a first scene, which is obtained by performing scene recognition on the preview images at the time of taking the picture, and the time of taking the picture is the time when the shutter button is received.

[0028] The scene recognition results may include the scene classification information described in the embodiments of this application.

[0029] The first image sequence can be the image sequence 1 described in the embodiments of this application.

[0030] In this way, electronic devices can perform scene recognition based on the preview image after launching the camera application, and the scene recognition process does not affect the generation of subsequent HDR or SDR images.

[0031] In one possible implementation, obtaining the first HDR image includes: obtaining a second image sequence, the second image sequence including one or more of the following: long frame images, short frame images, or short frame images; and performing image fusion on at least two frames in the second image sequence to obtain the first HDR image.

[0032] The second image sequence can be a cached image at the moment of taking the picture, as described in the embodiments of this application.

[0033] Electronic devices can obtain HDR images by fusing at least two frames of images with different exposure levels.

[0034] In one possible implementation, the electronic device includes: a camera application, a camera algorithm library, and a gallery application; upon receiving an operation on the shutter button, the method further includes: the camera application sending a message to the camera algorithm library indicating that shutter action should be initiated; acquiring a first HDR image, including: in response to the message indicating that shutter action should be initiated, the camera algorithm library acquiring the first HDR image; upon receiving an operation on the first thumbnail, the method further includes: the gallery application generating a second HDR image.

[0035] If the camera algorithm library has computing power, then electronic devices can perform steps that require high computing power, such as generating HDR images, within the camera algorithm library.

[0036] Secondly, embodiments of this application provide an image processing apparatus, which may be an electronic device, a chip, or a chip system within an electronic device. The image processing apparatus may include a display unit and a processing unit. When the image processing apparatus is an electronic device, the display unit may be a display screen. The display unit is used to perform display steps to cause the electronic device to implement an image processing method described in the first aspect or any possible implementation of the first aspect. When the image processing apparatus is an electronic device, the processing unit may be a processor. The image processing apparatus may further include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit to cause the electronic device to implement an image processing method described in the first aspect or any possible implementation of the first aspect. When the image processing apparatus is a chip or a chip system within an electronic device, the processing unit may be a processor. The processing unit executes the instructions stored in the storage unit to cause the electronic device to implement an image processing method described in the first aspect or any possible implementation of the first aspect. The storage unit can be a storage unit within the chip (e.g., a register, cache, etc.) or a storage unit located outside the chip within the electronic device (e.g., a read-only memory, random access memory, etc.).

[0037] Specifically, when the processing unit receives an operation to launch the camera application, the display unit displays a first interface, which includes a photo button.

[0038] The processing unit is further configured to: upon receiving an operation on the camera button, acquire a first HDR image, the first HDR image corresponding to a first scene, the first scene being used to characterize the category to which the shooting scene belongs; the processing unit is further configured to process the first HDR image into a first SDR image, and determine a first gain map based on the dynamic range difference between the first HDR image and the first SDR; upon receiving an operation to launch a gallery application, the display unit is further configured to display a second interface, the second interface including a first thumbnail; upon receiving an operation on the first thumbnail, the display unit is further configured to display a second HDR image, the second HDR image being an image obtained by dynamically expanding the first SDR image based on the first gain map and the gain value corresponding to the first scene.

[0039] Thirdly, embodiments of this application provide an electronic device, which includes: one or more processors and a memory; the memory is coupled to one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the electronic device to perform the methods described in the first aspect or any possible implementation of the first aspect.

[0040] Fourthly, embodiments of this application provide a computer-readable storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the methods described in the first aspect or any possible implementation thereof.

[0041] Fifthly, embodiments of this application provide a computer program product including a computer program. When the computer program product includes computer program code, when the computer program code is run on an electronic device, it causes the electronic device to perform the method described in the first aspect or any possible implementation of the first aspect.

[0042] Sixthly, this application provides a chip system applied to an electronic device. The chip system includes one or more processors, which are used to invoke computer instructions to cause the electronic device to perform the methods described in the first aspect or any possible implementation of the first aspect.

[0043] In one possible implementation, the chip system described above in this application further includes at least one memory storing instructions. The memory can be an internal storage unit of the chip system, such as a register or cache, or it can be a storage unit of the chip system itself (e.g., read-only memory, random access memory, etc.).

[0044] It should be understood that the second to sixth aspects of this application correspond to the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description

[0045] Figures 1A-1C A scenario diagram provided for an embodiment of this application;

[0046] Figure 2 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0047] Figure 3 A schematic diagram of the software structure of an electronic device provided in an embodiment of this application;

[0048] Figure 4 A schematic diagram of module interaction for an image processing method provided in an embodiment of this application;

[0049] Figure 5 This is a schematic diagram of a tone mapping process provided in an embodiment of this application;

[0050] Figure 6 A schematic flowchart illustrating gain map calculation provided in an embodiment of this application;

[0051] Figure 7 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0052] Figure 8 A flowchart illustrating another image processing method provided in an embodiment of this application;

[0053] Figure 9 This is a schematic diagram of the hardware structure of another electronic device provided in an embodiment of this application. Detailed Implementation

[0054] To facilitate a clear description of the technical solutions in the embodiments of this application, some terms and technologies involved in the embodiments of this application will be briefly introduced below:

[0055] 1. Standard Dynamic Range (SDR) Image

[0056] SDR images refer to images with a traditional dynamic range, as opposed to high dynamic range (HDR) images. Compared to HDR images, SDR images have a narrower dynamic range and color gamut, making them suitable for most traditional display devices and content. Furthermore, SDR images can be generated by tone mapping HDR images.

[0057] 2. High dynamic range (HDR) imaging

[0058] HDR refers to images that can display more brightness levels and a wider color gamut than SDR images. HDR images typically range in brightness from very dark blacks to very bright whites, more realistically reflecting lighting conditions in nature.

[0059] HDR images can display higher brightness and deeper blacks, thus providing richer details and a more realistic visual experience.

[0060] From a color space perspective, HDR images typically use a wider color space, such as Rec.2020 (BT.2020) or DCI-P3, while traditional SDR images typically use the Rec.709 color space. A wider color space means that HDR images can display more colors and richer color details.

[0061] 3. Tone mapping

[0062] Tone mapping maps HDR colors to the dynamic range that an SDR display device can display.

[0063] Because the color gamut of a display is smaller than that of the real world, tonemapping is needed to adjust the brightness and color of the image to ensure that the image is presented on a standard display device in a way that matches the original HDR scene.

[0064] Tone mapping involves adjusting the dynamic range of an image, and includes two methods: global tone mapping and local tone mapping. Global tone mapping uses a constant scaling parameter, while local tone mapping allows the scaling parameter to vary with spatial location to better simulate the adaptive characteristics of the human visual system.

[0065] 4. Gain map

[0066] Also known as gain information or gain image, it describes the brightness adjustment information for different regions in an HDR image. For example, gain information can be an image of the same size as the HDR image, where each pixel value represents the brightness gain for that region. Brightness gain can also be called gain or gain value, and is a coefficient or scaling factor that adjusts the brightness of an image.

[0067] Gain information is typically stored in floating-point format, representing the gain value in linear space.

[0068] 5. Normal frame image

[0069] Also known as a normal exposure image, N-frame image, N-frame, medium frame, or medium frame image, this is an image captured by a camera with an exposure of 0 EV. In other words, the exposure of a normal exposure image is 0 EV. Here, 0 EV is a relative value, not an absolute zero exposure. For example, exposure = exposure time * ISO. Assuming a normal exposure image is captured at ISO 200 and an exposure time of 50 milliseconds, the actual exposure corresponding to 0 EV is the product of 200 and 50 milliseconds.

[0070] 6. Short frame images

[0071] Also known as short frame, S-frame, or S-frame image, it is an image captured by a camera when the exposure is less than 0 EV.

[0072] 7. Long frame images

[0073] Also known as long frame, L-frame, or L-frame image, it is an image captured by a camera when the exposure is greater than 0 EV.

[0074] Normal frame images, short frame images, and long frame images can be understood as images with different levels of exposure. It is understandable that electronic devices can obtain HDR images by image fusion of at least two of these three types of images.

[0075] 8. Metadata

[0076] Metadata can be data that describes the key information and features needed in the image processing process. It can be understood that metadata is used to provide the information needed to correctly decode and display images.

[0077] For example, the metadata may also include one or more of the following: basic information such as image name, image size, image format, image creation time, or image modification time; technical information such as resolution, size, color mode, or compression type; and shooting information such as camera model, lens information, exposure time, aperture value, or ISO. This application embodiment does not limit these.

[0078] 9. Other terms

[0079] In the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with substantially the same function and purpose. For example, "first chip" and "second chip" are used only to distinguish different chips and do not limit their order of execution. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.

[0080] It should be noted that, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0081] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, and c can be single or multiple.

[0082] 10. Electronic equipment

[0083] The electronic devices in this application embodiment may include handheld devices with image recognition capabilities, vehicle-mounted devices, etc. For example, some electronic devices are: mobile phones, tablet computers, PDAs, laptops, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving cars, wireless terminals in remote medical surgery, etc., and this application embodiment is not limited to these.

[0084] The electronic devices in the embodiments of this application may also be referred to as: terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent, or user device, etc.

[0085] Figures 1A-1C This is a schematic diagram of a scenario provided in an embodiment of this application. Figures 1A-1C The corresponding embodiments use a mobile phone as an example for illustration, but this example does not constitute a limitation on the embodiments of this application.

[0086] In response to a user launching the camera app, the electronic device can display something like... Figure 1A The interface shown. Figure 1A This can be referred to as a camera interface, which may display one or more of the following: a preview window 101, a camera button 102, a camera mode option bar 103, and a button 104 for viewing the captured image, etc.

[0087] Preview window 101 can be used to display a preview image. The preview image can be an image displayed after the electronic device has performed simple processing on the image captured in real time by the camera.

[0088] Figure 1A The preview image can include multiple subjects, such as: people 10, buildings 20, and grass 30.

[0089] The camera button 102 can be used to receive the user's shooting operation.

[0090] The camera mode option bar 103 can display at least one shooting mode button. For example, from left to right, the camera mode option bar can display: aperture mode button, night scene mode button, portrait mode button, photo mode button, video mode button, short video mode, and a button for viewing more modes, etc.

[0091] A selection indicator can be displayed below the camera mode button, indicating that the camera mode button is selected. Figure 1A It can also be called the shooting interface corresponding to the photo mode.

[0092] Button 104 allows you to view the last captured image obtained from the camera application.

[0093] In response to a user's click on the camera button 102, the electronic device can acquire the original image using the camera and generate a captured image through image processing of the original image. If the original image contains at least two images with different exposure levels, the captured image can be obtained by image fusion processing based on at least two images with different exposure levels. In this case, the captured image can be understood as an image with HDR effect, or an HDR image. The difference in exposure level can be reflected in differences in exposure amount and / or exposure time, etc.

[0094] In response to the user opening the gallery app, the electronic device displays as follows: Figure 1B The interface shown is Figure 1B The screen can display thumbnail 105. In response to a user's click on thumbnail 105, the electronic device displays as follows: Figure 1C The interface shown is Figure 1C The image that can be displayed is the captured image 106 corresponding to the thumbnail 105, and the captured image 106 can be an HDR image.

[0095] Understandably, the image fusion process described above produces HDR images. However, electronic devices are limited by insufficient scene perception capabilities when displaying HDR images, leading to overcompensation and undercompensation issues during playback. This results in overly bright or underly dark images in the displayed image, failing to meet user needs in terms of quality. For example, if an electronic device photographs an object in a dark environment and then adds high dynamic range to the displayed image, the image will be overly bright, resulting in a discrepancy with the actual shooting environment.

[0096] In view of this, embodiments of this application provide an image processing method that enables an electronic device to recognize different shooting scenes. Upon receiving an operation to press the shutter button, the electronic device can acquire a first HDR image in a first scene, process the first HDR image into a first SDR image, and determine a first gain map based on the dynamic range difference between the first HDR image and the first SDR image. When receiving an operation to view the captured image, the electronic device can extend the dynamic range of the first SDR image using the first gain map and the first gain value corresponding to the first scene. In this way, the electronic device can acquire a first gain value related to the shooting scene and selectively extend the dynamic range of SDR images in different shooting scenes to different degrees, so that the image quality of the generated second HDR image matches the shooting scene and meets the user's shooting needs.

[0097] The image processing method provided in this application can be used not only in the field of photography, but also in the fields of video recording, live streaming, and video, etc. This application does not limit it.

[0098] To better understand the embodiments of this application, the structure of the electronic device according to the embodiments of this application is described below. For example, Figure 2 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application.

[0099] The electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, an indicator 192, a camera 193, and a display screen 194, etc.

[0100] Optionally, the sensor module 180 may also include one or more of the following: proximity sensor, ambient light sensor, touch sensor, pressure sensor, gyroscope sensor, barometric pressure sensor, magnetic sensor, accelerometer, distance sensor, fingerprint sensor, temperature sensor, or bone conduction sensor, etc. Figure 2 (not shown in the text), and this application does not specifically limit this in the embodiments.

[0101] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0102] The processor 110 may include one or more processing units. The processor 110 can implement the processing steps in the image processing method provided in the embodiments of this application. For example, the processor 110 can implement the scene recognition, tone mapping, gain map calculation and HDR enhancement processing steps in the image processing method.

[0103] USB interface 130 is an interface that conforms to the USB standard specification, specifically it can be a Mini USB interface, Micro USB interface, USB Type C interface, etc.

[0104] The charging management module 140 is used to receive charging input from the charger. The power management module 141 is used to connect the charging management module 140 and the processor 110.

[0105] The wireless communication function of electronic devices can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0106] Electronic devices utilize GPUs, displays (194), and application processors to achieve display functions. The GPU is a microprocessor for image processing, connecting the displays (194) and the application processor.

[0107] Display screen 194 is used to display images and videos, for example, display screen 194 can display the images provided in the embodiments of this application. Figures 1A-1C Any interface described herein, such as displaying an HDR image 2 obtained based on the image processing method provided in the embodiments of this application.

[0108] Electronic devices can achieve shooting functions through image signal processors (ISPs), cameras 193, video codecs, GPUs, displays 194, and application processors.

[0109] Camera 193 is used to capture still images or videos. Camera 193 may include a front-facing camera and / or a rear-facing camera. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, converting it into an image visible to the naked eye. In some embodiments, the ISP may be located in camera 193. Exemplarily, the electrical signal transmitted from the camera's photosensitive element to the ISP may be an image sequence, and the ISP may preprocess the image sequence (such as image preprocessing), etc. The image sequence after ISP processing may be a RAW format image.

[0110] The external storage interface 120 can be used to connect an external storage card.

[0111] Internal memory 121 can be used to store computer executable program code, which includes instructions.

[0112] Electronic devices can implement audio functions such as music playback and recording through audio modules 170, speakers 170A, receivers 170B, microphones 170C, headphone jacks 170D, and application processors.

[0113] Button 190 includes the power button, volume buttons, etc.

[0114] The software systems of electronic devices can adopt layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture, etc., which will not be elaborated here.

[0115] For example, Figure 3 This is a schematic diagram of the software structure of an electronic device provided in an embodiment of this application.

[0116] A layered architecture divides the system into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the system is divided into five layers, from top to bottom: application layer, application framework layer, hardware abstraction layer (HAL), driver layer, and hardware layer.

[0117] The application layer may include a series of application packages. In this embodiment, the application packages may include a camera application and a gallery application.

[0118] Camera apps are used to capture images using a webcam.

[0119] The gallery app can be used to view captured images; it may also be called a camera app. The gallery app is also used to perform dynamic range extension on SDR images using gainmaps and metadata to obtain target images (such as HDR image 2), as detailed in the steps described in S411.

[0120] The application framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer includes some predefined functions.

[0121] The application framework layer may include: camera access interface.

[0122] The camera access interface is used to provide camera applications with application programming interfaces and programming frameworks related to cameras. The camera access interface can include camera management and camera devices.

[0123] The hardware abstraction layer is an interface layer located between the application framework layer and the driver layer, providing a virtual hardware platform for the operating system. In this embodiment, the hardware abstraction layer may include: a camera hardware abstraction layer, a camera algorithm library, and an audio hardware abstraction layer.

[0124] The camera hardware abstraction layer can provide virtual hardware for camera device 1, camera device 2, or more camera devices. Camera devices may include the cameras described in the hardware layer.

[0125] The camera algorithm library may include runtime code and data for implementing the image processing methods provided in the embodiments of this application. For example, the camera algorithm library may include a scene recognition module and a gain map calculation module.

[0126] The scene recognition module can be used to classify scenes in preview images to obtain scene classification information, and to identify and segment objects in preview images to obtain scene segmentation information. The scene recognition module can implement the steps described in S404.

[0127] The gainmap calculation module can be used to calculate the gainmap by utilizing the dynamic range difference between standard dynamic range (SDR) images and HDR images. The gainmap calculation module can implement the gainmap calculation steps in S408.

[0128] The driver layer is the layer between hardware and software. It includes drivers for various hardware components, such as camera drivers, digital signal processor drivers, and image processor drivers.

[0129] The camera driver is used to drive the camera to acquire images and to drive the image signal processor to preprocess the images. The digital signal processor driver is used to drive the data signal processor to process digital signals. The image processor driver is used to drive the image processor to process images.

[0130] The hardware layer may include: camera, image signal processor, digital signal processor and image processor, etc.

[0131] The following is in conjunction with the above. Figure 3 The software architecture described herein provides a detailed description of the image processing methods in the embodiments of this application:

[0132] In response to a user's action of launching the camera application, such as clicking the camera application icon, the camera application calls the camera access interface in the application framework layer to launch the camera application, and then sends a command to start the camera by calling the camera device in the camera hardware abstraction layer. The camera hardware abstraction layer sends the command to start the camera to the camera device driver in the kernel layer. Upon receiving the command to start the camera, the camera device driver can launch the corresponding camera and acquire image light signals through the camera. One camera device in the camera hardware abstraction layer corresponds to one camera sensor in the hardware layer. Then, the camera sensor can transmit the acquired image light signals to the image signal processor for preprocessing (or image preprocessing) to obtain the image electrical signals (or raw image), and transmit the raw image to the camera hardware abstraction layer through the camera device driver.

[0133] The camera algorithm library stores program code that implements the image processing methods provided in the embodiments of this application. Based on a digital signal processor and an image processor, the camera algorithm library executes the above code to perform steps such as scene recognition, gain map calculation, and HDR image enhancement processing based on the received image.

[0134] The camera algorithm library can also send images captured by the camera to the camera hardware abstraction layer. The camera hardware abstraction layer can then display these images.

[0135] It is understood that the embodiments of this application do not specifically limit the software layers involved in the software architecture, the modules contained in the software layers, and the functions of the modules.

[0136] Combination Figure 3 The description of the software architecture in this application is followed by a detailed explanation of the technical solution and how it solves the aforementioned technical problems, using specific embodiments. These specific embodiments can be implemented independently or in combination with each other. Similar or identical concepts or processes may not be described again in some embodiments.

[0137] Figure 4 This is a schematic diagram illustrating the module interaction of an image processing method provided in an embodiment of this application. Figure 4 In a corresponding embodiment, the electronic device may include: a gallery application, a camera application, a camera HAL, a camera algorithm library, and a camera device driver. The function of each module can be found in [reference needed]. Figure 3 The description in the text will not be repeated here.

[0138] like Figure 4 As shown, the image processing method may include the following steps:

[0139] S401. After receiving the user's command to launch the camera application, the camera application sends a command to the camera device driver to launch the camera via the camera HAL.

[0140] For example, the specific process by which a camera application sends a command to the camera device driver to start the camera through modules such as the camera HAL can be found in [reference needed]. Figure 3 The description in the text will not be repeated here.

[0141] S402, Camera device driver acquires image sequence 1 through camera.

[0142] Image sequence 1 may include N images captured by the camera within a certain period of time (such as a first duration), where N can be an integer greater than or equal to 1.

[0143] For example, the N images can be consecutive images acquired by a camera, or the N images can be images whose quality meets preset conditions selected from consecutive images acquired by a camera, etc. This application does not limit this.

[0144] S403, The camera algorithm library obtains image sequence 1 from the camera device driver.

[0145] Optionally, the camera driver can also drive the image signal processor to perform image preprocessing on image sequence 1, obtaining preprocessed image sequence 1, and then pass the preprocessed image sequence 1 to the camera algorithm library to execute subsequent image processing, thereby improving the image quality of image sequence 1. After image preprocessing is completed, the camera algorithm library can also further perform RAW domain image processing, and subsequently, the camera algorithm library can further perform scene recognition on the RAW domain images.

[0146] Image preprocessing may include one or more of the following: bad pixel correction, RAW domain noise reduction, black level correction, optical shadow correction / automatic white balance, or gamma correction.

[0147] S404, the camera algorithm library performs scene recognition on image sequence 1 to obtain scene classification information and scene segmentation information.

[0148] Scene classification information can be used to describe the category to which the shooting scene belongs, while scene segmentation information can be used to describe the shooting objects (or shooting elements, objects, or elements, etc.) contained in the shooting scene.

[0149] Scene classification information can also be called scene category or scene recognition result, etc.

[0150] It is understandable that image sequence 1 can correspond to a scene classification information, such as scene classification information 0. Alternatively, each frame in image sequence 1 can correspond to a scene classification information. Subsequently, after the electronic device receives the user's action of clicking the shutter button, the electronic device can determine the preview image 1 at the moment of taking the picture, and perform scene recognition based on the preview image 1 to obtain scene classification information 1.

[0151] For example, in the shooting scene is Figure 1A As shown, the scene classification information can be 1, which can be understood as the label when the scene classification information represents the human image.

[0152] It should be noted that the steps shown in S401-S404 are steps that can be continuously executed after the camera application is launched. The camera algorithm library can store the scene classification information and scene segmentation information corresponding to each frame of image sequence 1. In this way, when the electronic device receives the user's operation of clicking the shutter button, the electronic device can obtain the cached image at the moment of shooting, as well as the scene classification information 1 and scene segmentation information 1 determined by scene recognition at the moment of shooting.

[0153] S405. After receiving the user's action of clicking the shutter button, the camera application sends a message to the camera algorithm library through the camera HAL to indicate that the photo should be taken.

[0154] S406, The camera algorithm library obtains the cached image at the moment of shooting, and obtains the HDR image 1 by image fusion based on the cached image.

[0155] A cached image may include at least two images with different exposure levels. For example, a cached image may include at least two frames from a long frame image, a short frame image, or a normal frame image.

[0156] For example, the camera algorithm library can continuously perform image preprocessing on the raw images acquired by the camera and cache the preprocessed images. After detecting that a user is taking a photo, the camera algorithm library can retrieve the cached image at the time of the photo capture from the cache, and then perform image fusion on at least two images with different exposure levels in the cached images to obtain HDR image 1.

[0157] S407, the camera algorithm library uses the scene segmentation information 1 at the moment of taking the picture to process the HDR image 1 into an SDR image 1.

[0158] For example, the camera algorithm library can use scene segmentation information 1 to determine the main subject region in the image, and perform different degrees of tone mapping processing on the main subject region and other regions respectively. The specific process can be found below. Figure 5 The description in the text.

[0159] In the case where scene segmentation information 1 indicates that the image includes multiple subjects, the camera algorithm library can determine one of the subjects as the main subject based on one or more of the following information: focus position, area occupied by each subject, or foreground (or background) conditions of each subject. The area occupied by the main subject is defined as the main subject region, and the areas in the image other than the main subject region are defined as other regions.

[0160] For example, if the scene segmentation information 1 determines that the HDR image contains people, grass, and buildings, the camera algorithm library can identify the person as the subject if the focus position is determined to be on the person. It can be understood that if only one subject (such as a person) is identified in the scene segmentation information 1, that subject can be assumed to be the main subject.

[0161] The person's label in the Segmentation_Map is 101. Using label 101 from the Segmentation_Map, HDR image 1 is adjusted to a binary image.

[0162] For example: If Tone_Map_Mask(i,j)=1, if Segmentation_Map(i,j)=Main_Object_tag.

[0163] Here, Main_Object_tag can be understood as the subject. For example, if an electronic device determines that the subject in an image is a person, it can identify the region containing tag 101 in the Segmentation_Map as the subject, set the subject region to 1, and set the non-subject regions (or other regions) to 0, thus obtaining a binary map. The binary map can be used to represent the subject region and non-subject regions in the HDR image 1.

[0164] S408, the camera algorithm library uses the scene classification information 1 at the moment of shooting to calculate the gainmap of HDR image 1 and SDR image 1, and obtain the gainmap.

[0165] For example, the camera algorithm library uses HDR image 1 and SDR image 1 to calculate the base gain, and then obtains the gainmap calculation parameters related to scene classification information 1. Based on the gainmap calculation parameters, the base gain is adjusted to obtain the gainmap. The process of calculating and adjusting the base gain can be seen below. Figure 6 The description in the text.

[0166] The gainmap calculation parameters may include one or more of the following: automatic exposure ambient brightness index AE_Lux_Index, scene histogram information Image_Hist, or scene color information Color_Saturation_Info.

[0167] Among them, AE_Lux_Index can determine whether the current scene is a high-light scene or a low-light scene, Image_Hist can determine the scene's light and shadow contrast information Scene_Contrast, and Color_Saturation_Info can determine the current scene's color gain and saturation information.

[0168] Understandably, the camera algorithm library can perform gainmap processing on both the SDR and HDR images using scene classification information 1 and Scene_Process_Params across three channels. Here, the three channels can be understood as the red, green, and blue channels. Alternatively, the camera algorithm library can perform gainmap processing on both the SDR and HDR images using scene classification information 1 and Scene_Process_Params across a single channel, see [link to relevant documentation]. Figure 6 The description in [the document] states that the camera algorithm library can perform a step of converting three-channel data into single-channel data before gainmap processing.

[0169] Understandably, current processing methods typically apply the same GainMap calculation method to all scenes. This same method is used for highlight areas, low-light areas within the same scene, and different scene semantics. Furthermore, the same processing method is applied to dynamic range compression for varying display scene brightness and color temperature. Consequently, without scene differentiation, semantic information differentiation within scenes, or awareness of the display environment, more refined image adjustments based on scene awareness are impossible during dynamic range compression and gainmap calculation.

[0170] Based on the descriptions in S407 and S408, it is understood that this application can use scene classification information to determine the adjustment of the gainmap calculation method for the scene. For example, Scene_Process_Params can be passed to the gainmap calculation module to realize gainmap calculation based on the shooting scene. Furthermore, in the dynamic range compression (i.e., tonemapping) part, by introducing Segmentation_Map and Display_Scene_Model, dynamic range compression operations based on pixels and / or regions are performed to achieve dynamic range compression processing related to the shooting object.

[0171] Following S408, the camera algorithm library can save the SDR image 1, gainmap, and metadata as image data (or image files), such as storing them in the image decoding module. This image decoding module can be set up in the gallery application for its use; alternatively, the image decoding module can be set up in other locations, and the gallery application can execute the steps shown in S409 by calling the image decoding module.

[0172] The following example illustrates how the camera algorithm library stores SDR image 1 and gainmap as metadata in a gallery application. This example does not constitute a limitation on the embodiments of this application.

[0173] S409, the camera algorithm library sends SDR image 1, gainmap, and metadata to the gallery application.

[0174] S410. After the gallery application receives the user's operation to view the image, the gallery application uses gain map and metadata to perform dynamic range enhancement processing on SDR image 1 to obtain HDR image 2.

[0175] The metadata may include scaling parameters, which can be determined during tone mapping, as described in S407. For example, after detecting the scaling parameters in the metadata, the electronic device can verify the enhanced HDR image based on the scaling parameters. For example, if scaling parameter 1 is included in the metadata, the camera algorithm library 1 can determine whether the HDR image 2 and the SDR image 1 satisfy scaling parameter 1.

[0176] The image library application can decode the metadata to parse the gain map according to the relevant parameters contained in the metadata, and also obtain the relevant data on which the reconstructed HDR image 2 depends. The application can then fuse the SDR image 1 and the gain map to obtain the HDR image 2.

[0177] The HDR image 2 can be called Display_Image, i.e., Display_Image = SDR_Image * GainMap.

[0178] Understandably, the camera algorithm library can use the gain map to extend the high dynamic range of SDR image 1, resulting in an enhanced HDR image 2. Since the gain map is based on scene segmentation information 1 and scene classification information 1, HDR image 2 can be understood as an image obtained after high dynamic range restoration of different types of scenes and different subjects.

[0179] S412, The gallery application displays HDR images on the monitor 2.

[0180] For example, a gallery app can provide... Figure 1C The image displayed is an HDR image 2.

[0181] Based on this, the electronic device can perform tone mapping processing on different regions of the HDR image 1 to varying degrees using scene segmentation information 1, resulting in an SDR image 1 that is adapted to the subject being photographed. The electronic device can also perform gain map calculations on both the HDR image 1 and the SDR image 1 using scene classification information 1, achieving dynamic range compression for different shooting scenarios (such as different scene brightness and color temperatures). Thus, on the display side, the electronic device can extend the dynamic range of the SDR image 1 based on the gain map to achieve an HDR image 2 that is adapted to both the shooting scene and the objects within it, meeting the user's shooting needs.

[0182] Optionally, in S404, the Scene_Index and Segmentation_Map tags can include more content.

[0183] For example, the Scene_Index tag can include one or more of the following:

[0184] 1-Portrait: Recognizes faces and bodies, optimizes skin tone and background blur;

[0185] 2-Night Scene: Identify low-light environments, improve brightness and reduce noise;

[0186] 3-Flowers: Identify flowers, enhance color saturation and detail;

[0187] 4-Food: Identify food, enhance color and contrast to make food look more appealing;

[0188] 5-Landscape: Identify natural or urban landscapes and optimize dynamic range and color balance;

[0189] 6-Pets: Recognizes pets such as cats and dogs, with optimized details and colors;

[0190] 7-Motion: Recognize motion scenes and increase shutter speed to capture action;

[0191] 8-Sunset / Sunrise: Identifies sunsets or sunrises and optimizes the colors of the sky and clouds;

[0192] 9-Blue Sky: Identify a clear sky, enhancing the blue and white of the clouds;

[0193] 10-Snow Scene: Identify the snow scene and adjust the exposure to avoid overexposure;

[0194] 11-Beach: Identify beach scenes and optimize the colors of the water and sky;

[0195] And 12-Fireworks: Identify fireworks, adjust exposure and color to capture their brilliance, etc.

[0196] For example, the Segmentation_Map tag can include one or more of the following:

[0197] 101-People: Segmenting out human figures in the image;

[0198] 102-Skin Tone: Specifically identifies skin tone areas on a person, used for beautification, skin tone adjustment, etc.;

[0199] 103-Blue Sky: Identifies and segments the blue sky for use in sky enhancement, replacement, etc.

[0200] 104-Vegetation: Delineates areas with vegetation such as trees and grasslands;

[0201] 105-Water Bodies: Identify and segment water bodies such as oceans, lakes, and rivers;

[0202] 106-Beach: Specially identifies beach areas for color enhancement, etc.

[0203] 107-Architecture: Separating buildings for purposes such as building enhancement and perspective correction;

[0204] 108-Roads: Identify and segment roads;

[0205] 109-Vehicles: Separates vehicles such as cars and bicycles;

[0206] 110-Pets: Identifies and segments pets such as cats and dogs;

[0207] 111-Food: Specially identifies food for enhanced color and detail in food photography;

[0208] 112-Sky: In addition to the blue sky, it may also include the segmentation of different sky scenes such as sunset and clouds;

[0209] And 113-Flowers: Flower areas in the scene.

[0210] It is understood that the Scene_Index and Segmentation_Map tags mentioned above are merely illustrative and do not constitute a limitation on the embodiments of this application.

[0211] In S404, the camera algorithm library can obtain scene classification information using module 1 and scene segmentation information using module 2. The following is a schematic explanation of how to obtain module 1 and module 2.

[0212] Scene classification information is implemented in camera applications using artificial intelligence (AI) scene classification functionality. AI scene classification automatically identifies the type of subject being photographed and applies different shooting settings based on different scenes to improve photo quality. This functionality typically relies on deep learning techniques, such as Model 1, which can be a convolutional neural network (CNN). CNNs are used in image recognition.

[0213] The generation and use of Model 1 can include the following steps: S1, Data collection: Collect a large number of images of different scenes, such as portrait images, night scene images, and flower images; S2, Data annotation: Manually annotate the collected images, indicating the scene category to which each image belongs; S3, Model training: Train the deep learning model using the annotated images, enabling the deep learning model to recognize the features of different scenes; S4, Model optimization and testing: Evaluate the recognition accuracy of the model through a test set and optimize the model to obtain the trained Model 1; S5, Deployment and application: Integrate the trained Model 1 into the camera application of electronic devices to realize scene recognition and corresponding shooting parameter adjustment functions.

[0214] Scene segmentation information, also known as semantic segmentation information or scene segmentation map information, can be the information obtained after semantic recognition and semantic segmentation of objects in an image. Scene segmentation information can include the object's label and the pixel position of the object in the image.

[0215] It is understandable that each pixel in each frame of image sequence 1 can correspond to a scene segmentation information. Subsequently, after the electronic device receives the user's operation of clicking the shutter button, the electronic device can obtain the preview image 1 at the moment of taking the picture, and perform scene recognition based on the preview image 1 to obtain scene segmentation information 1. Scene segmentation information 1 includes the scene segmentation information corresponding to each pixel.

[0216] For example, in the shooting scene is Figure 1A As shown, the scene segmentation information may include: the range of human figures and the corresponding label 101, the range of vegetation and the corresponding label 104, and the range of buildings and the corresponding label 107.

[0217] The character range can include the pixel position occupied by the character, the vegetation range can include the pixel position occupied by the vegetation, and the building range can include the pixel position occupied by the vegetation. The pixel position can be represented by two-dimensional coordinates (i, j).

[0218] Scene segmentation information can be achieved using AI scene segmentation technology in camera applications. AI segmentation technology can be understood as a more advanced image processing technique that can identify and segment different subjects or regions in an image, such as skin color, blue sky, and beach. This technique is usually based on Model 2, which can include semantic segmentation networks in deep learning.

[0219] The generation and use of Model 2 can include the following steps: S1, Data preparation: Collect a large number of images containing various scene elements and accurately annotate these images at the pixel level, indicating which category each pixel belongs to; S2, Model selection: Select a suitable deep learning model architecture, such as fully convolutional networks (FCN), U-Net, or DeepLab, which can be used to handle image segmentation tasks; S3, Model training: Train the model using the labeled dataset so that the model can recognize and segment different subjects and regions in the image; S4, Model optimization: Adjust the model parameters through the validation set to obtain the trained Model 2. The techniques used in the model optimization process can include data augmentation, transfer learning, etc. Model optimization can improve the accuracy and generalization ability of the model; S5, Deployment and application: Integrate the trained Model 2 into the camera application to achieve real-time image recognition and segmentation.

[0220] It is understood that S404 can be implemented based on both Model 1 and Model 2, or S404 can be implemented based on a single model. This application does not limit this.

[0221] In S407, the process by which the camera algorithm library performs tone mapping processing of varying degrees on the subject region and other regions can be found in [reference needed]. Figure 5 The description in the text. Figure 5 This is a schematic diagram of a tonemapping process provided in an embodiment of this application.

[0222] like Figure 5As shown, the camera algorithm library uses scene segmentation information 1 to divide the HDR image 1 into regions, obtaining the main region and other regions. Different tone mapping processes are performed on the main region and other regions, such as performing a more focused tone mapping process on the main region and performing a regular tone mapping process on other regions. Then, a smooth transition is performed between the regions to obtain the SDR image 1.

[0223] Targeted tone mapping of the main subject region can include applying tone mapping with a larger scaling parameter to the main subject region and recording the corresponding scaling parameter (or compression space) in the metadata headroom. A larger scaling parameter in tone mapping can be used to improve the dynamic range difference between HDR and SDR image data, thus calculating a larger gain map.

[0224] For example, for the main area, SDR_Image = ToneMapping(HDR_Image, Tone_Map_Mask).

[0225] Wherein, SDR_Image can be the SDR image 1 described in the embodiments of this application, and HDR_Image can be the HDR image 1 described in the embodiments of this application. In the description of S407-S412, the area marked as 1 in Tone_Map_Mask can be understood as the main area.

[0226] Performing regular tone mapping on other areas can include adjusting the brightness of other areas using a global tone mapping (GTM) table.

[0227] For other regions, SDR_Image = GTM(HDR_Image).

[0228] Among them, GTM can also be called a luminance mapping lookup table. The input data range of GTM is 0-MAX_VALUE_HDR, where MAX_VALUE_HDR can be the maximum luminance value allowed to be displayed in an HDR image. The output data range of GTM is 0-255.

[0229] It should be noted that when the scaling parameter is 1 when performing tonemapping on the main area, and 2 when performing tonemapping on other areas, scaling parameter 1 is greater than scaling parameter 2.

[0230] Furthermore, the electronic device can fuse the main area after tonemapping processing, as well as other areas after tonemapping processing, into an SDR image 1.

[0231] Understandably, the main subject area is determined by the Segmentation Map, which indicates the pixel positions occupied by each subject. Therefore, during tonemapping of the main subject area, pixel-level tonemapping can cause discontinuities between the main subject area and the surrounding areas. Thus, multi-scale processing can be introduced. For example, multi-scale processing could include downsampling the HDR image 1 by 2x, 4x, or 8x, then performing tonemapping on the main subject areas of the downsampled images, and finally merging the tone-mapped images to obtain the multi-scale processed image.

[0232] Optionally, electronic devices can also combine the Display_Scene_Model to perform tonemapping processing on HDR images.

[0233] The Display_Scene_Model model models the environment in which the screen is located during actual display. By using the Display_Scene_Model, electronic devices can utilize information such as the brightness (Display_Luminance) and color temperature (Display_Color_Temperature) of the display environment to perform more refined tone mapping, thereby obtaining an SDR image that better matches the brightness and color temperature of the current display environment.

[0234] The `Display_Scene_Model` can include lookup tables related to `Display_Luminance` and `Display_Color_Temperature`. This can be understood as the electronic device determining the lookup tables between different information elements of the display environment and tone mapping through pre-calibration. For example, combining `Display_Luminance` and `Display_Color_Temperature` can apply different levels of tone mapping to an image.

[0235] For example, SDR_Image=ToneMapping(HDR_Image,LUT(Display_Luminance),LUT(Display_Color_Temperature)).

[0236] Optionally, the electronic device can also combine Display_Scene_Model to perform tone mapping processing on the main area in HDR image 1, and perform regular tone mapping processing on other areas, and then merge the tone-mapped areas into SDR image 1. The specific tone mapping process is similar to that described above, and will not be repeated here.

[0237] In S408, such as Figure 6 As shown, the camera algorithm library can determine the corresponding Scene_Process_Params based on scene classification information 1.

[0238] For example, using scene segmentation information 1 and the display environment model in the camera algorithm library, HDR_Image (such as HDR image 1) is processed into SDR_Image (such as SDR image 1), see the description in S407.

[0239] The camera algorithm library can convert the three-channel information of an HDR_Image into grayscale information (i.e., convert it into single-channel data) to obtain a high dynamic range grayscale image HDR_Image_Gray, and convert the three-channel information of an SDR_Image into grayscale information to obtain a standard dynamic range grayscale image SDR_Image_Gray. The camera algorithm library uses HDR_Image, HDR_Image_Gray, SDR_Image, and SDR_Image_Gray to perform gainmap calculations.

[0240] For example, the gainmap can be calculated as follows: GainMap = Calculate_GainMap(HDR_Image, SDR_Image, HDR_Image_Gray, SDR_Image_Gray, Scene_Process_Params).

[0241] Among them, HDR_Image_Gray can be the image data when converting the three-channel data HDR_Image into single-channel data.

[0242] The conversion method can be found in the following formula:

[0243] HDR_Image_Gray(i,j)=Coeff1*HDR_Image(i,j,1)+Coeff2*HDR_Image(i,j,2)+Coeff3*HDR_Image(i,j,3).

[0244] Conversion coefficients: Coeff1, Coeff2, and Coeff can all be predefined conversion parameters, such as Coeff1 being 0.299, Coeff2 being 0.587, and Coeff3 being 0.114.

[0245] Scene_Process_Params can include gainmap calculation parameters related to scene classification information 1. For example, gainmap calculation parameters can include one or more of the following: automatic exposure ambient brightness index (or scene brightness information) AE_Lux_Index, scene histogram information Image_Hist, or scene color information Color_Saturation_Info.

[0246] Among them, AE_Lux_Index can determine whether the current scene is a high-light scene or a low-light scene, Image_Hist can determine the scene's light and shadow contrast information (or scene contrast information), Scene_Contrast, and Color_Saturation_Info can determine the current scene's color gain and saturation information.

[0247] Specifically, calculating the gain map Calculate_GainMap can include: calculating the basic gain Basic_Boot_Gain, and then adjusting the basic gain using the gainmap calculation parameters to obtain the final gainmap.

[0248] The basic gain can be calculated as follows: Basic_Boot_Gain = (HDR_Image_Gray / SDR_Image_Gray) * Scale_Ratio * 255.

[0249] The data range for HDR_Image_Gray can be 0-MAX_VALUE_HDR, and the data range for SDR_Image_Gray can be 0-255. Scale_Ratio is the adjustment coefficient for the base brightening gain. Scale_Ratio can be used for data normalization and can be preset, such as 1 / 2. 6 Equal values.

[0250] Furthermore, one or more gainmap calculation parameters from AE_Lux_Index, Image_Hist, or Color_Saturation_Info adjust the base gain.

[0251] For example, the final gainmap can be calculated using the following formula:

[0252] GainMap=Basic_Boot_Gain*LUT_Scene_Lux(AE_Lux_Index,Scene_Index)*LUT_Scene_Contrast(Scene_Contrast,Scene_Index)*LUT_Color_Saturation_Info(Color_Saturation_Info,Scene_Index).

[0253] In the descriptions of S408-S412, Scene_Index can be the scene classification information described in S404. LUT_Scene_Lux, LUT_Scene_Contrast, and LUT_Color_Saturation_Info can all be understood as gain lookup tables. Gain lookup tables can be used to perform gain data compensation for different shooting scenes to achieve gain map adjustment.

[0254] LUT_Scene_Lux can include a correspondence 1 between AE_Lux_Index, Scene_Index, and gain 1. This correspondence 1 is used to adjust the base gain under different AE_Lux_Index and / or different Scene_Index.

[0255] LUT_Scene_Contrast can include: Scene_Contrast, Scene_Index, and the correspondence 2 between gain 2. The correspondence 2 is used to adjust the base gain under different Scene_Contrast and / or different Scene_Index.

[0256] LUT_Color_Saturation_Info can include: Color_Saturation_Info, Scene_Index, and the correspondence 3 between gain 3. The correspondence 3 is used to adjust the base gain under different Color_Saturation_Info and / or different Scene_Index.

[0257] Under normal circumstances, the process of electronic devices performing gainmap calculations based on SDR and HDR images can be as follows.

[0258] GainMap=(HDR_Image / MAX_VALUE_HDR)*HDR_Ratio*Scale_Value*255

[0259] Where MAX_VALUE_HDR is the maximum value of the high dynamic range image, such as MAX_VALUE_HDR can be 2. 14 -1; where Scale_Value is the brightness parameter to adjust GainMap, such as Scale_Value can be 0.5; where HDR_Ratio is the maximum exposure ratio between long and short frames of the HDR image, such as HDR_Ratio can be 16.

[0260] After the gainmap is calculated, the electronic device can clamp the gainmap to the range of 0-255, that is, set values ​​less than 0 to zero and values ​​greater than 255 to 255.

[0261] Compared to the gainmap calculation process described in S408, the embodiments of this application can realize gainmap calculation based on the shooting scene. When the dynamic range of the image is subsequently restored using the gainmap, an HDR image that is more in line with the shooting scene can be restored (such as HDR image 2).

[0262] Understandable, combined Figures 4-6 The description in the text, Figure 7 This is a schematic flowchart illustrating an image processing method provided in an embodiment of this application. Figure 7 As shown, the image processing method may include the following steps:

[0263] ① Scene recognition. Electronic devices can acquire raw image sequences based on cameras, perform image preprocessing on the raw sequences, and then perform preview RAW domain processing and scene recognition to obtain scene classification information and scene segmentation information.

[0264] Simultaneously, the electronic device can continue performing preview YUV domain processing after executing RAW domain processing to obtain a preview image. This preview image can be displayed... Figure 1A In the preview window 101.

[0265] ① This can be implemented in the preview path.

[0266] ②HDR fusion. Electronic devices can cache the pre-processed image and use the cached image for HDR fusion to obtain HDR image 1. The cached image can include at least two images with different exposure levels.

[0267] ③tone mapping. Electronic devices can process HDR image 1 into SDR image 1 through tone mapping. The tone mapping process can be found in the description in S407.

[0268] Optionally, the electronic device can also determine AE Info through auto exposure (AE), use AE Info to perform HDR image preprocessing on HDR image 1, and then downsample the HDR image after HDR image preprocessing to obtain HDR image 3.

[0269] Optionally, the electronic device can also downsample SDR image 1 to obtain SDR image 2. It is understood that the electronic device can reduce computational load and thus gain power savings through downsampling.

[0270] After step ③, the electronic device can perform image post-processing on SDR image 1 and capture SDR frames.

[0271] ④ Gainmap calculation and metadata calculation. Based on scene classification information 1 and scene segmentation information 1, the electronic device performs gainmap calculation on SDR image 1 and HDR image 1 to obtain the gainmap.

[0272] Optionally, the electronic device may also perform gainmap calculation on the SDR image 2 and the HDR image 3 based on the scene classification information 1 and the scene segmentation information 1.

[0273] Following steps ③ and ④, the electronic device can encapsulate data from the SDR image (such as SDR image 1), gainmap, and metadata.

[0274] ②③④ can all be implemented in AI RAW.

[0275] ⑤ High Dynamic Range Enhancement Display Processing. Electronic devices can perform high dynamic range restoration processing on SDR image 1 based on gainmap and metadata to obtain HDR image 2. HDR image 2 can achieve HDR enhanced display.

[0276] In the above Figures 4-7 Based on the corresponding embodiments, Figure 8 This is a schematic flowchart illustrating an image processing method provided in an embodiment of this application. Figure 8 As shown, the image processing method may include the following steps:

[0277] S801: After receiving the command to launch the camera application, the first interface is displayed.

[0278] The first screen includes a camera button.

[0279] The first interface is Figure 1A The interface shown allows the camera button to be used. Figure 1A The camera button 102 in the image.

[0280] S802: After receiving an operation on the camera button, acquire the first HDR image.

[0281] The first HDR image corresponds to the first scene, which is used to characterize the category to which the shooting scene belongs.

[0282] The first HDR image can be HDR image 1 as described in the embodiments of this application, and the first scene can be scene classification information 1 as described in the embodiments of this application. The process of obtaining the first scene by scene recognition can be found in the description in S404.

[0283] S803. Process the first HDR image into a first SDR image, and determine a first gain map based on the dynamic range difference between the first HDR image and the first SDR.

[0284] The first SDR image can be SDR image 1 as described in the embodiments of this application. The first gain image can be the base gain as described in the embodiments of this application.

[0285] The specific process of processing the first HDR image into the first SDR image can be found in the description in S407.

[0286] The process of determining the first gain map based on the dynamic range difference between the first HDR image and the first SDR can be found in the description in S408.

[0287] S804. After receiving the command to launch the gallery application, the second interface is displayed.

[0288] The second interface includes: the first thumbnail.

[0289] The second interface can be Figure 1B The interface shown can have the first thumbnail as... Figure 1B Thumbnail 105.

[0290] S805. After receiving an operation on the first thumbnail, display the second HDR image.

[0291] The second HDR image is an image obtained by dynamically expanding the first SDR image based on the first gain map and the gain value corresponding to the first scene.

[0292] The second HDR image can be HDR image 2 as described in the embodiments of this application. The gain value corresponding to the first scene can include one or more of gain 1, gain 2 or gain 3 described in the embodiments.

[0293] Based on this, electronic devices can identify the first scene through scene recognition, and based on the gain value associated with the first scene, perform dynamic range expansion on the SDR image in the current shooting scene in a targeted manner, so that the image quality of the generated second HDR image matches the shooting scene and meets the user's shooting needs.

[0294] It should be noted that the interface described in the embodiments of this application is only an example and does not constitute a limitation on the embodiments of this application.

[0295] It should be noted that the module names involved in the embodiments of this application can all be defined as other names, as long as they can achieve the function of each module, and no specific restrictions are placed on the module names.

[0296] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0297] The image processing method of the embodiments of this application has been described above. The apparatus for performing the above method provided in the embodiments of this application is described below. Those skilled in the art will understand that the methods and apparatus can be combined with and referenced by each other, and the related apparatus provided in the embodiments of this application can perform the steps in the above list sorting method.

[0298] Figure 9 This is a schematic diagram of the hardware structure of another electronic device provided in an embodiment of this application.

[0299] The electronic device includes a processor 901, a communication line 904, and at least one communication interface. Figure 9 (The example provided uses communication interface 903 as an example.)

[0300] The processor 901 may be a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application.

[0301] Communication line 904 may include circuitry for transmitting information between the aforementioned components.

[0302] The communication interface 903 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, wireless local area networks (WLAN), etc.

[0303] Possibly, the electronic device may also include a memory 902.

[0304] The memory 902 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory may exist independently and be connected to the processor via communication line 904. The memory may also be integrated with the processor.

[0305] The memory 902 stores computer execution instructions for implementing the scheme of this application, and its execution is controlled by the processor 901. The processor 901 executes the computer execution instructions stored in the memory 902 to implement the method provided in the embodiments of this application.

[0306] It is possible that the computer execution instructions in the embodiments of this application may also be referred to as application code, and the embodiments of this application do not specifically limit this.

[0307] In a specific implementation, as one example, the processor 901 may include one or more CPUs, for example... Figure 9 CPU0 and CPU1 in the CPU.

[0308] In a specific implementation, as one example, an electronic device may include multiple processors, for example... Figure 9Processors 901 and 905 are mentioned. Each of these processors can be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. Here, "processor" can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0309] The image processing method provided in this application can be applied to electronic devices with communication functions. The electronic devices include terminal devices, and the specific device form of the terminal devices can be referred to the above-described related descriptions, which will not be repeated here.

[0310] This application provides a terminal device, which includes a processor and a memory; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, causing the terminal device to perform the above-described method.

[0311] This application provides a chip. The chip includes a processor, which is used to call a computer program in memory to execute the technical solutions in the above embodiments. Its implementation principle and technical effects are similar to those in the related embodiments described above, and will not be repeated here.

[0312] This application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, it implements the methods described above. The methods described in the above embodiments can be implemented wholly or partially by software, hardware, firmware, or any combination thereof. If implemented in software, the functionality can be stored as one or more instructions or code on or transmitted over the computer-readable medium. The computer-readable medium can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium accessible by a computer.

[0313] In one possible implementation, a computer-readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage or other magnetic storage devices, or any other medium targeted to carry or to store the required program code in the form of instructions or data structures, and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used herein, disks and optical discs include optical discs, laser discs, optical discs, Digital Versatile Discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs optically reproduce data using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0314] This application provides a computer program product, which includes a computer program that, when run, causes a computer to perform the above-described method.

[0315] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing device, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.

[0316] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image processing method, characterized by, The method is applied to an electronic device, and the method comprises: displaying a first interface after receiving an operation of starting a camera application, the first interface comprising a photographing button; acquiring a first HDR image after receiving an operation on the photographing button, the first HDR image corresponding to a first scene, the first scene being used to represent a category to which a photographing scene belongs; processing the first HDR image into a first SDR image and determining a first gain map based on a dynamic range difference between the first HDR image and the first SDR; displaying a second interface after receiving an operation of starting a gallery application, the second interface comprising a first thumbnail; displaying a second HDR image after receiving an operation on the first thumbnail, the second HDR image being obtained by performing dynamic range expansion on the first SDR image based on the first gain map and a gain value corresponding to the first scene.

2. The method of claim 1, wherein, The first HDR image comprises at least one photographing object, The method further comprises: determining a first region and a second region, the first region being determined by a first photographing object in the at least one photographing object, and the second region being a region other than the first region in the first HDR image; processing the first HDR image into a first SDR image comprises: performing different tone mapping processing on the first region and the second region to obtain a tone-mapped first region and a tone-mapped second region; and fusing the tone-mapped first region and the tone-mapped second region to obtain the first SDR image.

3. The method of claim 2, wherein, The different tone mapping processing on the first region and the second region comprises: tone mapping processing the first region by using a first scaling parameter and tone mapping processing the second region by using a second scaling parameter, the first scaling parameter being greater than the second scaling parameter.

4. The method according to claim 2 or 3, characterized in that, The determination of the first region comprises: when the first photographing object satisfies a preset condition, determining the first region containing the first photographing object, the preset condition comprising one or more of the following: the first photographing object being located at a focus position of the first HDR image, the first photographing object having a maximum area among all photographing objects in the first HDR image, or the first photographing object being located at a foreground of the first HDR image.

5. The method according to any one of claims 2 to 4, characterized in that, The at least one photographing object comprises one or more of the following: a person, a skin color, a blue sky, vegetation, water, a beach, a building, a road, a vehicle, a pet, a sky, food, or a flower.

6. The method according to any one of claims 2-5, characterized in that, The processing of the first HDR image into a first SDR image comprises: processing the first HDR image into the first SDR image by using a preset model, the preset model being used to tone map process the first HDR image according to luminance information and / or color temperature information.

7. The method according to any one of claims 1 to 6, characterized in that, The gain value corresponding to the first scene can include one or more of a first gain value related to illumination information of the first scene, a second gain value related to contrast information of the first scene, or a third gain value related to color information of the first scene.

8. The method of claim 7, wherein, The method further includes: adjusting the first gain map using at least one of the gain values corresponding to the first scene to obtain a second gain map; The second HDR image is obtained by performing dynamic range expansion on the first SDR image based on the first gain map and the gain values corresponding to the first scene, including: the second HDR image is obtained by performing dynamic range expansion on the first SDR image based on the second gain map.

9. The method according to any one of claims 1 to 8, characterized in that, The first scene includes one or more of a portrait, a night scene, flowers, food, scenery, pets, sports, sunset or sunrise, a blue sky, a snowy scene, a beach, or fireworks.

10. The method according to any one of claims 1 to 9, characterized in that, After receiving the operation of starting the camera application, the method further includes: obtaining a first image sequence; performing scene recognition on preview images in the first image sequence respectively to obtain scene recognition results, wherein the scene recognition results include the first scene, and the first scene is obtained by performing scene recognition on a preview image at a photographing time, and the photographing time is the time when the photographing button is received.

11. The method according to any one of claims 1 to 10, characterized in that, The method further includes: obtaining a second image sequence, the second image sequence including one or more of a long frame image, a short frame image, or a short frame image; performing image fusion on at least two images in the second image sequence to obtain the first HDR image.

12. The method according to any one of claims 1 to 11, characterized in that, The electronic device includes the camera application, a camera algorithm library, and the gallery application; After receiving the operation of the photographing button, the method further includes: the camera application sending a message indicating starting photographing to the camera algorithm library; The method further includes: After receiving the operation of the first thumbnail, the method further includes: the gallery application generating the second HDR image.

13. An electronic device, comprising: The electronic device includes a camera, a display, one or more processors, and a memory; The camera is configured to capture image data, the display is configured to perform the interface display, the memory is coupled to the one or more processors, the memory is configured to store computer program code, the computer program code includes computer instructions, and the one or more processors are configured to invoke the computer instructions to cause the electronic device to perform the method of any one of claims 1-12.

14. A chip system, characterized by The chip system is applied to an electronic device, and the chip system includes one or more processors configured to invoke computer instructions to cause the electronic device to perform the method of any one of claims 1-12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method of any of claims 1-12.

16. A computer program product, characterised in that, The computer program product includes computer program code that, when executed on an electronic device, causes the electronic device to perform the method of any of claims 1-12.

Images