Image processing method, electronic device, storage medium and program product

Abstract

The application provides an image processing method and an electronic device, a storage medium and a program product, and relates to the technical field of image processing. In the method, a plurality of images corresponding to different exposure parameters are acquired, the images being continuously collected for the same scene in a high dynamic range mode, the images including a reference image and a second image other than the reference image; the brightness of the second image is preliminarily adjusted according to the exposure parameters to obtain a third image; the brightness of the third image is closer to that of the reference image than the brightness of the second image; an exposure compensation value corresponding to a fifth image is determined according to difference information between pixel attribute values of the same pixel position of the fourth image and the fifth image; the brightness of the fifth image is adjusted according to the exposure compensation value to obtain a target image; and the brightness of the target image is closer to that of the reference image than the brightness of the fifth image. In this way, the brightness consistency of the reference image and the target image can be enhanced, and the technical effect of reducing the occurrence probability of a fusion boundary in an HDR image can be achieved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, electronic device, storage medium, and program product. Background Technology

[0002] With the development of electronic technology, high dynamic range (HDR) imaging technology has gradually become popular in electronic devices with photography functions such as smartphones and digital cameras. Compared with ordinary images, HDR images can provide more dynamic range and image details.

[0003] Currently, most devices, when capturing HDR images, often need to take multiple images with different exposure parameters for a high dynamic range scene, resulting in images with varying exposure levels. These images with different exposure levels are then fused together to obtain a single HDR image.

[0004] When combining multiple images with different exposure levels to obtain an HDR image, there is a tendency for the HDR image to have blending boundaries. Summary of the Invention

[0005] This application provides an image processing method, electronic device, storage medium, and program product, which can achieve the technical effect of reducing the probability of fusion boundary occurrence.

[0006] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0007] Firstly, an image processing method is provided, the method comprising:

[0008] In high dynamic range mode, N images to be processed (i.e., the first image) are acquired. These N images to be processed are obtained by the mobile phone continuously capturing the same target scene under N different exposure parameter settings. The N images to be processed include a reference image and N-1 images to be aligned (i.e., the second image). N is a positive integer greater than or equal to 2.

[0009] Based on the exposure parameters corresponding to the reference image and the image to be aligned, the brightness of the image to be aligned is adjusted to obtain the brightness-adjusted image to be aligned (i.e., the third image). Based on the difference information between the pixel attribute values ​​at the same pixel position in the fourth and fifth images, the exposure compensation value corresponding to the fifth image is determined; the brightness of the fifth image is adjusted according to the exposure compensation value to obtain the sixth image (i.e., the target image).

[0010] Among them, the pixel attribute values ​​include pixel values ​​or brightness values; the sixth image is closer to the brightness of the reference image than the fifth image; the fourth image is the reference image or an image preprocessed from the reference image; the fifth image is the third image or an image preprocessed from the third image.

[0011] Thus, the image processing method provided in this application, by determining a reference image from images to be processed obtained under different exposure parameters, and using images other than the reference image as images to be aligned, performs preliminary brightness adjustment on the images to be aligned according to the exposure parameters, which helps to reduce the brightness difference between the images to be aligned and the reference image. By performing exposure alignment on the images to be aligned according to the exposure compensation value to obtain a sixth image, it helps to further reduce the brightness difference between the sixth image and the reference image, enhancing the brightness consistency between the sixth image and the reference image. Furthermore, the image processing method provided in this application helps to optimize the exposure alignment effect between the sixth image and the reference image, thereby helping to optimize the image fusion effect and reduce the probability of fusion boundaries appearing.

[0012] In one possible implementation of the first aspect, determining the exposure compensation value corresponding to the fifth image based on the difference information between the pixel attribute values ​​at the same pixel location in the fourth and fifth images includes:

[0013] The difference information between the pixel attribute values ​​at the same pixel position in the fourth and fifth images is weighted and summed according to the weight corresponding to the pixel position to obtain the exposure compensation value corresponding to the fifth image.

[0014] The weight of each pixel location is negatively correlated with the noise intensity of that pixel location, or the weight of each pixel location is positively correlated with the pixel attribute value of the reference image.

[0015] Thus, by performing exposure alignment on the image to be aligned based on the exposure compensation value to obtain the sixth image, it helps to further reduce the brightness difference between the sixth image and the reference image, and enhance the brightness consistency between the sixth image and the reference image. Furthermore, the image processing method provided in this application embodiment helps to optimize the exposure alignment effect between the sixth image and the reference image, thereby helping to optimize the image fusion effect and reduce the probability of fusion boundaries appearing.

[0016] In another possible implementation of the first aspect, the difference information includes the difference between the pixel attribute values ​​of the fourth image and the fifth image at the same pixel location, or the difference information includes the ratio between the pixel attribute values ​​of the fifth image and the fourth image at the same pixel location.

[0017] Thus, the differences between pixel attribute values ​​at pixel locations can be characterized by the difference between pixel attribute values ​​at pixel locations or the ratio between pixel attribute values ​​at pixel locations.

[0018] In another possible implementation of the first aspect, before determining the exposure compensation value corresponding to the fifth image based on the difference information between the pixel attribute values ​​at the same pixel location in the fourth and fifth images, the method further includes:

[0019] For each pixel location, the weight corresponding to the pixel location is determined based on the first signal-to-noise ratio of the fourth image at the pixel location and the second signal-to-noise ratio of the fifth image at the pixel location.

[0020] Wherein, the first signal-to-noise ratio is the ratio between the first pixel attribute value and the first noise intensity information; the first pixel attribute value is the pixel attribute value corresponding to the pixel position in the fourth image; the first noise intensity information is used to characterize the noise intensity corresponding to the pixel position in the fourth image. The second signal-to-noise ratio is the ratio between the second pixel attribute value and the second noise intensity information; the second pixel attribute value is the pixel attribute value corresponding to the pixel position in the fifth image; the second noise intensity information is used to characterize the noise intensity corresponding to the pixel position in the fifth image.

[0021] In this way, the signal-to-noise ratio (SNR) at a pixel location indicates the ability of the reference image and the image to be aligned to reproduce the scene at that pixel location. A higher SNR indicates a stronger ability to reproduce the scene, while a lower SNR indicates a weaker ability. Since the weight is positively correlated with the SNR, the proportion of pixels with strong reproduction capabilities included in the exposure compensation value calculation can be increased, resulting in more accurate exposure compensation values.

[0022] In another possible implementation of the first aspect, for each pixel location, the weight corresponding to the pixel location is determined based on the first signal-to-noise ratio of the fourth image at the pixel location and the second signal-to-noise ratio of the fifth image at the pixel location, including:

[0023] For each pixel location, the weight corresponding to the pixel location is determined based on the product of the first signal-to-noise ratio and the second signal-to-noise ratio, or based on the sum of the first signal-to-noise ratio and the second signal-to-noise ratio.

[0024] In this way, the weight of the pixel position can be obtained by multiplying the first signal-to-noise ratio and the second signal-to-noise ratio, or by summing the first signal-to-noise ratio and the second signal-to-noise ratio. The weight is positively correlated with the signal-to-noise ratio. The pixel with the stronger ability to reproduce the shooting scene has a larger weight, which helps to obtain a more accurate exposure compensation value.

[0025] In another possible implementation of the first aspect, before determining the exposure compensation value corresponding to the fifth image based on the difference information between the pixel attribute values ​​at the same pixel location in the fourth and fifth images, the method further includes:

[0026] The pixel attribute value of the first pixel in the third image is set to the maximum pixel attribute value of the reference image to obtain the fifth image; where the first pixel is the pixel whose pixel attribute value is greater than the maximum pixel attribute value.

[0027] By truncating the reference image and the image to be aligned, the noise level of both images can be reduced and image details can be decreased.

[0028] In another possible implementation of the first aspect, before determining the exposure compensation value corresponding to the fifth image based on the difference information between the pixel attribute values ​​at the same pixel location in the fourth and fifth images, the method further includes:

[0029] The fourth and fifth images are obtained by performing at least one of the following preprocessing steps on the reference image and the third image, respectively:

[0030] Mean filtering;

[0031] The second pixel is removed from the reference image and the third pixel is removed from the third image; the second pixel and the third pixel correspond to the same pixel position, and the difference in pixel attribute values ​​between the second pixel and the third pixel exceeds a preset first range.

[0032] The reference image and the third image are subjected to pixel pair filtering to ensure that the remaining pixel pairs after filtering meet preset conditions. The preset conditions include that the remaining pixel pairs correspond to the same pixel position in the reference image and the third image, and that the difference information corresponding to each remaining pixel pair and the average difference satisfy a first preset proximity condition. The difference information corresponding to each remaining pixel pair refers to the difference information between the pixel attribute values ​​of the remaining pixel pairs. The average difference is the average value of the difference information corresponding to each remaining pixel pair after filtering.

[0033] In this way, by applying mean filtering to the reference image and the image to be aligned, the noise level of both images can be reduced and image details can be minimized. Mean filtering can also reduce brightness fluctuations. Removing the second pixel from the reference image and the third pixel from the mean-filtered image to be aligned improves the content consistency between the mean-filtered image and the reference image, thus enhancing brightness consistency. Furthermore, by performing pixel pair filtering on the reference image and the image to be aligned after removing the third pixel, the remaining pixel pairs meet preset conditions, helping to eliminate pixels whose difference information and average difference do not meet the first preset proximity condition, thus providing a basis for subsequent exposure compensation value calculation.

[0034] In another possible implementation of the first aspect, before determining the exposure compensation value corresponding to the fifth image based on the difference information between the pixel attribute values ​​at the same pixel location in the fourth and fifth images, the method further includes:

[0035] Pixel pair filtering is performed on the reference image and the third image to ensure that the remaining pixel pairs after filtering meet preset conditions, including:

[0036] Using a reference image as the first current image and a third image as the second current image, a fourth pixel is removed from the first current image and a fifth pixel is removed from the second current image; wherein, the fourth and fifth pixels are pixel pairs corresponding to the same pixel position, and the difference information of the pixel attribute values ​​of the fourth and fifth pixels does not satisfy a first preset proximity condition with respect to the current average difference; the current average difference is the average value of the difference information corresponding to the current remaining pixel pairs; the current remaining pixel pairs are the pixel pairs remaining in the first current image after removing the fourth pixel and the second current image after removing the fifth pixel;

[0037] The first current image from which the fourth pixel has been removed is taken as the new first current image, and the second current image from which the fifth pixel has been removed is taken as the new second current image. The process of removing the fourth pixel from the first current image and removing the fifth pixel from the second current image is repeated until the remaining pixel pairs after removal meet the preset conditions and the iteration stops.

[0038] In this way, by performing pixel pair filtering on the image, the remaining pixel pairs after filtering meet the preset conditions, which helps to filter out pixels whose corresponding difference information and average difference do not meet the first preset proximity condition, thus providing a basis for subsequent calculation of exposure compensation values.

[0039] In another possible implementation of the first aspect, for each second image, the brightness of the second image is adjusted according to the exposure parameters corresponding to the reference image and the second image, respectively, to obtain the third image, including:

[0040] For each second image, the theoretical exposure factor is calculated based on the exposure parameters corresponding to the reference image and the second image, respectively.

[0041] The brightness of the second image is adjusted according to the theoretical exposure factor to obtain the third image; the brightness of the third image is closer to that of the reference image than that of the second image.

[0042] In this way, by adjusting the brightness of the image to be aligned based on the exposure parameters of the reference image and the image to be aligned, the brightness difference between the image to be aligned and the reference image can be initially reduced, which helps to improve the brightness consistency between the image to be aligned and the reference image.

[0043] In another possible implementation of the first aspect, acquiring multiple first images of the same scene consecutively acquired in high dynamic range mode includes:

[0044] From multiple first images, the first image with the most sixth pixels is determined as the reference image. The sixth pixel is the pixel whose pixel attribute value satisfies a second preset proximity condition with the average of the pixel attribute values ​​of the multiple images; or,

[0045] From multiple first images, the first image whose exposure is closest to the average exposure is determined as the reference image. The average exposure is the average of the multiple exposures corresponding to the multiple first images.

[0046] In this way, by identifying the benchmark image from multiple first images whose exposure is closest to the average or whose pixel attribute value level is closest to the average, a reference standard can be provided for subsequent exposure alignment, which helps to optimize the fusion effect.

[0047] In a second aspect, this application provides an electronic device, which includes at least a memory and one or more processors; the memory is used to store computer instructions, and when the one or more processors execute the computer instructions, the electronic device performs the method described in any of the first aspects above.

[0048] Thirdly, this application provides a computer-readable storage medium storing computer instructions or programs that, when executed on a computer, cause the method described in any of the first aspects above to be performed.

[0049] Fourthly, this application provides a computer program product comprising computer instructions; when some or all of the computer instructions are executed on a computer, the method described in any of the first aspects above is performed. Attached Figure Description

[0050] Figure 1 A schematic diagram of the merging boundary;

[0051] Figure 2 A schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application;

[0052] Figure 3 A software structure block diagram of an electronic device provided in an embodiment of this application;

[0053] Figure 4 A schematic diagram illustrating the process of a mobile phone entering HDR shooting mode;

[0054] Figure 5 Flowchart of the image processing method provided in the embodiments of this application Figure 1 ;

[0055] Figure 6 Flowchart of the image processing method provided in the embodiments of this application Figure 2 ;

[0056] Figure 7 Flowchart of the image processing method provided in the embodiments of this application Figure 3 ;

[0057] Figure 8 Flowchart of the image processing method provided in the embodiments of this application Figure 4 ;

[0058] Figure 9 Flowchart of the image processing method provided in the embodiments of this application Figure 5 ;

[0059] Figure 10 Flowchart of the image processing method provided in the embodiments of this application Figure 6 ;

[0060] Figure 11 Flowchart of the image processing method provided in the embodiments of this application Figure 7 ;

[0061] Figure 12 This is a schematic diagram illustrating the conversion of a RAW image file to an RGB image file as provided in an embodiment of this application.

[0062] Figure 13 Flowchart of the image processing method provided in the embodiments of this application Figure 8 ;

[0063] Figure 14 This is a schematic diagram of the mean filtering process provided in an embodiment of this application;

[0064] Figure 15 Flowchart of the image processing method provided in the embodiments of this application Figure 9 ;

[0065] Figure 16 Flowchart of the image processing method provided in the embodiments of this application Figure 10 ;

[0066] Figure 17 Flowchart of the image processing method provided in the embodiments of this application Figure 10 one. Detailed Implementation

[0067] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0068] Before introducing specific embodiments, for ease of understanding, some concepts related to the embodiments of this application are explained by way of example for reference.

[0069] Image fusion is the process of integrating information from multiple images (i.e., two or more images) to generate a new image. The fusion result leverages the spatiotemporal correlation and information complementarity of the multiple images, resulting in a more comprehensive and clearer description of the scene. It should be understood that multiple images may have been acquired under different conditions. Therefore, image registration is often required during image fusion to align images acquired under different conditions. Furthermore, when processing images with different exposure levels, exposure alignment is often necessary to adjust the brightness of images with different exposures to the same level.

[0070] Image registration refers to the process of mapping one or more floating images onto a reference image through spatial transformation, so that points at the same spatial location in the one or more floating images and the reference image correspond one-to-one. The reference image serves as the baseline in the image registration process, while the floating images are the objects to be spatially transformed during the process.

[0071] Reference image: The image used as a reference during image registration to provide spatial positioning and orientation information for other images. The reference image remains unchanged during registration.

[0072] Floating image: In contrast to the reference image, the floating image is the image that needs to be registered onto the reference image. During the registration process, the floating image undergoes spatial transformations (such as translation, rotation, scaling, etc.) according to the registration algorithm to make it consistent with the reference image in spatial position.

[0073] Exposure alignment refers to adjusting images with different exposure levels to the same brightness level in image processing, so as to facilitate subsequent image fusion or analysis.

[0074] RAW image format (RAW image file): also known as digital negative, refers to the digital signal obtained by the image sensor of electronic devices such as digital cameras, scanners or film scanners, which converts the captured light source signal into a digital signal.

[0075] RGB image format (RGB image file) is an image format that uses a combination of three color channels—red, green, and blue—to represent colors. It is mainly used to display colors on monitors or screens.

[0076] Mean filtering is a linear smoothing filtering method mainly used to eliminate noise and details in an image. It replaces the pixel attribute value of the current pixel with the average brightness of the neighboring pixels in the image.

[0077] Neighborhood: In mean filtering, the neighborhood refers to the region within a preset range centered on the current pixel. This region is usually a square that includes the current pixel and the surrounding pixels. The size of the neighborhood is defined by the area of ​​the surrounding pixels (such as 3x3, 5x5, etc.).

[0078] Black level (bl): refers to the brightness level of the black area on a monitor or screen. Black level describes the minimum brightness that the darkest part of the screen can achieve.

[0079] Poisson noise, also known as shot noise, is noise generated due to the particle nature of light. It follows a Poisson distribution, and its main characteristic is that its intensity is positively correlated with the pixel value itself, introducing random variation into each pixel value in the image.

[0080] Gaussian noise, also known as white noise, is a common type of noise in image processing. The probability density function of Gaussian noise follows a Gaussian distribution (normal distribution). The characteristics of Gaussian noise are that its power is constant at any given frequency, and its amplitude is symmetrical in all directions.

[0081] The image processing method provided in the embodiments of this application is illustrated below.

[0082] When taking photos with electronic devices such as smartphones and digital cameras, images captured in a single exposure generally struggle to simultaneously capture detail in both bright and dark areas of a high dynamic range scene. For example, with low exposure, a single image can capture detail in bright areas, but dark areas are difficult to capture due to underexposure or high signal-to-noise ratio. With high exposure, a single image can better capture detail in dark areas, but bright areas will lose detail due to overexposure.

[0083] To capture all image details in both bright and dark areas of a high dynamic range (HDR) scene simultaneously, electronic devices can obtain HDR images in the camera's HDR shooting mode (i.e., high dynamic range mode or high dynamic range shooting mode). In HDR shooting mode, the electronic device often needs to take multiple images of the high dynamic range scene with different exposure settings, resulting in images with varying exposure levels. Among these images with different exposure levels, the lower-exposure image better recovers details in bright areas, while the higher-exposure image better recovers details in dark areas. Therefore, these multiple images with different exposure levels can be merged into a single HDR image, which displays rich details in both bright and dark areas.

[0084] In the process of fusing multiple images at different exposure levels into a single HDR image, it is often necessary to align the exposure of these images based on exposure parameters to ensure that their brightness levels are roughly consistent before image fusion. However, due to factors such as sensor nonlinearity in overexposed or underexposed areas, quantization errors in exposure parameters, or transmission errors in exposure parameters, there can be discrepancies between the exposure parameters input from the front end and the actual exposure parameters at the time of capture. This affects the accuracy of exposure alignment and consequently the image fusion effect. For example, the theoretical exposure calculated based on the front end's exposure parameters may deviate from the actual exposure at the time of capture, resulting in low accuracy of the theoretical exposure factor calculated from the theoretical exposure. Consequently, after aligning the images at different exposure levels according to the theoretical exposure factor, the brightness may not be consistent, leading to fusion boundaries in the resulting HDR image. The larger the dynamic range of the scene, the more pronounced the fusion boundaries in the HDR image. Figure 1 This is a schematic diagram of the merging boundary. Figure 1 of (a), Figure 1 (b) and Figure 1In (c), three fused HDR images are illustrated, with dashed boxes exemplarily marking several fusion boundaries in the HDR images. Figure 1 For example, in (c), Figure 1 In the HDR image shown in (c), the sky region has a relatively obvious blending boundary.

[0085] To address the aforementioned issues, this application provides an image processing method applicable to HDR photography scenarios (i.e., scenarios where photos are taken using HDR mode). In HDR photography scenarios, by executing the aforementioned image processing method, the electronic device can perform more accurate exposure alignment processing on multiple images with different exposure levels captured in HDR mode, that is, to make the brightness of the multiple images consistent or nearly consistent. Subsequently, the multiple images with exposure alignment can be used to fuse and generate an HDR image, making the boundary transitions of the HDR image more natural and improving the image quality of the HDR image.

[0086] In other words, the electronic device can perform preliminary brightness adjustments on multiple images at different exposure levels based on exposure parameters, and then further adjust the brightness based on the pixel attribute values ​​of the images after the preliminary brightness adjustment, so as to make the brightness of the multiple images consistent and achieve more accurate exposure alignment. Consequently, when using multiple images with more accurate exposure alignment for image fusion, the transition is more natural.

[0087] Specifically, in this embodiment, the electronic device uses one image from the images to be processed obtained under different exposure parameters as a reference image, and the other images to be processed as images to be aligned. Based on the exposure parameters, it performs initial brightness adjustment on the images to be aligned, using the reference image as a benchmark, which helps to initially reduce the brightness difference between the images to be aligned and the reference image. Further, the electronic device obtains exposure compensation values ​​based on the pixel values ​​or brightness values ​​of the images under different exposure levels, and then readjusts the brightness of the images to be aligned based on the exposure compensation values ​​to obtain the exposure-aligned target image, enhancing the brightness consistency between the reference image and the target image and improving the exposure alignment effect. Furthermore, the image processing method provided in this embodiment helps optimize the image fusion effect when subsequently synthesizing HDR images using the target image and the reference image, reducing the probability of fusion boundaries appearing in the HDR image.

[0088] The aforementioned electronic device can be used to implement the image processing method provided in the embodiments of this application. The electronic device can be a mobile phone, tablet computer, laptop computer, smart wearable device (smartwatch, smart bracelet, smart helmet, smart glasses, etc.), virtual reality device, or other terminal with image capture capabilities. It should be noted that the electronic device can also be a server; that is, after a device with image capture capabilities continuously captures multiple images in HDR shooting mode, it can send them to a server, which will then execute the image processing method provided in the embodiments of this application. It should be understood that the embodiments of this application do not impose special limitations on the specific form of the electronic device. The following illustration uses the electronic device as a terminal.

[0089] Figure 2 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application.

[0090] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, display screen 170, camera 180, etc.

[0091] It should be noted that, Figure 2 The structure shown does not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include... Figure 2 The components shown may include more or fewer components, or the electronic device 100 may include... Figure 2 The components shown may be a combination of certain components, or the electronic device 100 may include... Figure 2 Sub-components of some of the components shown. Figure 2 The components shown can be implemented in hardware, software, or a combination of both.

[0092] Processor 110 may include one or more processing units. For example, processor 110 may include at least one of the following processing units: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and neural network processing unit (NPU). These different processing units may be independent devices or integrated devices. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0093] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0094] Figure 2 The connection relationships between the modules shown are merely illustrative and do not constitute a limitation on the connection relationships between the modules of the electronic device 100. Optionally, the modules of the electronic device 100 may also adopt a combination of various connection methods described in the above embodiments.

[0095] The charging management module 140 is used to receive power from the charger.

[0096] The mobile communication module 150 can provide a wireless communication solution for use in the electronic device 100.

[0097] Similar to the mobile communication module 150, the wireless communication module 160 can also provide a wireless communication solution for use on the electronic device 100.

[0098] Electronic device 100 can perform shooting functions through ISP, camera 180, video codec, GPU, display 170 and application processor.

[0099] The Information Service Provider (ISP) is used to process data fed back from the camera 180. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimizations on image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be integrated into the camera 180.

[0100] Camera 180 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include one or N cameras 180, where N is a positive integer greater than 1.

[0101] In some embodiments of this application, the camera 180 can continuously capture multiple images of the same scene in HDR shooting mode. For example, in HDR shooting mode, when a user triggers a shooting operation, the camera continuously captures multiple images. Furthermore, the image processing method in this application embodiment achieves exposure alignment of the multiple images for subsequent synthesis of an HDR image.

[0102] The hardware system of electronic device 100 has been described in detail above. The software system of electronic device 100 is described below. The software system can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment takes a layered architecture as an example to exemplarily describe the software system of electronic device 100.

[0103] Figure 3 A software structure block diagram of an electronic device provided in an embodiment of this application.

[0104] A layered architecture divides the system into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the system is divided into five layers, from top to bottom: the application layer, the application framework layer, the hardware abstraction layer, the kernel layer, and the hardware layer.

[0105] The application layer may include a series of application packages. In this embodiment, the application package may include applications such as camera and gallery.

[0106] The application framework (FWK) layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions. In this embodiment, the application framework layer may include a camera service.

[0107] The hardware abstraction layer (HAL) is an interface layer located between the application framework layer and the kernel layer, providing a virtual hardware platform for the operating system. In this embodiment, the hardware abstraction layer may include a camera hardware abstraction layer and a camera algorithm module.

[0108] The camera hardware abstraction layer can provide virtual hardware for the camera device. The camera algorithm module may include the operating code and data required for the electronic device involved in the embodiments of this application to perform shooting.

[0109] For example, the camera algorithm module includes an image processing unit, which is used to execute the image processing method provided in the embodiments of this application.

[0110] The kernel layer is the layer between hardware and software. It includes drivers for various hardware components. For example, the kernel layer may include camera drivers.

[0111] The hardware layer may include camera modules.

[0112] To facilitate understanding, let's take the example of an electronic device capturing 100 HDR images in a single shot, combined with... Figure 3 The software system architecture in this application, and the image processing method in the embodiments thereof, are briefly described as follows:

[0113] 1. Respond to the photo-taking operation and generate a request to capture an HDR image.

[0114] For example, after the camera app launches, it detects that the user has selected the photo capture control in HDR mode. In response to the user's selection, the camera app sends a request to the camera service to capture an HDR image. Upon receiving the HDR image capture request, the camera service forwards it to the hardware abstraction layer.

[0115] 2. Respond to the request to capture HDR images, and collect image data under multiple sets of exposure parameters to obtain multiple image data.

[0116] For example, after receiving a request to capture an HDR image, the camera hardware abstraction module in the hardware abstraction layer initializes the camera module, sends multiple sets of exposure parameters to the camera driver, and triggers the camera module to acquire multiple image data. The camera driver sends multiple sets of exposure parameters to the camera module, and after the camera module acquires multiple image data under each of these exposure parameters, it sends the multiple image data to the camera driver. Upon receiving the multiple image data, the camera driver sends them to the image processing unit.

[0117] 3. Process multiple image data to obtain multiple images with consistent or nearly consistent brightness.

[0118] For example, after receiving multiple image data, the image processing unit can execute the image processing method proposed in this embodiment to process the multiple image data and achieve exposure alignment. After exposure alignment, the brightness of the multiple images is consistent or nearly consistent. Furthermore, based on the multiple images after exposure alignment, a higher quality HDR image can be generated by fusion, effectively improving the problem of fusion boundaries in HDR images. The image processing unit sends the processed HDR image to the image library for storage.

[0119] Next, this application will use a mobile phone as an example to illustrate the concept of electronic device 100, but it should be understood that the form of electronic device 100 is not limited to a mobile phone.

[0120] The following combination Figure 4 This section explains how to enter HDR shooting mode on your phone. It should be noted that... Figure 4 The illustrative representation of one way a mobile phone enters HDR shooting mode does not constitute a limitation on how a mobile phone enters HDR shooting mode.

[0121] Please see Figure 4 , Figure 4 This is a schematic diagram illustrating the process of a mobile phone entering HDR shooting mode. Figure 4 As shown in (a), on the phone's interface 101, the phone displays multiple applications, including a camera 102. In response to a user's selection of 102 on interface 101, the phone can display interface 103 of the camera application, i.e., as shown in (a). Figure 4 As shown in (b) of the diagram. In interface 103, the phone displays multiple shooting modes in the shooting mode area 104, which may include large aperture, night scene, portrait, photo, video, and multi-lens video. Figure 4 As shown in (c), the phone can display more shooting modes in response to a user's swipe-left operation in the shooting mode area 104, including HDR shooting 105. Figure 4As shown in (d), the mobile phone can display interface 106 in response to the user's selection operation of HDR photo 105. This interface 106 can be used to execute the image processing method provided in the embodiments of this application.

[0122] Figure 5 Flowchart of the image processing method provided in the embodiments of this application Figure 1 Please see. Figure 5 After the phone enters HDR shooting mode, it displays the HDR shooting interface. The phone can respond to the user's selection of the shooting control 107 and capture multiple images under different exposure parameters, resulting in multiple images with different exposure levels. For example, image 1 is captured under exposure parameter 1, image 2 is captured under exposure parameter 2, and image 3 is captured under exposure parameter 3. The phone can use the image processing method provided in this application embodiment to perform exposure alignment on multiple images with different exposure levels, improving the consistency of brightness levels of images 1, 2, and 3. For example, taking the pixel attribute value as the brightness value, the brightness values ​​of images 1, 2, and 3 at the same pixel position are brightness value 1, brightness value 2, and brightness value 3, respectively. After exposure alignment, the brightness value of images 1, 2, and 3 at the same pixel position is brightness value 2. The phone can then fuse multiple images with different exposure levels to obtain an HDR image with more natural boundary transitions.

[0123] The image processing method provided in this application embodiment is applicable to exposure alignment of multiple images. The image format can be a RAW image file, an RGB image file, a blue-green-red (BGR) image file, a YUV image file, a YCbCr image file, a CMYK image file, or a CIE L*a*b* image file.

[0124] Figure 6 Flowchart of the image processing method provided in the embodiments of this application Figure 2 .like Figure 6 As shown, the image processing method provided in this application includes:

[0125] Step 601: The mobile phone acquires N images to be processed (i.e., the first image) in high dynamic range mode; wherein, the N images to be processed are obtained by the mobile phone continuously acquiring the same target scene under N different exposure parameter settings, and the N images to be processed include a reference image and N-1 images to be aligned (i.e., the second image); N is a positive integer greater than or equal to 2.

[0126] Exposure parameters can refer to the settings of parameters related to exposure in a mobile phone camera when taking an image. For example, exposure parameters may include ISO sensitivity and / or exposure time.

[0127] If the exposure of the image to be aligned (i.e., the product of ISO and exposure time) is less than that of the reference image, the image to be aligned can be called an underexposed image. If the exposure of the image to be aligned is greater than that of the reference image, the image to be aligned can be called an overexposed image.

[0128] For example, the way a mobile phone determines a reference image from N images to be processed may include either method one or method two:

[0129] Method 1: The mobile phone can determine the reference image based on the pixel attribute values ​​of N images to be processed; where the pixel attribute values ​​include pixel values ​​or brightness values.

[0130] For example, the mobile phone can use the first image with the most sixth pixels from a plurality of first images as the reference image. The sixth pixel is a pixel whose pixel attribute value satisfies a second preset proximity condition with respect to the average of the pixel attribute values ​​of the plurality of images. For example, the second preset proximity condition could refer to a pixel whose sixth pixel falls within a third range interval, where the third range interval is the interval that is close to the average of the pixel attribute values ​​of the plurality of images. Another example is that the second preset proximity condition could refer to the difference between the sixth pixel and the average of the pixel attribute values ​​of the plurality of images being less than a certain threshold.

[0131] For example, the mobile phone can also normalize the pixel attribute values ​​of the pixels in N images to be processed, and select the image with the most pixels whose pixel attribute values ​​fall within a certain threshold range as the reference image. For example, the certain threshold range can be [0.45, 0.55].

[0132] Method 2: The mobile phone can determine the baseline image based on the exposure of N images to be processed.

[0133] For example, the mobile phone can determine the first image whose exposure is closest to the average exposure of the multiple first images from multiple first images as the reference image, where the average exposure is the average of the multiple exposures corresponding to the multiple first images.

[0134] Step 602: The mobile phone adjusts the brightness of the image to be aligned based on the exposure parameters corresponding to the reference image and the exposure parameters corresponding to the image to be aligned, to obtain the image to be aligned after brightness adjustment (i.e., the third image).

[0135] It should be noted that when the exposure of the image to be aligned is less than that of the reference image, the brightness is adjusted to increase the brightness of the image to be aligned based on the theoretical exposure factor. When the exposure of the image to be aligned is greater than that of the reference image, the brightness is adjusted to decrease the brightness of the image to be aligned based on the theoretical exposure factor.

[0136] For details, please refer to Figure 7The mobile phone can calculate the theoretical exposure factor based on the exposure parameters of the reference image and the image to be aligned. The phone then adjusts the brightness of the image to be aligned based on this theoretical exposure factor. For example, adjusting the brightness of the image to be aligned can be done by dividing the pixel attribute value of each pixel in the image by the theoretical exposure factor. The theoretical exposure factor can be obtained using the following formula.

[0137]

[0138] Where α represents the theoretical exposure factor, ISO S This refers to the ISO sensitivity parameter when the phone captures the image to be aligned. N t represents the sensitivity parameter when the mobile phone captures a reference image. S t represents the exposure time when the phone captures the image to be aligned. N ISO indicates the exposure time when the phone captures the reference image. S t S This indicates the exposure of the image to be aligned, in ISO. N t N This indicates the exposure level of the reference image.

[0139] In this way, by adjusting the brightness of the image to be aligned based on the exposure parameters of the reference image and the image to be aligned, the brightness difference between the image to be aligned and the reference image can be initially reduced, which helps to improve the brightness consistency between the image to be aligned and the reference image.

[0140] Step 603: The mobile phone determines the exposure compensation value corresponding to the fifth image based on the difference information between the pixel attribute values ​​at the same pixel position in the fourth and fifth images; the brightness of the fifth image is adjusted according to the exposure compensation value to obtain the sixth image (i.e., the target image).

[0141] The sixth image is closer in brightness to the reference image than the fifth image; the fourth image can be the reference image or an image preprocessed from the reference image; the fifth image can be the third image or an image preprocessed from the third image. Pixel attribute values ​​include pixel values ​​or brightness values.

[0142] It is understood that preprocessing may include one or more of the following: truncation, mean filtering, sieving, and removing second pixels from the reference image and third pixels from the third image.

[0143] It should be noted that pixel position can refer to the pixel coordinates of a pixel within an image. The pixel attribute values ​​at the same pixel position in the fourth and fifth images refer to the pixel attribute values ​​of the fourth and fifth images at that pixel position. The difference between the pixel attribute values ​​at the same pixel position in the fourth and fifth images can be represented by the difference between them, or by the ratio between them. This difference in pixel attribute values ​​at the same pixel position indicates the content consistency between the fourth and fifth images at that location; a larger difference indicates lower content consistency, and a smaller difference indicates higher content consistency.

[0144] Understandably, please refer to Figure 8 For each pixel location, the mobile phone can determine the weight corresponding to that pixel location based on the first signal-to-noise ratio (SNR) of the fourth image at that pixel location and the second SNR of the fifth image at that pixel location. Here, the first SNR is the ratio between the first pixel attribute value and the first noise intensity information; the first pixel attribute value is the pixel attribute value of the fourth image at that pixel location; the first noise intensity information is used to characterize the noise intensity of the fourth image at that pixel location; the second SNR is the ratio between the second pixel attribute value and the second noise intensity information; the second pixel attribute value is the pixel attribute value of the fifth image at that pixel location; and the second noise intensity information is used to characterize the noise intensity of the fifth image at that pixel location.

[0145] Please continue reading. Figure 8 The mobile phone determines the weight corresponding to the pixel position based on the first signal-to-noise ratio and the second signal-to-noise ratio, and performs a weighted summation of the differences between the pixel attribute values ​​of each pixel position to obtain the exposure compensation value.

[0146] Specifically, the mobile phone determines the weight corresponding to the pixel position based on the first signal-to-noise ratio and the second signal-to-noise ratio. This can be done by multiplying the first signal-to-noise ratio and the second signal-to-noise ratio, or by summing the first signal-to-noise ratio and the second signal-to-noise ratio.

[0147] It should be noted that the signal-to-noise ratio at a pixel location can indicate the ability of the reference image and the image to be aligned to reproduce the shooting scene (i.e., the target scene in this embodiment) at that pixel location. The higher the signal-to-noise ratio, the stronger the ability to reproduce the shooting scene, and the lower the signal-to-noise ratio, the weaker the ability to reproduce the shooting scene.

[0148] In this way, the weight of the pixel position can be obtained by multiplying the first signal-to-noise ratio and the second signal-to-noise ratio, or by summing the first signal-to-noise ratio and the second signal-to-noise ratio. The weight is positively correlated with the signal-to-noise ratio. The pixel with the stronger ability to reproduce the shooting scene has a larger weight, which helps to obtain a more accurate exposure compensation value.

[0149] For example, Figure 9 Taking the determination of the weight corresponding to the pixel position based on the product of the first signal-to-noise ratio and the second signal-to-noise ratio, and the difference information between the pixel attribute values ​​corresponding to the pixel position as the pixel attribute value ratio, as an example, the process of calculating the exposure compensation value is shown. The process of calculating the exposure compensation value may include the following steps 603-1 to 603-4:

[0150] Step 603-1: The mobile phone adjusts the ISO sensitivity parameter in the exposure parameters corresponding to the fourth image N. N The Poisson noise intensity K of the fourth image N was obtained by fitting. N and Gaussian noise intensity B N According to the ISO sensitivity parameter in the exposure parameters corresponding to the fifth image S S The Poisson noise intensity K of the fifth image S was obtained by fitting. S and Gaussian noise intensity B S .

[0151] Specifically, the Poisson noise intensity and Gaussian noise intensity of the fourth image, and the Poisson noise intensity and Gaussian noise intensity of the image to be aligned are calculated using the following formulas.

[0152] K N =K A ISO N +K B

[0153] B N =B A ISO N ISO N +B B ISO N +B C

[0154] K S =K A ISO S +K B

[0155] B S =B A ISO S ISO S +B B ISO S +B C

[0156] Among them, K N The fourth image represents the Poisson noise intensity, ISO. N B represents the ISO sensitivity parameter when capturing the fourth image. N K represents the Gaussian noise intensity of the fourth image. S B represents the Poisson noise intensity of the fifth image. S The ISO value represents the Gaussian noise intensity of the fifth image. S K represents the ISO sensitivity parameter when capturing the fifth image. A K represents the slope fitting term of the Poisson noise curve. B B represents the intercept fitting term of the Poisson noise curve. A B represents the quadratic parameter of the Gaussian noise curve. B B represents the parameter of the first-order term of the Gaussian noise curve. C This represents the intercept fitting term of the Gaussian noise curve.

[0157] Step 603-2: The mobile phone calculates the noise standard deviation of the seventh pixel in the fourth image based on the Poisson noise intensity, Gaussian noise intensity, and pixel attribute value of the seventh pixel; the mobile phone calculates the noise standard deviation of the eighth pixel in the fifth image based on the Poisson noise intensity, Gaussian noise intensity, and pixel attribute value of the eighth pixel. The seventh and eighth pixels are pixel pairs corresponding to the same pixel location.

[0158] It should be noted that the noise standard deviation is used to characterize the noise intensity of a pixel. The larger the noise standard deviation, the greater the noise intensity of the pixel; the smaller the noise standard deviation, the smaller the noise intensity of the pixel. The noise standard deviation can be calculated based on the following formula.

[0159]

[0160] in, This represents the noise standard deviation of the i-th seventh pixel in the fourth image. This represents the pixel attribute value of the i-th seventh pixel in the fourth image. This represents the noise standard deviation of the i-th eighth pixel in the fifth image. This represents the pixel attribute value of the i-th eighth pixel in the fifth image.

[0161] Step 603-3: The mobile phone determines the weight corresponding to the pixel position based on the first signal-to-noise ratio and the second signal-to-noise ratio.

[0162] Specifically, the phone obtains a first signal-to-noise ratio (SNR) based on the ratio of the pixel attribute value of the seventh pixel at the same pixel position to the noise standard deviation, and obtains a second SNR based on the ratio of the pixel attribute value of the eighth pixel at the same pixel position to the noise standard deviation. Based on the first and second SNRs, the phone calculates the weight of the first ratio corresponding to the pixel position.

[0163] Specifically, the weight of the first ratio corresponding to the pixel position can be calculated based on the following formula.

[0164]

[0165] Among them, W i This represents the weight of the first ratio corresponding to the i-th pixel position. The other parameters can be referred to the explanation in step 603-2, and will not be repeated here.

[0166] Step 603-4: The mobile phone calculates the exposure compensation value based on the first ratio corresponding to each pixel position and the weight of the first ratio.

[0167] Specifically, the mobile phone can perform a weighted summation of the first ratios corresponding to each pixel position according to the weights of each first ratio to obtain the exposure compensation value.

[0168] For example, the exposure compensation value can be calculated using the following formula:

[0169]

[0170] Where γ represents the exposure compensation value, r i This represents the first ratio corresponding to the i-th pixel position. The other parameters can be referred to in the explanation of step 603-2, and will not be repeated here.

[0171] Thus, the image processing method provided in this application, by determining a reference image from images to be processed obtained under different exposure parameters, and using images other than the reference image as images to be aligned, performs preliminary brightness adjustment on the images to be aligned according to the exposure parameters, which helps to reduce the brightness difference between the images to be aligned and the reference image. By performing exposure alignment on the images to be aligned according to the exposure compensation value to obtain a sixth image, it helps to further reduce the brightness difference between the sixth image and the reference image, enhancing the brightness consistency between the sixth image and the reference image. Furthermore, the image processing method provided in this application helps to optimize the exposure alignment effect between the sixth image and the reference image, thereby helping to optimize the image fusion effect and reduce the probability of fusion boundaries appearing.

[0172] Through in-depth research, the inventors of this application discovered that, based on the above-mentioned formula for noise standard deviation, it can be deduced that: since the pixel attribute values ​​of bright areas in the image to be processed are relatively large, the magnitude of the noise standard deviation is mainly determined by the coefficient of the pixel attribute value, i.e., Poisson noise, while the intensity of Gaussian noise is negligible relative to the pixel attribute value and the magnitude of Poisson noise. Therefore, the formulas involved in steps 603-1 to 603-4 above can be simplified based on this, and the simplified formula for exposure compensation value can be derived. Thus, the sixth image after exposure alignment can be obtained based on the simplified exposure compensation value, reducing the amount of calculation, improving the efficiency of exposure alignment, and without reducing the accuracy of exposure alignment.

[0173] Since the bright areas in the image are dominated by Poisson noise, the Gaussian noise intensity is negligible. Therefore, the Gaussian noise intensity B in the fifth image can be reduced. S The Gaussian noise intensity B in the fourth image N The intercept fitting term K of the Poisson noise curve B Ignoring these implications, the formulas involved in step 603-1 are simplified to obtain the following formula:

[0174] K N =K A ISO N

[0175] K S =K A ISO S

[0176] For an explanation of the parameters in Formula 1 above, please refer to step 603-1, which will not be repeated here.

[0177] In Formula 1 above, K N and K S Substituting the values ​​into the formulas involved in step 603-2, we obtain the following formula two:

[0178]

[0179] For an explanation of the parameters in Formula 2 above, please refer to step 603-2, which will not be repeated here.

[0180] Substituting the parameters from Formula 2 into the formula involved in step 603-3, we obtain the following Formula 3:

[0181]

[0182] For an explanation of the parameters in Formula 3 above, please refer to steps 603-2 and 603-3, which will not be repeated here.

[0183] Since the first ratio is the ratio of the pixel attribute value of the eighth pixel in the fifth image to the pixel attribute value of the seventh pixel in the corresponding fourth image, and the pixel positions of the eighth and seventh pixels are the same, we can obtain the following formula four:

[0184]

[0185] Where, r i This is the first ratio at the i-th pixel position. For explanations of other parameters, please refer to step 603-2, which will not be repeated here.

[0186] Dividing the pixel attribute value of the seventh pixel in the unprocessed fifth image by the product of the theoretical exposure multiplier and the exposure compensation value yields the pixel attribute value of the seventh pixel in the exposure-aligned sixth image. After exposure alignment, the pixel attribute value of the seventh pixel in the sixth image is considered to be the same as the pixel attribute value of the eighth pixel in the fourth image, which has the same pixel position as the seventh pixel. Based on this, combining the formula for the exposure multiplier and Formula 4, we can obtain the following Formula 5:

[0187]

[0188] For an explanation of the parameters in Formula 5 above, please refer to the previous explanation; it will not be repeated here.

[0189] Substituting the parameters from Formula 5 into Formula 3, we obtain Formula 6 as follows:

[0190]

[0191] For an explanation of the parameters in Formula 6 above, please refer to the previous explanation; it will not be repeated here.

[0192] Substitute the parameters from Formula 6 above into the formula involved in step 603-4, except... In addition, the remaining parameters on the right side of the equals sign in Formula 6 are all image-related constants. Therefore, the final weight is only related to the pixel attribute values ​​of the fourth image, and the weight corresponding to each pixel position is positively correlated with the pixel attribute value of the fourth image. After further simplification, the exposure alignment compensation value γ can be represented by Formula 7.

[0193]

[0194] Thus, by calculating the exposure compensation value using the simplified formula seven derived from the derivation, the computational load can be reduced, and the exposure compensation value can be obtained more quickly. Consequently, the sixth image after exposure alignment can be obtained more quickly based on the exposure compensation value, thereby improving the efficiency of exposure alignment.

[0195] To facilitate understanding, the image processing method provided in this application embodiment will be described below using two RAW image files as an example. However, it should be understood that this application is not limited to processing only two images; it can also process more acquired images, with the same principle as processing two images. These two RAW images were obtained by taking photos of the same scene with a mobile phone under different exposure parameters. Figure 10 As shown, the image processing method provided in this application includes:

[0196] Step 1001: The mobile phone acquires two images to be processed (i.e., the first image). The format of the two images to be processed is raw image file (RAW image file), one of which is the reference image and the other is the image to be aligned (i.e., the second image).

[0197] Understandably, the method by which the mobile phone determines the reference image can be found in the description of step 601, and will not be repeated here.

[0198] Step 1002: The mobile phone performs image registration on the image to be aligned based on the reference image to obtain the image to be aligned after image registration; the scene consistency between the image to be aligned after image registration and the reference image is higher than the scene consistency between the image to be aligned before image registration and the reference image.

[0199] It should be noted that the mobile phone can use the reference image as the reference image and the image to be aligned as the floating image. During the image registration process, the reference image remains unchanged, and the image to be aligned is transformed according to the reference image to obtain the image to be aligned after image registration.

[0200] For details, please refer to Figure 11 The image registration of the reference image and the image to be aligned by the mobile phone may include steps 1002-1 to 1002-2:

[0201] Step 1002-1: The mobile phone matches the features in the image to be aligned with the features in the reference image, and estimates the transformation model of the image to be aligned relative to the reference image based on the matched feature points.

[0202] To estimate the transformation model of the image to be aligned relative to the reference image, the parameters of the transformation model can be obtained by minimizing the error between the matched feature points, and then the estimated transformation model can be determined based on the parameters of the transformation model.

[0203] Step 1002-2: The mobile phone applies the estimated transformation model to the image to be aligned, and obtains the image to be aligned after image registration.

[0204] Common transformation models include translation, rotation, scaling, affine transformation, and perspective transformation.

[0205] In this way, by performing image registration on the image to be aligned based on the reference image, the scene consistency between the image to be aligned and the reference image can be improved, providing an accurate basis for subsequent format conversion and image fusion.

[0206] Step 1003: The mobile phone uses the RAW to RGB (RAW to RGB, RAW2RGB) algorithm to convert the reference image and the image to be aligned from the RAW image file to the RGB image file.

[0207] Figure 12 This is a schematic diagram illustrating the conversion of a RAW image file to an RGB image file, as provided in an embodiment of this application. Please refer to [link / reference]. Figure 12 Taking the smallest color unit of a RAW image file as a Bayer pattern as an example, a mobile phone can convert a RAW image file into an RGB image file.

[0208] Specifically, the mobile phone can convert the reference image and the image to be aligned, which are in RAW image file format, into RGB image files according to the following formula.

[0209] R′=(R-bl)gain R

[0210] G′=((Gr+Gb) / 2-bl)gain G

[0211] B′=(B-bl)gain B

[0212] Where R′ represents the red channel component in the RGB image file, G′ represents the green channel component in the RGB image file, and B′ represents the blue channel component in the RGB image file; R represents the red channel component in the RAW image file, Gb represents the green channel component in the RAW image file, Gr represents the green channel component in the RAW image file, and B represents the blue channel component in the RAW image file; bl represents the black level; gainR represents the AWB gain of the red channel, gainG represents the AWB gain of the green channel, and gainB represents the AWB gain of the blue channel.

[0213] In a color image, pixel values ​​represent the color of a pixel, typically using the values ​​of the three RGB components. When processing a color image, it's necessary to process the RGB components of each pixel to achieve the desired effect.

[0214] Understandably, after the reference image and the image to be aligned are converted to RGB image files, the pixel value of each pixel in the reference image and the image to be aligned can be represented by (R′, G′, B′), and the brightness value of each pixel in the reference image and the image to be aligned can be obtained by the following formula:

[0215] Y=0.299R′+0.578G′+0.114B′

[0216] Where Y is the brightness value of the pixel.

[0217] Step 1004: The mobile phone adjusts the brightness of the image to be aligned (i.e., the second image) according to the exposure parameters corresponding to the reference image and the exposure parameters corresponding to the image to be aligned, to obtain the image to be aligned after brightness adjustment (i.e., the third image).

[0218] Understandably, the method by which the phone adjusts the brightness of the image to be aligned can be found in the description of step 602, and will not be repeated here.

[0219] Step 1005: The mobile phone performs a truncation process on the image to be aligned after brightness adjustment, resulting in a truncated image to be aligned.

[0220] Specifically, the mobile phone sets the pixel attribute value of the first pixel in the image to be aligned after brightness adjustment to the maximum pixel attribute value of the reference image, thus obtaining the truncated image to be aligned; where the first pixel refers to the pixel with a pixel attribute value greater than the maximum pixel attribute value.

[0221] For example, such as Figure 13 As shown, when the maximum pixel attribute value of the reference image is 250, the brightness value of pixels with a brightness value greater than 250 in the brightness-adjusted image to be aligned can be set to 250, while the brightness value of pixels with a brightness value less than or equal to 250 in the brightness-adjusted image to be aligned remains unchanged, thus obtaining the truncated image to be aligned.

[0222] In this way, by truncating the image to be aligned after brightness adjustment, we can avoid excessive fluctuations in pixel attribute values ​​between the image to be aligned and the reference image, and reduce the noise level of the image to be aligned and the reference image.

[0223] Step 1006: The mobile phone performs mean filtering on the image to be aligned and the reference image to obtain the mean-filtered image to be aligned and the mean-filtered reference image.

[0224] Specifically, the mobile phone performs mean filtering on the truncated image to be aligned and the reference image. This can be done by setting the pixel value of each pixel in the truncated image to be aligned and the reference image to the average pixel value of the neighboring pixels, or by setting the brightness value of each pixel in the truncated image to be aligned and the reference image to the average brightness value of the neighboring pixels.

[0225] Taking a neighborhood size of 3×3 as an example, such as Figure 14 As shown, each pixel in the truncated image to be aligned or the reference image occupies a square, and the value in the square is the pixel attribute value of that pixel. Next, through... Figure 14 Figures (a), (b), (c), and (d) illustrate the process of mean filtering on the truncated image to be aligned or the reference image.

[0226] Please see Figure 14 In (a), for each pixel in the truncated image to be aligned or the reference image, if the distance between pixel 1401 and the edge in the truncated image to be aligned or the reference image is greater than or equal to 1 pixel, the neighborhood 1402 includes 9 pixels. By setting the pixel attribute value of the pixel 1401 at the center of the neighborhood 1402 to the average of the pixel attribute values ​​of all pixels in the neighborhood 1402, mean filtering is performed on the truncated image to be aligned or the reference image.

[0227] When the distance between pixel 1401 in the truncated image to be aligned or the reference image and the edge is less than 1 pixel, that is, when pixel 1401 is located on the boundary of the truncated image to be aligned or the reference image, the mobile phone performs mean filtering on the truncated image to be aligned or the reference image in the following four ways:

[0228] Method 1: Do not perform mean filtering on pixel 1401, and keep the pixel attribute value of pixel 1401 unchanged.

[0229] Method 2: Please refer to Figure 14 In (b), the neighborhood 1402 includes 6 pixels. By setting the pixel attribute value of the pixel 1401 at the center of the neighborhood 1402 to the average of the pixel attribute values ​​of all pixels in the neighborhood 1402, mean filtering is performed on the truncated image to be aligned or the reference image.

[0230] Method 3: Please refer to Figure 14In step (c), the mobile phone expands the boundary of the truncated image to be aligned or the reference image, filling the expanded boundary with pixel attribute values ​​of 0 to obtain neighborhood 1402, which includes 9 pixels. By setting the pixel attribute value of the center pixel 1401 of neighborhood 1402 to the average of the pixel attribute values ​​of all pixels in neighborhood 1402, mean filtering is performed on the truncated image to be aligned or the reference image.

[0231] Method 4: Please refer to Figure 14 In step (d), the boundary of the truncated image to be aligned or the reference image is expanded. The expanded boundary is filled with the pixel attribute value of the closest pixel, resulting in a neighborhood 1402, which includes 9 pixels. By setting the pixel attribute value of the center pixel 1401 of the neighborhood 1402 to the average of the pixel attribute values ​​of all pixels in the neighborhood 1402, mean filtering is performed on the truncated image to be aligned or the reference image.

[0232] In this way, by applying mean filtering to the truncated image to be aligned and the reference image, the pixel attribute values ​​of the pixels in the image can be averaged, which can reduce the noise level and image details of the truncated image to be aligned and the reference image, making the image smoother and softer, and helping to improve the brightness consistency of the truncated image to be aligned and the reference image.

[0233] Step 1007: Remove pixels whose pixel attribute values ​​exceed the second range in the baseline image after mean filtering, and obtain the baseline image after removing pixels that exceed the second range.

[0234] For details, please refer to Figure 15 The range of pixel attribute values ​​in the reference image is [c, d]. The second range interval can be obtained by subtracting the black level bl from the range [c, d], i.e., [c-bl, d-bl], and removing pixels whose pixel attribute values ​​are less than or equal to a certain threshold q from the endpoint. Since the pixel attribute values ​​cannot be negative, when c-bl or d-bl is less than 0, c-bl or d-bl is updated to 0.

[0235] For example, in the baseline image after mean filtering, when the pixel attribute value of a pixel is in the range of [0, 1023], the black level is 64. The second range interval is obtained by first subtracting the black level from the pixel attribute value of the pixel, i.e., [0, 959], and then removing pixels whose pixel attribute value is less than or equal to 10 from the endpoint of [0, 959], resulting in the second range interval of [10, 949].

[0236] In this way, by removing pixels whose pixel attribute values ​​in the mean-filtered reference image exceed the second range, the brightness fluctuation of the reference image can be reduced, which helps to provide a basis for subsequent removal of abnormal pixels.

[0237] Step 1008: Remove the second pixel from the mean-filtered reference image and remove the third pixel from the mean-filtered image to be aligned; the second pixel and the third pixel correspond to the same pixel position, and the difference in pixel attribute values ​​between the second pixel and the third pixel exceeds the preset first range.

[0238] Specifically, taking the difference in pixel attribute values ​​between the second and third pixels as an example, and the ratio of the pixel attribute values ​​between the second and third pixels, steps 1007 and 1008 will be explained. Please refer to [link / reference]. Figure 16 The mobile phone can iterate through the pixels of the reference image one by one. For the i-th pixel N(i) in the reference image, it determines whether the pixel attribute value of pixel N(i) is within the second range. If not, it updates i to i+1 and returns to the step of determining whether the pixel attribute value of pixel N(i) is within the second range. If it is, it calculates the ratio of the pixel attribute value of the i-th pixel S(i) in the mean-filtered image to the pixel attribute value of pixel N(i), i.e., the first ratio, where pixel S(i) and pixel N(i) correspond to the same pixel position. It determines whether the first ratio is within the first range. If it is, it retains both pixel S(i) and pixel N(i); otherwise, it removes pixel N(i) from the reference image and pixel S(i) from the mean-filtered image to be aligned. It updates i to i+1 and returns to the step of determining whether the pixel attribute value of pixel N(i) is within the second range.

[0239] It should be noted that since the pixel position of pixel S(i) in the mean-filtered image to be aligned is the same as the pixel position of pixel N(i) in the reference image, the closer the first ratio is to 1, the higher the brightness and content consistency between the mean-filtered image to be aligned and the reference image at that pixel position. If the first ratio exceeds the first range, for example, the first range is [0.5, 1.5], it indicates that the brightness and content consistency between the mean-filtered image to be aligned and the reference image at that pixel position is lower.

[0240] For example, a mobile phone can remove a second or third pixel by setting the pixel attribute value of the second or third pixel to 0 or black level.

[0241] In this way, by removing the second pixel from the reference image and the third pixel from the mean-filtered image to be aligned, the content consistency between the mean-filtered image to be aligned and the reference image can be improved, which helps to improve the brightness consistency between the mean-filtered image to be aligned and the reference image.

[0242] Step 1009: Perform pixel pair filtering on the reference image after removing the second pixel and the image to be aligned after removing the third pixel.

[0243] It should be understood that the remaining pixel pairs in the image to be aligned after screening and the reference image after screening meet the preset conditions.

[0244] The preset conditions include that the remaining pixel pairs correspond to the same pixel position in the reference image after removing the second pixel and the image to be aligned after removing the third pixel, and that the difference information corresponding to each remaining pixel pair and the average difference satisfy the first preset proximity condition; the difference information corresponding to each remaining pixel pair refers to the difference information between the pixel attribute values ​​of the remaining pixel pairs; the average difference is the average value of the difference information corresponding to each remaining pixel pair after filtering.

[0245] Specifically, taking the ratio between pixel attribute values ​​as an example to illustrate step 1009, please refer to [link to relevant documentation]. Figure 17 The mobile phone can use the reference image after removing the second pixel as the first current image and the image to be aligned after removing the third pixel as the second current image. It can then filter out the fourth pixel from the first current image and the fifth pixel from the second current image. The fourth and fifth pixels are pixel pairs corresponding to the same pixel position i, and the ratio of the pixel attribute values ​​of the fourth and fifth pixels (i.e., the first ratio r) is... i If the first ratio and the current average ratio μ do not meet the first preset proximity condition (e.g., the difference between the first ratio and the current average ratio μ is greater than λκ, where κ represents the standard deviation of the first ratio corresponding to the current remaining pixel pair, and λ is a preset coefficient), the current average ratio μ is the average of multiple first ratios corresponding to the current remaining pixel pair; the current remaining pixel pair is the pixel pair remaining in the first current image after removing the fourth pixel and the second current image after removing the fifth pixel. If r i If the pixel value satisfies the first preset proximity condition with the current average ratio μ, then the corresponding pixel is retained.

[0246] It can be understood that i represents the i-th pixel pair or the i-th pixel position.

[0247] Please continue reading. Figure 17The mobile phone can use the first current image after removing the fourth pixel as the new first current image and the second current image after removing the fifth pixel as the new second current image, update i = i + 1, and return to perform iterative processing to remove the fourth pixel from the first current image and the fifth pixel from the second current image until the remaining pixel pairs after removal meet the preset conditions and stop iterating. For example, the preset condition can be that k is less than a certain threshold t.

[0248] In this way, by performing pixel pair filtering on the reference image and the image to be aligned after removing the third pixel, the remaining pixel pairs after filtering meet the preset conditions. This helps to filter out pixels whose corresponding difference information and average difference do not meet the first preset proximity condition, thus providing a basis for subsequent calculation of exposure compensation values.

[0249] Step 1010: The mobile phone calculates the exposure compensation value corresponding to the fifth image based on the difference information between the pixel attribute values ​​at the same pixel position in the filtered reference image (i.e., the fourth image) and the filtered image to be aligned (i.e., the fifth image); and adjusts the brightness of the fifth image according to the exposure compensation value to obtain the sixth image.

[0250] It is understandable that the method by which the mobile phone determines the exposure compensation value corresponding to the fifth image can be found in the description of step 603, and will not be repeated here.

[0251] It should be understood that the above Figure 10 This application provides only one possible example of the image processing method provided in its embodiments. The image processing method provided in this application can also be more... Figure 10 The methods shown may have more or fewer steps. Figure 10 The order of the multiple steps shown can also be adjusted or replaced, and the embodiments of this application do not further limit this.

[0252] Thus, the image processing method provided in this application improves the scene consistency between the image to be aligned and the reference image by using the reference image as the reference image and the image to be aligned as the floating image; adjusting the brightness of the image to be aligned based on the reference image can help reduce the brightness difference between the image to be aligned and the reference image. By truncating and mean filtering the reference image and the image to be aligned, the noise level of both images can be reduced and image details can be minimized. Removing pixels whose attribute values ​​in the mean-filtered reference image exceed the second range reduces brightness fluctuations. Mean filtering of both the reference and image to be aligned further reduces brightness fluctuations. Removing a second pixel from the reference image and a third pixel from the mean-filtered image to be aligned improves the content consistency between the two images, contributing to improved brightness consistency. Pixel pair filtering of the reference and the image to be aligned after removing the third pixel ensures that the remaining pixel pairs meet preset conditions, helping to remove pixels whose difference information and average difference do not meet the first preset proximity condition, providing a basis for subsequent exposure compensation value calculation. Exposure alignment of the image to be aligned based on the exposure compensation value yields a sixth image, further reducing the brightness difference between the sixth and reference images and enhancing their brightness consistency. Furthermore, the image processing method provided in this application helps to optimize the exposure alignment effect between the sixth image and the reference image, thereby helping to optimize the image fusion effect and reduce the probability of fusion boundary occurrence.

[0253] This application also provides an electronic device, which includes at least a memory and one or more processors; the memory is used to store computer instructions, and when one or more processors execute the computer instructions, the electronic device performs the various functions or steps described in the above method embodiments.

[0254] This application also provides a computer-readable storage medium including computer instructions that, when executed on the electronic device, cause the electronic device to perform the functions or steps described in the method embodiments.

[0255] This application also provides a computer program product that, when run on an electronic device, causes the electronic device to perform the functions or steps described in the above method embodiments.

[0256] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0257] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0258] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0259] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0260] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0261] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. An image processing method, characterized in that, include: Acquire multiple first images of the same scene continuously captured in high dynamic range mode; The multiple first images correspond to different exposure parameters; The plurality of first images includes a reference image and a second image other than the reference image; For each second image, the brightness of the second image is adjusted according to the exposure parameters corresponding to the reference image and the second image respectively to obtain a third image; the brightness of the third image is closer to that of the reference image than that of the second image. For each pixel location, the weight corresponding to the pixel location is determined based on the first signal-to-noise ratio of the fourth image at the pixel location and the second signal-to-noise ratio of the fifth image at the pixel location. The fourth image is the reference image or an image preprocessed from the reference image; the fifth image is the third image or an image preprocessed from the third image; wherein, the signal in the signal-to-noise ratio of a pixel location is the pixel attribute value of the pixel location, and the noise in the signal-to-noise ratio of a pixel location is the noise standard deviation; The difference information between the pixel attribute values ​​of the fourth image and the fifth image at the same pixel position is weighted and summed according to the weight corresponding to the pixel position to obtain the exposure compensation value corresponding to the fifth image; the pixel attribute value includes pixel value or brightness value. The brightness of the fifth image is adjusted according to the exposure compensation value to obtain the sixth image; the brightness of the sixth image is closer to that of the reference image than that of the fifth image.

2. The method according to claim 1, characterized in that, The difference information includes the difference between the pixel attribute values ​​of the fourth image and the fifth image at the same pixel location, or the difference information includes the ratio between the pixel attribute values ​​of the fifth image and the fourth image at the same pixel location.

3. The method according to claim 1, characterized in that, The first signal-to-noise ratio is the ratio between the first pixel attribute value and the first noise standard deviation; the first pixel attribute value is the pixel attribute value of the fourth image at the pixel location; the first noise standard deviation is the noise standard deviation of the fourth image at the pixel location. The second signal-to-noise ratio is the ratio between the second pixel attribute value and the second noise standard deviation; The second pixel attribute value is the pixel attribute value of the fifth image at the pixel position; the second noise standard deviation is the noise standard deviation of the fifth image at the pixel position.

4. The method according to claim 3, characterized in that, For each pixel location, determining the weight corresponding to the pixel location based on the first signal-to-noise ratio of the fourth image at the pixel location and the second signal-to-noise ratio of the fifth image at the pixel location includes: For each pixel location, the weight corresponding to the pixel location is determined based on the product of the first signal-to-noise ratio and the second signal-to-noise ratio, or based on the sum of the first signal-to-noise ratio and the second signal-to-noise ratio.

5. The method according to any one of claims 1 to 4, characterized in that, Before performing a weighted summation of the difference information between the pixel attribute values ​​of the fourth image and the fifth image at the same pixel location according to the weight corresponding to the pixel location to obtain the exposure compensation value corresponding to the fifth image, the method further includes: The pixel attribute value of the first pixel in the third image is set to the maximum pixel attribute value of the reference image to obtain the fifth image; wherein, the first pixel refers to the pixel with a pixel attribute value greater than the maximum pixel attribute value.

6. The method according to any one of claims 1 to 4, characterized in that, Before performing a weighted summation of the difference information between the pixel attribute values ​​of the fourth image and the fifth image at the same pixel location according to the weight corresponding to the pixel location to obtain the exposure compensation value corresponding to the fifth image, the method further includes: The reference images are preprocessed sequentially to obtain the fourth image, and the third images are preprocessed sequentially to obtain the fifth image: Mean filtering; The second pixel is removed from the reference image and the third pixel is removed from the third image; the second pixel and the third pixel correspond to the same pixel position, and the difference in pixel attribute values ​​between the second pixel and the third pixel exceeds a preset first range. The reference image and the third image are subjected to pixel pair filtering processing, such that the remaining pixel pairs after filtering meet preset conditions; wherein, the preset conditions include that the remaining pixel pairs correspond to the same pixel position in the reference image and the third image, and that the difference information corresponding to each remaining pixel pair and the average difference satisfy a first preset proximity condition; the difference information corresponding to each remaining pixel pair refers to the difference information between the pixel attribute values ​​of the remaining pixel pairs; the average difference is the average value of the difference information corresponding to each of the remaining pixel pairs after filtering.

7. The method according to claim 6, characterized in that, The step of performing pixel pair filtering on the reference image and the third image, such that the remaining pixel pairs after filtering meet preset conditions, includes: Using the reference image as the first current image and the third image as the second current image, a fourth pixel is filtered out from the first current image and a fifth pixel is filtered out from the second current image; wherein, the fourth pixel and the fifth pixel are pixel pairs corresponding to the same pixel position, and the difference information of the pixel attribute values ​​of the fourth pixel and the fifth pixel does not satisfy a first preset proximity condition with respect to the current average difference; the current average difference is the average value of the difference information corresponding to the current remaining pixel pairs; the current remaining pixel pairs are the pixel pairs remaining in the first current image after filtering out the fourth pixel and the second current image after filtering out the fifth pixel; The first current image from which the fourth pixel is removed is taken as the new first current image, and the second current image from which the fifth pixel is removed is taken as the new second current image. The process of removing the fourth pixel from the first current image and removing the fifth pixel from the second current image is repeated until the remaining pixels after removal meet the preset condition and the iteration stops. The first preset proximity condition includes: the difference between the difference information and the current average difference is greater than λ*κ, where κ represents the standard deviation of the difference information corresponding to the current remaining pixel pairs, and λ is a preset coefficient.

8. The method according to any one of claims 1 to 4, characterized in that, The step of adjusting the brightness of the second image to obtain the third image for each second image according to the exposure parameters corresponding to the reference image and the second image respectively includes: For each of the second images, the theoretical exposure factor is calculated based on the exposure parameters corresponding to the reference image and the second image, respectively. The brightness of the second image is adjusted according to the theoretical exposure factor to obtain the third image; the brightness of the third image is closer to that of the reference image than that of the second image.

9. The method according to any one of claims 1 to 4, characterized in that, The acquisition of multiple first images continuously captured for the same scene in high dynamic range mode includes: From the plurality of first images, the first image with the most sixth pixels is determined as the reference image, wherein the sixth pixel is a pixel whose pixel attribute value satisfies a second preset proximity condition with the average value of the pixel attribute values ​​of the plurality of images; or, From the plurality of first images, the first image whose exposure is closest to the average exposure is determined as the reference image, wherein the average exposure is the average of the plurality of exposures corresponding to the plurality of first images; The second preset proximity condition includes: the sixth pixel falling into a third range interval, where the third range interval is the interval of the average value of the pixel attribute values ​​of the plurality of images; or, the difference between the sixth pixel and the average value of the pixel attribute values ​​of the plurality of images is less than a preset threshold.

10. An electronic device, characterized in that, The electronic device includes at least a memory and one or more processors; the memory stores computer instructions that, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 9.

12. A computer program product, characterized in that, When the computer program product is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1 to 9.

Images