Image processing method and related device

Abstract

Embodiments of the present application provide an image processing method and related equipment, the method comprises: acquiring a preview image, and detecting whether the preview image is overexposed; if the preview image is overexposed, determining the uneven degree of the overexposed area in the preview image; adjusting the exposure parameter of the preview image according to the uneven degree; generating an underexposed image according to the adjusted preview image; and fusing the adjusted preview image and the underexposed image to generate an HDR image corresponding to the preview image. When it is detected that the preview image is overexposed, the exposure parameter of the preview image is adjusted, and then the preview image and the corresponding underexposed image are fused to generate the corresponding HDR image, which avoids the defects of the generated HDR image caused by the uneven transition area in the preview image, and effectively improves the image quality of the HDR image.

Description

Technical Field

[0001] This application relates to the field of smart terminal technology, and in particular to an image processing method and related equipment. Background Technology

[0002] With the development of terminal technology, smart terminal devices such as smartphones and personal computers support High Dynamic Range (HDR) image display. Related technologies typically generate HDR images by fusing normally exposed and underexposed images, thereby expanding the dynamic range of the image. However, the brightness and color of the highlight areas in the preview image differ significantly from other areas, easily leading to imperfections such as color bars and blocks in the highlight areas of the fused HDR image. This results in poor image quality and negatively impacts the user's visual experience. Summary of the Invention

[0003] In view of the above, it is necessary to provide an image processing method and related equipment to solve the problem that the brightness and color of the highlight area of ​​the normally exposed image are significantly different from the brightness and color of other areas, which can easily lead to defects in the HDR image generated by fusion.

[0004] In a first aspect, this application provides an image processing method applied to an electronic device, the method comprising: acquiring a captured preview image and detecting whether the preview image is overexposed; if the preview image is overexposed, determining the degree of unevenness of the overexposed area in the preview image; adjusting the exposure parameters of the preview image according to the degree of unevenness; generating an underexposed image based on the adjusted preview image; and fusing the adjusted preview image and the underexposed image to generate an HDR image corresponding to the preview image.

[0005] The above technical solution detects whether the preview image is overexposed before fusing the preview image to generate an HDR image. If the preview image is overexposed, the exposure parameters of the preview image are adjusted, and then the preview image and the corresponding underexposed image are fused to generate the corresponding HDR image. This avoids the uneven transition areas (i.e., overexposed areas) in the preview image causing defects in the fused HDR image, effectively improving the image quality of the HDR image and enhancing the user's visual experience.

[0006] In one possible implementation, detecting whether the preview image is overexposed includes: converting the pixel value of each pixel in the preview image into a grayscale value; converting the grayscale value of each pixel in the preview image into a normalized grayscale value; determining that the pixel is an overexposed pixel if the normalized grayscale value of the pixel is greater than or equal to a preset overexposed threshold; counting the number of overexposed pixels in the preview image, and determining that the preview image is overexposed if the ratio between the number of overexposed pixels and the total number of pixels in the preview image is greater than or equal to a preset threshold ratio.

[0007] By using the above technical solution, overexposed pixels in the preview image are identified, and the presence of overexposure in the preview image is determined by judging whether the ratio between the number of overexposed pixels and the total number of pixels is greater than or equal to a preset threshold ratio. This allows for accurate overexposure judgment of the preview image.

[0008] In one possible implementation, converting the pixel value of each pixel in the preview image to a grayscale value includes: weighted summing of the R, G, and B components of the RGB pixel value of each pixel in the preview image to obtain the grayscale value of each pixel.

[0009] The above technical solution can accurately determine the grayscale value of each pixel in the preview image.

[0010] In one possible implementation, converting the grayscale value of each pixel in the preview image into a normalized grayscale value includes: calculating the normalized grayscale value of each pixel based on the grayscale value of each pixel in the preview image, the maximum grayscale value of all pixels, and the minimum grayscale value.

[0011] The above technical solution uses normalized grayscale values ​​as the brightness values ​​of pixels to compare with preset overexposure thresholds, thereby accurately determining whether a pixel is an overexposed pixel.

[0012] In one possible implementation, determining the non-uniformity of the overexposed area in the preview image includes: converting the color space of the preview image from RGB color space to YUV color space; calculating the gradients of the Y, U, and V components of the YUV pixel values ​​of each overexposed pixel in the overexposed area using a preset gradient algorithm; weighted summing of the gradients of the Y, U, and V components of the YUV pixel values ​​of each overexposed pixel to obtain the gradient of each overexposed pixel; calculating the mean of the gradients of all overexposed pixels to obtain the gradient of the overexposed area, and using the gradient of the overexposed area as the non-uniformity of the overexposed area.

[0013] By using the above technical solution, the gradient of the overexposed area can be calculated as the degree of unevenness in its brightness and color, and the degree of unevenness in the overexposed area can be accurately determined.

[0014] In one possible implementation, converting the color space of the preview image from RGB color space to YUV color space includes: converting the RGB pixel value of each pixel in the preview image to a YUV pixel value.

[0015] The above technical solution can accurately convert a preview image in the RGB color space into a preview image in the YUV color space.

[0016] In one possible implementation, adjusting the exposure parameters of the preview image according to the degree of non-uniformity includes: obtaining a target exposure value of the preview image using a preset inverse proportional function based on the degree of non-uniformity of the overexposed area and the current exposure value of the preview image; and adjusting the current exposure value of the preview image according to the target exposure value.

[0017] By using the above technical solution, the exposure parameters of the preview image after adjustment are determined based on the unevenness of the overexposed area and the preset inverse proportional function. This can effectively adjust the overexposed area in the preview image to a non-overexposed area and remove the uneven transition area in the preview image.

[0018] In one possible implementation, adjusting the exposure parameters of the preview image according to the degree of non-uniformity includes:

[0019] Based on the degree of unevenness and the correspondence between the current exposure value and the target exposure value of the preview image, the target exposure value of the preview image is determined according to the current exposure value of the preview image and the degree of unevenness of the overexposed area; the current exposure value of the preview image is adjusted according to the target exposure value.

[0020] By using the above technical solution, based on the degree of unevenness of the overexposed area, the correspondence between the degree of unevenness and the current exposure value of the preview image and the target exposure value, the exposure parameters of the preview image after adjustment can be determined. This can effectively adjust the overexposed area in the preview image to a non-overexposed area and remove the uneven transition area of ​​brightness in the preview image.

[0021] In one possible implementation, generating an underexposed image based on the adjusted preview image includes: obtaining the dynamic range value and exposure value of the preview image; determining the exposure value of the underexposed image based on the correspondence between the dynamic range and exposure value of the preview image and the exposure value of the underexposed image, and the dynamic range value and exposure value of the preview image; adjusting the exposure value of the adjusted preview image to the exposure value of the underexposed image to obtain the underexposed image.

[0022] By using the above technical solution, based on the dynamic range value and exposure value of the current preview image, as well as the correspondence between the dynamic range value and exposure value of the preview image and the exposure value of the underexposed image, the exposure value of the underexposed image to be generated can be accurately determined. Then, the exposure value of the current preview image can be adjusted to the exposure value of the underexposed image to be generated, and the corresponding underexposed image can be accurately obtained.

[0023] In one possible implementation, fusing the adjusted preview image and the underexposed image to generate an HDR image corresponding to the preview image includes: inputting the adjusted preview image and the underexposed image into an image fusion model, and outputting the HDR image corresponding to the preview image through the image fusion model.

[0024] By employing the above technical solutions and image fusion models, HDR images corresponding to preview images can be accurately generated, thereby improving the generation efficiency and image quality of HDR images.

[0025] In one possible implementation, the image fusion model is a Unet model, which includes an input layer, an encoder, a fusion unit, a decoder, and an output layer.

[0026] By using the above technical solution, the Unet model as an image fusion model can improve the efficiency of image feature extraction, reduce the loss of details in the image fusion process, and has strong scalability.

[0027] In one possible implementation, the step of inputting the adjusted preview image and the underexposed image into an image fusion model, and outputting an HDR image corresponding to the preview image through the image fusion model, includes: the input layer receiving the input preview image and the underexposed image, and passing the preview image and the underexposed image to the encoder; the encoder performing downsampling processing on the preview image and the underexposed image, extracting a first image feature of the preview image and a second image feature of the underexposed image respectively, and passing the first image feature and the second image feature to the fusion unit; the fusion unit fusing the first image feature and the second image feature to obtain a third image feature, and passing the third image feature to the decoder; the decoder performing upsampling processing on the third image feature to obtain a reconstructed image, and passing the reconstructed image to the output layer; the output layer performing tone mapping processing on the reconstructed image using a tone mapping function to obtain an HDR image corresponding to the preview image, and outputting the HDR image.

[0028] By employing the above technical solution and using the Unet model to perform feature extraction, feature fusion, image reconstruction, and tone mapping on the preview image and underexposed image, HDR images corresponding to the preview image can be generated accurately and efficiently.

[0029] In one possible implementation, the method further includes: creating training samples; and training a preset deep learning model using the training samples to obtain the image fusion model.

[0030] By using the above technical solutions, a precise image fusion model can be obtained by training a preset deep learning model using training samples, thereby improving the generation accuracy and efficiency of HDR.

[0031] In one possible implementation, creating training samples includes: controlling the camera of the electronic device to capture multiple preview images, using one preview image, the corresponding underexposed image, and the corresponding theoretical HDR image as a set of training data, creating multiple sets of training data, and obtaining the training samples.

[0032] By using the above technical solution, actual captured preview images are used as training data to avoid overfitting of the image fusion model and improve the generalization ability of the image fusion model.

[0033] In one possible implementation, the preview images of the training samples include a first preset percentage of preview images including overexposed areas and a second preset percentage of preview images excluding overexposed areas, wherein the first preset percentage is less than the second preset percentage.

[0034] By using the above technical solution, some preview images including overexposed areas can be used as training data, which can effectively avoid overfitting of the image fusion model and improve the generalization ability of the image fusion model.

[0035] In one possible implementation, the preset deep learning model is a Unet model. The step of training the preset deep learning model using the training samples to obtain the image fusion model includes: importing the Unet model into a deep learning framework and initializing the Unet model by setting its initial parameters; inputting a set of training data from the training samples into the Unet model, and outputting a predicted HDR image corresponding to the preview image in the training data; calculating the output value of the loss function of the Unet model based on the predicted HDR image and the theoretical HDR image corresponding to the preview image in the training data; if the output value of the loss function is greater than a preset value, adjusting the parameters of the Unet model, inputting another set of training data into the Unet model, and continuing to train the Unet model; or if the output value of the loss function is less than or equal to the preset value, determining that the Unet model training is complete, and obtaining the image fusion model.

[0036] The above technical solutions can improve the accuracy of the trained image fusion model.

[0037] In one possible implementation, the method further includes: if the preview image is not overexposed, generating an underexposed image based on the preview image.

[0038] By using the above technical solution, when the preview image is not overexposed, the corresponding underexposed image can be directly generated, thereby improving the generation efficiency of underexposed images.

[0039] Secondly, this application provides an electronic device, the electronic device including a memory and a processor: wherein the memory is used to store program instructions; the processor is used to read and execute the program instructions stored in the memory, and when the program instructions are executed by the processor, the electronic device performs the above-described image processing method.

[0040] Thirdly, this application provides a chip coupled to a memory in an electronic device, the chip being used to control the processor of the electronic device to execute the above-described image processing method.

[0041] Fourthly, this application provides a computer storage medium storing program instructions that, when executed on an electronic device, cause the processor of the electronic device to perform the above-described image processing method.

[0042] Furthermore, the technical effects brought about by the second to fourth aspects can be found in the descriptions of the methods in the above-mentioned method section, and will not be repeated here. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of a normally exposed image provided in an embodiment of this application.

[0044] Figure 2 This is a schematic diagram of an underexposed image provided in an embodiment of this application.

[0045] Figure 3 This is a schematic diagram of an HDR image provided in an embodiment of this application.

[0046] Figure 4 This is a software architecture diagram of an electronic device provided in an embodiment of this application.

[0047] Figure 5 This is a flowchart of an image processing method provided in an embodiment of this application.

[0048] Figure 6 This is a schematic diagram of a preview image provided in one embodiment of this application.

[0049] Figure 7 This is a schematic diagram of an adjusted preview image provided in one embodiment of this application.

[0050] Figure 8 This is a schematic diagram of the structure of an image fusion model provided in an embodiment of this application.

[0051] Figure 9 This is a schematic diagram of an HDR image corresponding to a preview image provided in an embodiment of this application.

[0052] Figure 10 This is a flowchart for determining whether a preview image is overexposed, provided in one embodiment of this application.

[0053] Figure 11 This is a flowchart for determining the degree of unevenness of overexposed areas in a preview image, provided in one embodiment of this application.

[0054] Figure 12 This is a flowchart of an image processing method provided in another embodiment of this application.

[0055] Figure 13 This is a hardware architecture diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0056] The terms "first" and "second" used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this application's specification is for the purpose of describing particular embodiments only and is not intended to limit the application. It should be understood that, unless otherwise stated, " / " in this application means "or". For example, A / B can mean A or B. "And / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. "At least one" refers to one or more. "More than one" refers to two or more. For example, at least one of a, b, or c can represent: a, b, c, a and b, a and c, b and c, and a, b, and c. Where there is no conflict, the following embodiments and features described herein can be combined with each other.

[0058] With the development of terminal technology, smart terminal devices such as smartphones and personal computers support high dynamic range (HDR) image display. Related technologies typically use a method of fusing a preview image with an underexposed image to generate an HDR image, thereby expanding the image's dynamic range. However, the brightness and color of the highlight areas in the preview image differ significantly from the brightness and color of other areas. This can easily lead to imperfections such as color bars and blocks in the highlight areas of the fused HDR image, resulting in poor image quality and negatively impacting the user's visual experience.

[0059] See Figure 1The image shown is a schematic diagram of a normally exposed image provided in an embodiment of this application. In scenes with strong contrast (such as backlit scenes), a high contrast ratio occurs (strong contrast between light and dark areas in the image). In this case, the brightness of the highlight areas in a normally exposed image may be significantly brighter than the brightness of other areas. Because the highlight areas of the sky are brighter, their color tends to be white, while other areas of the sky are blue. Therefore, the color of the highlight areas is also significantly different from the color of other areas.

[0060] See Figure 2 The image shown is a schematic diagram of an underexposed image provided in an embodiment of this application. An underexposed image has a shorter exposure time and is darker than the preview image. An image fusion model is used to fuse the preview image and the underexposed image to generate an image as shown. Figure 3 The HDR image shown is an example. Because the brightness and color of the highlight areas in the preview image differ significantly from those of other areas, the fused HDR image exhibits color bar defects in the highlight areas. Consequently, the resulting HDR image has poor image quality, negatively impacting the user's visual experience.

[0061] To avoid imperfections in the HDR image generated by fusion, this application provides an image processing method that can detect overexposed areas in a normally exposed image, adjust the brightness of the normally exposed image, and then fuse the normally exposed image with the underexposed image to generate an HDR image. This avoids fusion imperfections in the HDR image, effectively improves the image quality of the HDR image, and enhances the user's visual experience. This image processing method is applied to electronic devices, and the following describes its application in conjunction with... Figure 4 Illustrate the software architecture diagram of the electronic device.

[0062] See Figure 4 The diagram shown is a software architecture diagram of an electronic device provided in an embodiment of this application. The layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. For example, the Android system is divided into four layers, from top to bottom: application layer 101, framework layer 102, Android runtime and system library 103, hardware abstraction layer 104, kernel layer 105, and hardware layer 106.

[0063] Application layer 101 may include a series of application packages. For example, application packages may include applications such as camera, gallery, calendar, calling, map, navigation, WLAN, Bluetooth, music, video, SMS, device control services, etc.

[0064] The framework layer 102 provides an Application Programming Interface (API) and programming framework for applications in the application layer. The application framework layer includes predefined functions. For example, it may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0065] The window manager manages window programs. It can obtain screen size, determine the presence of a status bar, lock the screen, and capture screenshots. The content provider stores and retrieves data, making it accessible to applications. This data can include videos, images, audio, made and received calls, browsing history and bookmarks, phone books, etc. The view system includes visual controls, such as controls for displaying text and controls for displaying images. The view system can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon can include views for displaying text and views for displaying images. The phone manager provides communication functionality for electronic devices, such as managing call status (including connection and disconnection). The resource manager provides applications with various resources, such as localized strings, icons, images, layout files, and video files. The notification manager allows applications to display notifications in the status bar, conveying informational messages that disappear automatically after a short pause without user interaction. For example, the notification manager is used to notify of download completion or message alerts. The notification manager can also display notifications as icons or scrolling text in the system's top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting alert sounds, causing electronic devices to vibrate, and flashing indicator lights.

[0066] The Android Runtime consists of the core libraries and the virtual machine. The Android runtime is responsible for the scheduling and management of the Android system. The core libraries consist of two parts: one part contains the functionalities that the Java language needs to call, and the other part contains the core Android libraries.

[0067] Application layer 101 and framework layer 102 run in a virtual machine. The virtual machine executes the Java files of the application layer and framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0068] System library 103 may include multiple functional modules. For example, a surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0069] The Surface Manager manages the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The Media Library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG. The 3D Graphics Processing Library implements 3D graphics drawing, image rendering, compositing, and layer processing. The 2D Graphics Engine is the drawing engine for 2D graphics.

[0070] Hardware Abstraction Layer 104 runs in user space, encapsulates kernel-level drivers, and provides calling interfaces to the upper layers.

[0071] Kernel layer 105 is the layer between hardware and software. Kernel layer 105 contains at least the display driver, camera driver, audio driver, and sensor driver.

[0072] Kernel layer 105 is the core of the operating system for electronic devices. It is the first layer of software extension based on the hardware, providing the most basic functions of the operating system. It is the foundation for the operation of the operating system, responsible for managing system processes, memory, device drivers, files, and network systems, and determining the system's performance and stability. For example, the kernel can determine the timing of an application's operation on a certain part of the hardware.

[0073] Kernel layer 105 includes hardware-dependent programs such as interrupt handlers and device drivers, as well as basic, common, and frequently running modules such as clock management and process scheduling modules, and critical data structures. The kernel layer can be located within the processor or embedded in internal memory.

[0074] Hardware layer 106 includes the hardware of electronic devices, such as displays, buttons, cameras, etc.

[0075] See Figure 5 The diagram shown is a flowchart of an image processing method provided in an embodiment of this application. The method is applied in an electronic device, and the image processing method includes:

[0076] S101: Acquire the captured preview image and check if the preview image is overexposed. If the preview image is overexposed, proceed to S102; if the preview image is not overexposed, proceed to S106.

[0077] In one embodiment of this application, a user can click the camera application icon on the main interface of the electronic device, or click the camera activation control on the lock screen of the electronic device, to activate the camera application. The electronic device responds to the user's activation of the camera application by acquiring a preview image using its shooting device. In one embodiment, the electronic device can control the camera to use a fixed exposure mode (Fixed Auto Exposure, Fixed AE) to capture images of any scene and obtain a preview image. The preview image can also be called a normal exposure image, for example, an image captured by the camera with an exposure value of 0EV. The exposure value of the preview image is 0EV; 0EV is a relative value and does not mean the exposure value is zero. For example, the exposure value is calculated based on the combination of the camera's aperture and exposure time (i.e., shutter speed). A larger exposure value indicates a brighter image, and a smaller exposure value indicates a darker image.

[0078] In one embodiment of this application, after the camera application is started, a camera pipeline is established according to the shooting mode. The camera pipeline includes multiple image processing nodes. The camera service (CameraService) at the framework layer and the camera provider (CameraProvider) at the hardware abstraction layer are initialized. After initialization, the camera application sends a preview shooting request to the camera service, which in turn sends the preview shooting request to the camera provider. The camera provider sends the preview shooting request or a shooting request generated in response to user-triggered shooting controls to the camera driver at the kernel layer. The camera driver can then drive the camera to capture preview images and return the captured preview images to the camera provider. The camera provider then sends the preview images to the image processing nodes for processing.

[0079] In one embodiment of this application, it is determined whether the ratio between the number of overexposed pixels and the total number of pixels in the preview image is greater than or equal to a preset threshold ratio. If the ratio is greater than or equal to the preset threshold ratio, the preview image is determined to be overexposed; if the ratio is less than the preset threshold ratio, the preview image is determined not to be overexposed. For example, the preset threshold ratio can be set to 20%, 25%, 30%, or other percentages, which are merely illustrative examples and are not limited to these in practical applications.

[0080] In one embodiment of this application, the pixel value of each pixel in the preview image is converted into a grayscale value, and the grayscale value of each pixel is converted into a normalized grayscale value. It is then determined whether the normalized grayscale value of the pixel is greater than or equal to a preset overexposure threshold. If the normalized grayscale value of the pixel is greater than or equal to the preset overexposure threshold, the pixel is determined to be an overexposed pixel; if the normalized grayscale value of the pixel is less than the preset overexposure threshold, the pixel is determined not to be an overexposed pixel. After all pixels in the preview image have been determined to be overexposed, the number of overexposed pixels is counted. In one embodiment of this application, the preset overexposure threshold can be set between 0.8 and 0.95, for example, the preset overexposure threshold can be 0.8, 0.9, 0.95, or other values.

[0081] S102, determine the degree of unevenness in the overexposed areas of the preview image.

[0082] See Figure 6 The image shown is a schematic diagram of a preview image provided in an embodiment of this application. When the preview image is overexposed, the preview image contains elements such as... Figure 6 The overexposed areas are marked by the square box; these are areas with uneven brightness transitions.

[0083] In one embodiment of this application, the color space of the preview image is converted from the Red Green Blue (RGB) color space to the Luminance Color (YUV) color space. A preset gradient algorithm is used to calculate the gradients of the Y, U, and V components in the YUV pixel values ​​of each overexposed pixel. The gradients of the Y, U, and V components in the YUV pixel values ​​of each overexposed pixel are then weighted and summed to obtain the gradient of each overexposed pixel. The average gradient of all overexposed pixels is calculated to obtain the gradient of the overexposed region, and this gradient is determined as the degree of non-uniformity of the overexposed region. Non-uniformity refers to the degree of non-uniformity of brightness and color in the overexposed region. For example, the preset gradient algorithm can be the Sobel gradient algorithm or the Laplacian gradient algorithm.

[0084] S103, adjust the exposure parameters of the preview image according to the degree of unevenness.

[0085] In one embodiment of this application, the exposure parameters include exposure values. The unevenness G in the overexposed area and the current exposure value B0 of the preview image are input into a preset inverse proportional function. The preset inverse proportional function outputs a target exposure value B for the preview image, and the current exposure value of the preview image is adjusted to the target exposure value B. Based on the preset inverse proportional function, the greater the unevenness of the overexposed area, the smaller the exposure value of the preview image. The formula for calculating the exposure value B of the preview image based on the unevenness G of the overexposed area and the current exposure value B0 of the preview image is as follows:

[0086] B = f(G, B0) (1).

[0087] In formula (1), B is the target exposure value of the preview image, G is the unevenness of the overexposed area in the preview image, B0 is the current exposure value of the preview image, and f is a preset inverse proportional function. For example:

[0088]

[0089] In other embodiments of this application, f may also be other inverse proportional functions obtained by fitting empirical values ​​of multiple non-uniformities and the target exposure value and the current exposure value corresponding to each non-uniformity.

[0090] In another embodiment of this application, the electronic device pre-sets and stores a correspondence between unevenness, the current exposure value of the preview image, and the target exposure value. For example, a pre-set correspondence table records the correspondence between unevenness, the current exposure value of the preview image, and the target exposure value. Based on the correspondence table, the target exposure value B corresponding to the unevenness G and the current exposure value B0 in the overexposed area can be determined, and the current exposure value B0 of the preview image can be adjusted according to the target exposure value B. For example, the current exposure value B0 of the preview image can be adjusted to the target exposure value B.

[0091] See Figure 7 The image shown is a schematic diagram of the adjusted preview image provided in an embodiment of this application. After adjusting the exposure parameters of the overexposed areas in the preview image, that is, after reducing the exposure, as shown... Figure 7 The previously overexposed areas marked by the Chinese box will no longer be overexposed.

[0092] S104: Generate an underexposed image based on the adjusted preview image.

[0093] In one embodiment of this application, an underexposed image is generated according to the Smart Auto Exposure (Smart AE) mode. That is, an underexposed image is generated by reducing the exposure of a preview image after adjusting its exposure parameters. The electronic device also pre-stores the dynamic range (DR) value of the preview image, the correspondence between the preview image's exposure value and the underexposed image's exposure value, and the corresponding relationship between multiple factors, which can be recorded in the form of a table or list. Taking a correspondence table as an example, the dynamic range value and exposure value of the preview image are obtained. Based on the dynamic range value and exposure value of the preview image, the corresponding exposure value of the underexposed image is determined by looking up the corresponding relationship table between the dynamic range value of the preview image, the exposure value of the preview image, and the exposure value of the underexposed image. The exposure value of the adjusted preview image is then adjusted to the exposure value of the underexposed image to obtain the underexposed image. An underexposed image, also known as an S-frame image, is an image with an exposure value less than 0 EV. That is, the exposure value of the underexposed image is less than 0 EV, such as -2 EV, -4 EV, etc. The exposure value of the preview image is the target exposure value of the adjusted preview image.

[0094] In one embodiment of this application, the dynamic range value of the preview image is a value automatically returned by the camera driver after the preview image is captured by the camera. The dynamic range value is between 0 and 1, and can be, for example, 0.2, 0.3, 0.9, or other values. For example, in the correspondence table between the dynamic range value of the preview image, the exposure value of the preview image, and the exposure value of the underexposed image, the underexposed image with a dynamic range value of 0.2 and an exposure value of 0EV corresponds to an exposure value of -2EV; the underexposed image with a dynamic range value of 0.3 and an exposure value of 0EV corresponds to an exposure value of -3EV; and the underexposed image with a dynamic range value of 0.9 and an exposure value of 0EV corresponds to an exposure value of -6EV.

[0095] S105 merges the adjusted preview image and the underexposed image to generate an HDR image corresponding to the preview image.

[0096] In one embodiment of this application, the adjusted preview image and the underexposed image are input into an image fusion model, and the image fusion model outputs an HDR image corresponding to the preview image.

[0097] See Figure 8The diagram shown is a structural schematic of the image fusion model provided in an embodiment of this application. In one embodiment of this application, the image fusion model is the Unet model. The Unet model includes an input layer, an encoder, a fusion unit, a decoder, and an output layer. The input layer receives the input preview image and the underexposed image, and passes them to the encoder. The encoder performs downsampling processing on the preview image and the underexposed image. The downsampling processing includes convolution and pooling operations to extract the first image features of the preview image and the second image features of the underexposed image, respectively. The first image features and the second image features are then passed to the fusion unit. The fusion unit fuses the first image features and the second image features to obtain a third image feature, which is then passed to the decoder. The decoder performs upsampling processing on the third image features. The upsampling processing includes deconvolution to obtain a reconstructed image, which is then passed to the output layer. The output layer uses a tone mapping function to map the reconstructed image to obtain an HDR image corresponding to the preview image, and outputs the HDR image.

[0098] See Figure 9 As shown, after adjusting the exposure parameters of the preview image, the HDR image obtained by fusing the preview image and the underexposed image does not have defects such as color blocks or color bars.

[0099] S106, Generate an underexposed image based on the preview image.

[0100] S107, merges the preview image and the underexposed image to generate an HDR image corresponding to the preview image.

[0101] The specific implementation methods of S106 to S107 are the same as those of S104 to S105, and will not be described in detail here.

[0102] Through the above embodiments of this application, before fusing the preview image to generate an HDR image, it is detected whether the preview image is overexposed. If the preview image is overexposed, the exposure parameters of the preview image are adjusted, and then the preview image and the corresponding underexposed image are fused to generate the corresponding HDR image. This avoids the uneven transition areas (i.e., overexposed areas) in the preview image causing defects in the fused HDR image, effectively improving the image quality of the HDR image and enhancing the user's visual experience.

[0103] See Figure 10 The diagram shown is a flowchart for determining whether a preview image is overexposed, provided in an embodiment of this application.

[0104] S1011, convert the pixel value of each pixel in the preview image to a grayscale value.

[0105] In one embodiment of this application, the preview image captured by the camera is an RGB image. The R, G, and B components of the RGB pixel values ​​of each pixel in the preview image are weighted and summed to obtain the grayscale value of each pixel. The formula for calculating the grayscale value of a pixel is as follows:

[0106] Gray = x*R + y*G + z*B (3).

[0107] In the calculation formula (3), R is the R component of the pixel value, x is the weight of the R component, for example, 0.3, G is the G component of the pixel value, y is the weight of the G component, for example, 0.59, B is the B component of the pixel value, and z is the weight of the B component, for example, 0.11.

[0108] In another embodiment of this application, the grayscale value of each pixel can also be obtained by calculating the average value of the R component, G component, and B component of the RGB pixel value of each pixel in the preview image.

[0109] S1012, convert the grayscale value of each pixel in the preview image into a normalized grayscale value.

[0110] In one embodiment of this application, a normalized grayscale value for each pixel is calculated based on the grayscale value of each pixel in the preview image, the maximum grayscale value of all pixels, and the minimum grayscale value. The formula for calculating the normalized grayscale value Gray0 of a pixel is as follows:

[0111]

[0112] In the calculation formula (4), Gray is the gray value of each pixel. max The maximum grayscale value of all pixels, Gray min This is the minimum grayscale value of all pixels.

[0113] S1013, determine whether the normalized grayscale value of the pixel in the preview image is greater than or equal to the preset overexposure threshold. If the normalized grayscale value of the pixel in the preview image is greater than or equal to the preset overexposure threshold, proceed to S1014; if the normalized grayscale value of the pixel is less than the preset overexposure threshold, proceed to S1015.

[0114] S1014, the pixel is determined to be an overexposed pixel.

[0115] S1015, confirm that the pixel is not an overexposed pixel.

[0116] S1016: Count the number of overexposed pixels in the preview image and determine whether the ratio between the number of overexposed pixels and the total number of pixels in the preview image is greater than or equal to a preset threshold ratio. If the ratio between the number of overexposed pixels and the total number of pixels in the preview image is greater than or equal to the preset threshold ratio, proceed to S1017; if the ratio between the number of overexposed pixels and the total number of pixels in the preview image is less than the preset threshold ratio, proceed to S1018.

[0117] S1017, It has been determined that the preview image is overexposed.

[0118] S1018, confirm that the preview image is not overexposed.

[0119] Through the above embodiments of this application, overexposed pixels in the preview image can be identified, and whether the preview image is overexposed can be determined by judging whether the ratio between the number of overexposed pixels and the total number of pixels is greater than or equal to a preset threshold ratio. This allows for accurate overexposure judgment of the preview image.

[0120] See Figure 11 The diagram shown is a flowchart for determining the non-uniformity of overexposed areas in a preview image, provided in an embodiment of this application.

[0121] S1021 converts the color space of the preview image from RGB color space to YUV color space.

[0122] In one embodiment of this application, the RGB pixel value of each pixel in the preview image is converted to a YUV pixel value, thereby converting the color space of the preview image from the RGB color space to the YUV color space. The formula for calculating the conversion of the RGB pixel value to the YUV pixel value is as follows:

[0123]

[0124] In the calculation formula (5), R, G, and B are the R, G, and B components of the RGB pixel values ​​of the pixel points in the preview image, respectively, and Y, U, and V are the Y, U, and V components of the YUV pixel values ​​of the pixel points in the preview image, respectively.

[0125] S1022 uses a preset gradient algorithm to calculate the gradients of the Y, U, and V components in the YUV pixel values ​​of each overexposed pixel.

[0126] In one embodiment of this application, the preset gradient algorithm is the Sobel gradient algorithm. The formula for calculating the gradients of the Y, U, and V components in the YUV pixel values ​​of each overexposed pixel using the Sobel gradient algorithm is as follows:

[0127]

[0128] In one embodiment of this application, the calculation formula for the Sobel gradient algorithm is as follows:

[0129]

[0130]

[0131]

[0132] In the calculation formulas (7) and (8), I represents the Y component, U component, or V component in the YUV pixel value.

[0133] S1023, the gradients of the Y, U, and V components in the YUV pixel values ​​of each overexposed pixel are weighted and summed to obtain the gradient of each overexposed pixel.

[0134] In one embodiment of this application, the formula for calculating the gradient of a pixel is:

[0135] G = W_Y * G Y + W_U*G u + W_V*G v (10).

[0136] In the calculation formula (10), W_Y, W_U, and W_V are set between 0 and 1. Since the gradient of the overexposed area is mainly reflected in the brightness channel Y, W_Y is greater than W_U and W_V. For example, W_Y = 0.6, W_U = 0.2, and W_V = 0.2.

[0137] S1024, calculate the mean gradient of all overexposed pixels to obtain the gradient of the overexposed region, and determine the gradient of the overexposed region as the degree of non-uniformity of the overexposed region.

[0138] By using the above embodiments of this application, the gradient of overexposed pixels is calculated as the degree of unevenness in their brightness and color, which can accurately determine the degree of unevenness of overexposed pixels, and thus accurately determine the degree of unevenness in overexposed areas.

[0139] See Figure 12 The diagram shown is a flowchart of an image processing method provided in another embodiment of this application. The method is applied in an electronic device, and the image processing method includes:

[0140] S201, Create training samples.

[0141] In one embodiment of this application, a camera is controlled to capture multiple preview images. A preview image, its corresponding underexposed image, and its corresponding theoretical HDR image are used as a set of training data to create multiple sets of training data, thus obtaining training samples. In the preview images of the training samples, a first preset percentage of the preview image includes overexposed areas, while a second preset percentage of the preview image does not include overexposed areas. The first preset percentage is less than the second preset percentage; for example, the first preset percentage is 20%, and the second preset percentage is 80%.

[0142] In one embodiment of this application, the preview image including the overexposed area can be obtained by capturing it with a camera. In another embodiment of this application, the preview image including the overexposed area can also be obtained by increasing the brightness of a preset area in the preview image, making the preset area an overexposed area, thereby simulating a preview image with uneven brightness transition. The underexposed image corresponding to the preview image is simulated and generated using the method in step S104 of the above embodiment. That is, the underexposed image corresponding to the preview image is obtained by reducing the exposure value of the preview image.

[0143] S202, train the preset deep learning model using training samples to obtain the image fusion model.

[0144] In one embodiment of this application, the preset deep learning model is the Unet model. The Unet model is imported into the deep learning framework and initialized. Initial parameters of the Unet model are set. A set of training data from the training samples is input into the Unet model. The Unet model outputs a predicted HDR image. The output value of the loss function of the Unet model is calculated based on the predicted HDR image and the theoretical HDR image. If the output value of the loss function is greater than a preset value, another set of training data is input into the Unet model to continue training. If the output value of the loss function is less than or equal to the preset value, the Unet model training is considered complete, and an image fusion model is obtained.

[0145] In one embodiment of this application, the loss function is the mean squared error (MSE) between the theoretical HDR image and the predicted HDR image. That is, the mean squared error between the theoretical HDR image and the predicted HDR image is calculated as the difference between them. The formula for calculating the mean squared error is:

[0146]

[0147] In the calculation formula (11), y i y can be the grayscale value of the i-th pixel in a theoretical HDR image. i PTo predict the grayscale value of the i-th pixel in the HDR image, n is the number of pixels in the theoretical HDR image and the predicted HDR image.

[0148] In another embodiment of this application, the preset deep learning model may also be a ResNet model.

[0149] S203: Acquire a preview image and determine if the preview image is overexposed. If the preview image is overexposed, proceed to S204; if the preview image is not overexposed, proceed to S208.

[0150] S204, determine the degree of unevenness in overexposed areas in the preview image.

[0151] S205, adjusts the exposure parameters of the preview image according to the degree of unevenness.

[0152] S206, Generate an underexposed image based on the adjusted preview image.

[0153] S207 uses an image fusion model to fuse the adjusted preview image and the underexposed image to generate an HDR image corresponding to the preview image.

[0154] S208: Generate an underexposed image based on the preview image.

[0155] S209 uses an image fusion model to fuse the preview image and the underexposed image to generate an HDR image corresponding to the preview image.

[0156] The specific implementation methods of S203 to S209 are the same as those of S101 to S107, and will not be described in detail here.

[0157] Through the above embodiments of this application, by using preview images that are actually captured without overexposed areas and preview images that are actually captured or simulated with overexposed areas as training data, a preset deep learning model is trained to obtain an image fusion model. This can avoid overfitting of the image fusion model, improve the generalization ability of the image fusion model, and thus improve the efficiency of the image fusion model in generating HDR images, as well as improve the accuracy and image quality of the fused HDR images.

[0158] This application also provides an electronic device 100, see reference. Figure 13As shown, the electronic device 100 may be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, as well as cellular phone, personal digital assistant (PDA), augmented reality (AR) device, virtual reality (VR) device, artificial intelligence (AI) device, wearable device, in-vehicle device, smart home device and / or smart city device. The specific type of electronic device 100 is not specifically limited in the embodiments of this application.

[0159] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, Universal Serial Bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and Subscriber Identification Module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0160] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0161] Processor 110 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0162] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0163] The processor 110 may also include a memory for storing instructions and data. In one embodiment of this application, the memory in the processor 110 is a cache memory. The memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instructions or data again, it can directly retrieve them from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0164] In one embodiment of this application, the processor 110 may include one or more interfaces. These interfaces may include an Inter-integrated Circuit (I2C) interface, an Inter-integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver / Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI) interface, a General-Purpose Input / Output (GPIO) interface, a Subscriber Identity Module (SIM) interface, and / or a Universal Serial Bus (USB) interface, etc.

[0165] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In one embodiment of this application, the processor 110 may include multiple I2C buses. The processor 110 can couple to the touch sensor 180K, charger, flash, camera 193, etc., through different I2C bus interfaces. For example, the processor 110 can couple to the touch sensor 180K through the I2C interface, enabling the processor 110 and the touch sensor 180K to communicate through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0166] The I2S interface can be used for audio communication. In one embodiment of this application, the processor 110 may include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to realize communication between the processor 110 and the audio module 170. In one embodiment of this application, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface to realize the function of answering phone calls through a Bluetooth headset.

[0167] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In one embodiment of this application, the audio module 170 and the wireless communication module 160 can be coupled via the PCM bus interface. In another embodiment of this application, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering phone calls through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0168] The UART interface is a universal serial data bus used for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In one embodiment of this application, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In one embodiment of this application, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface to enable music playback via Bluetooth headphones.

[0169] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a Camera Serial Interface (CSI) and a Display Serial Interface (DSI). In one embodiment of this application, the processor 110 and the camera 193 communicate via the CSI interface to realize the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate via the DSI interface to realize the display function of the electronic device 100.

[0170] The GPIO interface can be configured via software. It can be configured as a control signal or a data signal. In one embodiment of this application, the GPIO interface can be used to connect the processor 110 to a camera 193, a display screen 194, a wireless communication module 160, an audio module 170, a sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0171] USB port 130 is a USB standard compliant interface, specifically a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge electronic device 100, and can also be used for data transfer between electronic device 100 and peripheral devices. It can also be used to connect headphones for audio playback. Furthermore, the interface can be used to connect other electronic devices 100, such as AR devices.

[0172] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0173] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via a USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device 100 via the power management module 141.

[0174] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0175] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0176] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0177] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low-noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In one embodiment of this application, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In another embodiment of this application, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0178] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through audio devices (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In one embodiment of this application, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and housed within the same device as the mobile communication module 150 or other functional modules.

[0179] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including Wireless Local Area Networks (WLANs) (such as Wireless Fidelity (Wi-Fi) networks), Bluetooth (BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), and Infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0180] In one embodiment of this application, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the Beidou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0181] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0182] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), an Active-Matrix Organic Light-Emitting Diode (AMOLED), a Flexible Light-Emitting Diode (FLED), a Minied, Microled, Micro-OLED, or a Quantum Dot Light-Emitting Diode (QLED), etc. In one embodiment of this application, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0183] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0184] The ISP is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, converting it into an image visible to the naked eye. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In one embodiment of this application, the ISP can be set in the camera 193.

[0185] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In one embodiment of this application, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0186] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP performs Fourier transforms on the frequency energy.

[0187] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0188] NPU stands for Neural Network (NN) computing processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0189] Internal memory 121 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM).

[0190] Random access memory can include static random-access memory (SRAM), dynamic random-access memory (DRAM), synchronous dynamic random-access memory (SDRAM), and double data rate synchronous dynamic random-access memory (DDR SDRAM, such as fifth-generation DDR SDRAM, which is generally called DDR5 SDRAM).

[0191] Non-volatile memory can include disk storage devices and flash memory.

[0192] Flash memory can be classified according to its operating principle, including NOR FLASH, NAND FLASH, 3D NAND FLASH, etc.; according to the level of the storage cell, including single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc.; and according to the storage specification, including universal flash storage (UFS) and embedded multi-media card (eMMC), etc.

[0193] The random access memory can be directly read and written by the processor 110. It can be used to store executable programs (such as machine instructions) of the operating system or other running programs, as well as user and application data.

[0194] Non-volatile memory can also store executable programs and user and application data, and can be pre-loaded into random access memory for direct reading and writing by the processor 110.

[0195] The external memory interface 120 can be used to connect to external non-volatile memory, thereby expanding the storage capacity of the electronic device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to perform data storage functions. For example, music, video, and other files can be stored in the external non-volatile memory.

[0196] Internal memory 121 or external memory interface 120 is used to store one or more computer programs. The one or more computer programs are configured to be executed by processor 110. The one or more computer programs include multiple instructions, which, when executed by processor 110, can implement the screen display detection method executed on electronic device 100 in the above embodiments, so as to realize the screen display detection function of electronic device 100.

[0197] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0198] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In one embodiment of this application, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0199] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0200] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0201] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Electronic device 100 may have at least one microphone 170C. In some embodiments, electronic device 100 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 100 may also have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0202] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0203] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0204] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to touch operations performed on different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations performed on different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0205] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0206] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and separate from the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to realize functions such as calls and data communication. In one embodiment of this application, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100. This application also provides a computer storage medium storing computer instructions. When the computer instructions are executed on the electronic device 100, the electronic device 100 performs the above-mentioned related method steps to implement the image processing method in the above embodiments.

[0207] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the image processing method described above.

[0208] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component or module. The apparatus may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and when the apparatus is running, the processor may execute the computer execution instructions stored in the memory to cause the chip to execute the image processing methods in the above-described method embodiments.

[0209] In this embodiment, the electronic device, computer storage medium, computer program product or chip are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding method provided above, and will not be repeated here.

[0210] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0211] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0212] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0213] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0214] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. The software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0215] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. An image processing method applied to electronic devices, characterized in that, The method includes: Acquire a captured preview image and detect whether the preview image is overexposed; If the preview image is overexposed, determine the degree of unevenness of the overexposed area in the preview image, including: converting the color space of the preview image from RGB color space to YUV color space; calculating the gradients of the Y, U, and V components of the YUV pixel values ​​of each overexposed pixel in the overexposed area using a preset gradient algorithm; determining the gradient of each overexposed pixel based on the gradients of the Y, U, and V components of the YUV pixel values ​​of each overexposed pixel; calculating the average of the gradients of all overexposed pixels to obtain the gradient of the overexposed area, and determining the degree of unevenness of the overexposed area based on the gradient of the overexposed area. Adjusting the exposure parameters of the preview image according to the degree of unevenness includes: obtaining a target exposure value of the preview image using a preset inverse proportional function based on the degree of unevenness and the current exposure value of the preview image; and adjusting the current exposure value of the preview image according to the target exposure value. The adjusted preview image was obtained by shooting based on the adjusted exposure parameters; Generating an underexposed image based on the adjusted preview image includes: determining the exposure value of the underexposed image based on the correspondence between the dynamic range and exposure value of the preview image and the exposure value of the underexposed image, and the dynamic range and exposure value of the adjusted preview image; and capturing the underexposed image based on the exposure value of the underexposed image. The adjusted preview image and the underexposed image are fused together to generate an HDR image corresponding to the preview image.

2. The image processing method as described in claim 1, characterized in that, The detection of whether the preview image is overexposed includes: Convert the pixel value of each pixel in the preview image to a grayscale value; Convert the grayscale value of each pixel in the preview image to a normalized grayscale value; If the normalized gray value of the pixel is greater than or equal to the preset overexposure threshold, the pixel is determined to be an overexposure pixel. The number of overexposed pixels in the preview image is counted. If the ratio between the number of overexposed pixels and the total number of pixels in the preview image is greater than or equal to a preset threshold ratio, the preview image is determined to be overexposed.

3. The image processing method as described in claim 2, characterized in that, The step of converting the pixel value of each pixel in the preview image to a grayscale value includes: The R, G, and B components of the RGB pixel values ​​of each pixel in the preview image are weighted and summed to obtain the grayscale value of each pixel.

4. The image processing method as described in claim 2, characterized in that, The step of converting the grayscale value of each pixel in the preview image into a normalized grayscale value includes: The normalized gray value of each pixel is calculated based on the gray value of each pixel in the preview image, the maximum gray value of all pixels, and the minimum gray value.

5. The image processing method as described in claim 1, characterized in that, The step of converting the color space of the preview image from RGB color space to YUV color space includes: Convert the RGB pixel value of each pixel in the preview image to a YUV pixel value.

6. The image processing method as described in claim 1, characterized in that, The step of adjusting the exposure parameters of the preview image according to the degree of unevenness includes: Based on the degree of unevenness and the correspondence between the current exposure value and the target exposure value of the preview image, the target exposure value of the preview image is determined according to the current exposure value of the preview image and the degree of unevenness of the overexposed area. Adjust the current exposure value of the preview image based on the target exposure value.

7. The image processing method as described in claim 1, characterized in that, The step of fusing the adjusted preview image and the underexposed image to generate an HDR image corresponding to the preview image includes: The adjusted preview image and the underexposed image are input into the image fusion model, and the image fusion model outputs the HDR image corresponding to the preview image.

8. The image processing method as described in claim 7, characterized in that, The image fusion model is the Unet model, which includes an input layer, an encoder, a fusion unit, a decoder, and an output layer.

9. The image processing method as described in claim 8, characterized in that, The step of inputting the adjusted preview image and the underexposed image into an image fusion model, and outputting an HDR image corresponding to the preview image through the image fusion model, includes: The input layer receives the adjusted preview image and the underexposed image, and transmits the adjusted preview image and the underexposed image to the encoder; The encoder performs downsampling processing on the adjusted preview image and the underexposed image, extracts the first image feature of the adjusted preview image and the second image feature of the underexposed image respectively, and transmits the first image feature and the second image feature to the fusion unit; The fusion unit fuses the first image feature and the second image feature to obtain a third image feature, and then transmits the third image feature to the decoder; The decoder upsamples the third image features to obtain a reconstructed image, and then passes the reconstructed image to the output layer. The output layer uses a tone mapping function to perform tone mapping processing on the reconstructed image to obtain an HDR image corresponding to the preview image, and then outputs the HDR image.

10. The image processing method as described in claim 7, characterized in that, The method further includes: Create training samples; The image fusion model is obtained by training a preset deep learning model using the training samples.

11. The image processing method as described in claim 10, characterized in that, The creation of training samples includes: The camera of the electronic device is controlled to capture multiple preview images. One preview image, the corresponding underexposed image, and the corresponding theoretical HDR image are used as a set of training data to create multiple sets of training data, thus obtaining the training samples.

12. The image processing method as described in claim 11, characterized in that, The preview images of the training samples include a first preset percentage of preview images that include overexposed areas and a second preset percentage of preview images that do not include overexposed areas, wherein the first preset percentage is less than the second preset percentage.

13. The image processing method as described in claim 10, characterized in that, The preset deep learning model is the Unet model. The step of training the preset deep learning model using the training samples to obtain the image fusion model includes: The Unet model is imported into the deep learning framework and initialized by setting the initial parameters of the Unet model. A set of training data from the training samples is input into the Unet model, and the Unet model outputs the predicted HDR image corresponding to the preview image in the training data. The output value of the loss function of the Unet model is calculated based on the predicted HDR image and the theoretical HDR image corresponding to the preview image in the training data; If the output value of the loss function is greater than a preset value, adjust the parameters of the Unet model, input another set of training data into the Unet model, and continue training the Unet model; or If the output value of the loss function is less than or equal to the preset value, the Unet model training is considered complete, and the image fusion model is obtained.

14. The image processing method as described in claim 1, characterized in that, The method further includes: If the preview image is not overexposed, an underexposed image is generated based on the preview image.

15. An electronic device, characterized in that, The electronic device includes a memory and a processor: The memory is used to store program instructions; The processor is configured to read and execute the program instructions stored in the memory, and when the program instructions are executed by the processor, cause the electronic device to perform the image processing method as described in any one of claims 1 to 14.

16. A computer storage medium, characterized in that, The computer storage medium stores program instructions that, when executed on the electronic device, cause the processor of the electronic device to perform the image processing method as described in any one of claims 1 to 14.

Images