Image processing method and related equipment

Abstract

The embodiment of the invention provides an image processing method and related equipment, and the method comprises the steps: obtaining a shot preview image, and detecting whether the preview image has overexposure or not; if overexposure exists in the preview image, the non-uniformity degree of an overexposure area in the preview image is determined; adjusting exposure parameters of the preview image according to the non-uniformity degree; generating an underexposure image according to the adjusted preview image; and fusing the adjusted preview image and the underexposure image to generate an HDR image corresponding to the preview image. According to the embodiment of the invention, when it is detected that overexposure exists in the preview image, the exposure parameter of the preview image is adjusted, and then the preview image and the corresponding underexposure image are fused to generate the corresponding HDR image, so that flaws of the HDR image generated by fusion due to a transition region with uneven brightness in the preview image are avoided, and the image quality is improved. And the image quality of the HDR image is effectively improved.

Description

Technical Field

[0001] This application relates to the technical field of intelligent terminals, and particularly to an image processing method and related devices. Background Art

[0002] With the development of terminal technologies, intelligent terminal devices such as smart phones and personal computers all support high dynamic range (HDR) image display. Related technologies usually generate HDR images by fusing a normally exposed image and an underexposed image, so as to expand the dynamic range of the image. However, there are significant differences in the brightness and color of the highlight area of the preview image compared to those of other areas, which easily causes defects such as color bars and color blocks in the highlight area of the fused HDR image, resulting in poor image quality of the HDR image and affecting the user's visual experience. Summary of the Invention

[0003] In view of the above, it is necessary to provide an image processing method and related devices to solve the problem that there are significant differences in the brightness and color of the highlight area of the normally exposed image compared to those of other areas, which easily causes defects in the fused HDR image.

[0004] In a first aspect, this application provides an image processing method applied to an electronic device. The method includes: obtaining a captured preview image and detecting whether the preview image is overexposed; if the preview image is overexposed, determining the unevenness degree of the overexposed area in the preview image; adjusting the exposure parameter of the preview image according to the unevenness degree; generating an underexposed image based on the adjusted preview image; and fusing the adjusted preview image and the underexposed image to generate an HDR image corresponding to the preview image.

[0005] Through the above technical solution, before fusing the preview image to generate an HDR image, it is detected whether the preview image is overexposed. If the preview image is overexposed, the exposure parameter of the preview image is adjusted, and then the preview image and the corresponding underexposed image are fused to generate the corresponding HDR image, avoiding defects in the fused HDR image caused by the uneven light and dark transition area (i.e., the overexposed area) in the preview image, effectively improving the image quality of the HDR image and enhancing the user's visual experience.

[0006] In a possible implementation, detecting whether the preview image is overexposed includes: converting the pixel value of each pixel point in the preview image into a grayscale value; converting the grayscale value of each pixel point in the preview image into a normalized grayscale value; if the normalized grayscale value of the pixel point is greater than or equal to a preset overexposure threshold, determining that the pixel point is an overexposed pixel point; counting the number of overexposed pixel points in the preview image, and if the ratio between the number of overexposed pixel points and the total number of pixel points in the preview image is greater than or equal to a preset threshold ratio, determining that the preview image is overexposed.

[0007] Through the above technical solution, the overexposed pixel points in the preview image are determined, and by judging whether the ratio between the number of overexposed pixel points and the total number of pixel points is greater than or equal to the preset threshold ratio, it can be accurately determined whether the preview image is overexposed, and the overexposure of the preview image can be accurately judged.

[0008] In a possible implementation, converting the pixel value of each pixel point in the preview image into a grayscale value includes: performing weighted summation on the R component, G component, and B component of the RGB pixel value of each pixel point in the preview image to obtain the grayscale value of each pixel point.

[0009] Through the above technical solution, the grayscale value of each pixel point in the preview image can be accurately determined.

[0010] In a possible implementation, converting the grayscale value of each pixel point in the preview image into a normalized grayscale value includes: calculating the normalized grayscale value of each pixel point according to the grayscale value of each pixel point in the preview image, the maximum grayscale value and the minimum grayscale value of all pixel points.

[0011] Through the above technical solution, the normalized grayscale value is used as the brightness value of the pixel point to facilitate comparison with the preset overexposure threshold, so as to accurately determine whether the pixel point is an overexposed pixel point.

[0012] In a possible implementation, determining the non-uniformity degree of the overexposed area in the preview image includes: converting the color space of the preview image from the RGB color space to the YUV color space; respectively calculating the gradients of the Y component, U component, and V component of the YUV pixel value of each overexposed pixel point in the overexposed area by using a preset gradient algorithm; performing weighted summation on the gradients of the Y component, U component, and V component of the YUV pixel value of each overexposed pixel point to obtain the gradient of each overexposed pixel point; calculating the mean value of the gradients of all overexposed pixel points to obtain the gradient of the overexposed area, and using the gradient of the overexposed area as the non-uniformity degree of the overexposed area.

[0013] Through the above technical solution, calculating the gradient of the overexposed area as the degree of non-uniformity of its brightness and color can accurately determine the degree of non-uniformity of the overexposed area.

[0014] In a possible implementation, the conversion of the color space of the preview image from the RGB color space to the YUV color space includes: converting the RGB pixel values of each pixel point in the preview image into YUV pixel values.

[0015] Through the above technical solution, the preview image in the RGB color space can be accurately converted into a preview image in the YUV color space.

[0016] In a possible implementation, the adjustment of the exposure parameter of the preview image according to the degree of non-uniformity includes: obtaining the target exposure value of the preview image by using a preset inverse proportional function according to the degree of non-uniformity of the overexposed area and the current exposure value of the preview image; adjusting the current exposure value of the preview image according to the target exposure value.

[0017] Through the above technical solution, determining the adjusted exposure parameter of the preview image according to the degree of non-uniformity of the overexposed area and the preset inverse proportional function can effectively adjust the overexposed area in the preview image to a non-overexposed area and remove the uneven transitional area of light and dark in the preview image.

[0018] In a possible implementation, the adjustment of the exposure parameter of the preview image according to the degree of non-uniformity includes:

[0019] Based on the correspondence relationship between the degree of non-uniformity, the current exposure value and the target exposure value of the preview image, determining the target exposure value of the preview image according to the current exposure value of the preview image and the degree of non-uniformity of the overexposed area; adjusting the current exposure value of the preview image according to the target exposure value.

[0020] Through the above technical solution, determining the adjusted exposure parameter of the preview image according to the degree of non-uniformity of the overexposed area, the correspondence relationship between the degree of non-uniformity, the current exposure value and the target exposure value of the preview image can effectively adjust the overexposed area in the preview image to a non-overexposed area and remove the uneven transitional area of light and dark in the preview image.

[0021] In a possible implementation, generating an underexposed image based on the adjusted preview image includes: obtaining the dynamic range value and the exposure value of the preview image; determining the exposure value of the underexposed image according to the correspondence between the dynamic range, the exposure value of the preview image and the exposure value of the underexposed image, and the dynamic range value and the exposure value of the preview image; and adjusting the exposure value of the adjusted preview image to the exposure value of the underexposed image to obtain the underexposed image.

[0022] Through the above technical solution, according to the dynamic range value and the exposure value of the current preview image and the correspondence between the dynamic range value, the exposure value of the preview image and the exposure value of the underexposed image, the exposure value of the underexposed image to be generated can be accurately determined, and then the exposure value of the current preview image can be adjusted to the exposure value of the underexposed image to be generated, and the corresponding underexposed image can be accurately obtained.

[0023] In a possible implementation, fusing the adjusted preview image and the underexposed image to generate the HDR image corresponding to the preview image includes: inputting the adjusted preview image and the underexposed image into an image fusion model, and outputting the HDR image corresponding to the preview image through the image fusion model.

[0024] Through the above technical solution, using an image fusion model can accurately generate the HDR image corresponding to the preview image, and improve the generation efficiency and image quality of the HDR image.

[0025] In a possible implementation, the image fusion model is a Unet model, and the image fusion model includes an input layer, an encoder, a fuser, a decoder and an output layer.

[0026] Through the above technical solution, using the Unet model as the image fusion model can improve the feature extraction efficiency of the image, reduce the loss of details in the image fusion process, and has strong scalability.

[0027] In a possible implementation, inputting the adjusted preview image and the underexposed image into an image fusion model, and outputting an HDR image corresponding to the preview image through the image fusion model includes: the input layer receives the input preview image and the underexposed image, and transfers the preview image and the underexposed image to the encoder; the encoder performs downsampling processing on the preview image and the underexposed image, respectively extracts the first image feature of the preview image and the second image feature of the underexposed image, and transfers the first image feature and the second image feature to the fuser; the fuser fuses the first image feature and the second image feature to obtain a third image feature, and transfers the third image feature to the decoder; the decoder performs upsampling processing on the third image feature to obtain a reconstructed image, and transfers the reconstructed image to the output layer; the output layer performs tone mapping processing on the reconstructed image by using a tone mapping function to obtain the HDR image corresponding to the preview image, and outputs the HDR image.

[0028] Through the above technical solution, using the Unet model to perform feature extraction, feature fusion, image reconstruction, and tone mapping on the preview image and the underexposed image can accurately and efficiently generate the HDR image corresponding to the preview image.

[0029] In a possible implementation, the method further includes: creating a training sample; training a preset deep learning model through the training sample to obtain the image fusion model.

[0030] Through the above technical solution, by creating a training sample to train a preset deep learning model, an accurate image fusion model can be obtained, thereby improving the generation accuracy and generation efficiency of HDR.

[0031] In a possible implementation, creating the training sample includes: controlling the camera of the electronic device to capture a plurality of preview images, and taking one preview image, the corresponding underexposed image, and the corresponding theoretical HDR image as a set of training data, creating multiple sets of training data to obtain the training sample.

[0032] Through the above technical solution, using the actually captured preview images as training data can avoid overfitting of the image fusion model and improve the generalization ability of the image fusion model.

[0033] In a possible implementation, the preview images in the training sample include preview images with a first preset percentage of overexposed regions and preview images with a second preset percentage of non-overexposed regions, and the first preset percentage is less than the second preset percentage.

[0034] Through the above technical solution, using a preview image including an overexposed area as training data can effectively prevent the image fusion model from overfitting and improve the generalization ability of the image fusion model.

[0035] In a possible implementation, the preset deep learning model is a Unet model. Training the preset deep learning model with the training samples to obtain the image fusion model includes: importing the Unet model into the deep learning framework, initializing the Unet model, and setting the initial parameters of the Unet model; inputting a set of training data in the training samples into the Unet model, and outputting a predicted HDR image corresponding to the preview image in the training data through the Unet model; calculating the output value of the loss function of the Unet model according to the predicted HDR image and the theoretical HDR image corresponding to the preview image in the training data; if the output value of the loss function is greater than a preset value, adjusting the parameters of the Unet model, inputting another set of training data into the Unet model, and continuing to train the Unet model; or if the output value of the loss function is less than or equal to the preset value, determining that the training of the Unet model is completed to obtain the image fusion model.

[0036] Through the above technical solution, the accuracy of the image fusion model established by training can be improved.

[0037] In a possible implementation, the method further includes: if the preview image does not have overexposure, generating an underexposed image according to the preview image.

[0038] Through the above technical solution, when the preview image does not have overexposure, directly generating an underexposed image corresponding to the preview image can improve the generation efficiency of the underexposed image.

[0039] In a second aspect, the present application provides an electronic device, which includes a memory and a processor: wherein, the memory is used to store program instructions; the processor is used to read and execute the program instructions stored in the memory, and when the program instructions are executed by the processor, the electronic device is enabled to execute the above image processing method.

[0040] In a third aspect, the present application provides a chip coupled to the memory in the electronic device, and the chip is used to control the processor of the electronic device to execute the above image processing method.

[0041] In a fourth aspect, the present application provides a computer storage medium, which stores program instructions, and when the program instructions are run on an electronic device, the processor of the electronic device is enabled to execute the above image processing method.

[0042] In addition, for the technical effects brought by the second to fourth aspects, reference may be made to the descriptions related to the methods of each design in the above method part, which will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic diagram of a normally exposed image provided by an embodiment of the present application.

[0044] Figure 2 It is a schematic diagram of an underexposed image provided by an embodiment of the present application.

[0045] Figure 3 It is a schematic diagram of an HDR image provided by an embodiment of the present application.

[0046] Figure 4 It is a software architecture diagram of an electronic device provided by an embodiment of the present application.

[0047] Figure 5 It is a flowchart of an image processing method provided by an embodiment of the present application.

[0048] Figure 6 It is a schematic diagram of a preview image provided by an embodiment of the present application.

[0049] Figure 7 It is a schematic diagram of an adjusted preview image provided by an embodiment of the present application.

[0050] Figure 8 It is a schematic structural diagram of an image fusion model provided by an embodiment of the present application.

[0051] Figure 9 It is a schematic diagram of an HDR image corresponding to a preview image provided by an embodiment of the present application.

[0052] Figure 10 It is a flowchart of determining whether a preview image is overexposed provided by an embodiment of the present application.

[0053] Figure 11 It is a flowchart of determining the non-uniformity degree of the overexposed area in a preview image provided by an embodiment of the present application.

[0054] Figure 12 It is a flowchart of an image processing method provided by another embodiment of the present application.

[0055] Figure 13 It is a hardware architecture diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0056] In the embodiments of the present application, the terms "first" and "second" are only used for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more features. In the description of the embodiments of the present application, words such as "exemplary" or "for example" are used to mean examples, illustrations, or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific manner.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. It should be understood that unless otherwise specified in this application, " / " means "or". For example, A / B may mean A or B. The "and / or" in this application is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may mean: A exists alone, A and B exist simultaneously, and B exists alone, three situations. "At least one" means one or more. "Multiple" means two or more than two. For example, at least one of a, b, or c may mean: a, b, c, a and b, a and c, b and c, a, b, and c, seven situations. Without conflict, the following embodiments and the features in the embodiments may be combined with each other.

[0058] With the development of terminal technology, intelligent terminal devices such as smart phones and personal computers all support high-dynamic-range image display. Related technologies usually generate HDR images by fusing preview images with underexposed images, thereby expanding the dynamic range of the images. However, there are significant differences in the brightness and color of the highlight areas of the preview images compared to those of other areas, which easily cause defects such as color bars and color blocks in the highlight areas of the fused HDR images, resulting in poor image quality of the HDR images and affecting the user's visual experience.

[0059] Refer to Figure 1As shown, it is a schematic diagram of a normal exposure image provided by an embodiment of the present application. In a scene with strong light and dark contrast (such as a backlight scene), a large contrast ratio phenomenon (strong light and dark contrast in the image) will occur. At this time, the brightness of the highlight area of the normal exposure image may also be significantly brighter than that of other areas. Since the highlight area of the sky is brighter, the color approaches white, and the other areas of the sky are blue. Therefore, there are also significant differences in the colors between the highlight area and other areas.

[0060] Refer to Figure 2 As shown, it is a schematic diagram of an underexposed image provided by an embodiment of the present application. The exposure time of the underexposed (ShortExposure) image is shorter than that of the preview image, and the brightness is darker. An image fusion model is used to fuse the preview image and the underexposed image to generate an HDR image as shown in Figure 3 Since there are significant differences in the brightness and color between the highlight area and other areas of the preview image, color bar defects appear in the highlight area of the fused HDR image. Thus, the image quality of the fused HDR image is poor, which affects the user's visual experience.

[0061] In order to avoid defects in the fused HDR image, an embodiment of the present application provides an image processing method that can detect the overexposed area of the normal exposure image, adjust the brightness of the normal exposure image, and then fuse the normal exposure image and the underexposed image to generate an HDR image, avoiding fusion defects in the HDR image, effectively improving the image quality of the HDR image, and enhancing the user's visual experience. The image processing method is applied to an electronic device. The following combines Figure 4 to illustrate the software architecture diagram of the electronic device.

[0062] Refer to Figure 4 As shown, it is the software architecture diagram of the electronic device provided by an embodiment of the present application. The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. For example, the Android system is divided into four layers, from top to bottom are the application layer 101, the framework layer 102, the Android runtime and system library 103, the hardware abstraction layer 104, the kernel layer 105, and the hardware layer 106.

[0063] The application layer 101 may include a series of application packages. For example, the application packages may include applications such as the camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, and device control service.

[0064] The framework layer 102 provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions. For example, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, etc.

[0065] Among them, the window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc. The content provider is used to store and obtain data, and make this data accessible to applications. The data may include videos, images, audio, dialed and received calls, browsing history and bookmarks, phone books, etc. The view system includes visible controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. For example, a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures. The phone manager is used to provide the communication functions of the electronic device. For example, the management of call states (including answering, hanging up, etc.). The resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, etc. The notification manager enables applications to display notification information in the status bar, can be used to convey notification-type messages, can disappear automatically after a short stay without user interaction. For example, the notification manager is used to inform that the download is completed, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as a notification of a background-running application, and can also be a notification that appears on the screen in the form of a dialogue window. For example, prompt text information in the status bar, emit a prompt tone, the electronic device vibrates, the indicator light flashes, etc.

[0066] Android Runtime includes a core library and a virtual machine. Android runtime is responsible for the scheduling and management of the Android system. The core library contains two parts: one part is the functional functions that the Java language needs to call, and the other part is the core library of Android.

[0067] The application layer 101 and the framework layer 102 run in the virtual machine. The virtual machine executes the Java files of the application layer and the framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0068] The system library 103 may include multiple functional modules. For example, a surface manager, media libraries, a 3D graphics processing library (e.g., OpenGL ES), a 2D graphics engine (e.g., SGL), etc.

[0069] Among them, the surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications. The media libraries support the playback and recording of multiple common audio and video formats, as well as static image files, etc. The media libraries can support multiple audio and video coding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing, etc. The 2D graphics engine is the drawing engine for 2D drawing.

[0070] The hardware abstraction layer 104 runs in the user space, encapsulates the kernel layer drivers, and provides call interfaces to the upper layer.

[0071] The kernel layer 105 is the layer between the hardware and the software. The kernel layer 105 at least includes a display driver, a camera driver, an audio driver, and a sensor driver.

[0072] The kernel layer 105 is the core of the operating system of the electronic device, is the first layer of software expansion based on the hardware, provides the most basic functions of the operating system, is the basis for the operation of the operating system, and is responsible for managing the system's processes, memory, device drivers, files, and network systems, and determines the performance and stability of the system. For example, the kernel can determine the operation time of an application for a certain part of the hardware.

[0073] The kernel layer 105 includes programs closely related to the hardware, such as interrupt handlers, device drivers, etc., and also includes basic, common, and frequently running modules, such as a clock management module, a process scheduling module, etc., and also includes key data structures. The kernel layer can be set in the processor or solidified in the internal memory.

[0074] The hardware layer 106 includes the hardware of the electronic device, such as a display screen, keys, a camera, etc.

[0075] See Figure 5 As shown, it is a flowchart of an image processing method provided by an embodiment of the present application. The method is applied to an electronic device, and the image processing method includes:

[0076] S101, obtain the captured preview image and detect whether the preview image is overexposed. If the preview image is overexposed, the process proceeds to S102; if the preview image is not overexposed, the process proceeds to S106.

[0077] In an embodiment of the present application, the user can click on the camera application icon on the main interface of the electronic device, or click on the camera activation control on the lock screen interface of the electronic device to perform the operation of activating the camera application. The electronic device responds to the user's operation of activating the camera application and uses the shooting device of the electronic device to obtain a preview image. In an embodiment, the electronic device can control the camera to capture an image of any scene in the fixed auto exposure (Fixed AE) mode and collect the preview image. The preview image can also be referred to as a normal exposure image. For example, it is an image captured by the camera when the exposure value is 0 EV. The exposure value of the preview image is 0 EV. 0 EV is a relative value and does not mean that the exposure value is 0. Exemplarily, the exposure value is calculated based on the combination of the aperture and exposure time (i.e., shutter speed) of the camera. The larger the exposure value, the brighter the image, and the smaller the exposure value, the darker the image.

[0078] In an embodiment of the present application, after the camera application is activated, a camera pipeline is established according to the shooting mode. The camera pipeline includes multiple image processing nodes. The CameraService in the framework layer and the CameraProvider in the hardware abstraction layer are initialized. After the initialization is completed, the camera application sends a preview shooting request to the CameraService. The CameraService sends the preview shooting request to the CameraProvider. The CameraProvider sends the preview shooting request or a shooting request generated in response to the user triggering the shooting control to the camera driver in the kernel layer. Through the camera driver, the camera can be driven to collect the preview image, and the collected preview image is returned to the CameraProvider. The CameraProvider sends the preview image to the image processing nodes for processing.

[0079] In an embodiment of the present application, it is determined whether the ratio between the number of overexposed pixel points and the total number of pixel points in the preview image is greater than or equal to a preset threshold ratio. If the ratio between the number of overexposed pixel points and the total number of pixel points in the preview image is greater than or equal to the preset threshold ratio, it is determined that the preview image is overexposed. If the ratio between the number of overexposed pixel points and the total number of pixel points in the preview image is less than the preset threshold ratio, it is determined that the preview image is not overexposed. For example, the preset threshold ratio can be set to 20%, 25%, 30% or other percentages, which are only for illustrative purposes and are not limited thereto in actual applications.

[0080] In an embodiment of the present application, the pixel value of each pixel in the preview image is converted into a grayscale value, and the grayscale value of each pixel is converted into a normalized grayscale value. It is determined whether the normalized grayscale value of the pixel is greater than or equal to a preset overexposure threshold. If the normalized grayscale value of the pixel is greater than or equal to the preset overexposure threshold, the pixel is determined as an overexposed pixel; if the normalized grayscale value of the pixel is less than the preset overexposure threshold, the pixel is determined not to be an overexposed pixel. After all pixels in the preview image have completed the determination of overexposed pixels, the number of overexposed pixels is counted. In an embodiment of the present application, the preset overexposure threshold can be set between 0.8 and 0.95. For example, the preset overexposure threshold can be 0.8, 0.9, 0.95 or other values.

[0081] S102. Determine the non-uniformity degree of the overexposed area in the preview image.

[0082] Refer to Figure 6 As shown, it is a schematic diagram of the preview image provided by the embodiment of the present application. When there is overexposure in the preview image, the preview image includes an overexposed area marked by a square box as shown in Figure 6 The overexposed area is a transition area with uneven brightness and darkness.

[0083] In an embodiment of the present application, the color space of the preview image is converted from the Red Green Blue (RGB) color space to the Luminance-Chrominance (YUV) color space. The gradients of the Y component, U component, and V component in the YUV pixel values of each overexposed pixel are calculated respectively using a preset gradient algorithm. The gradients of the Y component, U component, and V component in the YUV pixel values of each overexposed pixel are weighted and summed to obtain the gradient of each overexposed pixel. The average value of the gradients of all overexposed pixels is calculated to obtain the gradient of the overexposed area, and the gradient of the overexposed area is determined as the non-uniformity degree of the overexposed area. The non-uniformity degree refers to the non-uniformity degree of brightness and color in the overexposed area. For example, the preset gradient algorithm can be the Sobel gradient algorithm or the Laplace gradient algorithm.

[0084] S103. Adjust the exposure parameters of the preview image according to the non-uniformity degree.

[0085] In an embodiment of the present application, the exposure parameters include the exposure value. The non-uniformity degree G in the overexposed area and the current exposure value B0 of the preview image are input into a preset inverse proportional function, and the target exposure value B of the preview image is output through the preset inverse proportional function, and the current exposure value of the preview image is adjusted to the target exposure value B. Based on the preset inverse proportional function, if the non-uniformity degree of the overexposed area is greater, the exposure value of the preview image is smaller. Among them, the calculation formula for calculating the exposure value B of the preview image according to the non-uniformity degree G of the overexposed area and the current exposure value B0 of the preview image is:

[0086] B = f(G, B0) (1).

[0087] In the calculation formula (1), B is the target exposure value of the preview image, G is the non-uniformity degree of the overexposed area in the preview image, B0 is the current exposure value of the preview image, and f is a preset inverse proportional function. For example:

[0088]

[0089] In other embodiments of the present application, f may also be other inverse proportional functions obtained by fitting empirical values of multiple non-uniformity degrees and the corresponding target exposure values and current exposure values for each non-uniformity degree.

[0090] In another embodiment of the present application, the electronic device pre-sets and stores the correspondence relationship between the non-uniformity degree, the current exposure value of the preview image, and the target exposure value. For example, the correspondence relationship between the non-uniformity degree, the current exposure value of the preview image, and the target exposure value is recorded through a preset correspondence table. According to the correspondence table, the target exposure value B corresponding to the non-uniformity degree G and the current exposure value B0 in the overexposed area can be determined, and the current exposure value B0 of the preview image can be adjusted according to the target exposure value B. For example, the current exposure value B0 of the preview image is adjusted to the target exposure value B.

[0091] Refer to Figure 7 As shown, it is a schematic diagram of the adjusted preview image provided by the embodiment of the present application. After adjusting the exposure parameters of the overexposed area in the preview image, that is, after reducing the exposure, as Figure 7 the original overexposed area marked by the square frame is no longer overexposed.

[0092] S104, generate an underexposed image according to the adjusted preview image.

[0093] In an embodiment of the present application, an underexposed image is generated according to the Smart Auto Exposure (Smart AE) mode. That is to say, an underexposed image is generated by reducing the exposure of the preview image after adjusting the exposure parameters. The electronic device also pre-stores the correspondence between the dynamic range (DR) value of the preview image, the exposure value of the preview image, and the exposure value of the underexposed image. The correspondence can be recorded in the form of a table or a list to record the correspondence between multiple parties. Taking the correspondence table as an example, the dynamic range value and the exposure value of the preview image are obtained, and the exposure value of the corresponding underexposed image is determined by looking up the correspondence table between the dynamic range of the preview image, the exposure value of the preview image, and the exposure value of the underexposed image according to the dynamic range value and the exposure value of the preview image. The exposure value of the adjusted preview image is adjusted to the exposure value of the underexposed image to obtain the underexposed image. The underexposed image is also called the S-frame image, which is an image with an exposure value less than 0 EV. That is to say, the exposure value of the underexposed image is less than 0 EV, such as the exposure value of the underexposed image is -2 EV, -4 EV, etc. Among them, the exposure value of the preview image is the target exposure value of the adjusted preview image.

[0094] In an embodiment of the present application, the dynamic range value of the preview image is a value automatically returned by the camera driver after the camera captures the preview image. The dynamic range value is between 0 and 1, for example, it can be 0.2, 0.3, 0.9, or other values. For example, in the correspondence table between the dynamic range value of the preview image, the exposure value of the preview image, and the exposure value of the underexposed image, the exposure value of the underexposed image corresponding to the dynamic range value of 0.2 and the exposure value of 0 EV of the preview image is -2 EV; the exposure value of the underexposed image corresponding to the dynamic range value of 0.3 and the exposure value of 0 EV of the preview image is -3 EV; the exposure value of the underexposed image corresponding to the dynamic range value of 0.9 and the exposure value of 0 EV of the preview image is -6 EV.

[0095] S105, fuse the adjusted preview image and the underexposed image to generate an HDR image corresponding to the preview image.

[0096] In an embodiment of the present application, the adjusted preview image and the underexposed image are input into an image fusion model, and an HDR image corresponding to the preview image is output through the image fusion model.

[0097] Refer to Figure 8As shown in the figure, it is a schematic structural diagram of an image fusion model provided by an embodiment of the present application. In an embodiment of the present application, the image fusion model is a Unet model. The Unet model includes an input layer, an encoder, a fuser, a decoder, and an output layer. The input layer is used to receive the input preview image and underexposed image, and transfer the preview image and underexposed image to the encoder. The encoder performs downsampling on the preview image and underexposed image. The downsampling includes convolution operations and pooling operations, respectively extracting the first image features of the preview image and the second image features of the underexposed image, and transferring the first image features and the second image features to the fuser. The fuser fuses the first image features and the second image features to obtain third image features, and transfers the third image features to the decoder. The decoder performs upsampling on the third image features. The upsampling includes deconvolution operations to obtain a reconstructed image, and transfers the reconstructed image to the output layer. The output layer uses a tone mapping function to map the reconstructed image to obtain the HDR image corresponding to the preview image, and outputs the HDR image.

[0098] Referring to Figure 9 As shown in the figure, after adjusting the exposure parameters of the preview image, there are no defects such as color blocks and color bars in the HDR image obtained by fusing the preview image and the underexposed image.

[0099] S106. Generate an underexposed image according to the preview image.

[0100] S107. Fuse the preview image and the underexposed image to generate the HDR image corresponding to the preview image.

[0101] The specific implementation manners of S106 - S107 are the same as those of S104 - S105, and will not be elaborated here.

[0102] Through the above embodiments of the present application, before fusing the preview image to generate the HDR image, it is detected whether the preview image is overexposed. If the preview image is overexposed, the exposure parameters of the preview image are adjusted, and then the preview image and the corresponding underexposed image are fused to generate the corresponding HDR image, avoiding defects in the fused HDR image caused by the uneven light and dark transition area (i.e., the overexposed area) in the preview image, effectively improving the image quality of the HDR image and enhancing the user's visual experience.

[0103] Referring to Figure 10 As shown in the figure, it is a flowchart for determining whether the preview image is overexposed provided by an embodiment of the present application.

[0104] S1011. Convert the pixel value of each pixel point in the preview image into a grayscale value.

[0105] In an embodiment of the present application, the preview image captured by the camera is an RGB image. The R component, G component, and B component of the RGB pixel values of each pixel point in the preview image are weighted and summed to obtain the gray value of each pixel point. Among them, the calculation formula for the gray value of the pixel point is:

[0106] Gray = x * R + y * G + z * B (3).

[0107] In the calculation formula (3), R is the R component of the pixel value, x is the weight of the R component, for example, 0.3, G is the G component of the pixel value, y is the weight of the G component, for example, 0.59, B is the B component of the pixel value, and z is the weight of the B component, for example, 0.11.

[0108] In another embodiment of the present application, the gray value of each pixel point in the preview image can also be obtained by calculating the average value of the R component, G component, and B component of the RGB pixel values of each pixel point.

[0109] S1012, convert the gray value of each pixel point in the preview image into a normalized gray value.

[0110] In an embodiment of the present application, according to the gray value of each pixel point in the preview image, the maximum gray value and the minimum gray value of all pixel points, calculate the normalized gray value of each pixel point. Among them, the calculation formula for the normalized gray value Gray0 of the pixel point is:

[0111]

[0112] In the calculation formula (4), Gray is the gray value of each pixel point, Gray max is the maximum gray value of all pixel points, Gray min is the minimum gray value of all pixel points.

[0113] S1013, determine whether the normalized gray value of the pixel point in the preview image is greater than or equal to a preset overexposure threshold. If the normalized gray value of the pixel point in the preview image is greater than or equal to the preset overexposure threshold, the process proceeds to S1014; if the normalized gray value of the pixel point is less than the preset overexposure threshold, the process proceeds to S1015.

[0114] S1014, determine that the pixel point is an overexposed pixel point.

[0115] S1015, determine that the pixel point is not an overexposed pixel point.

[0116] S1016. Count the number of overexposed pixels in the preview image, and determine whether the ratio between the number of overexposed pixels and the total number of pixels in the preview image is greater than or equal to a preset threshold ratio. If the ratio between the number of overexposed pixels and the total number of pixels in the preview image is greater than or equal to the preset threshold ratio, the process proceeds to S1017; if the ratio between the number of overexposed pixels and the total number of pixels in the preview image is less than the preset threshold ratio, the process proceeds to S1018.

[0117] S1017. Determine that the preview image is overexposed.

[0118] S1018. Determine that the preview image is not overexposed.

[0119] Through the above embodiments of the present application, the overexposed pixels in the preview image can be determined, and by judging whether the ratio between the number of overexposed pixels and the total number of pixels is greater than or equal to a preset threshold ratio, it can be determined whether the preview image is overexposed, and the overexposure of the preview image can be accurately judged.

[0120] See Figure 11 As shown, it is a flowchart for determining the non-uniformity degree of the overexposed area in the preview image provided by the embodiment of the present application.

[0121] S1021. Convert the color space of the preview image from the RGB color space to the YUV color space.

[0122] In an embodiment of the present application, the RGB pixel values of each pixel point in the preview image are converted into YUV pixel values, so as to convert the color space of the preview image from the RGB color space to the YUV color space. Among them, the calculation formula for converting the RGB pixel value of a pixel point into a YUV pixel value is:

[0123]

[0124] In the calculation formula (5), R, G, and B are respectively the R component, G component, and B component of the RGB pixel value of the pixel point in the preview image, and Y, U, and V are respectively the Y component, U component, and V component of the YUV pixel value of the pixel point in the preview image.

[0125] S1022. Use a preset gradient algorithm to calculate the gradients of the Y component, U component, and V component in the YUV pixel values of each overexposed pixel point respectively.

[0126] In an embodiment of the present application, the preset gradient algorithm is the Sobel gradient algorithm. The calculation formula for the Sobel gradient algorithm to calculate the gradients of the Y component, U component, and V component in the YUV pixel values of each overexposed pixel point is:

[0127]

[0128] In an embodiment of the present application, the calculation formula of the Sobel gradient algorithm is as follows:

[0129]

[0130]

[0131]

[0132] In calculation formulas (7) and (8), I is the Y component, U component, or V component in the YUV pixel value.

[0133] S1023. Perform weighted summation on the gradients of the Y component, U component, and V component in the YUV pixel value of each overexposed pixel point to obtain the gradient of each overexposed pixel point.

[0134] In an embodiment of the present application, the calculation formula of the gradient of a pixel point is as follows:

[0135] G = W_Y * G Y + W_U * G u + W_V * G v (10).

[0136] In calculation formula (10), W_Y, W_U, and W_V are set between 0 and 1. Since the gradient of the overexposed area is mainly reflected in the luminance channel Y, W_Y is greater than W_U and W_V. For example, W_Y = 0.6, W_U = 0.2, and W_V = 0.2.

[0137] S1024. Calculate the mean value of the gradients of all overexposed pixel points to obtain the gradient of the overexposed area, and determine the gradient of the overexposed area as the degree of non-uniformity of the overexposed area.

[0138] Through the above embodiments of the present application, calculating the gradient of the overexposed pixel point as the degree of non-uniformity of its brightness and color can accurately determine the degree of non-uniformity of the overexposed pixel point, and further accurately determine the degree of non-uniformity of the overexposed area.

[0139] Refer to Figure 12 As shown in the figure, it is a flowchart of an image processing method provided by another embodiment of the present application. The method is applied to an electronic device, and the image processing method includes:

[0140] S201. Create a training sample.

[0141] In an embodiment of the present application, the camera is controlled to capture multiple preview images. One preview image, the corresponding underexposed image, and the corresponding theoretical HDR image are used as a set of training data. Multiple sets of training data are created to obtain training samples. In the preview images of the training samples, the preview images with the first preset percentage include overexposed regions, and the preview images with the second preset percentage do not include overexposed regions. The first preset percentage is less than the second preset percentage. For example, the first preset percentage is 20%, and the second preset percentage is 80%.

[0142] In an embodiment of the present application, the preview image including the overexposed region can be obtained by capturing with the camera. In another embodiment of the present application, the preview image including the overexposed region can also be obtained by increasing the brightness of a preset region in the preview image, making the preset region an overexposed region, so as to simulate a preview image with uneven bright transition. The underexposed image corresponding to the preview image is generated by simulating the method in step S104 of the above embodiment. That is, the underexposed image corresponding to the preview image is obtained by reducing the exposure value of the preview image.

[0143] S202. Train a preset deep learning model with the training samples to obtain an image fusion model.

[0144] In an embodiment of the present application, the preset deep learning model is a Unet model. The Unet model is imported into the deep learning framework, and the Unet model is initialized. The initial parameters of the Unet model are set. A set of training data in the training samples is input into the Unet model, and the predicted HDR image is output through the Unet model. The output value of the loss function of the Unet model is calculated according to the predicted HDR image and the theoretical HDR image. If the output value of the loss function is greater than the preset value, another set of training data is input into the Unet model to continue training the Unet model. If the output value of the loss function is less than or equal to the preset value, it is determined that the training of the Unet model is completed, and an image fusion model is obtained.

[0145] In an embodiment of the present application, the loss function is the mean square error (MSE) between the theoretical HDR image and the predicted HDR image. That is, by calculating the mean square error between the theoretical HDR image and the predicted HDR image as the difference value between the theoretical HDR image and the predicted HDR image. Among them, the calculation formula of the mean square error is:

[0146]

[0147] In calculation formula (11), y i can be the gray value of the i-th pixel point of the theoretical HDR image, y i PTo predict the gray value of the i-th pixel in the HDR image, where n is the number of pixels in the theoretical HDR image and the predicted HDR image.

[0148] In another embodiment of the present application, the preset deep learning model can also be a ResNet model.

[0149] S203, Collect a preview image and determine whether the preview image is overexposed. If the preview image is overexposed, the process proceeds to S204; if the preview image is not overexposed, the process proceeds to S208.

[0150] S204, Determine the non-uniformity degree of the overexposed area in the preview image.

[0151] S205, Adjust the exposure parameters of the preview image according to the non-uniformity degree.

[0152] S206, Generate an underexposed image according to the adjusted preview image.

[0153] S207, Fuse the adjusted preview image and the underexposed image through an image fusion model to generate an HDR image corresponding to the preview image.

[0154] S208, Generate an underexposed image according to the preview image.

[0155] S209, Fuse the preview image and the underexposed image through an image fusion model to generate an HDR image corresponding to the preview image.

[0156] The specific implementation manners of S203 to S209 are the same as those of S101 to S107, and will not be elaborated here.

[0157] Through the above embodiments of the present application, using the actually captured preview image without overexposed areas and the actually captured or simulated preview image containing overexposed areas as training data to train the preset deep learning model to obtain an image fusion model can avoid overfitting of the image fusion model, improve the generalization ability of the image fusion model, and further improve the efficiency of the image fusion model in generating HDR images, and improve the accuracy and image quality of the fused HDR images.

[0158] The embodiments of the present application also provide an electronic device 100, refer to Figure 13As shown, the electronic device 100 may be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an Ultra-mobile Personal Computer (UMPC), a netbook, a cellular phone, a Personal Digital Assistant (PDA), an Augmented Reality (AR) device, a Virtual Reality (VR) device, an Artificial Intelligence (AI) device, a wearable device, a vehicle-mounted device, a smart home device, and / or a smart city device. The specific type of the electronic device 100 is not particularly limited in the embodiments of this application.

[0159] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a Universal Serial Bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a Subscriber Identification Module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0160] It can be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than those illustrated, or combine certain components, or split certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0161] The processor 110 may include one or more processing units. For example, the processor 110 may include an Application Processor (AP), a modem processor, a Graphics Processing Unit (GPU), an Image Signal Processor (ISP), a controller, a video codec, a Digital Signal Processor (DSP), a baseband processor, and / or a Neural-network Processing Unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0162] The controller may generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.

[0163] A memory may also be provided in the processor 110 for storing instructions and data. In an embodiment of the present application, the memory in the processor 110 is a cache memory. The memory may save the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instructions or data again, it can directly call them from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0164] In an embodiment of the present application, the processor 110 may include one or more interfaces. The interfaces may include an Inter-integrated Circuit (I2C) interface, an Inter-integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a General-Purpose Input / Output (GPIO) interface, a Subscriber Identity Module (SIM) interface, and / or a Universal Serial Bus (USB) interface, etc.

[0165] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In an embodiment of the present application, the processor 110 may include multiple groups of I2C buses. The processor 110 may be respectively coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K through the I2C interface, enabling the processor 110 and the touch sensor 180K to communicate through the I2C bus interface, and implementing the touch function of the electronic device 100.

[0166] The I2S interface can be used for audio communication. In an embodiment of the present application, the processor 110 may include multiple groups of I2S buses. The processor 110 may be coupled to the audio module 170 through the I2S bus to implement communication between the processor 110 and the audio module 170. In an embodiment of the present application, the audio module 170 may transmit an audio signal to the wireless communication module 160 through the I2S interface to implement the function of answering a call through a Bluetooth headset.

[0167] The PCM interface can also be used for audio communication to sample, quantize, and encode analog signals. In an embodiment of the present application, the audio module 170 and the wireless communication module 160 may be coupled through the PCM bus interface. In an embodiment of the present application, the audio module 170 may also transmit an audio signal to the wireless communication module 160 through the PCM interface to implement the function of answering a call through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0168] The UART interface is a general-purpose serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In an embodiment of the present application, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In an embodiment of the present application, the audio module 170 may transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0169] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a Camera Serial Interface (CSI), a Display Serial Interface (DSI), etc. In an embodiment of the present application, the processor 110 and the camera 193 communicate through the CSI interface to implement the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate through the DSI interface to implement the display function of the electronic device 100.

[0170] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or a data signal. In an embodiment of the present application, the GPIO interface can be used to connect the processor 110 to the camera 193, the display screen 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0171] The USB interface 130 is an interface that conforms to the USB standard specification, and can specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used for data transmission between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio. The interface can also be used to connect other electronic devices 100, such as AR devices, etc.

[0172] It can be understood that the interface connection relationship between the modules illustrated in the embodiments of the present invention is only for illustrative purposes and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 can also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0173] The charging management module 140 is used to receive a charging input from a charger. Among them, the charger can be a wireless charger or a wired charger. In some embodiments of wired charging, the charging management module 140 can receive the charging input from the wired charger through the USB interface 130. In some embodiments of wireless charging, the charging management module 140 can receive the wireless charging input through the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device 100 through the power management module 141.

[0174] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives inputs from the battery 142 and / or the charging management module 140 and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, the wireless communication module 160, etc. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be disposed in the processor 110. In some other embodiments, the power management module 141 and the charging management module 140 can also be disposed in the same device.

[0175] The wireless communication function of the electronic device 100 can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modulation and demodulation processor, and the baseband processor, etc.

[0176] The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example, the antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0177] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc. applied to the electronic device 100. The mobile communication module 150 can include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves by the antenna 1, filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor and convert it into electromagnetic waves through the antenna 1 for radiation. In an embodiment of the present application, at least some function modules of the mobile communication module 150 can be disposed in the processor 110. In an embodiment of the present application, at least some function modules of the mobile communication module 150 and at least some modules of the processor 110 can be disposed in the same device.

[0178] The modulation and demodulation processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. Subsequently, the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.), or displays an image or video through the display screen 194. In an embodiment of the present application, the modulation and demodulation processor may be an independent device. In other embodiments, the modulation and demodulation processor may be independent of the processor 110 and be disposed in the same device as the mobile communication module 150 or other functional modules.

[0179] The wireless communication module 160 may provide solutions for wireless communications applied to the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite systems (GNSSs), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signals, and transmits the processed signals to the processor 110. The wireless communication module 160 may also receive the signals to be transmitted from the processor 110, perform frequency modulation and amplification on them, and convert them into electromagnetic waves through the antenna 2 for radiation.

[0180] In an embodiment of the present application, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technologies. The wireless communication technologies may include Global System For Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), Beidou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0181] The electronic device 100 implements the display function through the GPU, the display screen 194, and the application processor, etc. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0182] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled, a Microled, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In an embodiment of the present application, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0183] The electronic device 100 can implement the shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, an application processor, etc.

[0184] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and light passes through the lens and is transmitted to the camera photosensitive element. The optical signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In an embodiment of the present application, the ISP can be set in the camera 193.

[0185] The camera 193 is used to capture static images or videos. An object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in standard RGB, YUV, etc. formats. In an embodiment of the present application, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0186] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0187] The video codec is used to compress or decompress digital videos. The electronic device 100 can support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple coding formats, such as: Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0188] The NPU is a Neural-Network (NN) computing processor. By drawing on the structure of the biological neural network, such as the transmission pattern between human brain neurons, it can quickly process the input information and can also continuously self-learn. Through the NPU, applications such as intelligent cognition of the electronic device 100 can be realized, such as: image recognition, face recognition, speech recognition, text understanding, etc.

[0189] The internal memory 121 may include one or more Random Access Memories (RAM) and one or more Non-Volatile Memories (NVM).

[0190] The random access memory may include Static Random-Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM, for example, the fifth-generation DDR SDRAM is generally called DDR5 SDRAM), etc.;

[0191] The non-volatile memory may include disk storage devices, flash memory.

[0192] Flash memory can be classified according to its operating principle into NOR Flash, NAND Flash, 3D NAND Flash, etc., according to the number of potential levels of storage cells into Single-Level Cell (SLC), Multi-Level Cell (MLC), Triple-Level Cell (TLC), Quad-Level Cell (QLC), etc., and according to storage specifications into Universal Flash Storage (UFS), embedded Multi Media Card (eMMC), etc.

[0193] The random access memory can be directly read and written by the processor 110, and can be used to store the operating system or executable programs (such as machine instructions) of other running programs, and can also be used to store data of users and application programs, etc.

[0194] The non-volatile memory can also store executable programs and data of users and application programs, etc., and can be pre-loaded into the random access memory for direct reading and writing by the processor 110.

[0195] The external memory interface 120 can be used to connect to an external non-volatile memory to expand the storage capacity of the electronic device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to implement the data storage function. For example, files such as music and videos are saved in the external non-volatile memory.

[0196] The internal memory 121 or the external memory interface 120 is used to store one or more computer programs. One or more computer programs are configured to be executed by the processor 110. One or more computer programs include a plurality of instructions, and when the plurality of instructions are executed by the processor 110, the screen display detection method executed on the electronic device 100 in the above embodiments can be implemented to realize the screen display detection function of the electronic device 100.

[0197] The electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor, etc. Such as music playback, recording, etc.

[0198] The audio module 170 is used to convert digital audio information into an analog audio signal for output, and is also used to convert an analog audio input into a digital audio signal. The audio module 170 can also be used for encoding and decoding audio signals. In an embodiment of the present application, the audio module 170 can be disposed in the processor 110, or some functional modules of the audio module 170 can be disposed in the processor 110.

[0199] The speaker 170A, also known as the "loudspeaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or a hands-free call through the speaker 170A.

[0200] The receiver 170B, also known as the "earpiece", is used to convert an audio electrical signal into a sound signal. When the electronic device 100 answers a call or a voice message, the voice can be listened to by placing the receiver 170B close to the human ear.

[0201] The microphone 170C, also known as the "microphone" or "transmitter", is used to convert a sound signal into an electrical signal. When making a call or sending a voice message, the user can speak by bringing the mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In some other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also implement a noise reduction function. In some other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to implement functions such as collecting sound signals, noise reduction, identifying the sound source, and implementing a directional recording function.

[0202] The headphone jack 170D is used to connect a wired headphone. The headphone jack 170D can be a USB interface 130, or a 3.5 mm Open Mobile Terminal Platform (OMTP) standard interface, or a Cellular Telecommunications Industry Association of the USA (CTIA) standard interface.

[0203] The keys 190 include a power-on key, volume keys, etc. The keys 190 can be mechanical keys or touch keys. The electronic device 100 can receive key inputs to generate key signal inputs related to the user settings and function controls of the electronic device 100.

[0204] The motor 191 can generate vibration prompts. The motor 191 can be used for incoming call vibration prompts and also for touch vibration feedback. For example, touch operations applied to different applications (such as taking pictures, playing audio, etc.) can correspond to different vibration feedback effects. For touch operations applied to different areas of the display screen 194, the motor 191 can also correspond to different vibration feedback effects. Different application scenarios (such as time reminder, receiving messages, alarm clock, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0205] The indicator 192 can be an indicator light and can be used to indicate the charging state, power change, and can also be used to indicate messages, missed calls, notifications, etc.

[0206] The SIM card interface 195 is used to connect the SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation from the electronic device 100. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communication. In an embodiment of the present application, the electronic device 100 uses an eSIM, that is, an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100. The embodiment of the present application also provides a computer storage medium, in which computer instructions are stored. When the computer instructions run on the electronic device 100, the electronic device 100 is enabled to execute the above-related method steps to implement the image processing method in the above embodiment.

[0207] The embodiment of the present application also provides a computer program product. When the computer program product runs on a computer, the computer is enabled to execute the above-related steps to implement the image processing method in the above embodiment.

[0208] In addition, the embodiment of the present application also provides a device, which can specifically be a chip, a component or a module. The device can include a processor and a memory connected to each other; wherein, the memory is used to store computer execution instructions. When the device runs, the processor can execute the computer execution instructions stored in the memory to enable the chip to execute the image processing method in each of the above method embodiments.

[0209] Among them, the electronic device, computer storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be elaborated here.

[0210] Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and simplicity of description, only the above division of each functional module is used as an example. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0211] In several embodiments provided in the present application, it should be understood that the disclosed device and method can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in electrical, mechanical or other forms.

[0212] The unit described as a separated component may or may not be physically separated. The component displayed as a unit may be a physical unit or multiple physical units, that is, it can be located in one place, or it can be distributed to multiple different places. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0213] In addition, each functional unit in each embodiment of the present application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0214] When an integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The software product is stored in a storage medium and includes several instructions for causing a device (which can be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs that can store program codes.

[0215] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and not to limit them. Although this application has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of this application can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of this application.

Claims

1. An image processing method, applied to an electronic device, characterized in that, The method includes: Obtain a captured preview image and detect whether the preview image is overexposed; If the preview image is overexposed, determine the non-uniformity degree of the overexposed area in the preview image; Adjust the exposure parameter of the preview image according to the non-uniformity degree; Generate an underexposed image according to the adjusted preview image; Fuse the adjusted preview image and the underexposed image to generate an HDR image corresponding to the preview image.

2. The image processing method according to claim 1, wherein The detection of whether the preview image is overexposed includes: Convert the pixel value of each pixel point in the preview image into a grayscale value; Convert the grayscale value of each pixel point in the preview image into a normalized grayscale value; If the normalized grayscale value of the pixel point is greater than or equal to a preset overexposure threshold, determine that the pixel point is an overexposed pixel point; Count the number of overexposed pixel points in the preview image. If the ratio between the number of overexposed pixel points and the total number of pixel points in the preview image is greater than or equal to a preset threshold ratio, determine that the preview image is overexposed.

3. The image processing method according to claim 2, wherein The conversion of the pixel value of each pixel point in the preview image into a grayscale value includes: Perform weighted summation on the R component, G component, and B component of the RGB pixel value of each pixel point in the preview image to obtain the grayscale value of each pixel point.

4. The image processing method according to claim 2, wherein The conversion of the grayscale value of each pixel point in the preview image into a normalized grayscale value includes: Calculate the normalized grayscale value of each pixel point according to the grayscale value of each pixel point in the preview image, the maximum grayscale value and the minimum grayscale value of all pixel points.

5. The image processing method according to claim 1, wherein The determination of the non-uniformity degree of the overexposed area in the preview image includes: Convert the color space of the preview image from the RGB color space to the YUV color space; Use a preset gradient algorithm to calculate the gradients of the Y component, U component, and V component of the YUV pixel value of each overexposed pixel point in the overexposed area respectively; Perform weighted summation on the gradients of the Y component, U component, and V component of the YUV pixel value of each overexposed pixel point to obtain the gradient of each overexposed pixel point; Calculate the mean value of the gradients of all overexposed pixel points to obtain the gradient of the overexposed area, and use the gradient of the overexposed area as the non-uniformity degree of the overexposed area.

6. The image processing method according to claim 5, wherein, The conversion of the color space of the preview image from the RGB color space to the YUV color space includes: Convert the RGB pixel value of each pixel point in the preview image into a YUV pixel value.

7. The image processing method according to claim 1, wherein The adjustment of the exposure parameter of the preview image according to the non-uniformity degree includes: According to the non-uniformity degree of the overexposed area and the current exposure value of the preview image, use a preset inverse proportional function to obtain the target exposure value of the preview image; Adjust the current exposure value of the preview image according to the target exposure value.

8. The image processing method according to claim 1, wherein The adjustment of the exposure parameter of the preview image according to the non-uniformity degree includes: Based on the correspondence between the non-uniformity degree, the current exposure value and the target exposure value of the preview image, determine the target exposure value of the preview image according to the current exposure value of the preview image and the non-uniformity degree of the overexposed area; Adjust the current exposure value of the preview image according to the target exposure value.

9. The image processing method according to claim 1, wherein The generating an underexposed image according to the adjusted preview image includes: Obtain the dynamic range value and the exposure value of the preview image; Determine the exposure value of the underexposed image according to the correspondence between the dynamic range, the exposure value of the preview image and the exposure value of the underexposed image, and the dynamic range value and the exposure value of the preview image; Adjust the exposure value of the adjusted preview image to the exposure value of the underexposed image to obtain the underexposed image.

10. The image processing method according to claim 1, wherein, The fusing the adjusted preview image and the underexposed image to generate the HDR image corresponding to the preview image includes: Input the adjusted preview image and the underexposed image into an image fusion model, and output the HDR image corresponding to the preview image through the image fusion model.

11. The image processing method according to claim 10, wherein The image fusion model is a Unet model, and the image fusion model includes an input layer, an encoder, a fuser, a decoder and an output layer.

12. The image processing method according to claim 11, wherein, The inputting the adjusted preview image and the underexposed image into an image fusion model, and outputting the HDR image corresponding to the preview image through the image fusion model includes: The input layer receives the input preview image and the underexposed image, and transfers the preview image and the underexposed image to the encoder; The encoder performs downsampling processing on the preview image and the underexposed image, respectively extracts the first image feature of the preview image and the second image feature of the underexposed image, and transfers the first image feature and the second image feature to the fuser; The fuser fuses the first image feature and the second image feature to obtain a third image feature, and transfers the third image feature to the decoder; The decoder performs upsampling processing on the third image feature to obtain a reconstructed image, and transfers the reconstructed image to the output layer; The output layer performs tone mapping processing on the reconstructed image by using a tone mapping function to obtain the HDR image corresponding to the preview image, and outputs the HDR image.

13. The image processing method according to claim 10, wherein The method further includes: Create training samples; Train a preset deep learning model through the training samples to obtain the image fusion model.

14. The image processing method according to claim 13, wherein The creating training samples includes: Control the camera of the electronic device to capture a plurality of preview images, and use one preview image, the corresponding underexposed image and the corresponding theoretical HDR image as a set of training data, create multiple sets of training data to obtain the training samples.

15. The image processing method according to claim 14, wherein The preview images in the training samples include preview images with overexposed areas accounting for a first preset percentage and preview images without overexposed areas accounting for a second preset percentage, and the first preset percentage is less than the second preset percentage.

16. The image processing method according to claim 13, wherein The preset deep learning model is a Unet model. Training the preset deep learning model with the training samples to obtain the image fusion model includes: Import the Unet model into the deep learning framework, initialize the Unet model, and set the initial parameters of the Unet model; Input a set of training data in the training samples into the Unet model, and output a predicted HDR image corresponding to the preview image in the training data through the Unet model; Calculate the output value of the loss function of the Unet model according to the predicted HDR image and the theoretical HDR image corresponding to the preview image in the training data; If the output value of the loss function is greater than the preset value, adjust the parameters of the Unet model, input another set of training data into the Unet model, and continue to train the Unet model; or If the output value of the loss function is less than or equal to the preset value, determine that the training of the Unet model is completed to obtain the image fusion model.

17. The image processing method according to claim 1, wherein The method further includes: If there is no overexposure in the preview image, generate an underexposed image according to the preview image.

18. An electronic device, characterized in that, The electronic device includes a memory and a processor: Wherein, the memory is used to store program instructions; The processor is used to read and execute the program instructions stored in the memory. When the program instructions are executed by the processor, the electronic device executes the image processing method according to any one of claims 1 to 17.

19. A chip, coupled to a memory in an electronic device, characterized in that, The chip is used to control the electronic device to execute the image processing method according to any one of claims 1 to 17.

20. A computer storage medium, characterized in that, The computer storage medium stores program instructions. When the program instructions run on the electronic device, the processor of the electronic device executes the image processing method according to any one of claims 1 to 17.

Images