Image processing method, model training method and related device

Abstract

Embodiments of the present application provide an image processing method, a model training method and related equipment, the method comprising: in response to an opening operation applied to a camera, acquiring a preview image; in response to a shooting operation, determining a first image from the preview image, and acquiring a dynamic range compression (DRC) image corresponding to the first image as a second image; performing pyramid decomposition on the first image through a preset image processing model to obtain a first low-frequency image, and performing pyramid decomposition on the second image to obtain a second low-frequency image and a plurality of high-frequency images; performing pyramid decomposition on a mask image corresponding to the first image to obtain a third low-frequency image; fusing the first low-frequency image, the second low-frequency image and the third low-frequency image to obtain a first fused image; and obtaining a target DRC image based on an image reconstruction algorithm and according to the first fused image and the plurality of high-frequency images. Through the preset image processing model, the DRC image can be optimized using the preview image, and the brightness difference between the preview image and the target DRC image is reduced.

Description

Technical Field

[0001] This application relates to the field of terminal equipment, and in particular to an image processing method, a model training method, and related equipment. Background Technology

[0002] With the widespread adoption of mobile devices (such as smartphones), the performance of camera applications within these devices has also improved. For example, mobile devices have increasingly more built-in algorithm modules and parameters that can be used to improve the final image quality of photos taken by camera applications. When a user takes a photo using a mobile device, the device provides a preview image, allowing the user to adjust the shooting angle, parameters, and mode. Typically, the user can trigger the camera application to take a photo based on the preview image's effect, thus obtaining a displayed image with the same effect as the preview.

[0003] However, in some shooting scenarios with a large range of brightness, after the camera application is triggered to take a picture, the image captured by the camera will be processed by various algorithm modules. This may result in a large difference in brightness between the processed image and the unprocessed image. As a result, the displayed image generated based on the processed image will have a large brightness difference from the preview image, which will affect the user's visual experience. Summary of the Invention

[0004] In view of the above, it is necessary to provide an image processing method, a model training method, and related equipment that can solve the technical problem of a large brightness difference between the displayed image and the preview image.

[0005] In a first aspect, embodiments of this application provide an image processing method applied to an electronic device. The method includes: acquiring a preview image in response to an operation to open a camera application; determining a first image from the preview image in response to a shooting operation, and acquiring a dynamic range compressed (DRC) image corresponding to the first image as a second image; performing pyramid decomposition on the first image using a preset image processing model to obtain a first low-frequency image, performing pyramid decomposition on the second image to obtain a second low-frequency image and multiple high-frequency images; performing pyramid decomposition on a mask image corresponding to the first image to obtain a third low-frequency image; fusing the first low-frequency image, the second low-frequency image, and the third low-frequency image to obtain a first fused image; and obtaining a target DRC image based on an image reconstruction algorithm, according to the first fused image and the multiple high-frequency images.

[0006] The above technical solution allows for the input of a first image and a second image into a preset image processing model. The model then generates a first fused image, composed of a first low-frequency image, a second low-frequency image, and a third low-frequency image. This first fused image contains significant brightness information. Pyramid reconstruction is then performed using the first fused image and multiple high-frequency images to obtain the target DRC image, thereby reducing the brightness difference between the target DRC image and the preview image.

[0007] In one possible implementation, the step of obtaining the target DRC image based on the image reconstruction algorithm according to the first fused image and the multiple high-frequency images includes: converting the attributes of the first fused image according to the style feature map corresponding to the first fused image to obtain a second fused image; using the second fused image to perform high-frequency reconstruction on the multiple high-frequency images to obtain multiple reconstructed images; and performing pyramid reconstruction based on the second fused image and the multiple reconstructed images to obtain the target DRC image.

[0008] The above technical solution enhances the brightness of the first fused image by utilizing attribute transformation. High-frequency reconstruction based on the obtained second fused image accelerates the processing efficiency of the image processing model.

[0009] Secondly, embodiments of this application provide a model training method applied to electronic devices, which trains a Laplacian pyramid fusion network using a sample dataset, wherein the sample dataset includes a first training image, a second training image, and a desired image, the first training image being a preview image obtained using a camera application, and the second training image being a DRC image corresponding to the preview image; the trained Laplacian pyramid fusion network is used as an image processing model.

[0010] The above technical solution uses a first training image and a second training image as inputs to a Laplacian pyramid fusion network. This allows the network to continuously learn and adjust based on the first and second training images, resulting in a trained Laplacian pyramid fusion network. The first training image contains more low-frequency structural features, while the second training image contains more high-frequency detail features. Combining the first and second training images during the learning process effectively improves the image processing performance of the obtained image processing model. Furthermore, using the desired image as a sample label for supervised training enhances the generalization ability of the image processing model. Applying this image processing model to electronic devices can improve the brightness distribution of images captured in shooting scenarios (e.g., HDR scenarios), thus enhancing the visual experience.

[0011] In one possible implementation, training the Laplacian pyramid fusion network using a sample dataset includes: inputting the first training image and the second training image into the Laplacian pyramid fusion network for training to obtain a predicted image; and using the Laplacian pyramid fusion network that meets preset conditions as the image processing model.

[0012] Through the above technical solution, the Laplacian pyramid fusion network can continuously learn and adjust the network based on the first training image and the second training image before the preset conditions are met.

[0013] In one possible implementation, using the Laplacian pyramid fusion network that meets preset conditions as the image processing model includes: calculating the loss value of the Laplacian pyramid fusion network based on the first training image, the second training image, the predicted image, and the expected image; if the loss value is less than or equal to a preset loss threshold, determining that the preset conditions are met, completing the training of the Laplacian pyramid fusion network, and obtaining the image processing model.

[0014] The above technical solution is used to train based on the loss value, thereby reducing the brightness difference between the predicted image and the expected image, so as to obtain an image processing model when the preset loss threshold is met.

[0015] In one possible implementation, the step of inputting the first training image and the second training image into the Laplacian pyramid fusion network for training includes: obtaining a mask image based on the first training image; performing pyramid decomposition on the first training image, the second training image, and the mask image respectively to obtain a first low-frequency image corresponding to the first training image, a second low-frequency image corresponding to the second training image, multiple high-frequency images, and a third low-frequency image corresponding to the mask image; and fusing the first low-frequency image, the second low-frequency image, and the third low-frequency image to obtain a first fused image.

[0016] Through the above technical solution, a first fused image is obtained by fusing the first low-frequency image, the second low-frequency image, and the third low-frequency image, so that the Laplacian pyramid network can focus on learning the brightness information of the first fused image.

[0017] In one possible implementation, the step of inputting the first training image and the second training image into the Laplacian pyramid fusion network for training further includes: converting the attributes of the first fusion image according to the style feature map corresponding to the first fusion image to obtain a second fusion image; using the second fusion image to perform high-frequency reconstruction on the multiple high-frequency images to obtain multiple reconstructed images; and performing pyramid reconstruction on the second fusion image and the multiple reconstructed images to obtain the predicted image.

[0018] By employing the aforementioned technical solution, attribute transformation is performed on the first fused image to obtain the second fused image. This preserves the image details of the first fused image while adaptively adjusting image brightness and controlling contrast. Introducing a high-frequency reconstruction step accelerates the convergence speed of network training. Pyramid reconstruction is performed based on the second fused image and multiple reconstructed images to obtain a predicted image. This predicted image retains the detailed features of the second training image and also possesses superior brightness performance. Therefore, through training the aforementioned modules / network layers, network training efficiency can be ensured while optimizing image brightness.

[0019] In one possible implementation, the step of using the second fused image to perform high-frequency reconstruction on the multiple high-frequency images to obtain multiple high-frequency reconstructed images includes: upsampling the second fused image to obtain an upsampled second fused image, the resolution of the upsampled second fused image being a first resolution; performing high-frequency reconstruction on the upsampled second fused image and the first high-frequency image to obtain a first reconstructed image, the first high-frequency image being one of the multiple high-frequency images, and the resolution of the first high-frequency image being the first resolution; upsampling the first reconstructed image to obtain an upsampled first reconstructed image, the resolution of the upsampled first reconstructed image being a second resolution. The second resolution is greater than the first resolution; the upsampled first reconstructed image and the second high-frequency image are subjected to high-frequency reconstruction to obtain a second reconstructed image, the second high-frequency image being one of the plurality of high-frequency images, and the resolution of the second high-frequency image being the second resolution; the second reconstructed image is upsampled to obtain an upsampled second reconstructed image, the resolution of the upsampled second reconstructed image being a third resolution, the third resolution being greater than the second resolution; the upsampled second reconstructed image and the third high-frequency image are subjected to high-frequency reconstruction to obtain a third reconstructed image; the first reconstructed image, the second reconstructed image, and the third reconstructed image are used as the plurality of high-frequency reconstructed images.

[0020] The above technical solution utilizes high-frequency reconstruction to accelerate network convergence.

[0021] In one possible implementation, the step of performing pyramid reconstruction on the second fused image and the multiple reconstructed images to obtain the predicted image includes: obtaining a first-layer reconstructed image based on the sum of pixel values ​​corresponding to pixels at the same position in the upsampled second fused image and the first reconstructed image; obtaining a second-layer reconstructed image based on the sum of pixel values ​​corresponding to pixels at the same position in the upsampled first-layer reconstructed image and the second reconstructed image; and obtaining the predicted image based on the sum of pixel values ​​corresponding to pixels at the same position in the upsampled second-layer reconstructed image and the third reconstructed image.

[0022] By using the above technical solutions to generate predicted images through pyramid reconstruction, image artifacts can be avoided to a certain extent.

[0023] In one possible implementation, obtaining a mask image based on the first training image includes: determining a first brightness value and a second brightness value based on the brightness values ​​of pixels in the first training image, wherein the first brightness value is less than the second brightness value; determining a target pixel in the first training image based on the first brightness value and the second brightness value, wherein the brightness value of the target pixel is greater than the first brightness value and less than the second brightness value; and generating the mask image based on the target pixel.

[0024] By introducing a mask image, the learning effect of brightness information can be improved through the above technical solution.

[0025] Thirdly, embodiments of this application provide an electronic device, the electronic device including a memory and a processor: wherein the memory is used to store program instructions; the processor is used to read and execute the program instructions stored in the memory, and when the program instructions are executed by the processor, the electronic device performs the above-described image processing method or the above-described model training method.

[0026] Fourthly, embodiments of this application provide a chip system applied to an electronic device. The chip system includes one or more processors, which are used to invoke computer instructions to cause the electronic device to input an image to be processed into the chip system and execute the image processing method described above, or execute the model training method described above.

[0027] Fifthly, embodiments of this application provide a computer storage medium storing program instructions that, when executed on an electronic device, cause the processor of the electronic device to perform the aforementioned image processing method or the aforementioned model training method.

[0028] Furthermore, the technical effects brought about by the third to fifth aspects can be found in the descriptions of the methods in the above-mentioned method section, and will not be repeated here. Attached Figure Description

[0029] Figure 1 This is a flowchart of a model training method provided in an embodiment of this application.

[0030] Figure 2 This is a schematic diagram of the structure of a Laplace pyramid fusion network provided in an embodiment of this application.

[0031] Figure 3 This is a flowchart illustrating the generation process of a predicted image according to an embodiment of this application.

[0032] Figure 4 This is a schematic diagram illustrating the generation of a mask image provided in an embodiment of this application.

[0033] Figure 5 This is a schematic diagram of the Laplace pyramid decomposition module and fusion layer provided in an embodiment of this application.

[0034] Figure 6 This is a schematic diagram of the structure of a style transfer module provided in an embodiment of this application.

[0035] Figure 7 This is a reconstruction schematic diagram of a high-frequency reconstruction module provided in an embodiment of this application.

[0036] Figure 8 This is another schematic diagram of the structure of the Laplace pyramid fusion network provided in one embodiment of this application.

[0037] Figure 9 This is a schematic diagram of brightness comparison of a first training image, a second training image, and a predicted image provided in an embodiment of this application.

[0038] Figure 10 This is a flowchart of an image processing method provided in an embodiment of this application.

[0039] Figure 11 This is a software structure diagram of an electronic device provided in an embodiment of this application.

[0040] Figure 12 This is a software workflow diagram of an electronic device provided in an embodiment of this application.

[0041] Figure 13 This is a hardware architecture diagram of an electronic device provided in an embodiment of this application.

[0042] Figure 14 This is a schematic diagram of the structure of a chip system provided in an embodiment of this application. Detailed Implementation

[0043] In one embodiment of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more features. In the description of embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in one embodiment of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this application's specification is for the purpose of describing particular embodiments only and is not intended to limit the application. It should be understood that, unless otherwise stated, " / " in this application means "or". For example, A / B can mean A or B. "And / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. "At least one" refers to one or more. "More than one" refers to two or more. For example, at least one of a, b, or c can represent: a, b, c, a and b, a and c, b and c, and a, b, and c. Where there is no conflict, the following embodiments and features described herein can be combined with each other.

[0045] The embodiments of this application can be applied to scenarios where electronic devices with camera functions take photos.

[0046] Optionally, the electronic device can be a mobile terminal such as a mobile phone, tablet computer, wearable device, in-vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), or it can be a professional shooting device such as a digital camera, SLR camera / mirrorless camera, action camera, gimbal camera, or drone. The specific type of electronic device is not limited in the embodiments of this application.

[0047] Taking a mobile phone as an example, a typical mobile phone camera module (or camera assembly) includes a lens and a sensor. When taking a picture, the lens of the camera module captures the light signal corresponding to the subject in the scene. The sensor converts this light signal into an electrical signal, then performs analog-to-digital (A / D) conversion on the electrical signal, outputting the corresponding digital signal to the intermediate processing module. The digital signal output by the sensor to the intermediate processing module is the raw image captured by the camera module, also known as a RAW image or digital negative. The intermediate processing module performs a series of processes on the received RAW image to obtain an image that can be displayed, such as a Joint Photographic Experts Group (JPEG) image. This JPEG image can be transmitted to the mobile phone's display screen for display, and / or transmitted to the mobile phone's memory for storage.

[0048] The intermediate processing module processes the RAW image to generate a JPEG image. This process includes: performing image signal processing (ISP) on the RAW image to convert it from the RAW domain to the YUV domain (the YUV image is called a YUV image); then, performing post-processing on the YUV image; finally, encoding the post-processed YUV image using JPEG encoding to obtain the JPEG image. YUV is a color coding system used for video compression and transmission, where Y represents luminance information, and U and V represent chrominance information.

[0049] ISP processing may include: Bad Pixel Correction (BPC), RAW domain noise reduction, Black Level Correction (BLC), Lens Shading Correction (LSC), Auto White Balance (AWB), Demosica color interpolation, Color Correction Matrix (CCM), Dynamic Range Compression (DRC), Gamma, Look-Up Table (LUT), YUV domain noise reduction, Sharpen, and Detail Enhancement. Post-processing algorithms may include: multi-frame registration, fusion, and noise reduction for High Dynamic Range (HDR) images, as well as one or more algorithms combined to improve sharpness, such as Super Resolution (SR), skin smoothing, distortion correction, and bokeh.

[0050] Because the brightness range (also known as dynamic range) of RAW images captured by camera modules often exceeds the display capabilities of electronic device screens. The dynamic range of an image refers to the grayscale ratio between the brightest and darkest parts of the image. In other words, the dynamic range of an image is a measure of the brightness levels that can be presented in an image. Generally, the higher the dynamic range of an image, the richer the details in the image, and the closer it is to the real environment. In some embodiments, if the original image is displayed directly on the screen, the bright areas may be too bright and the dark areas too dark. Therefore, in the aforementioned ISP processing, DRC can be used to adjust and optimize the dynamic range of the RAW image, ensuring that the brightness range of the image after DRC adjustment can adapt to the brightness range of a low dynamic range (LDR) display, not exceeding the display capabilities of the screen, presenting appropriate brightness variations on the screen, while retaining more image details.

[0051] However, in some shooting scenarios with a wide range of brightness (also known as HDR scenarios), the RAW images captured by the camera module have a high dynamic range. During ISP processing, after remapping the image's brightness levels using DRC, there may be a significant difference in brightness between the image before and after DRC processing. In cases of significant brightness differences, the final displayed image in the user's camera library will also differ considerably in brightness from the preview image.

[0052] Based on this, embodiments of this application propose an image processing method, a model training method, and related equipment. A Laplacian pyramid fusion network is trained using the DRC image generated after DRC processing and the preview image to obtain an image processing model for optimizing the DRC image. Applying this image processing model to electronic devices can avoid significant brightness differences between the image before and after DRC processing, ensuring that the brightness level of the displayed image is close to that of the preview image, thus improving the user's visual experience.

[0053] The embodiments of this application are described in detail below.

[0054] See Figure 1 The diagram shown is a flowchart of a model training method provided in an embodiment of this application. This model training method is applied to electronic devices and specifically includes the following process. Depending on different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0055] S101 uses a sample dataset to train a Laplacian pyramid fusion network.

[0056] In some embodiments of this application, the sample dataset may be an image dataset, which includes multiple sets of training images, each set of training images including a first training image, a second training image, and a desired image.

[0057] The first training image can be a preview image. The electronic device can launch and run the camera application in response to a user's click or touch operation on the camera application icon. In other embodiments, the camera application can also be invoked through other applications, such as a sharing application or an instant messaging application, to obtain captured images or videos. After launching the camera application, the electronic device displays a photo preview interface on its screen, showing the image of the current shooting scene captured by the camera, as the initial preview image. To enhance the brightness of the initial preview image, it is tone-mapped using multiple pre-defined tone mapping regions to obtain a preview image suitable for model training.

[0058] The second training image can be a DRC image. The camera application's preview interface also includes a shooting control. In other embodiments, the function of the shooting control can also be implemented through other physical buttons on the electronic device; this application does not limit this. The electronic device responds to a shooting operation triggered by a user's click or touch operation on the shooting control by acquiring a RAW image through the electronic device's camera. Then, the electronic device's ISP processes the RAW image. During ISP processing, the image generated after DRC processing can be called a DRC image. In this embodiment, the preview image and the DRC image correspond to the same shooting scene.

[0059] To clarify the training objective and improve the network's generalization ability, a desired image is introduced during the training process. In some embodiments of this application, a DRC image with added brightness information is used as the desired image. The desired image is a sample label generated based on the preview image and the DRC image. In embodiments of this application, brightness information from the preview image is added to the DRC image, making the brightness of the DRC image with added brightness information similar to that of the preview image.

[0060] In some embodiments of this application, a Laplacian pyramid fusion network is trained using a sample dataset, such as... Figure 2 As shown, the network structure of the Laplacian pyramid fusion network can include, but is not limited to: an input layer, a data processing module, a Laplacian pyramid decomposition module, a fusion layer, a style transfer module, a high-frequency reconstruction module, a pyramid reconstruction module, and an output layer. The data processing module preprocesses the input sample dataset. The Laplacian pyramid decomposition module decomposes the output of the data processing module. The fusion layer performs channel fusion on a portion of the output of the Laplacian pyramid decomposition module (e.g., the low-frequency portion). The style transfer module performs style transfer on the output of the fusion layer. The high-frequency reconstruction module performs high-frequency reconstruction on another portion of the output of the Laplacian pyramid decomposition module (e.g., the high-frequency portion). The pyramid reconstruction module reconstructs the outputs of the high-frequency reconstruction module and the style transfer module.

[0061] In some embodiments of this application, the process of training a Laplacian pyramid fusion network using a sample dataset may include at least four stages: forward propagation, loss calculation, backpropagation, and model optimization.

[0062] In the forward propagation phase, the sample dataset sequentially passes through the input layer, data processing module, Laplacian pyramid decomposition module, fusion layer, style transfer module, high-frequency reconstruction module, pyramid reconstruction module, and output layer to obtain the predicted image output by the Laplacian pyramid fusion network. The specific process for generating the predicted image can be found below. Figure 3 The description.

[0063] During the loss calculation phase, the loss value of the Laplacian pyramid fusion network is calculated based on the first training image, the second training image, the predicted image, and the desired image. This loss value reflects the accuracy of the current network's prediction; that is, the smaller the loss value, the higher the prediction accuracy; and the larger the loss value, the lower the prediction accuracy. The loss function can include a combination of one or more of the following: a fusion loss function, a contrast loss function, and a gradient descent loss function. Specifically, the fusion loss function and the contrast loss function can be calculated based on the predicted image and the desired image. Alternatively, the gradient descent loss function can be calculated based on either the first training image and the predicted image, or it can be calculated based on the second training image and the predicted image.

[0064] Specifically, the calculation process of the fusion loss function, contrast loss function, and gradient descent loss function is as follows:

[0065] Fusion loss function L blending Expressed as a formula:

[0066]

[0067] In the formula, N represents the number of samples, which is the number of training image sets, where the first training image, the second training image, and the desired image form a set. M represents the feature matrix of the mask image of the first training image, used to filter pixel values ​​to control the fusion region during training. The generation of the mask image can refer to the following... Figure 3 As shown. Y label_avg Y represents the expected value of the desired image after average pooling. out_avg This represents the predicted values ​​of the image after average pooling. Σ represents the summation over all samples.

[0068] Contrast loss function L cs Referring to the MS-SSIM loss function, the contrast loss across M levels is calculated. The formula is as follows:

[0069]

[0070] In the formula, j represents the pyramid level of the Laplace pyramid fusion network, with a total of M levels. C j (X,Y) represents the contrast factor, where X represents the desired image, Y represents the predicted image, and β represents the parameters at each scale, which are empirical values.

[0071] Gradient descent loss function L deshoot Expressed as a formula:

[0072] L desho =∑(ΔI) h *ΔOh )+(ΔI v *ΔO v );

[0073] In the formula, ΔI h ΔO represents the horizontal difference in pixel values ​​between the first and second training images. h ΔI represents the difference in pixel values ​​of the predicted image in the horizontal direction. v ΔO represents the vertical difference in pixel values ​​between the first and second training images. v This represents the difference in pixel values ​​of the predicted image in the vertical direction.

[0074] In some embodiments of this application, the fusion loss function, contrast loss function, and gradient descent loss function are weighted and summed to obtain the loss function of the Laplacian pyramid fusion network, thereby allowing the calculation of the loss value of this loss function. If the loss value is greater than a preset loss threshold, the backpropagation phase continues.

[0075] In the backpropagation phase, the direction of backpropagation is opposite to that of forward propagation. That is, starting from the output layer, backpropagation is performed layer by layer back to the input layer to calculate the gradient of each parameter with respect to the loss value.

[0076] In the model optimization phase, the network weights and biases are updated based on gradients using a pre-defined optimization algorithm (such as gradient descent), and the learning rate is adaptively adjusted, or the weights of the fusion loss function, contrast loss function, and gradient descent loss function are adjusted to accelerate convergence and improve network performance. Through optimization in the model optimization phase, a smaller loss value is obtained in the next forward propagation phase. Then, the above four phases are iterated until preset conditions are met. These preset conditions may include the loss value of the loss function being less than or equal to a preset loss threshold, or reaching a preset number of training iterations; this application does not impose any restrictions on these conditions.

[0077] In some embodiments of this application, the first training image is a preview image, which contains more brightness information. The second training image is a DRC image generated after dynamic range compression (DRC), which contains more image detail information (such as texture). Therefore, by training the Laplacian pyramid fusion network using the first and second training images, the resulting predicted image can fuse the brightness information of the preview image and the image detail information of the DRC image. During the training of the Laplacian pyramid fusion network using the first and second training images, the Laplacian pyramid fusion network can learn how to combine the first and second training images to reduce the brightness difference between the predicted image and the desired image. Furthermore, the Laplacian pyramid fusion network can effectively capture detail information at different resolutions and consider this detail information during the fusion process, thereby reducing artifacts generated during reconstruction and improving the quality of the predicted image.

[0078] S102 uses the trained Laplacian pyramid fusion network as an image processing model.

[0079] In some embodiments of this application, during training, the network parameters of the Laplacian pyramid fusion network need to be adjusted according to the loss value of the loss function. These network parameters may include, but are not limited to, one or more combinations of: learning rate, number of iterations, number and size of convolutional kernels, weights and biases of convolutional layers, batch size, optimizer, and regularization parameters.

[0080] In one example, if the loss value is greater than a preset loss threshold, fine-tuning is performed on the current network parameters of the Laplacian Pyramid Fusion Network (LPFF) to enable it to converge as quickly as possible, thus achieving better performance with less training resources and time. If the loss value is less than or equal to the preset loss threshold, the LPFF is considered to have converged, and the trained LPFF is used as an image processing model. Applying this image processing model to electronic devices can avoid significant brightness differences between the images before and after DRC processing, ensuring that the brightness level of the displayed image matches the brightness level of the preview image, thus improving the user's visual experience.

[0081] Through the above embodiments, supervised learning can be used to input the first and second training images into the Laplacian pyramid fusion network to generate a predicted image. The predicted image is then compared with the desired image, and further compared with the first and second training images to calculate the loss value between the predicted and desired images. If the loss value exceeds a preset loss threshold, the network parameters of the Laplacian pyramid fusion network are adjusted to optimize the network output. During training, the brightness difference between the predicted and desired images is gradually reduced, improving the accuracy and generalization ability of the Laplacian pyramid fusion network.

[0082] Figure 3 A flowchart illustrating the generation process of a predicted image according to an embodiment of this application. The process of generating the predicted image may include:

[0083] S301, Based on the first training image, obtain the mask image.

[0084] In some embodiments of this application, the mask image is an image generated by partially or completely occluding a preset image. In this embodiment, a mask image is used to partially occlude a first training image, thereby obtaining the region of interest (ROI) of the first training image. Since the first training image contains more brightness information from the actual shooting scene, in order for the Laplacian network to learn the brightness information of the RIO during training, a mask image can be used... Figure 2 The data processing module shown counts the brightness value of each pixel in the first training image and obtains the mask image based on the counted brightness value.

[0085] Specifically, the brightness value of each pixel in the first training image is statistically analyzed and sorted. Based on the sorting result, the maximum and median values ​​are determined. The brightness value corresponding to the median value is taken as the first brightness value, and the brightness value corresponding to the maximum value is taken as the second brightness value. The second brightness value is greater than the first brightness value. Therefore, based on the brightness value of each pixel in the first training image, pixels in the first training image that are greater than the first brightness value and less than the second brightness value are identified, and the mask image is determined based on these pixels. Figure 4 As shown, the first brightness value and the second brightness value are determined by the brightness values ​​of the pixels in the first training image. Pixels in the first training image whose pixel values ​​are greater than the first brightness value and less than the second brightness value are retained, while pixels whose pixel values ​​are less than or equal to the first brightness value and greater than or equal to the second brightness value are discarded, thus obtaining the mask image.

[0086] In one example, the brightness value of each pixel in the first training image is counted, and the maximum value is determined to be 240 nits, and the median value is 128 nits. Then the brightness value of the mask image is between 128 nits and 240 nits.

[0087] S302, pyramid decomposition is performed on the first training image, the second training image and the mask image respectively to obtain the first low-frequency image corresponding to the first training image, the second low-frequency image corresponding to the second training image and multiple high-frequency images, and the third low-frequency image corresponding to the mask image.

[0088] In some embodiments of this application, methods such as Figure 2 The Laplacian pyramid decomposition module shown performs pyramid decomposition. The Laplacian pyramid decomposition module is used to extract low-frequency structural feature maps and high-frequency detail feature maps from the first training image, the second training image, and the mask image.

[0089] The first training image is decomposed using a Laplacian pyramid decomposition module to obtain a first low-frequency map, which can be used to describe the low-frequency features of the first training image. Furthermore, during the pyramid decomposition of the first training image, a corresponding high-frequency map can also be extracted, which is used to describe the high-frequency features of the first training image.

[0090] The second training image is decomposed using the Laplacian pyramid decomposition module, resulting in a second low-frequency image and multiple high-frequency images. The second low-frequency image describes the low-frequency features of the second training image, while the different high-frequency images describe the high-frequency features of the first training image at different resolutions. Each high-frequency image corresponding to the second training image has a different resolution.

[0091] The mask image is decomposed using the Laplacian pyramid decomposition module to obtain a third low-frequency map, which is used to describe the low-frequency features of the mask image. Furthermore, during the pyramid decomposition process, a corresponding high-frequency map can also be extracted, which is used to describe the high-frequency features of the mask image.

[0092] S303, the first low-frequency image, the second low-frequency image and the third low-frequency image are fused to obtain the first fused image.

[0093] In some embodiments of this application, in order to enhance the feature representation of low-frequency features, methods such as Figure 2 The fusion layer shown performs channel fusion on the first, second, and third low-frequency maps. This channel fusion can be achieved by pixel stacking or channel stacking. The following section will combine... Figure 5 Describe, such as Figure 5The diagram shown is a schematic of a Laplacian pyramid decomposition module and a fusion layer provided in an embodiment of this application. The first training image, the second training image, and the mask image are input into the Laplacian pyramid decomposition module for decomposition, resulting in a first low-frequency image corresponding to the first training image, a second low-frequency image corresponding to the second training image, multiple high-frequency images, and a third low-frequency image corresponding to the mask image. Furthermore, the first training image and the mask image are also decomposed into multiple high-frequency images (…). Figure 5 (Not shown). The first low-frequency image, the second low-frequency image, and the third low-frequency image are fused using a fusion layer to generate a first fused image.

[0094] S304, transform the attributes of the first fusion map according to the style feature map corresponding to the first fusion map to obtain the second fusion map.

[0095] In some embodiments of this application, methods such as Figure 2 The style transfer module shown transforms the attributes of the first fused image. The style transfer module deploys a lightweight style transfer network, which is used to transfer style features from one image to another, thereby achieving image attribute transformation.

[0096] In some embodiments of this application, since the first fused image is obtained by fusing the first low-frequency image, the second low-frequency image, and the third low-frequency image, the first fused image contains more brightness information. To further enhance the representation of brightness information, a style transfer module can be added during the training process of the Laplacian pyramid fusion network to integrate the style feature map extracted from the first fused image into the first fused image, so that the second fused image generated by the style transfer module has better brightness performance. The following is combined with... Figure 6 Please provide an explanation. Figure 6 This is a schematic diagram of the structure of a style transfer module provided in an embodiment of this application.

[0097] like Figure 6 As shown, the style transfer module is a lightweight style transfer network, consisting of six network layers. The first network layer consists of a convolutional layer (CONV), a normalization layer (IN), and an activation function (RELU); the second network layer consists of a convolutional layer and an activation function; the third network layer consists of a residual block (RESBLOCK); the fourth network layer consists of a convolutional layer and an activation function; and the fifth and sixth network layers are convolutional layers.

[0098] The first fused image is input into the first network layer. In the first network layer, the first fused image is convolved with a convolution kernel, then normalized, and then nonlinearity is added using an activation function to obtain the basic feature map of the first fused image, such as an image containing edge, texture, and brightness information.

[0099] In the second network layer, features are further extracted and transformed from the basic feature map, and the extracted and transformed features are subjected to nonlinear transformation to obtain the output image of the second network layer. The output image of the second network layer has richer feature expression.

[0100] In the third network layer, residual blocks are used to extract multi-layer information, prevent gradient explosion, and avoid destroying the main features of the image. The output image of the second network layer is processed in the third network layer to improve the network's ability to learn style features without compromising the main structural and content information of the second network layer's output image.

[0101] In the fourth network layer, the output image of the third network layer is used as input and then passed through convolution and activation functions again in order to extract and transform the output image of the third network layer.

[0102] In the fifth network layer, more features are extracted from the output image of the fourth network layer using convolutional layers to obtain the output image of the fifth network layer. The output image of the fifth network layer is then used as a style feature map.

[0103] To achieve attribute transformation, the style feature map and the first fused map are merged to generate an attribute transformation map. During the fusion process, the pixel values ​​of pixels at the same position in the style feature map and the first fused map are added together to obtain the attribute transformation map. This pixel value addition operation enhances the visibility of style features in the attribute transformation map. This ensures that the attribute transformation map contains both the main content and structural information of the first fused map and the style feature maps obtained from the first to fifth network layers of the style transfer module.

[0104] In the sixth network layer, convolutional layers are used to further adjust the size, number of channels, or feature transformation of the feature map of the attribute transformation map to obtain the second fusion map.

[0105] In the embodiments of this application, multiple network layers in the style transfer module are combined and transformed. By extracting the style feature map of the first fused image and transforming the attributes of the first fused image using the style feature map, the second fused image not only possesses the detailed information of the first fused image but also exhibits superior brightness performance. During training, the ability of the style transfer module to learn style features can be optimized, the effect of attribute transformation can be improved, and the brightness performance of the second fused image can be enhanced.

[0106] S305, the second fusion image is used to perform high-frequency reconstruction on multiple high-frequency images to obtain multiple reconstructed images.

[0107] In some embodiments of this application, methods such as Figure 2The high-frequency reconstruction module shown performs high-frequency reconstruction on multiple high-frequency images. The module includes a connection module (CONCAT) and structural units composed of multiple convolutional layers (CONV) and residual blocks (RESBLOCK). The following section combines... Figure 7 To explain, Figure 7 This is a schematic diagram of a high-frequency reconstruction module provided in an embodiment of this application. Figure 7 The multiple high-frequency reconstruction modules shown can be the same high-frequency reconstruction module or multiple separate high-frequency reconstruction modules; this application does not impose any restrictions on this.

[0108] In some embodiments of this application, since the second fused map is the feature map corresponding to the top layer of the Laplacian pyramid, high-frequency reconstruction can be performed starting from the second fused map. For example... Figure 7 As shown, after the style transfer module transforms the attributes of the first fused image, it generates a second fused image. The second fused image is then upsampled to obtain an upsampled second fused image with a resolution equal to the first resolution. Upsampling is the process of increasing the image resolution. A first high-frequency image H1 with the first resolution is found from multiple high-frequency images. The upsampled second fused image and the first high-frequency image are fused using a connect block, and then processed sequentially through a convolutional layer, a residual block, and another convolutional layer to generate a first intermediate result. The product of the first intermediate result and the pixel values ​​of pixels at the same position in the first high-frequency image H1 is calculated to obtain the first reconstructed image H11.

[0109] The first reconstructed image H11 is upsampled to obtain an upsampled first reconstructed image. The resolution of the upsampled first reconstructed image is the second resolution, which is greater than the first resolution. A second high-frequency image H2 with the second resolution is found from multiple high-frequency images. The upsampled first reconstructed image and the second high-frequency image H2 are fused together using a connect block, and then processed sequentially through a convolutional layer, a residual block, and another convolutional layer to generate a second intermediate result. The product of the second intermediate result and the pixel values ​​of pixels at the same position in the second high-frequency image H2 is calculated to obtain the second reconstructed image H22.

[0110] The second reconstructed image H22 is upsampled to obtain the upsampled second reconstructed image. The resolution of the upsampled second reconstructed image is the third resolution, which is greater than the second resolution. A third high-frequency image H3 with the third resolution is found from multiple high-frequency images. The upsampled second reconstructed image and the third high-frequency image H3 are fused together using a connect block, and then processed sequentially through a convolutional layer, a residual block, and another convolutional layer to generate a third intermediate result. The product of the pixel values ​​of the third intermediate result and the pixels at the same position in the third high-frequency image H3 is calculated to obtain the third reconstructed image H33.

[0111] In some embodiments of this application, in order to ensure computational efficiency and fusion effect, a high-frequency image with a third resolution is used as the feature image corresponding to the bottom layer of the Laplacian pyramid, and the first reconstruction image, the second reconstruction image, and the third reconstruction image are used as multiple high-frequency reconstruction images.

[0112] Furthermore, the high-frequency image corresponding to the bottom layer can be determined based on a preset number of pyramid levels. In one example, if the preset number of pyramid levels has not been reached during the reconstruction process, high-frequency reconstruction of the high-frequency image with the fourth resolution can continue until the preset number of pyramid levels is reached, at which point the reconstruction of all high-frequency images is considered complete. This application does not restrict the conditions for stopping the pyramid reconstruction process.

[0113] By building a lightweight high-frequency reconstruction module, the convergence speed of the Laplacian pyramid fusion network during training can be accelerated, which can improve the training efficiency to a certain extent.

[0114] S306, perform pyramid reconstruction on the second fused image and multiple reconstructed images to obtain the predicted image.

[0115] In some embodiments of this application, multiple reconstructed images include a first reconstructed image, a second reconstructed image, and a third reconstructed image, which can be utilized using, for example... Figure 2 The pyramid reconstruction module shown performs pyramid reconstruction. It calculates the sum of pixel values ​​of pixels at the same position in the upsampled second fused image and the first reconstructed image to obtain the first layer reconstructed image, including: determining each pixel at the same position in the upsampled second fused image and the first reconstructed image; based on each same position, determining the first pixel value of the first pixel corresponding to that same position in the upsampled second fused image, and determining the second pixel value of the first pixel corresponding to that same position in the first reconstructed image; calculating the sum of the first pixel value and the second pixel value to obtain the first target pixel value of the first pixel at each same position; and determining the first layer reconstructed image based on the first target pixel value.

[0116] The second-layer reconstruction image is obtained by summing the pixel values ​​of pixels at the same position in the upsampled first and second reconstructed images. This includes: determining the pixels at the same position in the upsampled first and second reconstructed images; determining the third pixel value of the second pixel corresponding to that position in the upsampled first reconstructed image, and determining the fourth pixel value of the second pixel corresponding to that position in the second reconstructed image; calculating the sum of the third and fourth pixel values ​​to obtain the second target pixel value of the second pixel at each position; and determining the second-layer reconstruction image based on the second target pixel value.

[0117] The predicted image is obtained by summing the pixel values ​​of pixels at the same position in the upsampled second-layer reconstructed image and the third-layer reconstructed image. This includes: determining the pixels at the same position in the upsampled second-layer reconstructed image and the third-layer reconstructed image; determining the fifth pixel value of the third pixel corresponding to that position in the upsampled second-layer reconstructed image and the sixth pixel value of the third pixel corresponding to that position in the third-layer reconstructed image; calculating the sum of the fifth and sixth pixel values ​​to obtain the third target pixel value of the third pixel at each position; and determining the predicted image based on the third target pixel value.

[0118] Through the above embodiments, by introducing a mask image, the Laplacian pyramid fusion network focuses more on learning the region of interest corresponding to the mask image during training. Furthermore, the process of fusing the first, second, and third low-frequency images concentrates more on the first and second brightness values, improving the brightness performance of the first fused image. Further, a lightweight style transfer network is used to transform the attributes of the first fused image, which can adaptively adjust the image brightness while controlling contrast, while preserving the image details of the first fused image. In addition, to accelerate the convergence speed during network training, a high-frequency reconstruction module is introduced to obtain multiple reconstructed images. Pyramid reconstruction is performed based on these multiple reconstructed images and the second fused image to obtain a predicted image. This predicted image retains the detailed features of the second training image and also has better brightness performance than the second training image. Therefore, through training the above modules / network layers, the efficiency of network training can be ensured while optimizing image brightness.

[0119] In some embodiments of this application, Figure 8 This is another schematic diagram of the Laplace pyramid fusion network provided in one embodiment of this application. (See diagram below.) Figure 8 As shown, after acquiring the initial preview image, the electronic device performs tone mapping preprocessing on the initial preview image using multiple tone mapping curves to obtain the first training image (also called the preview image). The electronic device also acquires the DRC image generated after DRC processing as the second training image. Figure 8 As shown, the Laplace pyramid fusion network may include an input layer, a data processing module, a Laplace pyramid decomposition module, a fusion layer, a style transfer module, a high-frequency reconstruction module, a pyramid reconstruction module, and an output layer.

[0120] The input layer receives a first training image and a second training image. The data processing module performs pixel brightness value statistics on the first training image to obtain a mask image. The Laplacian pyramid decomposition module performs pyramid decomposition on the first training image, the second training image, and the mask image respectively to obtain a first low-frequency image corresponding to the first training image, a second low-frequency image corresponding to the second training image, multiple high-frequency images, and a third low-frequency image corresponding to the mask image.

[0121] The first fused image is obtained by fusing the first, second, and third low-frequency images through a fusion layer. Introducing the third low-frequency image during the fusion process enhances low-frequency structural information, such as brightness information.

[0122] By performing attribute transformation on the first fused image using the style transfer module, the brightness performance of the first fused image can be improved, thus obtaining the second fused image.

[0123] By using a high-frequency reconstruction module to perform high-frequency reconstruction on multiple high-frequency images and the second fused image, the convergence speed of the network during training can be accelerated, and multiple reconstructed images can be generated.

[0124] The pyramid reconstruction module performs pyramid reconstruction on multiple reconstructed images to generate a predicted image.

[0125] Since the embodiments of this application employ a supervised training method, a supervised network loss is introduced to ensure that the predicted image generated by the Laplacian pyramid fusion network has the same brightness performance as the desired image. The desired image is generated based on a first training image and a second training image, and includes brightness information from the first training image and high-frequency detail information from the second training image. Figure 8 As shown in A, it can be seen that the predicted image generated by the trained Laplacian pyramid fusion network has a brightness performance close to that of the first training image, and also has the local features of the second training image.

[0126] Furthermore, in order to better compare the brightness information among the first training image, the second training image, and the predicted image, embodiments of this application provide, as follows: Figure 9 The diagram shows a brightness comparison of the first training image, the second training image, and the predicted image. (See attached diagram.) Figure 9 As shown, Figure 9 Figure (a) is a preview image. Figure 9 Figure (b) is a DRC image, such as Figure 9 Figure (c) shows the predicted image generated when the Laplacian pyramid fusion network converges. Figure 9 The top left corner of images (a), (b), and (c) shows the brightness distribution, with the horizontal axis representing brightness and the vertical axis representing the proportion of pixels. Figure 9It can be seen that the Laplacian pyramid fusion network trained using the above embodiments can avoid a large brightness difference between the DRC image generated by DRC and the preview image.

[0127] Figure 10 This is a flowchart illustrating an image processing method according to an embodiment of this application. The image processing method is applied in an electronic device and specifically includes the following process. Depending on different requirements, the order of the steps in this flowchart can be changed, and some steps can be omitted.

[0128] S1001, in response to the opening of the camera application, acquires a preview image.

[0129] In some embodiments of this application, the opening operation can be triggered by a user clicking or touching the camera application's icon. In response to the opening operation, the electronic device launches and runs the camera application. In other embodiments, the camera application can also be invoked through other applications, such as a sharing application or an instant messaging application, to acquire captured images or videos. After launching the camera application, the electronic device displays a photo preview interface on its screen, showing the image of the current shooting scene captured by the electronic device's camera as a preview image.

[0130] S1002, in response to the shooting operation, determines the first image from the preview image and obtains the dynamic range compressed (DRC) image corresponding to the first image as the second image.

[0131] In some embodiments of this application, when a user determines to take a photo based on the photo preview interface, the electronic device responds to the user's shooting operation by capturing a RAW image through its camera. The electronic device then processes the RAW image using an ISP. During ISP processing, the image generated after DRC processing can be called a DRC image, i.e., the second image. Furthermore, the electronic device searches for the most recently captured preview image from the preview images captured after the triggering of the shooting operation, and uses this most recently captured preview image as the first image.

[0132] S1003, using a preset image processing model, performs pyramid decomposition on the first image to obtain a first low-frequency image, and performs pyramid decomposition on the second image to obtain a second low-frequency image and multiple high-frequency images.

[0133] In some embodiments of this application, a preset image processing model can be trained using the model training method provided in the embodiments of this application. The image processing model may include at least a Laplacian pyramid decomposition module.

[0134] The process of performing pyramid decomposition on the first image using the Laplacian pyramid decomposition module includes: performing pyramid decomposition on the first image using the Laplacian pyramid decomposition module to obtain a first low-frequency image corresponding to the first image. The first low-frequency image can be used to describe the low-frequency structural features of the first image. Additionally, multiple high-frequency images corresponding to the first image are also obtained. In some embodiments of this application, the process of performing pyramid decomposition on the first image using the Laplacian pyramid decomposition module can be referred to as follows: Figure 3 The diagram shows the pyramid decomposition process for the first training image.

[0135] The second image is decomposed using a Laplacian pyramid decomposition module, including: performing pyramid decomposition on the second image using the Laplacian pyramid decomposition module to obtain a second low-frequency image and multiple high-frequency images corresponding to the second image. The second low-frequency image is used to describe the low-frequency structural features of the second image, and the multiple high-frequency images are used to describe the high-frequency detail features of the second image. In some embodiments of this application, the pyramid decomposition of the second image using the Laplacian pyramid decomposition module can be referred to as follows: Figure 3 The diagram shows the pyramid decomposition process for the second training image.

[0136] S1004, perform pyramid decomposition on the mask image corresponding to the first image to obtain the third low-frequency image.

[0137] In some embodiments of this application, the image processing model may further include at least a data processing module. The data processing module processes the first image to determine a first brightness value and a second brightness value based on the brightness values ​​of the pixels in the first image, wherein the second brightness value is greater than the first brightness value. Based on the first brightness value and the second brightness value, a target pixel in the first image is determined, wherein the brightness value of the target pixel is greater than the first brightness value and less than the second brightness value. A mask image is generated based on the target pixel in the first image.

[0138] The mask image is decomposed into a pyramid using the Laplacian pyramid decomposition module, resulting in a third low-frequency image corresponding to the mask image. This third low-frequency image describes the low-frequency structural features of the mask image. Additionally, multiple high-frequency images corresponding to the mask image are also obtained. In some embodiments of this application, the pyramid decomposition of the mask image using the Laplacian pyramid decomposition module can be referred to as follows: Figure 3 The diagram illustrates the pyramid decomposition process for the mask image.

[0139] S1005, the first low-frequency image, the second low-frequency image, and the third low-frequency image are fused to obtain the first fused image.

[0140] In some embodiments of this application, the image processing model further includes at least a fusion layer, which performs channel fusion of the first low-frequency image, the second low-frequency image, and the third low-frequency image. This fusion can be achieved by pixel stacking or channel stacking to obtain the first fused image.

[0141] In the embodiments of this application, introducing a third low-frequency image during the fusion of the first and second low-frequency images can further enhance the brightness information, resulting in a first fused image with stronger brightness performance. In some embodiments of this application, the fusion of the first, second, and third low-frequency images to obtain the first fused image can be referred to as follows: Figure 3 The method shown is to fuse the first low-frequency image, the second low-frequency image, and the third low-frequency image.

[0142] S1006, based on the image reconstruction algorithm, obtains the target DRC image according to the first fused image and multiple high-frequency images.

[0143] In some embodiments of this application, the image processing model further includes at least a style transfer module, a high-frequency reconstruction module, and a pyramid reconstruction module. The image reconstruction algorithm can be an algorithm that performs the functions of the corresponding modules of the style transfer module, the high-frequency reconstruction module, and the pyramid reconstruction module.

[0144] In some embodiments of this application, the style feature map corresponding to the first fused image can be used to perform attribute transformation on the first fused image to obtain a second fused image. A style transfer module is used to further enhance the brightness performance of the first fused image. By extracting the style feature map of the first fused image, the brightness performance of the first fused image is enhanced, thereby obtaining the second fused image. The style transfer module is a lightweight style transfer network, including six network layers. The first network layer consists of a convolutional layer (CONV), a normalization layer (IN), and an activation function (RELU); the second network layer consists of a convolutional layer and an activation function; the third network layer consists of a residual block (RESBLOCK); the fourth network layer consists of a convolutional layer and an activation function; and the fifth and sixth network layers are convolutional layers. In the style transfer module, the style feature map extracted from the first to fifth network layers is used to perform attribute transformation on the first fused image to obtain an attribute transformation map. This attribute transformation map is processed by the sixth network layer to obtain the second fused image. In some embodiments of this application, the method of using the style feature map corresponding to the first fused image to perform attribute transformation on the first fused image to obtain the second fused image can refer to... Figure 3 The processing method shown is used to obtain the second fused image.

[0145] In some embodiments of this application, after determining the second fused image, multiple high-frequency images can be reconstructed using the second fused image to obtain multiple reconstructed images. To accelerate the generation efficiency of subsequent target DRC images, the image processing model at least includes a high-frequency reconstruction module for reconstructing multiple high-frequency images. In some embodiments of this application, the second fused image is upsampled to obtain an upsampled second fused image, the resolution of which is a first resolution. A first high-frequency image with the first resolution is found from the multiple high-frequency images. The upsampled second fused image and the first high-frequency image are input into the high-frequency reconstruction module for high-frequency reconstruction to obtain a first reconstructed image.

[0146] The first reconstructed image is upsampled to obtain an upsampled first reconstructed image, which has a second resolution. A second high-frequency image with the second resolution is found from multiple high-frequency images. High-frequency reconstruction is performed between the upsampled first reconstructed image and the second high-frequency image to obtain the second reconstructed image. The second resolution is greater than the first resolution.

[0147] The second reconstructed image is upsampled to obtain an upsampled second reconstructed image, which has a resolution equal to the third resolution. A third high-frequency image with the third resolution is then found from multiple high-frequency images. High-frequency reconstruction is performed between the upsampled second reconstructed image and the third high-frequency image to obtain the third reconstructed image. The third resolution is greater than the second resolution.

[0148] In some embodiments of this application, the high-frequency reconstruction process can be referred to as follows: Figure 3 The diagram shows a method for high-frequency reconstruction of multiple high-frequency images using a second fusion image.

[0149] In some embodiments of this application, after determining multiple reconstructed images, pyramid reconstruction can be performed based on the second fused image and the multiple reconstructed images to obtain the target DRC image. The image processing model also includes at least a pyramid reconstruction module and an output layer. The second fused image, the first reconstructed image, the second reconstructed image, and the third reconstructed image can be input into the pyramid reconstruction module for pyramid reconstruction.

[0150] In some embodiments of this application, the sum of pixel values ​​of pixels at the same position in the upsampled second fused image and the first reconstructed image is calculated to obtain the first layer reconstructed image. The sum of pixel values ​​of pixels at the same position in the upsampled first layer reconstructed image and the second reconstructed image is calculated to obtain the second layer reconstructed image. The sum of pixel values ​​of pixels at the same position in the upsampled second layer reconstructed image and the third reconstructed image is calculated to obtain the target DRC image. In some embodiments of this application, the pyramid reconstruction process can refer to... Figure 3The diagram shows the process of pyramid reconstruction for the second fused image and multiple reconstructed images.

[0151] Through the above embodiments, an image processing model integrating an input layer, a data processing module, a Laplacian pyramid decomposition module, a fusion layer, a style transfer module, a high-frequency reconstruction module, a pyramid reconstruction module, and an output layer is constructed. This model can optimize the low-frequency structural features of the second image using the low-frequency structural features of the first image while preserving the high-frequency detail features of the second image, thus avoiding significant brightness differences between the images before and after DRC processing. Furthermore, based on the target DRC image, a display image stored in the camera's image library can be obtained, ensuring that the brightness level of the display image is close to that of the preview image, reducing visual differences between the preview and display images.

[0152] Furthermore, the image processing model proposed in this application embodiment can also be applied to scenarios of single-channel multi-exposure image fusion, solving the problem of poor brightness and contrast performance of the fused image.

[0153] In other embodiments of this application, the image processing model described above may further consist of a data processing module, a Gaussian pyramid module, a Laplacian module, and a stitching module. The third and fourth images are used as inputs to this image processing model, and the data processing module, Gaussian pyramid module, Laplacian module, and stitching module are used to generate the target DRC image. The third image is a preview image, and the fourth image is the DRC image.

[0154] In the data processing module, a fusion curve with adjustable parameters is constructed. The third image is processed into grayscale to obtain its corresponding grayscale image. The fusion curve is then used to map the grayscale image to the third image, followed by a 5x5 window filter to generate the first weight map. The fourth image is also processed into grayscale to obtain its corresponding grayscale image. The fusion curve is then used to map the grayscale image to the fourth image, followed by a 5x5 window filter to generate the second weight map.

[0155] In the Gaussian pyramid module, the first weight graph is decomposed into a multi-level Gaussian pyramid, and the second weight graph is also decomposed into a multi-level Gaussian pyramid, wherein the levels of the first weight graph decomposition are the same as those of the second weight graph decomposition.

[0156] In the Laplacian module, the third image is decomposed into a multi-level Laplacian pyramid, and the fourth image is also decomposed into a multi-level Laplacian pyramid. The levels of the third and fourth image decompositions are the same as the levels of the first and second weighted graph decompositions.

[0157] In the stitching module, the multi-level Gaussian pyramids corresponding to the first weight image, the second weight image, the third image, and the fourth image are stitched together according to their corresponding levels to generate the target pyramid. Pyramid reconstruction is then performed on the target pyramid to generate the target DRC image.

[0158] In the above embodiments of this application, by using the preview image and the DRC image as input, the image brightness can be optimized. By stitching the results generated by the Gaussian pyramid module and the Laplacian module, artifacts can be effectively reduced and the contrast of the target DRC image can be improved.

[0159] In other embodiments of this application, after the image processing model is applied to the dynamic range compression (DRC) module of the electronic device, the target RDC image still needs to be processed by the post-processing module of the electronic device. The post-processing module may include other processing modules of the ISP, such as the YUV domain. The following is in conjunction with... Figure 11 and Figure 12 Please provide an explanation.

[0160] Figure 11 This is a software architecture diagram of an electronic device provided in an embodiment of this application. The layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. For example, the Android system, from top to bottom, consists of the application layer 1101, framework layer 1102, Android runtime and system libraries 1103, hardware abstraction layer 1104, kernel layer 1105, and hardware layer 1106.

[0161] Application layer 1101 may include a series of application packages. For example, application packages may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, SMS, device control services, etc.

[0162] The framework layer 1102 provides an Application Programming Interface (API) and programming framework for applications in the application layer. The application framework layer includes predefined functions. For example, it may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0163] The window manager manages window programs. It can obtain screen size, determine the presence of a status bar, lock the screen, and capture screenshots. The content provider stores and retrieves data, making it accessible to applications. This data can include videos, images, audio, made and received phone calls, browsing history and bookmarks, and phone books. The view system includes visual controls, such as controls for displaying text and controls for displaying images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon can include views for displaying text and views for displaying images. The phone manager provides communication functionality for electronic devices, such as managing call status (including connection and disconnection). The resource manager provides applications with various resources, such as localized strings, icons, images, layout files, and video files. The notification manager allows applications to display notifications in the status bar, conveying informational messages that disappear automatically after a short pause without user interaction. For example, the notification manager is used to notify of download completion or message alerts. The notification manager can also display notifications as icons or scrolling text in the system's top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting alert sounds, causing electronic devices to vibrate, and flashing indicator lights.

[0164] The Android Runtime consists of the core libraries and the virtual machine. The Android runtime is responsible for the scheduling and management of the Android system. The core libraries consist of two parts: one part contains the functionalities that the Java language needs to call, and the other part contains the core Android libraries.

[0165] The application layer 1101 and the framework layer 1102 run in a virtual machine. The virtual machine executes the Java files of the application layer and the framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0166] System library 1103 may include multiple functional modules. For example, a surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGLES), 2D graphics engines (e.g., SGL), etc.

[0167] The Surface Manager manages the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The Media Library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG. The 3D Graphics Processing Library implements 3D graphics drawing, image rendering, compositing, and layer processing. The 2D Graphics Engine is the drawing engine for 2D graphics.

[0168] The Hardware Abstraction Layer 1104 runs in user space, encapsulates kernel-level drivers, and provides calling interfaces to the upper layers.

[0169] Kernel layer 1105 is the layer between hardware and software. Kernel layer 1105 includes at least display drivers, touch drivers, audio drivers, and sensor drivers.

[0170] The kernel layer (1105) is the core of the operating system for electronic devices. It is the first layer of software extension based on the hardware, providing the most basic functions of the operating system. It is the foundation for the operation of the operating system, responsible for managing system processes, memory, device drivers, files, and network systems, and determining the system's performance and stability. For example, the kernel layer can determine the timing of an application's operation on a certain part of the hardware.

[0171] The kernel layer 1105 includes hardware-dependent programs such as interrupt handlers and device drivers, as well as basic, common, and frequently running modules such as clock management and process scheduling modules, and critical data structures. The kernel layer can be located in the processor or embedded in internal memory.

[0172] The hardware layer 1106 includes the hardware of electronic devices, such as displays, buttons, cameras, etc.

[0173] Figure 12 This is a software workflow diagram of an electronic device provided in one embodiment of this application. Figure 12As shown, in response to a preset operation, the electronic device can send a shooting request corresponding to the preset operation to the hardware abstraction layer through the camera access interface. The hardware abstraction layer then calls the camera driver in the kernel layer based on the received shooting request. The camera driver then drives the camera and other hardware devices to respond to the shooting request and begin capturing the image, i.e., capturing a preview image. The camera can then transmit the captured preview image to the camera hardware abstraction layer through the camera driver. Optionally, after receiving the preview image captured by the camera, the camera hardware abstraction layer can report the preview image to the camera application through the camera access interface, or it can call an image processing algorithm library to process the captured preview image. Specifically, the camera hardware abstraction layer processes the captured preview image using a separate processor or an integrated processor. Specifically, it calls the dynamic range compression module in the image processing algorithm library to process the preview image and generate a DRC image. The DRC image and the preview image without dynamic range compression are then input into an image processing model for processing to generate a target DRC image. The target DRC image is then input into a post-processing module to obtain the display image. After obtaining the display image, the camera hardware abstraction layer can pass the display image to the camera application for display through the camera access interface, or it can pass it to the image management module for storage through the camera access interface.

[0174] This application also provides an electronic device 100, see reference. Figure 13 As shown, the electronic device 100 can be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, as well as cellular phone, personal digital assistant (PDA), augmented reality (AR) device, virtual reality (VR) device, artificial intelligence (AI) device, wearable device, in-vehicle device, smart home device and / or smart city device. The embodiments of this application do not impose special restrictions on the specific type of electronic device 100.

[0175] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, Universal Serial Bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and Subscriber Identification Module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0176] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0177] Processor 110 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0178] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0179] The processor 110 may also include a memory for storing instructions and data. In one embodiment of this application, the memory in the processor 110 is a cache memory. The memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instructions or data again, it can directly retrieve them from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0180] In one embodiment of this application, the processor 110 may include one or more interfaces. These interfaces may include an Inter-integrated Circuit (I2C) interface, an Inter-integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver / Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI) interface, a General-Purpose Input / Output (GPIO) interface, a Subscriber Identity Module (SIM) interface, and / or a Universal Serial Bus (USB) interface, etc.

[0181] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In one embodiment of this application, the processor 110 may include multiple I2C buses. The processor 110 can couple to the touch sensor 180K, charger, flash, camera 193, etc., through different I2C bus interfaces. For example, the processor 110 can couple to the touch sensor 180K through the I2C interface, enabling the processor 110 and the touch sensor 180K to communicate through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0182] The I2S interface can be used for audio communication. In one embodiment of this application, the processor 110 may include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to realize communication between the processor 110 and the audio module 170. In one embodiment of this application, the audio module 170 can transmit audio signals to the wireless communication module 160 via the I2S interface to realize the function of answering phone calls through a Bluetooth headset.

[0183] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In one embodiment of this application, the audio module 170 and the wireless communication module 160 can be coupled via the PCM bus interface. In another embodiment of this application, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface, enabling the function of answering phone calls through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0184] The UART interface is a universal serial data bus used for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In one embodiment of this application, the UART interface is typically used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 via the UART interface to implement Bluetooth functionality. In one embodiment of this application, the audio module 170 can transmit audio signals to the wireless communication module 160 via the UART interface to enable music playback via Bluetooth headphones.

[0185] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a Camera Serial Interface (CSI) and a Display Serial Interface (DSI). In one embodiment of this application, the processor 110 and the camera 193 communicate via the CSI interface to realize the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate via the DSI interface to realize the display function of the electronic device 100.

[0186] The GPIO interface can be configured via software. It can be configured as a control signal or a data signal. In one embodiment of this application, the GPIO interface can be used to connect the processor 110 to a camera 193, a display screen 194, a wireless communication module 160, an audio module 170, a sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0187] USB port 130 is a USB standard compliant interface, specifically a Mini USB port, MicroUSB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge electronic device 100, and can also be used for data transfer between electronic device 100 and peripheral devices. It can also be used to connect headphones for audio playback. Furthermore, the interface can be used to connect other electronic devices 100, such as AR devices.

[0188] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0189] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via a USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device 100 via the power management module 141.

[0190] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0191] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0192] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0193] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low-noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In one embodiment of this application, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In another embodiment of this application, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0194] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through audio devices (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In one embodiment of this application, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and housed within the same device as the mobile communication module 150 or other functional modules.

[0195] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including Wireless Local Area Networks (WLANs) (such as Wireless Fidelity (Wi-Fi) networks), Bluetooth (BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), and Infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0196] In one embodiment of this application, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the Beidou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0197] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0198] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), an Active-Matrix Organic Light-Emitting Diode (AMOLED), a Flexible Light-Emitting Diode (FLED), a Minied, Microled, Micro-OLED, or a Quantum Dot Light-Emitting Diode (QLED), etc. In one embodiment of this application, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0199] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0200] The ISP is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, converting it into an image visible to the naked eye. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In one embodiment of this application, the ISP can be set in the camera 193.

[0201] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In one embodiment of this application, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0202] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0203] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0204] NPU stands for Neural Network (NN) computing processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, information prompts, speech recognition, and text understanding.

[0205] Internal memory 121 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM).

[0206] Random access memory can include static random-access memory (SRAM), dynamic random-access memory (DRAM), synchronous dynamic random-access memory (SDRAM), double data rate synchronous dynamic random-access memory (DDRSDRAM, for example, fifth-generation DDRSDRAM is generally called DDR5SDRAM), etc.

[0207] Non-volatile memory can include disk storage devices and flash memory.

[0208] Flash memory can be classified according to its operating principle, including NORFLASH, NANDFLASH, 3D NANDFLASH, etc.; according to the level of the storage cell, including single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc.; and according to the storage specification, including universal flash storage (UFS) and embedded multi-media card (eMMC), etc.

[0209] The random access memory can be directly read and written by the processor 110. It can be used to store executable programs (such as machine instructions) of the operating system or other running programs, as well as user and application data.

[0210] Non-volatile memory can also store executable programs and user and application data, and can be pre-loaded into random access memory for direct reading and writing by the processor 110.

[0211] The external memory interface 120 can be used to connect to external non-volatile memory, thereby expanding the storage capacity of the electronic device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to perform data storage functions. For example, music, video, and other files can be stored in the external non-volatile memory.

[0212] Internal memory 121 or external memory interface 120 is used to store one or more computer programs. The one or more computer programs are configured to be executed by processor 110. The one or more computer programs include multiple instructions, which, when executed by processor 110, can implement the screen display detection method executed on electronic device 100 in the above embodiments, so as to realize the screen display detection function of electronic device 100.

[0213] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0214] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In one embodiment of this application, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0215] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0216] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0217] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Electronic device 100 may have at least one microphone 170C. In some embodiments, electronic device 100 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 100 may also have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0218] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0219] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0220] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to touch operations performed on different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations performed on different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0221] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0222] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and separate from the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to realize functions such as calls and data communication. In one embodiment of this application, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100. This application also provides a computer storage medium storing computer instructions. When the computer instructions are executed on the electronic device 100, the electronic device 100 performs the aforementioned related method steps to implement the methods in the above embodiments.

[0223] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the methods described in the above embodiments.

[0224] See Figure 14As shown in the illustration, this application also provides a chip system 1400, which is applied to an electronic device 100. The chip system includes one or more processors 1410, which are used to invoke computer instructions to cause the electronic device 100 to input an image to be processed into the chip system. The chip system processes the image to be processed based on the method provided in this application and outputs a processed image. For example, the image to be processed includes a preview image and a DRC image, and the processed image is a target DRC image or a display image.

[0225] In this embodiment, the electronic device, computer storage medium, computer program product, or chip system are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0226] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0227] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0228] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0229] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0230] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. The software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0231] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. An image processing method applied to electronic devices, characterized in that, The method includes: In response to the opening of the camera app, a preview image is obtained; In response to the shooting operation, a first image is determined from the preview image, and the dynamic range compressed (DRC) image corresponding to the first image is obtained as the second image; Using a preset image processing model, the first image is decomposed into a pyramid to obtain a first low-frequency image, and the second image is decomposed into a pyramid to obtain a second low-frequency image and multiple high-frequency images. Pyramid decomposition is performed on the mask image corresponding to the first image to obtain the third low-frequency image; The first low-frequency image, the second low-frequency image, and the third low-frequency image are fused to obtain a first fused image; Based on the image reconstruction algorithm, the target DRC image is obtained according to the first fused image and the multiple high-frequency images.

2. The image processing method according to claim 1, characterized in that, The image reconstruction algorithm, which obtains the target DRC image based on the first fused image and the multiple high-frequency images, includes: The attributes of the first fusion map are transformed based on the style feature map corresponding to the first fusion map to obtain the second fusion map; The second fusion image is used to perform high-frequency reconstruction on the multiple high-frequency images to obtain multiple reconstructed images; Based on the second fused image and the multiple reconstructed images, pyramid reconstruction is performed to obtain the target DRC image.

3. A model training method applied to electronic devices, characterized in that, The method includes: A Laplacian pyramid fusion network is trained using a sample dataset, wherein the sample dataset includes a first training image, a second training image, and a desired image. The first training image is a preview image acquired using a camera application, and the second training image is a DRC image corresponding to the preview image. The training of the Laplacian pyramid fusion network using the sample dataset includes: inputting the first training image and the second training image into the Laplacian pyramid fusion network for training to obtain a predicted image; using the Laplacian pyramid fusion network that meets preset conditions as an image processing model. The inputting the first training image and the second training image into the Laplacian pyramid fusion network for training includes: obtaining a mask image based on the first training image; performing pyramid decomposition on the first training image, the second training image, and the mask image respectively to obtain a first low-frequency image corresponding to the first training image, a second low-frequency image corresponding to the second training image, multiple high-frequency images, and a third low-frequency image corresponding to the mask image; fusing the first low-frequency image, the second low-frequency image, and the third low-frequency image to obtain a first fused image. The trained Laplacian pyramid fusion network was used as an image processing model.

4. The model training method according to claim 3, characterized in that, The step of using a Laplacian pyramid fusion network that meets preset conditions as the image processing model includes: Based on the first training image, the second training image, the predicted image, and the expected image, the loss value of the Laplacian pyramid fusion network is calculated; If the loss value is less than or equal to a preset loss threshold, the preset condition is met, the training of the Laplacian pyramid fusion network is completed, and the image processing model is obtained.

5. The model training method according to claim 4, characterized in that, The step of inputting the first training image and the second training image into the Laplacian pyramid fusion network for training further includes: The attributes of the first fusion map are transformed based on the style feature map corresponding to the first fusion map to obtain the second fusion map; The second fusion image is used to perform high-frequency reconstruction on the multiple high-frequency images to obtain multiple reconstructed images; The second fused image and the multiple reconstructed images are used to perform pyramid reconstruction to obtain the predicted image.

6. The model training method according to claim 5, characterized in that, The step of using the second fusion map to perform high-frequency reconstruction on the multiple high-frequency maps to obtain multiple high-frequency reconstructed maps includes: The second fused image is upsampled to obtain an upsampled second fused image, wherein the resolution of the upsampled second fused image is the first resolution; The upsampled second fused image and the first high-frequency image are reconstructed at high frequency to obtain a first reconstructed image. The first high-frequency image is one of the multiple high-frequency images, and the resolution of the first high-frequency image is the first resolution. The first reconstructed image is upsampled to obtain an upsampled first reconstructed image, wherein the resolution of the upsampled first reconstructed image is a second resolution, and the second resolution is greater than the first resolution; The upsampled first reconstructed image and the second high-frequency image are subjected to high-frequency reconstruction to obtain the second reconstructed image. The second high-frequency image is one of the multiple high-frequency images, and the resolution of the second high-frequency image is the second resolution. The second reconstructed image is upsampled to obtain an upsampled second reconstructed image, the resolution of the upsampled second reconstructed image is a third resolution, the third resolution is greater than the second resolution; The upsampled second reconstructed image and the third high-frequency image are then subjected to high-frequency reconstruction to obtain the third reconstructed image; The first reconstruction image, the second reconstruction image, and the third reconstruction image are used as the multiple high-frequency reconstruction images.

7. The model training method according to claim 6, characterized in that, The step of performing pyramid reconstruction on the second fused image and the multiple reconstructed images to obtain the predicted image includes: The first layer reconstruction image is obtained by summing the pixel values ​​corresponding to each pixel point at the same position in the upsampled second fused image and the first reconstructed image. The second-layer reconstruction image is obtained by summing the pixel values ​​corresponding to each pixel at the same position in the upsampled first-layer reconstruction image and the second-layer reconstruction image. The predicted image is obtained by summing the pixel values ​​corresponding to pixels at the same position in the upsampled second-layer reconstruction image and the third reconstruction image.

8. The model training method according to claim 5, characterized in that, The process of obtaining the mask image based on the first training image includes: Based on the brightness values ​​of the pixels in the first training image, a first brightness value and a second brightness value are determined, wherein the first brightness value is less than the second brightness value; Based on the first brightness value and the second brightness value, the target pixel of the first training image is determined, wherein the brightness value of the target pixel is greater than the first brightness value and less than the second brightness value; The mask image is generated based on the target pixels.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor: The memory is used to store program instructions; The processor is configured to read and execute the program instructions stored in the memory, and when the program instructions are executed by the processor, cause the electronic device to perform the image processing method as described in any one of claims 1 to 2, or to perform the model training method as described in any one of claims 3 to 8.

10. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the processors being configured to invoke computer instructions to cause the electronic device to input an image to be processed into the chip system and execute the image processing method as described in any one of claims 1 to 2, or execute the model training method as described in any one of claims 3 to 8.

11. A computer storage medium, characterized in that, The computer storage medium stores program instructions that, when executed on an electronic device, cause the processor of the electronic device to perform the image processing method as described in any one of claims 1 to 2, or to perform the model training method as described in any one of claims 3 to 8.

Images