Image processing method, electronic device, computer program product and storage medium

Abstract

The application provides an image processing method, an electronic device, a computer program product and a storage medium, and relates to the technical field of terminals. The electronic device collects a first image in a dark-light shooting mode or a night scene shooting mode, and in response to a shooting operation of a shooting preview interface, performs high-frequency enhancement processing on the first image according to a high-frequency information component of the first image and a low-frequency information component obtained based on the high-frequency information component of the first image to obtain a second image. The electronic device performs high-frequency enhancement processing on the collected first image, so that the second image contains more high-frequency characteristics than the first image, and the obtained second image has higher definition than the first image, thereby improving the quality of an image obtained by dark-light shooting.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to an image processing method, electronic device, computer program product, and storage medium. Background Technology

[0002] With the increasing popularity of mobile phone camera photography, mobile phone camera shooting environments can be divided into bright light environments and low light environments. Mobile phone shooting environments are low light environments. Due to the limitations of the camera shutter and lens components, the raw image data (RAW image) collected by the sensor often has dense noise and completely black dark areas, resulting in poor signal-to-noise ratio and loss of high-frequency information in the image, and poor image quality. Summary of the Invention

[0003] This application provides an image processing method, an electronic device, a computer program product, and a storage medium to solve the technical problem of poor image quality when the electronic device acquires images in low-light environments.

[0004] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0005] Firstly, an image processing method is provided, applied to an electronic device. The electronic device can be a mobile phone, tablet, wearable device, etc., equipped with a camera. The image processing method provided in this embodiment processes a first image captured by the camera of the electronic device.

[0006] The electronic device responds to the first operation by displaying a shooting preview interface. This first operation can be varied. For example, it could be a click on the night mode control of the camera application, or a click on the camera application icon. Upon receiving this first operation, the electronic device opens the camera application and begins the shooting process. Alternatively, the first operation could be a voice control operation. If the electronic device receives a voice control command containing keywords such as "start taking pictures" or "turn on night mode," it can also begin the shooting process.

[0007] The shooting preview interface displayed on the electronic device can be used to preview shots in low-light or night scene shooting modes. When the electronic device is in low-light or night scene shooting mode, the preview interface may or may not display a corresponding prompt text.

[0008] In low-light or night scene shooting modes, electronic devices capture the first image. In these modes, the ambient light level is typically low, meaning the device is in a dimly lit environment with limited light intake. Consequently, the first image captured by the electronic device has fewer high-frequency features and relatively lower sharpness. This first image can be the raw image data, or RAW image, captured by the camera sensor of the electronic device.

[0009] The electronic device receives a shooting operation applied to the shooting preview interface and performs high-frequency enhancement processing on the captured first image to obtain a second image with more high-frequency features. The shooting operation received by the electronic device on the shooting preview interface can be a click operation on the shutter control, instructing the camera to capture a frame. In this scenario, the electronic device can use the most recently captured frame as the first image and perform high-frequency enhancement processing on it to obtain the second image.

[0010] In other cases, the shooting operation received by the electronic device can also be a click operation on the shutter control, instructing the camera to start capturing video. The electronic device can treat each frame captured by the camera as a first image and perform high-frequency enhancement processing to obtain a second image corresponding to each first image. In this case, when the electronic device receives a click operation from the user on the shutter control to end the video capture process, the final video generated by the electronic device can contain images that have undergone high-frequency enhancement processing, thus allowing the electronic device to directly obtain a video with higher clarity.

[0011] The electronic device performs high-frequency enhancement processing on the acquired first image. This mainly involves first acquiring the high-frequency information components of the first image, then obtaining the low-frequency information components based on the high-frequency information components, and finally performing high-frequency enhancement processing on the first image based on the high-frequency and low-frequency information components to obtain the corresponding second image. For example, the electronic device can extract a high-frequency feature map from the high-frequency information components, obtain a low-frequency feature map from the low-frequency information components, and then fuse the high-frequency and low-frequency feature maps to obtain the second image.

[0012] The electronic device performs high-frequency enhancement processing on the first image to obtain a second image. Because the second image contains more high-frequency features, its clarity is higher than that of the first image. Thus, when the electronic device captures images in low-light or night scene shooting modes, even with limited light intake, it can obtain clearer images with more detailed features, improving the quality of the acquired images.

[0013] In one possible implementation of the first aspect, an application scenario for the electronic device to perform image processing is defined. The electronic device automatically invokes an ambient light sensor to detect the current ambient light brightness value and selects whether to trigger the execution of the image processing method.

[0014] Specifically, in response to the first operation, the electronic device can invoke its ambient light sensor to collect ambient light brightness values. If the electronic device detects that the ambient light brightness value is less than or equal to a preset low-light threshold, it can determine that the current shooting environment brightness is low, which may cause the camera to capture images with lost high-frequency features due to limited light intake. Based on this, the electronic device can display the shooting preview interface in low-light shooting mode or night scene shooting mode. In response to the photo-taking operation in the shooting preview interface of low-light shooting mode or night scene shooting mode, it can perform highlight enhancement processing on the first image to obtain a second image with more high-frequency features.

[0015] In one possible implementation of the first aspect, another application scenario for image processing by the electronic device is defined. In this implementation, if the electronic device detects that the first operation is a click operation on the night scene shooting control, it can also display a shooting preview interface in low light shooting mode or night scene shooting mode. In response to the photo-taking operation in the shooting preview interface in low light shooting mode or night scene shooting mode, the acquired first image is subjected to highlight enhancement processing to obtain a second image with more high-frequency features.

[0016] In one possible implementation of the first aspect, a scheme is further defined in which an electronic device performs high-frequency enhancement processing on a first image to obtain a second image based on the high-frequency information components of the first image and the low-frequency information components obtained based on the high-frequency information components of the first image.

[0017] Specifically, the electronic device uses a convolutional neural network to obtain the high-frequency information components of the first image, and obtains the low-frequency information components based on the high-frequency information components. The electronic device extracts a high-frequency feature map based on the high-frequency information components, and extracts a low-frequency feature map based on the low-frequency information components. Finally, the electronic device uses a feature fusion network to process the high-frequency and low-frequency feature maps to generate a second image corresponding to the first image.

[0018] The electronic device uses a convolutional neural network to obtain high-frequency information components from the first image. These high-frequency information components cover a large number of pixel regions containing high-frequency features, so that the high-frequency feature map extracted from them also contains more high-frequency features. The electronic device then fuses the obtained high-frequency feature maps to obtain the second image, thereby achieving a high-frequency enhancement effect.

[0019] In one possible implementation of the first aspect, the implementation scheme of the electronic device using a convolutional neural network to obtain the high-frequency information components of the first image is further defined. The electronic device performs local variance statistics on the first image to obtain a first variance result. The obtained first variance result may include the brightness difference between each pixel and its adjacent pixel regions in the first image, and the pixel regions where the high-frequency features are located.

[0020] The electronic device uses the first variance result as prior information to input the first image into a convolutional neural network to obtain the high-frequency information components of the first image.

[0021] The electronic device calculates the first variance of the first image, which serves as prior information to guide the convolutional neural network to obtain more accurate high-frequency information components, thereby improving the efficiency of the electronic device in high-frequency enhancement processing.

[0022] In one possible implementation of the first aspect, the convolutional neural network used by the electronic device to acquire high-frequency information components from the first image is further defined. Specifically, the convolutional neural network includes a first convolutional layer, a second convolutional layer, and an activation layer. The kernels of the first and second convolutional layers have different sizes and dilatations to perform different convolutional processing on the input images, resulting in images with the same pixel size. The activation layer includes an activation function used to calculate the contrast of each pixel in the two input images.

[0023] The step of an electronic device inputting a first image into a convolutional neural network to obtain high-frequency information components of the first image specifically includes: the electronic device inputting the first image into a first convolutional layer and a second convolutional layer respectively; the first convolutional layer performing convolution processing on the first image to obtain a first feature map; and the second convolutional layer performing convolution processing on the first image to obtain a second feature map. The first feature map and the second feature map have the same pixel size but different features.

[0024] The electronic device then inputs the first feature map and the second feature map into the activation layer, calculates the difference in pixel features contained in the first feature map and the second feature map for each pixel, and selects the pixel features of the pixels with greater differences to obtain the high-frequency information components of the first image.

[0025] In one possible implementation of the first aspect, a further defined scheme for the electronic device to obtain low-frequency information components based on high-frequency information components of a first image is provided. The first image includes both high-frequency and low-frequency information components. Since the electronic device has already extracted the high-frequency information components of the first image, it can perform an inverse operation only on the high-frequency information components to obtain an inverse mapping of the high-frequency information components. Based on the first image and the inverse mapping of the high-frequency information components, the low-frequency information components of the first image can be obtained.

[0026] In this way, the electronic device does not need to provide a separate low-pass filter to extract the low-frequency information components of the first image, saving computational costs.

[0027] In one possible implementation of the first aspect, a scheme for extracting high-frequency feature maps from high-frequency information components by the electronic device is further defined. The electronic device utilizes a generative adversarial network to extract high-frequency feature maps from the high-frequency information components.

[0028] In this embodiment, the generative adversarial network includes a generator and a discriminator. The generator includes a multi-level processing network. Each level of the processing network includes a wavelet transform network and an inverse wavelet transform network with the same scale and symmetrical arrangement. The generator sequentially includes multiple wavelet transform networks with progressively decreasing scales and inverse wavelet transform networks with progressively increasing scales. The input of the first-level wavelet transform network is a high-frequency information component, and the output of the first-level inverse wavelet transform network is a high-frequency feature map. The input of other levels of wavelet transform networks is the first-class feature map output by the wavelet transform network of the adjacent level above. The input of each level of inverse wavelet transform is the first-class feature map output by the wavelet transform network of the same level and the second-class feature map output by the inverse wavelet transform network of the next level.

[0029] Electronic devices can obtain high-frequency feature maps corresponding to high-frequency information components by using multi-scale wavelet transform for downsampling and then using inverse wavelet transform for upsampling.

[0030] In one possible implementation of the first aspect, the implementation scheme for the electronic device to extract low-frequency feature maps based on low-frequency information components is further defined. The low-frequency information components acquired by the electronic device contain a large number of low-frequency features, which are redundant and do not need to be enhanced during high-frequency enhancement processing.

[0031] Based on this, electronic devices can extract only the low-frequency features from a portion of the low-frequency information components, thus obtaining the low-frequency features of the first image while reducing the computational load of the electronic device in extracting low-frequency features.

[0032] The electronic device splits the low-frequency information component into a first low-frequency component and a second low-frequency component, extracts a third feature map from the first low-frequency component, and fuses the third feature map and the second low-frequency component to obtain a low-frequency feature map.

[0033] In one possible implementation of the first aspect, both the first low-frequency component and the second low-frequency component are defined to include half of the low-frequency information component.

[0034] The more first low-frequency components an electronic device retains, the more low-frequency information components participate in the extraction of low-frequency feature maps, resulting in more retained low-frequency features and a closer match to the original image. Conversely, the more second low-frequency components an electronic device retains, the more low-frequency information components do not participate in the extraction of low-frequency feature maps, and the less data needs to be calculated.

[0035] Electronic devices divide low-frequency information components into a first low-frequency component and a second low-frequency component. This not only preserves more low-frequency features but also reduces the amount of computation, thereby further improving the quality and efficiency of image processing.

[0036] Secondly, another image processing method is provided for application in electronic devices. The differences between the provided image processing method and the image processing method provided in the first aspect include: the first image is not limited to an image captured in real time by the camera of the electronic device in low-light shooting mode or night scene shooting mode.

[0037] Specifically, the electronic device first displays a first image, which can be an image pre-stored in the electronic device's photo album or an image obtained from another terminal. The electronic device can receive click operations from the user on the editing controls of the interface of the first image and perform high-frequency enhancement processing on the first image.

[0038] The electronic device performs high-frequency enhancement processing on the acquired first image. This mainly involves first acquiring the high-frequency information components of the first image, then obtaining the low-frequency information components based on the high-frequency information components, and finally performing high-frequency enhancement processing on the first image based on the high-frequency and low-frequency information components to obtain the corresponding second image. For example, the electronic device can extract a high-frequency feature map from the high-frequency information components, obtain a low-frequency feature map from the low-frequency information components, and then fuse the high-frequency and low-frequency feature maps to obtain the second image.

[0039] The electronic device performs high-frequency enhancement processing on the first image to obtain a second image. Because the second image contains more high-frequency features, its clarity is higher than that of the first image. Thus, when the electronic device captures images in low-light or night scene shooting modes, even with limited light intake, it can obtain clearer images with more detailed features, improving the quality of the acquired images.

[0040] In another possible implementation of the second aspect, a scheme is further defined in which the electronic device performs high-frequency enhancement processing on the first image to obtain the second image based on the high-frequency information components of the first image and the low-frequency information components obtained based on the high-frequency information components of the first image.

[0041] Specifically, the electronic device uses a convolutional neural network to obtain the high-frequency information components of the first image, and obtains the low-frequency information components based on the high-frequency information components. The electronic device extracts a high-frequency feature map based on the high-frequency information components, and extracts a low-frequency feature map based on the low-frequency information components. Finally, the electronic device uses a feature fusion network to process the high-frequency and low-frequency feature maps to generate a second image corresponding to the first image.

[0042] The electronic device uses a convolutional neural network to obtain high-frequency information components from the first image. These high-frequency information components cover a large number of pixel regions containing high-frequency features, so that the high-frequency feature map extracted from them also contains more high-frequency features. The electronic device then fuses the obtained high-frequency feature maps to obtain the second image, thereby achieving a high-frequency enhancement effect.

[0043] In one possible implementation of the second aspect, the electronic device further specifies a scheme for acquiring high-frequency information components of the first image using a convolutional neural network. The electronic device performs local variance statistics on the first image to obtain a first variance result. The obtained first variance result may include the brightness difference between each pixel and its adjacent pixel regions in the first image, and the pixel regions where the high-frequency features are located.

[0044] The electronic device uses the first variance result as prior information to input the first image into a convolutional neural network to obtain the high-frequency information components of the first image.

[0045] The electronic device calculates the first variance of the first image, which serves as prior information to guide the convolutional neural network to obtain more accurate high-frequency information components, thereby improving the efficiency of the electronic device in high-frequency enhancement processing.

[0046] In one possible implementation of the second aspect, the convolutional neural network used by the electronic device to acquire high-frequency information components from the first image is further defined. Specifically, the convolutional neural network includes a first convolutional layer, a second convolutional layer, and an activation layer. The kernels of the first and second convolutional layers have different sizes and dilatations to perform different convolutional processing on the input images, resulting in images with the same pixel size. The activation layer includes an activation function used to calculate the contrast of each pixel in the two input images.

[0047] The step of an electronic device inputting a first image into a convolutional neural network to obtain high-frequency information components of the first image specifically includes: the electronic device inputting the first image into a first convolutional layer and a second convolutional layer respectively; the first convolutional layer performing convolution processing on the first image to obtain a first feature map; and the second convolutional layer performing convolution processing on the first image to obtain a second feature map. The first feature map and the second feature map have the same pixel size but different features.

[0048] The electronic device then inputs the first feature map and the second feature map into the activation layer, calculates the difference in pixel features contained in the first feature map and the second feature map for each pixel, and selects the pixel features of the pixels with greater differences to obtain the high-frequency information components of the first image.

[0049] In one possible implementation of the second aspect, the implementation scheme for the electronic device to obtain low-frequency information components based on high-frequency information components of the first image is further defined. The first image includes high-frequency information components and low-frequency information components. The electronic device has already extracted the high-frequency information components of the first image, so it can perform inverse operations only on the high-frequency information components to obtain the inverse mapping of the high-frequency information components. Based on the first image and the inverse mapping of the high-frequency information components, the low-frequency information components of the first image can be obtained.

[0050] In this way, the electronic device does not need to provide a separate low-pass filter to extract the low-frequency information components of the first image, saving computational costs.

[0051] In one possible implementation of the second aspect, the electronic device extracts high-frequency feature maps based on high-frequency information components. The electronic device utilizes a generative adversarial network to extract high-frequency feature maps from the high-frequency information components.

[0052] In this embodiment, the generative adversarial network includes a generator and a discriminator. The generator includes a multi-level processing network. Each level of the processing network includes a wavelet transform network and an inverse wavelet transform network with the same scale and symmetrical arrangement. The generator sequentially includes multiple wavelet transform networks with progressively decreasing scales and inverse wavelet transform networks with progressively increasing scales. The input of the first-level wavelet transform network is a high-frequency information component, and the output of the first-level inverse wavelet transform network is a high-frequency feature map. The input of other levels of wavelet transform networks is the first-class feature map output by the wavelet transform network of the adjacent level above. The input of each level of inverse wavelet transform is the first-class feature map output by the wavelet transform network of the same level and the second-class feature map output by the inverse wavelet transform network of the next level.

[0053] Electronic devices can obtain high-frequency feature maps corresponding to high-frequency information components by using multi-scale wavelet transform for downsampling and then using inverse wavelet transform for upsampling.

[0054] In one possible implementation of the second aspect, the implementation scheme for the electronic device to extract low-frequency feature maps based on low-frequency information components is further defined. The low-frequency information components acquired by the electronic device contain a large number of low-frequency features, which are redundant and do not need to be enhanced during high-frequency enhancement processing.

[0055] Based on this, electronic devices can extract only the low-frequency features from a portion of the low-frequency information components, thus obtaining the low-frequency features of the first image while reducing the computational load of the electronic device in extracting low-frequency features.

[0056] The electronic device splits the low-frequency information component into a first low-frequency component and a second low-frequency component, extracts a third feature map from the first low-frequency component, and fuses the third feature map and the second low-frequency component to obtain a low-frequency feature map.

[0057] In one possible implementation of the second aspect, both the first low-frequency component and the second low-frequency component are defined to include half of the low-frequency information component.

[0058] The more first low-frequency components an electronic device retains, the more low-frequency information components participate in the extraction of low-frequency feature maps, resulting in more retained low-frequency features and a closer match to the original image. Conversely, the more second low-frequency components an electronic device retains, the more low-frequency information components do not participate in the extraction of low-frequency feature maps, and the less data needs to be calculated.

[0059] Electronic devices divide low-frequency information components into a first low-frequency component and a second low-frequency component. This not only preserves more low-frequency features but also reduces the amount of computation, thereby further improving the quality and efficiency of image processing.

[0060] Thirdly, an electronic device is provided, which includes a memory and a processor, wherein the memory is coupled to the processor;

[0061] The memory stores the instructions that the computer executes;

[0062] The processor executes computer execution instructions stored in memory, causing the electronic device to perform an image processing method such as that described in either the first or second aspect.

[0063] Fourthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, which, when executed on a computer, causes the computer to perform an image processing method as described in either the first or second aspect.

[0064] Fifthly, a computer program product is provided, including a computer program that, when executed by a processor, implements an image processing method as described in either the first or second aspect.

[0065] The technical effects of any of the design methods in aspects two through five can be found in the technical effects of different design methods in aspect one, and will not be repeated here. Attached Figure Description

[0066] Figure 1 A comparative illustration of images captured by the camera of an electronic device in different environments;

[0067] Figure 2 A schematic diagram of the interface involved in the image processing method provided in the embodiments of this application;

[0068] Figure 3 This is another schematic diagram of an interface involved in the image processing method provided in the embodiments of this application;

[0069] Figure 4 A schematic flowchart of the image processing method provided in the embodiments of this application;

[0070] Figure 5 A schematic flowchart illustrating the high-frequency enhancement processing involved in the image processing method provided in the embodiments of this application;

[0071] Figure 6 This is a schematic diagram of the structure of the convolutional neural network involved in the image processing method provided in the embodiments of this application;

[0072] Figure 7 A schematic diagram illustrating the process of extracting high-frequency and low-frequency information components in the image processing method provided in this application embodiment;

[0073] Figure 8 A schematic diagram illustrating the principles of standard convolution and dilated convolution operations involved in the image processing method provided in the embodiments of this application;

[0074] Figure 9 A schematic diagram illustrating the principle of the generative adversarial network involved in the image processing method provided in the embodiments of this application;

[0075] Figure 10 A schematic diagram of the process of obtaining low-frequency feature maps by decomposing low-frequency information components in the image processing method provided in the embodiments of this application;

[0076] Figure 11 A schematic diagram of the software framework and process involved in the image processing method provided in the embodiments of this application;

[0077] Figure 12 This is a hardware structure diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0078] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0079] To facilitate understanding, some technical common sense involved in the embodiments of this application will be introduced first.

[0080] Image quality, or texture, refers to the details and layers of an image.

[0081] Grayscale value refers to the color depth of a point in a black and white image, typically ranging from 0 to 255, where white is 255 and black is 0. A black and white image can also be called a grayscale image. By mapping the brightness values ​​of each pixel in a color image to grayscale values, the corresponding grayscale image can be obtained.

[0082] Noise refers to random, discrete, and isolated pixels in an image caused by factors such as camera sensor noise, image graininess, and transmission errors during image processing. Noise is visually distinct from its neighboring pixels; for example, white dots in a black area, or black dots in a white area.

[0083] The frequency of an image is an indicator of the drastic changes in its grayscale values; it is the gradient of grayscale values ​​in a two-dimensional space. The larger the gradient in an image, or the greater the change in grayscale values ​​of individual pixels, the higher the frequency. More cluttered parts of an image, such as edges, tend to have higher frequencies. Smoother parts of an image, such as a large expanse of sky, tend to have lower frequencies.

[0084] Low-frequency regions refer to areas in an image where the grayscale values ​​of pixels change slowly. The information contained in low-frequency regions of an image can be called low-frequency information components.

[0085] High-frequency regions refer to areas in an image where the grayscale values ​​of pixels change rapidly. These regions contain information that needs to be clearly displayed to show details. The information contained in high-frequency regions of an image can be called high-frequency information components.

[0086] An image frame can be divided into high-frequency information components and low-frequency information components. The high-frequency information components form the edges and details of the image, while the low-frequency information components form the background and subject. During image processing, the low-frequency and high-frequency information in the image can be separated according to the actual processing requirements and the desired level of detail.

[0087] RAW images represent raw images captured by the camera's sensor, that is, images that have not been processed.

[0088] An electronic device is equipped with a camera, which can control the camera to capture images of objects within the camera's field of view in the environment in which the electronic device is located. The environment within the camera's field of view is referred to as the shooting environment.

[0089] A camera includes a sensor, such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor, where all pixels work synchronously. The principle of camera shooting is as follows: an electronic shutter controls the sensor's exposure; all pixels accumulate photons during the shutter's exposure time. When a photon strikes the sensor surface, it is converted into an electron and stored. The stronger the light, the more electrons are generated per unit time, thus allowing the determination of the illuminance at each pixel. When the accumulated electron time reaches a preset time (exposure time), the stored electrons are transferred and read out, completing one exposure cycle. In other words, the ambient brightness of the shooting environment is a significant factor affecting the quality of images captured by the camera.

[0090] Ambient light intensity refers to the ratio of the luminous intensity of a light source in the environment to the area of ​​the light source. The unit of ambient light intensity is candela per square meter (cd / m²). 2 The higher the ambient brightness, the greater the intensity of colors reflected by objects in the environment, and the clearer the images of the objects in the environment appear to the human eye and in the camera. Conversely, the lower the ambient brightness, the lower the intensity of colors reflected by objects in the environment, and the more blurred the images of the objects in the environment appear to the human eye and in the camera.

[0091] Based on ambient brightness, the shooting environment of electronic devices can be divided into bright light environments and low light environments. A bright light environment refers to an environment with brightness greater than or equal to a preset brightness threshold, while a low light environment refers to an environment with brightness less than the preset brightness threshold.

[0092] When electronic devices are photographed in bright light, the camera's pixels accumulate more photons during the shutter's exposure time, resulting in more pixel features collected by each pixel and a clearer image. Conversely, when electronic devices are photographed in dark light, the camera's pixels accumulate fewer photons during the shutter's exposure time, resulting in fewer pixel features collected by each pixel and a blurrier image.

[0093] like Figure 1 The image shown is a schematic diagram of images captured by an electronic device in different shooting environments. Figure 1 As shown in (1) and (2), these are images captured for the same static scene, where (1) is an image captured in bright light and (2) is an image captured in dark light. Compared to images captured in bright light, images captured in dark light have fewer details and lower clarity. For example... Figure 1 Images corresponding to another low-light environment, shown in (3) and (4) in the image, can be real images of the scene. Figure 1 As shown in (3) above, the image captured by the camera is Figure 1As shown in (4) in the figure. Among them, the Z1 pixel region in (3) corresponds to the Z2 pixel region in (4). The Z2 pixel region loses more detailed features than the Z1 pixel region, making the image more blurry.

[0094] From the perspective of the frequency of pixels contained in an image, an image includes high-frequency information components and low-frequency information components. Low-frequency information components correspond to pixel areas in the image where the brightness of objects is uniform or changes slowly, while high-frequency information components correspond to pixel areas in the image where the brightness of objects changes more significantly, especially the edges, details, and noise of feature objects in the image. For example... Figure 1 As shown in (4), the high-frequency information components in the Z2 pixel region are relatively few, resulting in lower image clarity.

[0095] In other words, when the camera of an electronic device is shooting in a low-light environment, the high-frequency features of the captured image are lost, resulting in lower image clarity.

[0096] Based on this, this embodiment provides an image processing method applied to an electronic device. The electronic device performs high-frequency enhancement processing on a first image acquired in low-light or night-scene shooting mode to obtain a second feature map with more high-frequency features. Compared to the first image acquired by the electronic device, the second image has more high-frequency features and higher clarity.

[0097] The image processing method provided in this embodiment can be applied to electronic devices. Electronic devices may include personal computers (PCs), tablet computers, laptops, portable computers (such as mobile phones), wearable electronic devices (such as smartwatches), augmented reality (AR) / virtual reality (VR) devices, in-vehicle computers, and other electronic devices. The following embodiments do not impose any special limitations on the specific form of the electronic device.

[0098] The image processing method provided in this embodiment can be divided into an image acquisition operation and an image processing operation that performs high-frequency enhancement processing on the first image. The electronic device first performs an image acquisition operation to acquire a first image, and then performs an image processing operation to process the acquired first image to obtain a second image. There can be multiple ways for the electronic device to acquire the first image and trigger the image processing operation on the first image.

[0099] In one scenario, the first image acquired by the electronic device can be an image pre-stored by the electronic device.

[0100] For example, an electronic device can acquire a first image from another terminal and store it in the electronic device's photo album, or the electronic device can pre-control its built-in camera to acquire a first image and store it in the electronic device's photo album.

[0101] In this case, the electronic device can respond to the first operation and process the first image.

[0102] In one example, such as Figure 2 As shown in (1), the electronic device displays the first interface (such as...). Figure 2 As shown in (1) U1), the first interface can be the interface of a photo album application on an electronic device. The first interface includes a first image (such as... Figure 2 P1) shown in (1) and multiple processing controls, including a high-frequency enhancement control. Figure 2 As shown in (1), the multiple processing controls displayed on the first interface may include cropping controls, filter controls, adjustment controls, and smart controls, wherein the highlight enhancement control may be a smart control (such as...). Figure 2 A as shown in (1) of the text. Or as shown in the text. Figure 2 As shown in (2), the processing controls displayed on the first interface may include a cropping control, a filter control, an adjustment control, and a low-light enhancement control. Among them, the low-light enhancement control can be a control that triggers the electronic device to perform high-frequency enhancement processing. In specific implementations, the specific name and display method of the high-frequency enhancement control that triggers high-frequency enhancement processing can be implemented in other ways and are not limited.

[0103] like Figure 2 As shown in (1), if the electronic device receives a click operation applied to the high-frequency enhancement control, it can perform high-frequency enhancement processing on the first image to obtain the second image. Figure 2 As shown in (2) above, the interface of the electronic device (such as...) Figure 2 U2, as shown in (2) in the figure, includes a second image (such as... Figure 2 As shown in (2) P2), the detailed features of some pixel regions in the second image are increased. Figure 2 As shown in (3), the electronic device can receive user actions on the save control (such as...). Figure 2 The click operation shown in (3) B) stores the generated second image.

[0104] After the electronic device processes the first image pre-stored in the photo album to obtain the corresponding second image, it can store the second image separately or store the second image by overwriting the first image.

[0105] In another scenario, the first image processed by the electronic device can be an image captured in real time by the camera controlled by the electronic device.

[0106] The electronic device is equipped with a camera. The electronic device can control the camera to acquire a first image. When the first image is acquired, the first image is subjected to high-frequency enhancement processing, and the high-frequency enhanced second image is stored or displayed.

[0107] In one example, such as Figure 3 As shown, the electronic device opens the camera application and displays the shooting preview interface (such as...). Figure 3 (1) U3). The shooting preview interface displayed by the electronic device can have multiple shooting modes, for example, it can be... Figure 3 The shooting mode shown in (1) can also be used for... Figure 3 The night scene shooting mode is shown in (2) above. Of course, the electronic device can also perform the image processing operations provided in this embodiment in video recording mode or other shooting modes, without limitation.

[0108] like Figure 3 As shown in (2), the electronic device responds to the user's action on the night scene shooting control (such as...). Figure 3 Clicking A) as shown in (2) enters the night scene shooting mode and captures images within the field of view. Figure 3 The interface shown in (3) (as shown in the image) Figure 3 (3) As shown in U4), if the electronic device receives a user action on the camera control (such as...) Figure 3 The click operation shown in (3) B) allows for the acquisition of a first image frame. The electronic device then performs high-frequency enhancement processing on the first image to obtain the second image. Figure 3 As shown in (4) above, the electronic device can be used in the shooting interface (such as...) Figure 3 The second image obtained after image processing is displayed on U5) as shown in (4) in the image processing diagram. In other cases, the electronic device may also store the second image obtained after image processing in the album, or the electronic device may also update the thumbnail of the second image displayed in the album thumbnail display area of ​​the shooting preview interface. If the electronic device receives a click operation from the user on the album thumbnail display area, it can display the second image.

[0109] In another example, upon receiving a click operation on the shooting preview control, the electronic device enters various shooting modes, including photo, video, night scene, low light, and portrait modes. In each shooting mode, the electronic device can utilize the ambient light sensor to collect the ambient light intensity value of the shooting environment. The electronic device compares the ambient light intensity value with a preset low light threshold. If the ambient light intensity value is less than or equal to the preset low light threshold, the current shooting environment is determined to be a low light shooting environment. In one scenario, the electronic device can enter either the low light or night scene shooting mode, capture a first image in either mode, and then perform high-frequency enhancement processing on the first image to obtain a corresponding second image. In other scenarios, the electronic device can perform high-frequency enhancement processing on the first image captured in the current shooting mode without switching shooting modes to obtain a corresponding second image.

[0110] In other examples, the electronic device may also perform image processing operations on any frame of the first image captured, regardless of the currently running shooting mode, so that the final generated second image contains more high-frequency information.

[0111] See Figure 4 This is a flowchart illustrating the image processing method provided in this embodiment. The high-frequency enhancement processing involved in the provided image processing method mainly includes the following steps:

[0112] S41: The electronic device uses a convolutional neural network to obtain the high-frequency information components of the first image, and obtains the low-frequency information components of the first image based on the high-frequency information components of the first image.

[0113] Each pixel in the first image possesses corresponding feature parameters, which can be of various types, such as brightness, grayscale, or chroma values. The feature parameters of each pixel enable it to display a corresponding visual effect, and the feature parameters of all pixels contribute to the overall display effect of the first image.

[0114] Different pixels may have the same or different feature parameters. In the first image, if the feature parameters of pixels in some pixel regions are the same or similar, the visual effect of these pixel regions is more uniform or less varied. In other words, the grayscale values ​​of pixels in these pixel regions change less frequently and contain more low-frequency information.

[0115] In the first image, if the feature parameters of pixels in certain pixel regions differ significantly, the visual effect of these pixel regions is less consistent or more variable. In other words, the grayscale values ​​of pixels in these pixel regions change more frequently, containing more high-frequency information.

[0116] Based on this, electronic devices can determine the first pixel region corresponding to the high-frequency information component by acquiring pixel regions with significantly different feature parameters between adjacent pixels in the first image, and then extract the high-frequency information component from the first pixel region of the first image. For example... Figure 5 The diagram shown illustrates another process for high-frequency enhancement processing of the first image by an electronic device. Figure 4 and Figure 5 As shown, the electronic device acquires the high-frequency information components of the first image, mainly through two steps:

[0117] S411, the electronic device performs local variance statistics on the first image and obtains the first variance result.

[0118] Variance is a statistical indicator that measures the dispersion of data. A smaller variance means that the data points are more concentrated, the distribution is narrower, the fluctuation is smaller, and the data is more stable. Conversely, a larger variance means that the data points are more widely distributed, the fluctuation is greater, and the data is less stable. The electronic device performs local variance statistics on the first image to determine the degree of deviation between the feature parameters of pixels in each local region of the first image. The feature parameters of the pixels mentioned here can include the brightness value or grayscale value of the pixels.

[0119] The electronic device performs local variance statistics on the first image. If the variance of a pixel is larger, it means that the feature parameters of that pixel deviate more from the average feature parameters of the pixel region, and therefore the pixel contains more high-frequency information. Conversely, if the variance of a pixel is smaller, it means that the feature parameters of that pixel deviate less from the average feature parameters of the pixel region, and therefore the corresponding high-frequency information is less, while low-frequency information is more.

[0120] In other words, the variance of each pixel is positively correlated with the high-frequency information contained in that pixel. Based on this, an electronic device can use local variance statistics to calculate the variance of each pixel within the first pixel, and use the variance of all pixels in the first image as the first variance result. The first variance result obtained by the electronic device can include the correspondence between the coordinates of each pixel and its variance.

[0121] In one specific implementation, such as Figure 5 As shown, the electronic device can construct an adaptive local difference operator to statistically analyze the local variance of the first image by sliding a window.

[0122] The electronic device constructs an adaptive local difference operator, which has the ability to perform feature parameter difference statistics on each pixel within each pixel region of the input image. The electronic device inputs the first image into the adaptive local difference operator and calculates the local variance of each pixel within each pixel region. Pixels with larger variances contain more high-frequency information, while pixels with smaller variances contain less high-frequency information.

[0123] S412, the electronic device uses the first variance result as prior information to input the first image into the convolutional neural network to obtain the high-frequency information components of the first image.

[0124] The first variance result acquired by the electronic device indicates the variance of each pixel in the first image, and the variance of each pixel is positively correlated with the high-frequency information contained in each pixel. By using the first variance result of the first image as prior information, the electronic device can acquire the high-frequency information components of the first image more quickly and accurately.

[0125] In one scenario, electronic devices can directly use the first variance result as prior information.

[0126] In another scenario, the electronic device can also pre-set a variance threshold. The electronic device assigns pixels with a variance greater than or equal to the variance threshold to a first pixel region, and pixels with a variance less than the variance threshold to a second pixel region. The first pixel region determined by the electronic device is a pixel region containing high-frequency information, and the second pixel region is a pixel region containing low-frequency information.

[0127] An electronic device loads a Convolutional Neural Network (CNN), which has the ability to extract high-frequency information from an input image. Specifically, for example... Figure 6 As shown, a convolutional neural network (CNN) mainly consists of convolutional layers and activation layers. The convolutional layers include a first convolutional layer and a second convolutional layer arranged side-by-side, with the outputs of both layers connected to the inputs of the activation layers. Furthermore, the CNN also includes an input layer and an output layer. The input layer is connected to both the first and second convolutional layers, and the activation layers are connected to the output layer. In other words, a CNN sequentially comprises an input layer, a first convolutional layer, a second convolutional layer, an activation layer, and an output layer.

[0128] The first and second convolutional layers in a convolutional neural network are different convolutional networks. A convolutional network includes one or more convolutional kernels, and the parameters involved in the convolution operation of the kernels can include kernel size, weights, stride, etc.

[0129] A convolution kernel, also known as a convolution template or convolution window, is an operation matrix. The kernel size refers to the number of rows and columns in the operation matrix, the weights are the sum of the operational values ​​of each element in the matrix, and the stride is the number of pixels the kernel moves during the operation. The kernel size determines the range of the convolution operation, the weights determine the result of the convolution operation, and the stride determines the number of convolution operations.

[0130] The process of an electronic device performing convolution operations on a first image using a convolutional network involves inputting the first image into the convolutional network. The convolutional window of the convolutional kernel slides sequentially across the first image, determining a pixel region for convolution calculation at each slide. The pixels within this region correspond to elements of the convolutional kernel. The electronic device calculates a weighted sum based on the weights of each pixel within this region and its corresponding element to obtain a new grayscale value. This new grayscale value is then used as the grayscale value of the center pixel of this region and mapped onto a new feature map. The electronic device continues this process of sliding and convolution operations on the first image until the convolutional window of the kernel covers the entire pixel region of the first image. The resulting feature map is the feature map obtained after performing the convolution operation on the first image.

[0131] In this embodiment, the convolutional network includes a first convolutional layer and a second convolutional layer, which are different convolutional networks. These different convolutional networks can be understood as having different or partially different numbers of convolutional kernels, kernel sizes, weights, or strides in the first and second convolutional layers. This results in different convolutional operations performed by the first and second convolutional layers on the same input first image, leading to different output feature maps.

[0132] like Figure 7 The diagram illustrates how an electronic device processes a first image using a convolutional neural network. The electronic device inputs the first image x, and the first variance result σ2(x) of the first image is used. The feature map output by the first convolutional layer is the first feature map F. c1 (x), where the feature map output by the second convolutional layer is F. C2 (x).

[0133] In one example, in the convolutional neural network loaded by the electronic device, the convolutional kernels of the first and second convolutional layers are both standard convolutional kernels, and their convolution operation principle is as follows: Figure 8 As shown in (1) of the table.

[0134] Figure 8 In (1), the convolution kernel size is 3*3, and the corresponding elements are a1, a2...a8 and a9. The pixel region size of the first image is 9*6, and the pixels included in the first image are b11, b12...b68 and b69 in sequence.

[0135] Figure 8 In (1), the process of convolution operation performed by the electronic device includes: the pixel region corresponding to the first convolution operation performed by the electronic device with the convolution kernel includes the following pixels: b11, b12, b13, b21, b22, b23, b31, b32 and b33. The first convolution operation obtains the updated grayscale value c11 corresponding to the center pixel b22, where:

[0136] C11=a1*b11+a2*b12+a3*b13+a4*b21+a5*b22+a6*b23+a7*b31+a8*b32+a9*b33.

[0137] Figure 8 In (2), the electronic device slides the convolution window of the convolution kernel to perform the next convolution operation. The corresponding pixel region includes the following pixels: b12, b13, b14, b22, b23, b24, b32, b33, and b34. The updated grayscale value corresponding to the center pixel b23 obtained in this convolution operation is c12, where:

[0138] c12=a1*b12+a2*b13+a3*b14+a4*b22+a5*b23+a6*b24+a7*b32+a8*b33+a9*b34.

[0139] In another example, in the convolutional neural network loaded on an electronic device, the first and second convolutional layers can also perform dilated convolution operations on the input first image using convolutional kernels. Dilated convolution is an algorithm that adds a receptive field to a convolutional network. By introducing holes (or gaps) into the standard convolutional kernel, it expands the receptive field of the convolutional network without increasing the number of parameters or using pooling layers. Compared to standard convolution, dilated convolution adds a parameter called dilation rate, which refers to the spacing between data objects when the convolutional kernel processes data, or the gaps between pixels. Thus, when an electronic device uses dilated convolution for convolution operations, the convolutional kernel is not applied to the input first image perfectly, but rather by skipping fixed pixel gaps.

[0140] The principle of dilated convolution operation in electronic devices can be as follows: Figure 8 As shown in (3) and (4) in the example. In this example, the kernel size is still 3*3, the pixel area size of the first image is still 9*6, and the dilatancy of the kernel is 1, which means that 1 pixel gap is skipped during the convolution operation.

[0141] Specifically, Figure 8 In (3), the process of the electronic device performing the first dilated convolution operation includes: the pixel points included in the pixel region corresponding to the convolution kernel are: b12, b14, b16, b32, b34, b36, b52, b54, and b56. The electronic device performs the first dilated convolution operation to obtain the updated grayscale value d11 corresponding to the center pixel b34, where:

[0142] d11=a1*b12+a2*b14+a3*b16+a4*b32+a5*b34+a6*b36+a7*b52+a8*b54+a9*b56.

[0143] Figure 8 In step (4), the electronic device slides the convolution window of the convolution kernel to perform the next dilated convolution operation. The corresponding pixel region includes the following pixels: b13, b15, b17, b33, b35, b37, b53, b55, and b57. The updated grayscale value corresponding to the center pixel b35 obtained in this convolution operation is d12, where:

[0144] d12=a1*b13+a2*b15+a3*b17+a4*b33+a5*b35+a6*b37+a7*b53+a8*b55+a9*b57.

[0145] The parameters involved in the convolutional neural networks used in electronic devices can include kernel size, weights, stride, and dilation rate. Different values ​​of one or more of these parameters will cause the convolutional layer to perform different convolution operations on the input image, resulting in different output feature maps.

[0146] In this embodiment, the neural network loaded by the electronic device has different kernel sizes and dilation rates for the first and second convolutional layers. The electronic device inputs a first image into the first and second convolutional layers respectively. The first and second convolutional layers perform different dilated convolution operations on the first image, and the first feature map output by the first convolutional layer is also different from the feature map output by the second convolutional layer. It should be noted that the first feature map obtained by the electronic device after performing dilated convolution operation on the first image using the first convolutional layer and the second feature map obtained by performing dilated convolution operation on the second image using the second convolutional layer have the same pixel size. This allows for the determination of whether each pixel contains high-frequency or low-frequency features based on the first and second feature maps, thereby obtaining more accurate high-frequency information components.

[0147] In practice, the kernels and dilation rates of the first and second convolutional layers are different, so the pixel sizes of the actual output feature maps may differ. Electronic devices can also add a zero-padding operation to pad the feature maps output by the first and second convolutional layers with zeros, ensuring that the pixel sizes of the first and second feature maps are the same. In other cases, electronic devices can directly limit the pixel sizes of the input and output images of the first and second convolutional layers, making the pixel sizes of the input and output feature maps of each convolutional layer consistent. Therefore, by inputting the same first image into the first and second convolutional layers respectively, first and second feature maps with the same pixel size can be obtained.

[0148] The first image includes high-frequency information components and low-frequency information components. The electronic device performs different convolution operations on the first image. After different convolution operations, the differences in high-frequency information components increase, while the differences in low-frequency information components decrease. By performing different convolution operations on the same first image, the electronic device effectively preserves the high-frequency information components with large pixel differences in the first image.

[0149] Based on this, the electronic device then inputs the first feature map and the second feature map into the activation layer to perform contrast calculation pixel by pixel, thereby obtaining the high-frequency information components of the first image.

[0150] In one example, continue as follows Figure 7 As shown, the activation function of the convolutional neural network used in the electronic device can be the Sigmoid function. The Sigmoid function essentially compresses a real number to the range of 0 to 1. When z is a very large positive number, g(z) approaches 1, while when z is a very small negative number, g(z) approaches 0. The electronic device performs contrast calculation pixel by pixel by inputting the first feature map and the second feature map into the activation layer. The greater the pixel difference between the two feature maps, the greater the contrast and the more high-frequency information; the smaller the pixel difference, the smaller the contrast and the less high-frequency information. In this way, the electronic device can obtain the high-frequency information component F of the first image. H , of which F H =Sigmoid[F c1 (x)-F C2 (x)].

[0151] When an electronic device uses an activation layer to obtain high-frequency information components, it can set a contrast threshold, retain pixel features in the first image whose pixel contrast is greater than the contrast threshold, and discard pixel features whose pixel contrast is less than or equal to the contrast threshold, thereby obtaining the high-frequency information components of the first image.

[0152] S413, the electronic device performs an inverse operation on the high-frequency information component to obtain the inverse mapping of the high-frequency information component, and obtains the low-frequency information component of the first image based on the inverse mapping of the first image and the high-frequency information component.

[0153] The pixel features of the first image can be divided into high-frequency information components and low-frequency information components. Once the electronic device determines the high-frequency information components, it can reversely determine the low-frequency information components of the first image.

[0154] In one example, the electronic device targets the high-frequency information component F. H Perform the inverse operation to obtain the inverse mapping. Inverse mapping Multiplying it by the first image x yields the low-frequency information component F of the first image. L ,

[0155] S42: The electronic device extracts a high-frequency feature map based on the high-frequency information component, and extracts a low-frequency feature map based on the low-frequency information component.

[0156] After acquiring the high-frequency and low-frequency information components of the first image, the electronic device performs operations to extract high-frequency feature maps from the high-frequency information components and low-frequency feature maps from the low-frequency information components, respectively. Specifically, the steps include:

[0157] S421: The electronic device inputs high-frequency information components into the generative adversarial network to obtain a high-frequency feature map.

[0158] Electronic devices load a Generative Adversarial Network (GAN) to extract high-frequency feature maps from high-frequency information components. Specifically, for example... Figure 9 As shown in (1), a Generative Adversarial Network (GAN) consists of a generator (or generative model) and a discriminator (or discriminator network). The generator learns the features of the training set data and, under the guidance of the discriminator, tries to fit the random noise distribution as closely as possible to the true distribution of the training data, thus generating similar data with the features of the training set. The discriminator is responsible for distinguishing whether the input data is real or fake data generated by the generator and feeding this information back to the generator. The two networks are trained alternately, and their capabilities improve synchronously until the data generated by the generative network can be indistinguishable from real data and reaches a certain balance with the capabilities of the discriminator network.

[0159] The training process of a generative adversarial model typically includes: fixing the parameters of the generator and training the parameters of the discriminator. First, some vectors are randomly input, and the generator produces random training values. Then, some real values ​​are retrieved from a real database, and both the training and real values ​​are used as input to the discriminator to train it. The goal of training is to enable the discriminator to distinguish between training and real values, assigning a score as close to 0 as possible to training values ​​and as close to 1 as possible to real values. Afterward, the discriminator parameters are fixed, and the generator parameters are adjusted. Specifically, a random input value is fed into the generator to produce an image, which is then fed into the discriminator to obtain a score. The goal is to ensure that the final output score of the discriminator is close to 1. This embodiment uses a generative adversarial network to extract high-frequency feature maps of high-frequency information components to achieve high-frequency feature enhancement.

[0160] In this embodiment, the generator is based on multi-scale wavelet transform (WT). It utilizes the characteristics of multi-scale wavelet transform to obtain the frequency and spatial location information of the feature map, thereby further improving high-frequency details.

[0161] The generative adversarial network (GAN) includes a generator and a discriminator. The generator comprises a multi-level processing network, each of which includes a wavelet transform network and an inverse wavelet transform network of the same scale and symmetrically arranged. The generator sequentially includes multiple wavelet transform networks with progressively decreasing scales and inverse wavelet transform networks with progressively increasing scales. The input to the first-level wavelet transform network is a high-frequency information component, and the output of the first-level inverse wavelet transform network is a high-frequency feature map. The input to other levels of wavelet transform networks is the first-class feature map output by the wavelet transform network of the adjacent level above. The input to each level of inverse wavelet transform is the first-class feature map output by the wavelet transform network of the same level and the second-class feature map output by the inverse wavelet transform network of the next level below.

[0162] like Figure 9 As shown in (2), the generator includes multiple pairs of networks. Each pair of networks includes a wavelet transform network and an inverse wavelet transform network of the same scale, and the scales of adjacent wavelet transform networks are different. Each wavelet transform network includes a downsampling layer, a convolutional layer, a normalization layer, and an activation layer. Correspondingly, each inverse wavelet transform network also includes an upsampling layer, a convolutional layer, a normalization layer, and an activation layer.

[0163] The electronic device will transfer the high-frequency information components (F) of the first image. H The input is processed by the first-level wavelet transform network, which then performs downsampling through the downsampling layer, convolution operation through the convolution layer, normalization processing through the normalization layer, and finally processing through the activation function of the activation layer to obtain the first type of feature map (T1).

[0164] The electronic device inputs the first type of feature map (T1) into a second-level wavelet transform network with reduced scale and a first-level inverse wavelet transform network with the same scale. The electronic device then inputs the first feature map into the second-level wavelet transform network, performing downsampling, convolution, normalization, and activation function processing in sequence to obtain the first type of feature map (T2).

[0165] The electronic device then inputs the first type of feature map (T2) into a third-level wavelet transform network with reduced scale and a second-level inverse wavelet transform network with the same scale. The electronic device then inputs the second feature map into the third-level wavelet transform network, and performs downsampling, convolution, normalization, and activation function processing in sequence to obtain the first type of feature map (T3).

[0166] The electronic device inputs the third type feature map (T3) into the fourth-level wavelet transform network with a reduced scale and the third-level inverse wavelet transform network with the same scale, and so on, until the Nth-level wavelet transform network with the smallest scale outputs the first feature map (Tn).

[0167] The electronic device inputs the first feature map (Tn) into the Nth-level inverse wavelet transform network with the same scale. The Nth-level inverse wavelet transform network processes the Nth-class feature map sequentially as follows: upsampling layer performs upsampling, convolutional layer performs convolution operation, normalization layer performs normalization processing, and activation layer performs activation function processing to obtain the second-class feature map (Tn).

[0168] The electronic device inputs the second type of feature map into a (N-1)th level inverse wavelet transform network with increased input scale. The (N-1)th level inverse wavelet transform network combines the first type of feature map (Tn-1) and the second type of feature map (Tn) and performs upsampling, convolution, normalization and activation function processing respectively to obtain the second type of feature map (Tn-1).

[0169] The electronic device then transmits the second type of feature map (Tn-1) to the next level of the inverse wavelet transform network at the (N-2)th level with an increased scale. This process is repeated. The second-level inverse wavelet transform network obtains the first type of feature map (T2) and the second type of feature map (T3), and performs upsampling, convolution, normalization, and activation function processing in sequence to obtain the second type of feature map (T2).

[0170] The electronic device transmits the second type of feature map (T2) to the first-level inverse wavelet transform network. The first-level inverse wavelet transform network combines the first type of feature map (T1) and the second type of feature map (T2), and sequentially performs upsampling, convolution, normalization, and activation function processing to obtain the first and second type of feature maps. At this point, the electronic device can obtain the high-frequency feature maps corresponding to the high-frequency information components of the first image.

[0171] The electronic device uses a multi-scale wavelet transform network included in the generator to extract high-frequency features. The discriminator then discriminates the output high-frequency features and feeds the discrimination results back to the generator to optimize its parameters. After multiple rounds of feedback training, the electronic device can obtain an enhanced high-frequency feature map.

[0172] In practice, electronic devices can use the ESRGAN loss function as the loss function for the generative adversarial network during training, while the discriminator uses the Visual Geometry Group Network (VGGNet) to obtain more accurate high-frequency feature maps.

[0173] S422: The electronic device splits the low-frequency information component into a first low-frequency component and a second low-frequency component, extracts a third feature map from the first low-frequency component, and fuses the third feature map and the second low-frequency component to obtain a low-frequency feature map.

[0174] Electronic devices extract low-frequency features from low-frequency information components and perform calculations using only a portion of these low-frequency information components. Specifically, for example... Figure 10 As shown, the electronic device performs channel separation on the low-frequency information components to obtain a first low-frequency component and a second low-frequency component. The first low-frequency component is used to extract low-frequency features, while the second low-frequency component is used for feature fusion. The electronic device can split the low-frequency information components in half, meaning that both the first and second low-frequency components are half of the total low-frequency information components. In other embodiments, the electronic device can also split the low-frequency information components at other ratios, as long as the splitting ratio satisfies the requirement of reducing the computational load of low-frequency feature extraction while preserving the accuracy of low-frequency features. For example, the electronic device can also split one-third of the low-frequency information components into the first low-frequency component and use the remaining two-thirds as the second low-frequency information component, thus reducing the computational load of low-frequency feature extraction. As another example, the electronic device can also split two-thirds of the low-frequency information components into the first low-frequency component and use the remaining one-third as the second low-frequency component, thus increasing the accuracy of low-frequency features.

[0175] The electronic device inputs the first low-frequency component into a Convolutional Neural Network (CNN) to extract low-frequency features from this component, denoted as the first feature. Specifically, the electronic device uses an attention mechanism (CNN) to extract the low-frequency features. The electronic device then superimposes the first feature and the second low-frequency component to obtain a feature map containing both low-frequency information and features; this is the low-frequency enhanced feature.

[0176] S43: The electronic device uses a feature fusion network to process high-frequency feature maps and low-frequency feature maps to generate a second image corresponding to the first image.

[0177] An electronic device loads a pre-trained feature fusion network, which is capable of outputting a high-frequency enhanced image based on high-frequency and low-frequency feature maps. The electronic device inputs the high-frequency and low-frequency feature maps of the first image obtained in the preceding steps into the feature fusion network to obtain a new image, denoted as the second image. The electronic device processes the second image obtained from the first image, enhancing its high-frequency features relative to the first image.

[0178] The electronic device applies the solutions described in S41-S43 to the aforementioned... Figure 3 The image shown can be acquired from a scene and processed in real time using RAW images captured by the camera of an electronic device in low-light conditions. This process enhances high-frequency features, yields images with more detailed features, and improves the quality of the images output by the electronic device.

[0179] The pre-trained feature fusion network used in electronic devices is essentially a network obtained by iteratively training a neural network using samples. A neural network is a network formed by connecting multiple individual neural units together. The output of one neural unit can be the input of one or more other neural units. The input of each neural unit can be connected to the local receptive field of the previous layer to extract features from that local receptive field, which can be a region composed of several neural units.

[0180] Electronic devices used to train feature fusion networks typically include Deep Neural Networks (DNNs) or Convolutional Neural Networks (CNNs). Deep Neural Networks, also known as multi-layer neural networks, can be understood as neural networks with multiple hidden layers. Based on the position of different layers, DNNs can be divided into three categories: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the multiple layers in between are hidden layers. Layers can be fully connected; that is, any neuron in the i-th layer can be connected to any neuron in the (i+1)-th layer.

[0181] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers; this feature extractor can be viewed as a filter. A convolutional layer is a layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to a subset of neurons in its neighboring layers. A convolutional layer typically contains several feature planes, each of which can consist of a series of rectangularly arranged neural units.

[0182] Electronic devices typically require multiple sets of sample images for network training. Each set includes two types of images: input sample images containing features to be classified, and target sample images containing target features. It's important to note that the features to be classified refer to those requiring recognition, classification, and labeling, such as features exhibiting characteristics like defocus or motion blur. Target features are the specific types or categories of features that need to be identified or processed, such as features indicating focus on specific pixel areas or sharpness. The image content of the input and target sample images is correlated.

[0183] In this system, the input sample image serves as the input value to the neural network, while the target sample image serves as the target value. The input sample image contains various basic elements that the image to be processed by the feature fusion network may include. The target sample image can be an image obtained by processing the input sample image according to user requirements, such as an out-of-focus image or an image with motion blur. This target sample image may be obtained by removing unwanted, unclear, or motion-blurred elements from the input sample image. In other words, the elemental content of the input sample image and the target sample image are basically the same, but their feature attributes may differ.

[0184] In this embodiment, the electronic device can train a neural network to obtain a feature fusion network, and then use the feature fusion network to perform image processing operations. Alternatively, the electronic device can directly acquire feature fusion networks trained by other devices and use them to execute the image processing method provided in this embodiment.

[0185] The process of training a neural network in an electronic device to obtain a feature fusion network mainly includes: the electronic device acquiring sample images and a neural network without setting relevant parameters of the weight matrix; inputting the sample images into the neural network for iterative training; gradually optimizing the weight matrix in the neural network according to the loss function until the loss function converges, so that the weight matrix in the neural network reaches the optimal value; at this point, the neural network can be used as a feature fusion network.

[0186] like Figure 5 As shown in (2), the electronic device performs high-frequency and low-frequency feature fusion through a feature fusion network consisting of two convolutional layers, and finally calculates the L1 loss function with the output and the ground truth image.

[0187] Specifically, the electronic device uses a loss function to calculate the difference between the predicted value and the true value, and then uses the difference to backpropagate and modify the weight matrix. After that, the training sample data is input into the neural network after the weight matrix is ​​modified, and multiple iterations of training are performed until the difference is minimized, that is, the neural network converges.

[0188] When a neural network iterates through input and target values, it employs a backpropagation algorithm to correct the parameters of the initial weight matrix during training, minimizing the reconstruction error loss. Specifically, the difference between the predicted and target values ​​obtained in each iteration is calculated and then propagated back into the neural network to modify the weight matrix. The modified neural network is then iterated through again, and the difference from each iteration is used to update the weight matrix. This process of iterative training and weight matrix modification continues until the difference between the predicted and target values ​​falls within the acceptable loss range of the loss function, indicating convergence. This is how neural network convergence is achieved through multiple iterative training iterations. The backpropagation algorithm, primarily driven by error loss, is used to obtain the optimal parameters of the neural network model. Of course, other similar algorithms can also be used for iterative optimization, without limitation.

[0189] The above embodiments explain the specific implementation process of the image processing method from the perspective of electronic devices. The following will explain the specific implementation process of the image processing method from the perspective of the internal software architecture of electronic devices.

[0190] like Figure 11 The diagram shown illustrates the internal hardware and software architecture of the electronic device. The following section will explain in detail the image processing flow of the electronic device, taking into account its internal architecture.

[0191] Specifically, the internal architecture of an electronic device can be divided into four layers, from top to bottom: the application layer (APP), the framework layer (FWK), the hardware abstraction layer (HAL), and the kernel layer (or driver layer). It should be noted that in addition to these main functional layers, other functional modules may also be included, without limitation.

[0192] The application layer may include a series of application packages, such as the camera application involved in this embodiment. In addition, the application layer also includes applications such as a gallery and image processing applications with camera functionality. Application packages may also include applications such as calling, calendar, maps, navigation, music, video, and text messaging.

[0193] The framework layer provides the Application Programming Interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0194] The framework layer runs a camera service, which can be called by camera applications to implement shooting-related functions. In addition, the framework layer may include a window manager, content providers, a view system, a resource manager, and a notification manager. The window manager manages window applications. It can obtain the screen size, determine if a status bar is present, lock the screen, and capture the screen. The content provider stores and retrieves data, making this data accessible to applications. Data can include video, images, and audio. The view system includes visual controls, such as controls for displaying text and images. The view system can be used to build applications. The display interface can consist of one or more views. The resource manager provides applications with various resources, such as localized strings, icons, images, layout files, video files, etc. The notification manager allows applications to display notifications in the status bar. These notifications can be used to convey informational messages and can disappear automatically after a short pause without user interaction. For example, notifications can be used to announce download completion or message alerts. Notifications can also appear as icons or scrollbar text in the system's top status bar, such as notifications from background applications, or as dialog windows on the screen. Notifications can include text messages displayed in the status bar, sound alerts, vibrations from the terminal, flashing indicator lights, etc. It should be noted that the camera application can also invoke content providers, resource managers, notification managers, window managers, view systems, etc., according to actual business needs; this embodiment does not impose any restrictions on this.

[0195] The kernel layer is the layer between hardware and software. The kernel layer contains at least a camera driver. This camera driver can be used to drive hardware modules with shooting capabilities, such as a camera sensor. In other words, the camera driver is responsible for data interaction with the camera sensor. The kernel layer may also include display drivers, audio drivers, sensor drivers, etc., but this embodiment does not impose any limitations on this.

[0196] The Hardware Abstraction Layer (HAL) can encapsulate drivers in the kernel layer and provide an interface for the framework layer to call them, shielding the underlying hardware implementation details. For example... Figure 11 As shown, the aforementioned hardware abstraction layer may include a camera call processing module (Camera HAL), etc. The camera call processing module is the core software framework of the camera, and it includes an interface module, a sensor call module (Sensor Node), a high-frequency enhancement processing module, etc. These interface module, sensor call module, and high-frequency enhancement processing module are components in the image data and control command transmission pipeline within the camera call processing module.

[0197] Specifically, the sensor module can be a control node for the camera sensor, which can control the camera sensor through the camera driver. The interface module can be a software interface for the application framework layer, used for data interaction with the application framework layer. Of course, the interface module can also interact with high-frequency enhancement processing modules, etc., in the camera call processing module.

[0198] like Figure 11 As shown, the camera application can receive the user's first operation, pass the user-selected camera shooting mode (such as night scene shooting mode) to the camera service in the framework layer, and then the camera service passes it to the sensor invocation module through the interface module of the hardware abstraction layer. The camera application can also receive the user's second operation, generate a shooting command from the user-triggered photo-taking operation, and pass it to the camera service in the framework layer, which then passes it to the sensor invocation module through the interface module of the hardware abstraction layer.

[0199] Upon receiving the second operation, the camera application generates a photo-taking request and sends it to the camera invocation module, which in turn sends it to the sensor invocation module via the interface module. The sensor invocation module can then use the camera driver to invoke the camera sensor to acquire raw image data, i.e., the first image. The first image is then transmitted to the high-frequency enhancement processing module for high-frequency enhancement processing to obtain the second image.

[0200] The high-frequency enhancement processing module uploads the second image to the camera service for image rendering processing, and then transmits it to the camera application to display the second image.

[0201] Understandable Figure 11 The layers in the illustrated software structure and the components contained in each layer do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer layers than illustrated, and each layer may include more or fewer components; this application does not impose any limitations.

[0202] Furthermore, it is understood that the electronic device, in order to implement the image processing method of this embodiment, includes hardware and / or software modules that perform the respective functions. Based on the algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of this application.

[0203] The image processing method provided in this embodiment uses an adaptive local difference descriptor as prior information to input the first image into a frequency domain separation module to extract high-frequency information components. These high-frequency information components are then input into the high-frequency generation branch of a generative adversarial network based on multi-scale wavelet transform to extract high-frequency feature maps. Furthermore, the electronic device performs an inverse operation on the high-frequency information components and multiplies them with the first image to obtain low-frequency information components. A portion of the low-frequency information components is then separated through channel separation for low-frequency feature map extraction, thus reducing computational load while preserving low-frequency features. Subsequently, the electronic device inputs the extracted high-frequency and low-frequency feature maps into a feature fusion network for fusion to obtain a second image corresponding to the first image. This scheme uses multi-scale wavelet transform downsampling to construct a UNet for high-frequency information feature extraction. This significantly increases the content of high-frequency features and improves the clarity of images acquired in low-light or night scene shooting modes.

[0204] Furthermore, embodiments of this application also provide an electronic device, including a memory and a processor;

[0205] The memory stores the instructions that the computer executes;

[0206] The processor executes computer execution instructions stored in the memory, causing the electronic device to perform the image processing method provided in the above embodiments. In addition to these main components, the electronic device also includes components for implementing basic functions, which will be discussed below. Figure 12 Please provide a detailed explanation.

[0207] like Figure 12 The diagram shown is a structural schematic of an electronic device 1200 provided in an embodiment of this application. The electronic device 1200 may include a processor 1210, an external memory interface 1220, an internal memory 1221, a Universal Serial Bus (USB) interface 1230, a charging management module 1240, a power management module 1241, a battery 1242, antenna 1, antenna 2, a mobile communication module 1250, a wireless communication module 1260, an audio module 1270, a speaker, a receiver, a microphone, a headphone jack, a sensor module 1280, buttons 1290, a motor 1291, an indicator 1292, a camera 1293, a display screen 1294, and a SIM card module 1295, etc. The sensor module 1280 may include a pressure sensor 1280A, a gyroscope sensor 1280B, an ambient light sensor 1280C, a touch sensor 1280D, etc.

[0208] The illustrated structure of this embodiment does not constitute a limitation on the electronic device 1200. It may include more or fewer components than illustrated, or combine or separate certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of both.

[0209] Processor 1210 may include one or more processing units. For example, processor 1210 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0210] The aforementioned controller can be the decision-maker that directs the various components of the electronic device 1200 to work in a coordinated manner according to instructions. It is the nerve center and command center of the electronic device 1200. The controller generates operation control signals based on the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0211] The processor 1210 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 1210 is a cache memory, which can store instructions or data that the processor 1210 has just used or that are used repeatedly. If the processor 1210 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 1210, and thus improves the efficiency of the system.

[0212] In some embodiments, the processor 1210 may include interfaces. These interfaces may include an Inter-Integrated Circuit (I2C) interface, an Inter-Integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver / Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI) interface, a General Purpose Input / Output (GPIO) interface, a SIM interface, and / or a USB interface, etc.

[0213] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDL) and a serial clock line (SCL). In some embodiments, the processor 1210 may include multiple I2C buses. The processor 1210 can couple to the touch sensor 1280D, charger, flash, camera 1293, etc., through different I2C bus interfaces. For example, the processor 1210 can couple to the touch sensor 1280D through the I2C interface, enabling the processor 1210 and the touch sensor 1280D to communicate through the I2C bus interface, thereby realizing the touch function of the electronic device 1200.

[0214] The I2S interface can be used for audio communication. In some embodiments, the processor 1210 may include multiple I2S buses. The processor 1210 can be coupled to the audio module 1270 via the I2S bus to enable communication between the processor 1210 and the audio module 1270. In some embodiments, the audio module 1270 can transmit audio signals to the wireless communication module 1260 via the I2S interface to enable the function of answering phone calls through a Bluetooth headset.

[0215] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 1270 and the wireless communication module 1260 can be coupled via the PCM bus interface. In some embodiments, the audio module 1270 can also transmit audio signals to the wireless communication module 1260 via the PCM interface, enabling the function of answering phone calls through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication, but the two interfaces have different sampling rates.

[0216] The UART interface is a universal serial data bus used for asynchronous communication. This bus is bidirectional, converting the data to be transmitted between serial and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 1210 and the wireless communication module 1260. For example, the processor 1210 communicates with a Bluetooth module via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 1270 can transmit audio signals to the wireless communication module 1260 via the UART interface to enable music playback through Bluetooth headphones.

[0217] The MIPI interface can be used to connect the processor 1210 to peripheral devices such as the display screen 1294 and the camera 1293. The MIPI interface includes a Camera Serial Interface (CSI) and a Display Serial Interface (DSI). In some embodiments, the processor 1210 and the camera 1293 communicate via the CSI interface to enable the electronic device 1200 to perform its shooting function. The processor 1210 and the display screen 1294 communicate via the DSI interface to enable the electronic device 1200 to perform its display function.

[0218] The GPIO interface is configurable via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 1210 to a camera 1293, a display screen 1294, a wireless communication module 1260, an audio module 1270, a sensor module 1280, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0219] USB port 1230 can be a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 1230 can be used to connect a charger to charge electronic device 1200, or to transfer data between electronic device 1200 and peripheral devices. It can also be used to connect headphones for audio playback. Furthermore, it can be used to connect other electronic devices, such as AR devices.

[0220] The interface connection relationships between the modules illustrated in this embodiment of the invention are merely illustrative and do not constitute a structural limitation on the electronic device 1200. The electronic device 1200 may employ different interface connection methods or combinations of multiple interface connection methods as described in this embodiment of the invention.

[0221] The charging management module 1240 can be a rechargeable battery or a disposable battery. If it's a rechargeable battery, it can receive charging input via a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 1240 can receive charging input from the wired charger via the USB interface 1230. In some wireless charging embodiments, the charging management module 1240 can receive wireless charging input via the wireless charging coil of the electronic device 1200. While charging the battery 1242, the charging management module 1240 can also supply power to the electronic device 1200 via the power management module 1241.

[0222] The power management module 1241 connects the battery 1242, the charging management module 1240, and the processor 1210. The power management module 1241 receives input from the battery 1242 and / or the charging management module 1240, providing power to the processor 1210, memory 1221, external memory interface 1220, display 1294, camera 1293, and wireless communication module 1260, etc. The power management module 1241 can also monitor parameters such as the charging management module capacity, charging management module cycle count, and charging management module health status (leakage current, impedance). In some embodiments, the power management module 1241 may also be located within the processor 1210. In some embodiments, the power management module 1241 and the battery 1242 may also be located in the same device.

[0223] The wireless communication function of electronic device 1200 can be realized through antenna 1, antenna 2, mobile communication module 1250, wireless communication module 1260, modem, and baseband processor.

[0224] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 1200 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, a cellular antenna can be multiplexed as a wireless local area network diversity antenna. In some embodiments, the antenna can be used in conjunction with a tuning switch.

[0225] The mobile communication module 1250 provides a communication processing module for wireless communication solutions, including 2G / 3G / 4G / 5G, applied to the electronic device 1200. The mobile communication module 1250 may include at least one filter, switch, power amplifier, low-noise amplifier (LNA), etc. The mobile communication module 1250 receives electromagnetic waves via antenna 1, filters and amplifies the received electromagnetic waves, and transmits them to a modem for demodulation. The mobile communication module 1250 can also amplify the signal modulated by the modem and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 1250 may be housed in the processor 1210. In some embodiments, at least some functional modules of the mobile communication module 1250 and at least some modules of the processor 1210 may be housed in the same device.

[0226] A modem may include a modulator and a demodulator. The modulator modulates a low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates a received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to a baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to an application processor. The application processor outputs sound signals through an audio device (not limited to a speaker, receiver, etc.) or displays images or videos through a display screen 1294. In some embodiments, the modem may be a standalone device. In some embodiments, the modem may be independent of the processor 1210 and may be housed in the same device as the mobile communication module 1250 or other functional modules.

[0227] The wireless communication module 1260 provides a communication processing module for wireless communication solutions applied to the electronic device 1200, including Wireless Local Area Networks (WLANs) (such as Wireless Fidelity (Wi-Fi) networks), Bluetooth (BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), and Infrared (IR) technologies. The wireless communication module 1260 can be one or more devices integrating at least one communication processing module. The wireless communication module 1260 receives electromagnetic waves via antenna 2, modulates and filters the electromagnetic wave signals, and sends the processed signal to processor 1210. The wireless communication module 1260 can also receive signals to be transmitted from processor 1210, modulate and amplify them, and then convert them into electromagnetic waves for radiation via antenna 2.

[0228] In some embodiments, antenna 1 of electronic device 1200 is coupled to mobile communication module 1250, and antenna 2 is coupled to wireless communication module 1260, enabling electronic device 1200 to communicate with networks and other devices via wireless communication technology. Wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. GNSS can include Global Positioning System (SBAS), Global Navigation Satellite System (GLONASS), BeiDou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation System (SBAS).

[0229] Electronic device 1200 implements display functions through a GPU, a display screen 1294, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 1294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 1210 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0230] Display screen 1294 is used to display images, videos, etc. Display screen 1294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 1200 may include one or N displays 1294, where N is a positive integer greater than 1.

[0231] Electronic device 1200 can perform shooting functions through ISP, camera 1293, video codec, GPU, display screen and application processor.

[0232] The ISP (Image Signal Processor) is used to process data fed back from the camera 1293. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's image sensor. The light signal is converted into an electrical signal, and the image sensor transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and color. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set within the camera 1293.

[0233] Camera 1293 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through a lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, electronic device 1200 may include one or N cameras 1293, where N is a positive integer greater than 1.

[0234] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 1200 is selecting a frequency point, the DSP is used to perform Fourier transforms on the frequency energy.

[0235] Video codecs are used to compress or decompress digital video. Electronic device 1200 can support one or more video codecs. Thus, electronic device 1200 can play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0236] NPU stands for Neural Network (NN) computing processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs can enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0237] The external storage interface 1220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 1200. The external memory card communicates with the processor 1210 through the external storage interface 1220 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0238] Internal memory 1221 can be used to store computer executable program code, which includes instructions. Processor 1210 executes various functional applications and data processing of electronic device 1200 by running the instructions stored in internal memory 1221. Memory 1221 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 1200 (such as audio data, phonebook, etc.). Furthermore, memory 1221 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, other volatile solid-state storage devices, Universal Flash Storage (UFS), etc.

[0239] Electronic device 1200 can implement audio functions such as music playback and recording through audio module 1270, speaker, receiver, microphone, headphone jack, and application processor.

[0240] The audio module 1270 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 1270 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 1270 may be located in the processor 1210, or some functional modules of the audio module 1270 may be located in the processor 1210.

[0241] A loudspeaker, also known as a "speaker," is used to convert audio electrical signals into sound signals. Electronic devices 1200 can listen to music or make hands-free calls through loudspeakers.

[0242] A receiver, also known as a "handpiece," is used to convert audio electrical signals into sound signals. When an electronic device answers a phone call or voice message, the receiver can be brought close to the listener's ear to hear the voice.

[0243] A microphone, also known as a "voice transducer," is used to convert sound signals into audio electrical signals. When making a phone call or sending a voice message, the user speaks by bringing their mouth close to the microphone, inputting the sound signal into the microphone. Electronic device 1200 may have at least one microphone. In some embodiments, electronic device 1200 may have two microphones, which, in addition to acquiring sound signals, can also perform noise reduction. In some embodiments, electronic device 1200 may also have three, four, or more microphones, enabling sound signal acquisition, noise reduction, sound source identification, and directional recording, among other functions.

[0244] The headphone jack is used to connect wired headphones. The headphone jack can be a USB interface, or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, or a Cellular Telecommunications Communications Industry Association (CTIA) standard interface.

[0245] Buttons 1290 include a power button, volume buttons, etc. Buttons 1290 can be mechanical buttons or touch-sensitive buttons. Electronic device 1200 receives input from buttons 1290 and generates key signal inputs related to user settings and function control of electronic device 1200.

[0246] Motor 1291 can generate vibration alerts. Motor 1291 can be used for incoming call vibration alerts or for touch vibration feedback. For example, touch operations applied to different applications (such as taking photos, playing audio, etc.) can correspond to different vibration feedback effects. Touch operations applied to different areas of the display screen 1294 can also correspond to different vibration feedback effects. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0247] Indicator 1292 can be an indicator light, which can be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0248] The SIM card module 1295 is used to implement the communication function of the SIM card. The SIM card module 1295 may include a SIM card interface, SIM card circuitry, and related auxiliary components. The SIM card can be inserted into or removed from the SIM card interface to make contact with and separate from the electronic device 1200. The electronic device 1200 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card module 1295 can support Nano SIM cards, Micro SIM cards, and other SIM cards. Multiple cards can be inserted into the same SIM card interface simultaneously. The multiple cards can be of the same or different types. The SIM card interface can also be compatible with different types of SIM cards. The SIM card interface can also be compatible with external memory cards. The electronic device 1200 interacts with the network through the SIM card to realize functions such as voice calls and data communication. In some embodiments, the electronic device 1200 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 1200 and cannot be separated from the electronic device 1200.

[0249] The image processing methods described in the foregoing embodiments can all be implemented in the electronic device 1200 having the above-described hardware structure.

[0250] Based on the above embodiments, this application also provides an image processing apparatus, which includes a processor for executing the image processing method provided in the above embodiments.

[0251] This application also provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the image processing method provided in the above embodiments.

[0252] This application also provides a computer program product containing instructions that, when run on a computer, enable the computer to perform the image processing method provided in the above embodiments.

[0253] The specific implementation methods and technical effects of the electronic devices, computer-readable storage media, and computer program products containing instructions provided in this application can be found in the specific implementation process and technical effects of the image processing method provided in the foregoing embodiments, which will not be repeated here.

[0254] In some embodiments, as described above, those skilled in the art will clearly understand that, for the sake of convenience and brevity, the division of the functional modules described above is merely an example. In practical applications, the functions described above can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0255] In the embodiments of this application, the functional units can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0256] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as flash memory, portable hard disk, read-only memory, random access memory, magnetic disk, or optical disk.

[0257] The above are merely specific embodiments of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. An image processing method, characterized in that, This technology is applied to electronic devices, which include an application layer, a framework layer, a hardware abstraction layer, and a kernel layer; the framework layer runs a camera service; and the hardware abstraction layer includes an interface module, a sensor calling module, and a high-frequency enhancement processing module. The image processing method includes: In response to the first operation, a shooting preview interface is displayed; wherein, the shooting preview interface is the shooting preview interface of the electronic device in low light shooting mode or night scene shooting mode, and the electronic device captures the first image in the low light shooting mode or the night scene shooting mode; In response to the shooting operation on the shooting preview interface, the camera service generates a photo-taking request and sends it to the interface module; the interface module sends the photo-taking request to the sensor invocation module; the sensor invocation module invokes the camera sensor through the camera driver to acquire a first image and transmits the first image to the high-frequency enhancement processing module; The high-frequency enhancement processing module performs the following steps: The first image is input into an adaptive local difference operator to calculate the local variance of each pixel in each pixel region and obtain the first variance result. The pixels with larger local variance contain more high-frequency information, and the pixels with smaller local variance contain less high-frequency information. The first variance result includes the brightness difference between each pixel in the first image and the adjacent pixel region. Using the first variance result as prior information, the first image is input into a convolutional neural network to obtain the high-frequency information components of the first image. The low-frequency information component is obtained based on the high-frequency information component of the first image. High-frequency feature maps are extracted based on the high-frequency information components, and low-frequency feature maps are extracted based on the low-frequency information components. The high-frequency feature map and the low-frequency feature map are processed using a feature fusion network to generate a second image corresponding to the first image; the clarity of the second image is higher than that of the first image.

2. The image processing method according to claim 1, characterized in that, The convolutional neural network includes a first convolutional layer, a second convolutional layer, and an activation layer; the step of inputting the first image into the convolutional neural network to obtain the high-frequency information components of the first image includes: The first image is input into the first convolutional layer and the second convolutional layer respectively to obtain a first feature map and a second feature map; the kernel size and dilation rate of the first convolutional layer and the second convolutional layer are different, and the pixel size of the first feature map and the second feature map are the same; The first feature map and the second feature map are input into the activation layer to obtain the high-frequency information components of the first image; the activation layer includes an activation function, which is used to compare the contrast of each pixel in the first feature map and the second feature map.

3. The image processing method according to claim 1, characterized in that, The step of obtaining the low-frequency information component based on the high-frequency information component of the first image includes: Perform inverse operations on the high-frequency information components to obtain the inverse mapping of the high-frequency information components; The low-frequency information component of the first image is obtained by inverse mapping of the first image and the high-frequency information component.

4. The image processing method according to any one of claims 1 to 3, characterized in that, The step of extracting high-frequency feature maps based on the high-frequency information components includes: The high-frequency information components are input into a generative adversarial network to obtain the high-frequency feature map; The generative adversarial network (GAN) includes a generator and a discriminator. The generator comprises a multi-level processing network, each level of which includes a wavelet transform network and an inverse wavelet transform network of the same scale and symmetrically arranged. The generator sequentially includes multiple wavelet transform networks with progressively decreasing scales and inverse wavelet transform networks with progressively increasing scales. The input to the first-level wavelet transform network is the high-frequency information component, and the output of the first-level inverse wavelet transform network is the high-frequency feature map. The input to other levels of wavelet transform networks is the first-class feature map output by the wavelet transform network of the adjacent level above. The input to each level of inverse wavelet transform is the first-class feature map output by the wavelet transform network of the same level and the second-class feature map output by the inverse wavelet transform network of the next level below.

5. The image processing method according to any one of claims 1 to 3, characterized in that, The step of extracting the low-frequency feature map based on the low-frequency information components includes: The low-frequency information component is split into a first low-frequency component and a second low-frequency component. Extract the third feature map from the first low-frequency component; The third feature map and the second low-frequency component are fused to obtain the low-frequency feature map.

6. The image processing method according to claim 5, characterized in that, Both the first low-frequency component and the second low-frequency component include half of the low-frequency information component.

7. The image processing method according to any one of claims 1 to 3, characterized in that, The image processing method further includes: In response to the first operation, the ambient light sensor of the electronic device is invoked to collect the ambient light brightness value; The step of displaying the shooting preview interface includes: If the ambient light brightness value is less than or equal to a preset low light threshold, the shooting preview interface in the low light shooting mode or the night scene shooting mode will be displayed.

8. The image processing method according to any one of claims 1 to 3, characterized in that, The first operation is a click operation applied to the night scene shooting control.

9. An image processing method, characterized in that, Applied to electronic devices, the electronic devices include an application layer, a framework layer, a hardware abstraction layer, and a kernel layer; the hardware abstraction layer includes a high-frequency enhancement processing module; the image processing method includes: Display the first image; The high-frequency enhancement processing module performs the following steps: The first image is input into an adaptive local difference operator to calculate the local variance of each pixel in each pixel region and obtain the first variance result. The pixels with larger local variance contain more high-frequency information, and the pixels with smaller local variance contain less high-frequency information. The first variance result includes the brightness difference between each pixel in the first image and the adjacent pixel region. Using the first variance result as prior information, the first image is input into a convolutional neural network to obtain the high-frequency information components of the first image. The low-frequency information component is obtained based on the high-frequency information component of the first image. High-frequency feature maps are extracted based on the high-frequency information components, and low-frequency feature maps are extracted based on the low-frequency information components. The high-frequency feature map and the low-frequency feature map are processed using a feature fusion network to generate a second image corresponding to the first image.

10. The image processing method according to claim 9, characterized in that, The convolutional neural network includes a first convolutional layer, a second convolutional layer, and an activation layer; the step of using the first variance result as prior information, inputting the first image into the convolutional neural network, and obtaining the high-frequency information components of the first image includes: Using the first variance result as prior information, the first image is input into the first convolutional layer and the second convolutional layer of the convolutional neural network to obtain a first feature map and a second feature map; the kernel size and dilation rate of the first convolutional layer and the second convolutional layer are different, and the pixel size of the first feature map and the second feature map are the same; The first feature map and the second feature map are input into the activation layer of the neural network to obtain the high-frequency information components of the first image; the activation layer includes an activation function, which is used to compare the contrast of each pixel in the first feature map and the second feature map; And / or, The step of obtaining the low-frequency information component based on the high-frequency information component of the first image includes: Perform inverse operations on the high-frequency information components to obtain the inverse mapping of the high-frequency information components; The low-frequency information component of the first image is obtained by inverse mapping of the first image and the high-frequency information component.

11. The image processing method according to claim 9 or 10, characterized in that, The step of extracting high-frequency feature maps based on the high-frequency information components includes: The generative adversarial network (GAN) includes a generator and a discriminator. The generator comprises a multi-level processing network, each level of which includes a wavelet transform network and an inverse wavelet transform network of the same scale and symmetrically arranged. The generator sequentially includes multiple wavelet transform networks with progressively decreasing scales and inverse wavelet transform networks with progressively increasing scales. The input of the first-level wavelet transform network is the high-frequency information component, and the output of the first-level inverse wavelet transform network is the high-frequency feature map. The input of other levels of wavelet transform networks is the first-class feature map output by the wavelet transform network of the adjacent level above. The input of each level of inverse wavelet transform is the first-class feature map output by the wavelet transform network of the same level and the second-class feature map output by the inverse wavelet transform network of the next level below. And / or, The step of extracting the low-frequency feature map based on the low-frequency information components includes: The low-frequency information component is split into a first low-frequency component and a second low-frequency component. Extract the first feature from the first low-frequency component; The first feature and the second low-frequency component are fused to obtain the low-frequency feature map.

12. An electronic device, characterized in that, The electronic device includes a memory and a processor, wherein the memory is coupled to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the electronic device to perform the image processing method as described in any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on a computer, causes the computer to perform the image processing method as described in any one of claims 1 to 11.

14. A computer program product, characterized in that, It includes a computer program, which, when executed by a processor, implements the image processing method as described in any one of claims 1 to 11.

Images