Image processing method, electronic device, computer program product and storage medium

Abstract

The invention provides an image processing method, electronic equipment, a computer program product and a storage medium, and relates to the technical field of terminals. The electronic equipment collects a first image in a dark light shooting mode or a night scene shooting mode, and responds to a shooting operation of the shooting preview interface, and the shooting preview interface is shot according to a high-frequency information component of the first image and a low-frequency information component obtained based on the high-frequency information component of the first image. And performing high-frequency enhancement processing on the first image to obtain a second image. The electronic equipment performs high-frequency enhancement processing on the acquired first image, so that the high-frequency characteristics of the second image are more than those contained in the first image, the definition of the obtained second image is higher than that of the first image, and the quality of the image obtained through dark light shooting is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of terminal technology, and in particular to an image processing method, an electronic device, a computer program product, and a storage medium. Background Art

[0002] Mobile phone cameras are becoming more and more popular. The shooting environment of mobile phone cameras is divided into bright light environment and dark light environment. The shooting environment of mobile phones is dark light environment. Mobile phone cameras are limited by the amount of light entering the camera shutter and lens components. The original image data (RAW image) collected by the sensor often has dense noise and dark areas, resulting in poor signal-to-noise ratio and loss of high-frequency information, and the quality of the collected image is poor. Summary of the invention

[0003] The embodiments of the present application provide an image processing method, an electronic device, a computer program product, and a storage medium, which are used to solve the technical problem that the image quality captured by the electronic device in a dark light environment is poor.

[0004] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0005] In a first aspect, an image processing method is provided, which is applied to an electronic device. The electronic device may be a mobile phone, a tablet, a wearable device, etc. equipped with a camera. The image processing method provided in this embodiment processes a first image captured by the camera of the electronic device.

[0006] The electronic device displays a shooting preview interface in response to the first operation. There may be multiple first operations received by the electronic device. For example, the first operation may be a click operation on a night scene shooting mode control of a camera application received by the electronic device, or the first operation may be a click operation on an icon of a camera application. The electronic device receives the first operation, opens the camera application and enters the shooting process. Alternatively, the first operation received by the electronic device may be a voice control operation. If the electronic device receives a voice control instruction containing keywords such as "start shooting" or "open night scene shooting mode", it may also enter the shooting process.

[0007] The shooting preview interface displayed by the electronic device may be a shooting preview interface of the electronic device in a low-light shooting mode or a night scene shooting mode. When the electronic device is in the low-light shooting mode or the night scene shooting mode, the prompt text of the corresponding shooting mode may be displayed on the shooting preview interface, or the corresponding prompt text may not be displayed on the shooting preview interface.

[0008] The electronic device captures a first image in a low-light shooting mode or a night scene shooting mode. When the electronic device is in a low-light shooting mode or a night scene shooting mode, the ambient light brightness value of the shooting environment in which the camera of the electronic device is located is usually low, that is, the electronic device is in a low-light shooting environment, the amount of light entering is limited, and the first image captured by the electronic device has fewer high-frequency features and is relatively low in clarity. The first image mentioned here can be the original image data or RAW image captured by the camera sensor of the electronic device.

[0009] The electronic device receives a shooting operation on the shooting preview interface, and performs high-frequency enhancement processing on the captured first image to obtain a second image with more high-frequency features. The shooting operation on the shooting preview interface received by the electronic device may be a click operation on the camera shutter control to instruct the camera to capture a frame of image. For this scenario, the electronic device may use the latest captured frame of image as the first image, and perform high-frequency enhancement processing on the first image to obtain the second image.

[0010] In other cases, the shooting operation received by the electronic device may also be a click operation of a shooting shutter control to instruct the camera to start capturing video. The electronic device may use each frame of an image obtained in the video captured by the camera as a first image and perform high-frequency enhancement processing to obtain a second image corresponding to each first image. In this case, when the electronic device receives a click operation of the user on the shooting shutter control to end the video capture process, each frame of the image contained in the video finally generated by the electronic device may be an image obtained after high-frequency enhancement processing, so that the electronic device can directly obtain a video with higher definition.

[0011] The electronic device performs high-frequency enhancement processing on the collected first image, mainly by first acquiring the high-frequency information component of the first image, and then obtaining the low-frequency information component of the first image according to the high-frequency information component, and performing high-frequency enhancement processing on the first image according to the high-frequency information component and the low-frequency information component, so as to obtain a second image corresponding to the first image. For example, the electronic device can extract a high-frequency feature map through the high-frequency information component, obtain a low-frequency feature map through the low-frequency information component, and fuse the high-frequency feature map and the low-frequency feature map to obtain the second image.

[0012] The electronic device performs high-frequency enhancement processing on the first image to obtain a second image. Since the second image contains more high-frequency features, the clarity of the second image is higher than that of the first image. In this way, when the electronic device collects images in low-light shooting mode or night scene shooting mode, even if it is limited by a lower amount of light input, it can obtain a clear image with more detailed features, thereby improving the quality of the acquired image.

[0013] In a possible implementation of the first aspect, an application scenario of an electronic device performing image processing is defined. The electronic device automatically calls an ambient light sensor to detect a current ambient light brightness value, and selects whether to trigger execution of an image processing method.

[0014] Specifically, the electronic device can call the ambient light sensor of the electronic device to collect the ambient light brightness value in response to the first operation. If the electronic device detects that the ambient light brightness value is less than or equal to the preset dark light threshold, it can be determined that the current shooting environment brightness value is low, which may cause the camera to collect an image with lost high-frequency features due to the limited amount of light entering. Based on this, the electronic device can display a shooting preview interface in a dark light shooting mode or a night scene shooting mode, and in response to a photo operation on the shooting preview interface in the dark light shooting mode or the night scene shooting mode, perform highlight enhancement processing on the collected first image to obtain a second image with more high-frequency features.

[0015] In a possible implementation of the first aspect, another application scenario in which the electronic device performs image processing is defined. In this implementation, if the electronic device detects that the first operation is a click operation on a night scene shooting control, it may also display a shooting preview interface in a low-light shooting mode or a night scene shooting mode, and in response to a photo operation on the shooting preview interface in the low-light shooting mode or the night scene shooting mode, perform highlight enhancement processing on the captured first image to obtain a second image with more high-frequency features.

[0016] In a possible implementation of the first aspect, a scheme is further defined in which the electronic device performs high-frequency enhancement processing on the first image to obtain a second image based on the high-frequency information component of the first image and the low-frequency information component obtained based on the high-frequency information component of the first image.

[0017] Specifically, the electronic device uses a convolutional neural network to obtain the high-frequency information component of the first image, and obtains the low-frequency information component based on the high-frequency information component of the first image. The electronic device extracts a high-frequency feature map based on the high-frequency information component, and extracts a low-frequency feature map based on the low-frequency information component. Finally, the electronic device uses a feature fusion network to process the high-frequency feature map and the low-frequency feature map to generate a second image corresponding to the first image.

[0018] The electronic device uses a convolutional neural network to obtain a high-frequency information component of the first image, and the obtained high-frequency information component covers a pixel area where more high-frequency features are located, so that a high-frequency feature map extracted according to the high-frequency information component also contains more high-frequency features. The electronic device fuses the obtained high-frequency feature map to obtain a second image to achieve a high-frequency enhancement effect.

[0019] In a possible implementation of the first aspect, an implementation scheme of the electronic device using a convolutional neural network to obtain a high-frequency information component of the first image is further defined. The electronic device performs local variance statistics on the first image to obtain a first variance result, and the obtained first variance result may include a brightness difference between each pixel point and an adjacent pixel area in the first image, a brightness difference between each pixel point and an adjacent pixel area, and a pixel area where a high-frequency feature is located.

[0020] The electronic device uses the first variance result as prior information, inputs the first image into a convolutional neural network, and obtains a high-frequency information component of the first image.

[0021] The electronic device calculates a first variance result of the first image and uses it as prior information to guide the convolutional neural network to obtain more accurate high-frequency information components, so as to improve the efficiency of the electronic device in performing high-frequency enhancement processing.

[0022] In a possible implementation of the first aspect, a convolutional neural network used by an electronic device to obtain a high-frequency information component from a first image is further limited. Specifically, the convolutional neural network includes a first convolutional layer, a second convolutional layer, and an activation layer, and the convolution kernel specifications and void ratios of the first convolutional layer and the second convolutional layer are different to implement different convolution processing on the input image to obtain an image with the same pixel size. The activation layer includes an activation function, and the activation function is used to calculate the contrast of each pixel in the two input images.

[0023] The electronic device inputs the first image into the convolutional neural network, and the step of obtaining the high-frequency information component of the first image specifically includes: the electronic device inputs the first image into the first convolutional layer and the second convolutional layer respectively, the first convolutional layer performs convolution processing on the first image to obtain a first feature map, and the second convolutional layer performs convolution processing on the first image to obtain a second feature map. The first feature map and the second feature map have the same pixel size and different features;

[0024] The electronic device then inputs the first feature map and the second feature map into the activation layer, calculates the difference between the pixel features of each pixel point in the first feature map and the second feature map pixel by pixel, selects the pixel features of the pixel points with larger differences, and obtains the high-frequency information component of the first image.

[0025] In a possible implementation of the first aspect, an implementation scheme for the electronic device to obtain the low-frequency information component based on the high-frequency information component of the first image is further defined. The first image includes a high-frequency information component and a low-frequency information component, and the electronic device has extracted the high-frequency information component of the first image, and can perform an inverse operation on the high-frequency information component to obtain an inverse mapping of the high-frequency information component. According to the inverse mapping of the first image and the high-frequency information component, the low-frequency information component of the first image can be obtained.

[0026] In this way, the electronic device does not need to provide an additional low-pass filter to extract the low-frequency information component of the first image, thereby saving calculation costs.

[0027] In a possible implementation of the first aspect, an implementation scheme for the electronic device to extract the high-frequency feature map according to the high-frequency information component is further defined. The electronic device extracts the high-frequency feature map from the high-frequency information component using a generative adversarial network.

[0028] In this embodiment, the generative adversarial network includes a generator and a discriminator, and the generator includes a multi-level processing network. Each level of the processing network includes a wavelet transform network and an inverse wavelet transform network of the same scale and symmetrically arranged, and the generator includes a plurality of wavelet transform networks with successively decreasing scales and inverse wavelet transform networks with successively increasing scales, the input of the first-level wavelet transform network is a high-frequency information component, the output of the first-level inverse wavelet transform network is a high-frequency feature map, the input of the wavelet transform network of other levels is the first type of feature map output by the wavelet transform network of the previous adjacent level, and the input of the inverse wavelet transform of each level is the first type of feature map output by the wavelet transform network of the same level and the second type of feature map output by the inverse wavelet transform network of the next level.

[0029] The electronic device can obtain a high-frequency feature map corresponding to the high-frequency information component by performing multi-scale wavelet transform, performing multi-scale down-sampling using wavelet transform, and then performing multi-scale up-sampling using inverse wavelet transform.

[0030] In a possible implementation of the first aspect, an implementation scheme of the electronic device extracting a low-frequency feature map based on the low-frequency information component is further defined. The low-frequency information component obtained by the electronic device contains a large number of low-frequency features, which are redundant and do not need to be enhanced during high-frequency enhancement processing.

[0031] Based on this, the electronic device can only extract the low-frequency features from part of the low-frequency information components, which can not only obtain the low-frequency features of the first image, but also reduce the amount of calculation of the electronic device in extracting the low-frequency features.

[0032] The electronic device splits the low-frequency information component into a first low-frequency component and a second low-frequency component, extracts a third feature map from the first low-frequency component, and fuses the third feature map and the second low-frequency component to obtain a low-frequency feature map.

[0033] In a possible implementation manner of the first aspect, it is defined that the first low-frequency component and the second low-frequency component both include half of the low-frequency information of the low-frequency information component.

[0034] The more first low-frequency components the electronic device retains, the more low-frequency information components participate in extracting the low-frequency feature map, and the more low-frequency features are retained to make the image more consistent with the original image. The more second low-frequency components the electronic device retains, the more low-frequency information components do not participate in extracting the low-frequency feature map, and the less data needs to be calculated.

[0035] The electronic device divides the low-frequency information component into a first low-frequency component and a second low-frequency component, so that more low-frequency features can be retained and the amount of calculation can be reduced, thereby further improving the quality and efficiency of image processing.

[0036] In a second aspect, another image processing method is provided, which is applied to an electronic device. The difference between the provided image processing method and the image processing method provided in the first aspect includes: the first image may not be limited to an image captured in real time by a camera of the electronic device in a low-light shooting mode or a night scene shooting mode.

[0037] Specifically, the electronic device first displays a first image, which may be an image pre-stored in an album by the electronic device, or an image obtained from another terminal. The electronic device may receive a click operation of an editing control of an interface of the first image by a user, and perform high-frequency enhancement processing on the first image.

[0038] The electronic device performs high-frequency enhancement processing on the collected first image, mainly by first acquiring the high-frequency information component of the first image, and then obtaining the low-frequency information component of the first image according to the high-frequency information component, and performing high-frequency enhancement processing on the first image according to the high-frequency information component and the low-frequency information component, so as to obtain a second image corresponding to the first image. For example, the electronic device can extract a high-frequency feature map through the high-frequency information component, obtain a low-frequency feature map through the low-frequency information component, and fuse the high-frequency feature map and the low-frequency feature map to obtain the second image.

[0039] The electronic device performs high-frequency enhancement processing on the first image to obtain a second image. Since the second image contains more high-frequency features, the clarity of the second image is higher than that of the first image. In this way, when the electronic device collects images in low-light shooting mode or night scene shooting mode, even if it is limited by a lower amount of light input, it can obtain a clear image with more detailed features, thereby improving the quality of the acquired image.

[0040] In another possible implementation of the second aspect, a scheme is further defined in which the electronic device performs high-frequency enhancement processing on the first image to obtain the second image based on the high-frequency information component of the first image and the low-frequency information component obtained based on the high-frequency information component of the first image.

[0041] Specifically, the electronic device uses a convolutional neural network to obtain the high-frequency information component of the first image, and obtains the low-frequency information component based on the high-frequency information component of the first image. The electronic device extracts a high-frequency feature map based on the high-frequency information component, and extracts a low-frequency feature map based on the low-frequency information component. Finally, the electronic device uses a feature fusion network to process the high-frequency feature map and the low-frequency feature map to generate a second image corresponding to the first image.

[0042] The electronic device uses a convolutional neural network to obtain a high-frequency information component of the first image, and the obtained high-frequency information component covers a pixel area where more high-frequency features are located, so that a high-frequency feature map extracted according to the high-frequency information component also contains more high-frequency features. The electronic device fuses the obtained high-frequency feature map to obtain a second image to achieve a high-frequency enhancement effect.

[0043] In a possible implementation of the second aspect, an implementation scheme of the electronic device using a convolutional neural network to obtain a high-frequency information component of the first image is further defined. The electronic device performs local variance statistics on the first image to obtain a first variance result, and the obtained first variance result may include a brightness difference between each pixel point and an adjacent pixel area in the first image, a brightness difference between each pixel point and an adjacent pixel area, and a pixel area where a high-frequency feature is located.

[0044] The electronic device uses the first variance result as prior information, inputs the first image into a convolutional neural network, and obtains a high-frequency information component of the first image.

[0045] The electronic device calculates a first variance result of the first image and uses it as prior information to guide the convolutional neural network to obtain more accurate high-frequency information components, so as to improve the efficiency of the electronic device in performing high-frequency enhancement processing.

[0046] In a possible implementation of the second aspect, a convolutional neural network used by the electronic device to obtain a high-frequency information component from a first image is further limited. Specifically, the convolutional neural network includes a first convolutional layer, a second convolutional layer, and an activation layer, and the convolution kernel specifications and void ratios of the first convolutional layer and the second convolutional layer are different to implement different convolution processing on the input image to obtain an image with the same pixel size. The activation layer includes an activation function, and the activation function is used to calculate the contrast of each pixel in the two input images.

[0047] The electronic device inputs the first image into the convolutional neural network, and the step of obtaining the high-frequency information component of the first image specifically includes: the electronic device inputs the first image into the first convolutional layer and the second convolutional layer respectively, the first convolutional layer performs convolution processing on the first image to obtain a first feature map, and the second convolutional layer performs convolution processing on the first image to obtain a second feature map. The first feature map and the second feature map have the same pixel size and different features;

[0048] The electronic device then inputs the first feature map and the second feature map into the activation layer, calculates the difference between the pixel features of each pixel point in the first feature map and the second feature map pixel by pixel, selects the pixel features of the pixel points with larger differences, and obtains the high-frequency information component of the first image.

[0049] In a possible implementation manner of the second aspect, an implementation scheme for the electronic device to obtain the low-frequency information component based on the high-frequency information component of the first image is further defined. The first image includes a high-frequency information component and a low-frequency information component. The electronic device has extracted the high-frequency information component of the first image, and can perform an inverse operation on the high-frequency information component to obtain an inverse mapping of the high-frequency information component. According to the inverse mapping of the first image and the high-frequency information component, the low-frequency information component of the first image can be obtained.

[0050] In this way, the electronic device does not need to provide an additional low-pass filter to extract the low-frequency information component of the first image, thereby saving calculation costs.

[0051] In a possible implementation of the second aspect, an implementation scheme for the electronic device to extract the high-frequency feature map according to the high-frequency information component is further defined. The electronic device uses a generative adversarial network to extract the high-frequency feature map from the high-frequency information component.

[0052] In this embodiment, the generative adversarial network includes a generator and a discriminator, and the generator includes a multi-level processing network. Each level of the processing network includes a wavelet transform network and an inverse wavelet transform network of the same scale and symmetrically arranged, and the generator includes a plurality of wavelet transform networks with successively decreasing scales and inverse wavelet transform networks with successively increasing scales, the input of the first-level wavelet transform network is a high-frequency information component, the output of the first-level inverse wavelet transform network is a high-frequency feature map, the input of the wavelet transform network of other levels is the first type of feature map output by the wavelet transform network of the previous adjacent level, and the input of the inverse wavelet transform of each level is the first type of feature map output by the wavelet transform network of the same level and the second type of feature map output by the inverse wavelet transform network of the next level.

[0053] The electronic device can obtain a high-frequency feature map corresponding to the high-frequency information component by performing multi-scale wavelet transform, performing multi-scale down-sampling using wavelet transform, and then performing multi-scale up-sampling using inverse wavelet transform.

[0054] In a possible implementation of the second aspect, an implementation scheme of the electronic device extracting a low-frequency feature map based on the low-frequency information component is further defined. The low-frequency information component obtained by the electronic device contains more low-frequency features, which are redundant and do not need to be enhanced during high-frequency enhancement processing.

[0055] Based on this, the electronic device can only extract the low-frequency features from part of the low-frequency information components, which can not only obtain the low-frequency features of the first image, but also reduce the amount of calculation of the electronic device in extracting the low-frequency features.

[0056] The electronic device splits the low-frequency information component into a first low-frequency component and a second low-frequency component, extracts a third feature map from the first low-frequency component, and fuses the third feature map and the second low-frequency component to obtain a low-frequency feature map.

[0057] In a possible implementation manner of the second aspect, it is defined that the first low-frequency component and the second low-frequency component both include half of the low-frequency information of the low-frequency information component.

[0058] The more first low-frequency components the electronic device retains, the more low-frequency information components participate in extracting the low-frequency feature map, and the more low-frequency features are retained to make the image more consistent with the original image. The more second low-frequency components the electronic device retains, the more low-frequency information components do not participate in extracting the low-frequency feature map, and the less data needs to be calculated.

[0059] The electronic device divides the low-frequency information component into a first low-frequency component and a second low-frequency component, so that more low-frequency features can be retained and the amount of calculation can be reduced, thereby further improving the quality and efficiency of image processing.

[0060] In a third aspect, an electronic device is provided, the electronic device comprising a memory and a processor, the memory being coupled to the processor;

[0061] Memory stores computer-executable instructions;

[0062] The processor executes the computer-executable instructions stored in the memory, so that the electronic device performs the image processing method as described in any one of the first aspect or the second aspect.

[0063] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the image processing method of any one of the first aspect or the second aspect.

[0064] In a fifth aspect, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the image processing method as described in any one of the first aspect or the second aspect is implemented.

[0065] Among them, the technical effects brought about by any design method in the second to fifth aspects can refer to the technical effects brought about by different design methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A schematic diagram showing the comparison of images captured by a camera of an electronic device in different environments;

[0067] Figure 2 A schematic diagram of an interface involved in the image processing method provided in an embodiment of the present application;

[0068] Figure 3 A schematic diagram of another interface involved in the image processing method provided in an embodiment of the present application;

[0069] Figure 4 A schematic diagram of the process of an image processing method provided in an embodiment of the present application;

[0070] Figure 5 A schematic diagram of a high-frequency enhancement process involved in the image processing method provided in an embodiment of the present application;

[0071] Figure 6 A schematic diagram of the structure of a convolutional neural network involved in the image processing method provided in an embodiment of the present application;

[0072] Figure 7 A schematic diagram of a process for extracting high-frequency information components and low-frequency information components involved in the image processing method provided in an embodiment of the present application;

[0073] Figure 8 A schematic diagram of the principles of the standard convolution operation and the dilated convolution operation involved in the image processing method provided in the embodiment of the present application;

[0074] Fig. 9 A schematic diagram of the principle of a generative adversarial network involved in the image processing method provided in an embodiment of the present application;

[0075] Fig.10 A schematic diagram of a process for splitting low-frequency information components involved in the image processing method provided in an embodiment of the present application to obtain a low-frequency feature map;

[0076] Fig.11 A software framework and flowchart diagram of the image processing method provided in the embodiment of the present application;

[0077] Fig.12 A hardware structure diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0078] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0079] To facilitate understanding, some technical common sense involved in the embodiments of the present application is first introduced.

[0080] The quality or texture of an image refers to the details and layers of the image.

[0081] Grayscale refers to the color depth of a point in a black and white image, and generally ranges from 0 to 255, with white being 255 and black being 0. Black and white images can also be called grayscale images. By mapping the brightness value of each pixel in a color image to a grayscale value, you can convert the grayscale image corresponding to the color image.

[0082] Noise refers to the image sensor noise of the camera that collects the image, the photo grain noise, the channel transmission error during the image transmission process, and other factors that will cause some random, discrete, and isolated pixels to appear on the image. Noise is usually visually different from its adjacent pixels. For example, white dots in a black area, black dots in a white area, etc.

[0083] The frequency of an image is an indicator of how drastically the grayscale value of the image changes, and is the gradient of the grayscale in the plane space. The greater the gradient of the image, or the part where the grayscale value of the pixel changes greatly, the higher the frequency. The more chaotic the image is, the higher the frequency, such as the edge. The softer the image is, the lower the frequency, such as a large expanse of sky.

[0084] The low-frequency area refers to the area where the gray value of the pixel in the image changes slowly. The information included in the low-frequency area in the image can be called the low-frequency information component.

[0085] The high-frequency region refers to the region where the grayscale value of the pixel in the image changes rapidly, which includes information in the image that needs to be clearly displayed in detail. The information included in the high-frequency region in the image can be called the high-frequency information component.

[0086] A frame of image can be divided into high-frequency information components and low-frequency information components. The high-frequency information components form the edges and details of the image, while the low-frequency information components form the background and main body of the image. When processing images, the low-frequency information and high-frequency information in the image can be divided according to the actual processing requirements and the degree of sophistication.

[0087] RAW images represent the original images captured by the camera's sensor, that is, images that have not been processed.

[0088] The electronic device is equipped with a camera, and the electronic device can control the camera to collect images corresponding to the target object within the camera's framing range in the environment where the electronic device is located. The environment within the camera's framing range in the environment where the electronic device is located is recorded as the shooting environment.

[0089] The camera includes a sensor, such as a complementary metal oxide semiconductor (CMOS) image sensor, and all pixels on the CMOS image sensor work synchronously. The principle of camera shooting is: the electronic shutter controls the exposure of the sensor, and all pixels accumulate photons during the shutter exposure period. When the photon is incident on the sensor surface, it will be converted into an electron and stored. The stronger the light, the more electrons are generated per unit time, so the illumination at each pixel can be obtained. When the accumulated electron time reaches the preset time (exposure time), the stored electrons are transferred and read out to complete the exposure process. In other words, the ambient brightness of the shooting environment is a relatively important factor affecting the quality of the image captured by the camera of the electronic device.

[0090] Ambient brightness refers to the ratio of the light intensity of the luminous body in the environment to the area of ​​the light source. The unit of ambient brightness is candela / square meter (cd / m 2 ). The higher the ambient brightness, the greater the intensity of the reflected color of the objects in the environment, and the clearer the features of the target objects in the environment are in the human eye and the image in the camera. Conversely, at the lowest ambient brightness, the smaller the intensity of the reflected color of the objects in the environment, and the more blurred the features of the target objects in the environment are in the human eye and the image in the camera.

[0091] According to the ambient brightness, the shooting environment of the electronic device can be divided into a bright light environment and a dark light environment, wherein the bright light environment refers to an ambient brightness greater than or equal to a preset brightness threshold, and the dark light environment refers to an ambient brightness less than the preset brightness threshold.

[0092] When the shooting environment of an electronic device is a bright light environment, all the pixels of the camera accumulate more photons during the shutter exposure period, and each pixel collects more pixel features, and the corresponding image display effect is clearer. When the shooting environment of an electronic device is a dark environment, all the pixels of the camera accumulate fewer photons during the shutter exposure period, and each pixel collects fewer pixel features, and the corresponding image display effect is blurrier.

[0093] like Figure 1 As shown in FIG. , it is a schematic diagram of images collected by an electronic device in different shooting environments. Figure 1 As shown in (1) and (2) in the figure, they are images captured in the same static scene, where (1) is an image captured in a bright light environment, and (2) is an image captured in a dark light environment. Compared with images captured in a bright light environment, images captured in a dark light environment have fewer details and lower clarity. Figure 1 The image corresponding to another dark light environment shown in (3) and (4) in FIG. 1 can be the real image of the scene. Figure 1 As shown in (3) in the figure, the image captured by the camera is Figure 1As shown in (4) in the figure, the pixel region Z1 in (3) corresponds to the pixel region Z2 in (4). The pixel region Z2 loses more detailed features than the pixel region Z1, making the image blurry.

[0094] From the perspective of the frequency of the pixels contained in the image, the image includes high-frequency information components and low-frequency information components. Among them, the low-frequency information component corresponds to the pixel area where the brightness of the object in the image is uniform or changes slowly, and the high-frequency information component corresponds to the brightness of the object in the image with large changes, especially the edges, details and noise of the characteristic objects in the image. Figure 1 As shown in (4), the high-frequency information component of the Z2 pixel area is small and the image clarity is low.

[0095] That is to say, when the camera of an electronic device is shooting in a dark environment, the high-frequency features of the image captured by the camera are lost and the clarity of the captured image is low.

[0096] Based on this, this embodiment provides an image processing method, which is applied to an electronic device. The electronic device performs high-frequency enhancement processing on a first image acquired in a low-light shooting mode or a night scene shooting mode to obtain a second feature map with more high-frequency features. Compared with the first image acquired by the electronic device, the second image has more high-frequency features and higher clarity.

[0097] The image processing method provided in this embodiment can be applied to electronic devices. The electronic devices may include personal computers (PCs), tablet computers, laptop computers, portable computers (such as mobile phones), wearable electronic devices (such as smart watches), augmented reality (AR) and virtual reality (VR) devices, vehicle-mounted computers and other electronic devices. The following embodiments do not impose any special restrictions on the specific form of the electronic devices.

[0098] The image processing method provided in this embodiment can be divided into an image acquisition operation and an image processing operation for performing high-frequency enhancement processing on the first image. The electronic device first performs an image acquisition operation to acquire a first image, and then performs an image processing operation to process the acquired first image to obtain a second image. There can be multiple ways for the electronic device to acquire the first image and for triggering the image processing operation to be performed on the first image.

[0099] In one case, the first image acquired by the electronic device may be an image pre-stored in the electronic device.

[0100] For example, the electronic device may obtain the first image from other terminals and store it in a photo album of the electronic device, or the electronic device may control a camera installed on the electronic device in advance to capture the first image and store it in a photo album of the electronic device.

[0101] In this case, the electronic device may process the first image in response to the first operation.

[0102] In one example, if Figure 2 As shown in (1) in FIG. 1 , the electronic device displays a first interface (such as Figure 2 In (1) shown in U1), the first interface may be an interface of an album application of an electronic device. The first interface includes a first image (such as Figure 2 P1) and multiple processing controls shown in (1) in the figure, wherein the multiple processing controls include a high frequency enhancement control. Figure 2 As shown in (1), the multiple processing controls displayed on the first interface may include a cropping control, a filter control, an adjustment control, and an intelligent control, wherein the highlight enhancement control may be an intelligent control (such as Figure 2 A shown in (1) in the figure). Or Figure 2 As shown in (2), the processing controls displayed on the first interface may include a cropping control, a filter control, an adjustment control, and a dimming enhancement control, wherein the dimming enhancement control may be a control for triggering the electronic device to perform high-frequency enhancement processing. In specific implementations, the specific name and display method of the high-frequency enhancement control that triggers the high-frequency enhancement processing may have other implementation methods and are not limited thereto.

[0103] like Figure 2 As shown in (1) in FIG. 1 , if the electronic device receives a click operation on the high-frequency enhancement control, it can perform high-frequency enhancement processing on the first image to obtain a second image. Figure 2 As shown in (2) in FIG. 1 , the interface of the electronic device (such as Figure 2 U2) shown in (2) in FIG. 1 includes a second image (such as Figure 2 In P2) shown in (2), the detail features of some pixel areas in the second image increase. Figure 2 As shown in (3) in FIG. 1 , the electronic device can receive the user's action on the save control (such as Figure 2 The click operation of B) shown in (3) in the above embodiment stores the generated second image.

[0104] After the electronic device processes the first image pre-stored in the album to obtain the corresponding second image, the second image may be stored separately or stored by overwriting the first image.

[0105] In another case, the first image processed by the electronic device may be an image captured in real time by a camera controlled by the electronic device.

[0106] The electronic device is equipped with a camera, and the electronic device can control the camera to acquire a first image, perform high-frequency enhancement processing on the first image when acquiring the first image, and store or display a second image after the high-frequency enhancement processing.

[0107] In one example, if Figure 3 As shown, the electronic device opens the camera application and displays the shooting preview interface (such as Figure 3 The shooting preview interface displayed by the electronic device may correspond to a variety of shooting modes, for example, Figure 3 The photographing mode shown in (1) can also be Figure 3 Of course, the electronic device can also perform the image processing operation provided in this embodiment in a video recording mode or other shooting modes, without limitation.

[0108] like Figure 3 As shown in (2) in FIG. 1 , the electronic device responds to the user's action on the night scene shooting control (such as Figure 3 Click the A) shown in (2) in the figure to enter the night scene shooting mode and collect images within the viewing range. Figure 3 The interface shown in (3) (such as Figure 3 U4) shown in (3) of FIG. 1 , if the electronic device receives a user's action on the photo control (such as Figure 3 By performing a click operation of B) shown in (3) in the figure, a frame of the first image can be captured. The electronic device performs high-frequency enhancement processing on the first image to obtain a second image. Figure 3 As shown in (4) of FIG. 1 , the electronic device can be in the shooting interface (such as Figure 3 The second image obtained after the image processing is displayed on U5) shown in (4) in the figure. In other cases, the electronic device may also store the second image obtained after the image processing operation in the album, or the electronic device may also update the thumbnail of the second image displayed in the album thumbnail display area of ​​the shooting preview interface. If the electronic device receives a click operation from the user on the album thumbnail display area, the second image may be displayed.

[0109] In another example, upon receiving a click operation for acting on a shooting preview control, the electronic device enters multiple shooting modes such as photo taking, video recording, night scene, dark light, and portrait. In multiple shooting modes, the electronic device can call the ambient light sensor to collect the ambient light brightness value of the shooting environment in which the electronic device is located. The electronic device compares the ambient light brightness value with the preset dark light threshold. If it is determined that the ambient light brightness value is less than or equal to the preset dark light threshold, it can be determined that the current shooting environment is a dark light shooting environment. In one case, the electronic device can enter a dark light shooting environment or a night scene shooting mode, capture a first image in the dark light shooting environment or the night scene shooting mode, and perform high-frequency enhancement processing on the first image to obtain a corresponding second image. In other cases, the electronic device may not switch the shooting mode, and perform high-frequency enhancement processing on the first image captured in the current shooting mode to obtain the corresponding second image.

[0110] In other examples, the electronic device may not distinguish the currently running shooting mode, and perform image processing operations on any captured frame of the first image, so that the second image finally generated contains more high-frequency information.

[0111] See also Figure 4 , is a flow chart of the image processing method provided in this embodiment. The high-frequency enhancement processing involved in the provided image processing method mainly includes the following steps:

[0112] S41: The electronic device obtains a high-frequency information component of the first image using a convolutional neural network, and obtains a low-frequency information component of the first image based on the high-frequency information component of the first image.

[0113] Each pixel of the first image has a corresponding characteristic parameter, which may be a variety of characteristic parameters, such as brightness value, gray value or chromaticity value. The characteristic parameter of each pixel enables each pixel to display a corresponding visual effect, and the characteristic parameters of all pixels enable the first image to present an overall display effect.

[0114] The characteristic parameters of different pixels may be the same or different. In the first image, if the characteristic parameters of the pixels in a part of the pixel area are the same or close, the visual effect of this part of the pixel area is relatively uniform or has a small change. In other words, the grayscale value of the pixels in this part of the pixel area changes less frequently and contains more low-frequency information.

[0115] In the first image, if the characteristic parameters of the pixels in some pixel areas are quite different, the visual effects of these pixel areas are not uniform or vary greatly. In other words, the grayscale values ​​of the pixels in these pixel areas vary frequently and contain more high-frequency information.

[0116] Based on this, the electronic device can determine the first pixel area corresponding to the high-frequency information component by acquiring the pixel area with large differences in characteristic parameters of adjacent pixels in the first image, and then extract the high-frequency information component from the first pixel area of ​​the first image. Figure 5 As shown in FIG. 1 , it is another schematic diagram of a process of an electronic device performing high frequency enhancement processing on a first image. Figure 4 and Figure 5 As shown, the electronic device acquires the high-frequency information component of the first image, mainly including two steps:

[0117] S411: The electronic device performs local variance statistics on the first image to obtain a first variance result.

[0118] Variance is a statistical indicator to measure the degree of data dispersion. The smaller the variance, the more concentrated the data points, the narrower the distribution, the smaller the fluctuation, and the more stable the data. On the contrary, the larger the variance, the wider the distribution of the data points, the greater the fluctuation, and the more unstable the data. The electronic device performs local variance statistics on the first image to count the degree of deviation between the characteristic parameters of the pixel points in each local area of ​​the first image. The characteristic parameters of the pixel points mentioned here may include the brightness value or grayscale value of the pixel points.

[0119] The electronic device performs local variance statistics on the first image. If the variance of a certain pixel point is larger, it means that the deviation between the characteristic parameter of the pixel point and the average characteristic parameter of the pixel area is larger, and the pixel point contains more high-frequency information. Conversely, if the variance of a certain pixel point is smaller, it means that the deviation between the characteristic parameter of the pixel point and the average characteristic parameter of the pixel area is smaller, and the corresponding high-frequency information is less and the low-frequency information is more.

[0120] That is to say, the variance of each pixel is positively correlated with the high-frequency information contained in the pixel. Based on this, the electronic device can count the variance of each pixel in the first pixel by means of local variance statistics, and use the variance of all pixels of the first image as the first variance result. The first variance result obtained by the electronic device may include the corresponding relationship between the coordinates of each pixel and the variance.

[0121] In a specific embodiment, Figure 5 As shown, the electronic device can construct an adaptive local difference operator to calculate the local variance of the first image through window sliding.

[0122] The electronic device constructs an adaptive local difference operator, which has the ability to perform feature parameter difference statistics on each pixel point in each pixel area of ​​the input image. The electronic device inputs the first image into the adaptive local difference operator and calculates the local variance of each pixel point in each pixel area. The pixel point with a larger variance contains more high-frequency information, and the pixel point with a smaller variance contains less high-frequency information.

[0123] S412, the electronic device uses the first variance result as prior information, inputs the first image into a convolutional neural network, and obtains a high-frequency information component of the first image.

[0124] The first variance result obtained by the electronic device indicates the variance of each pixel in the first image, and the variance of each pixel is positively correlated with the high-frequency information contained in each pixel. The electronic device uses the first variance result of the first image as prior information to obtain the high-frequency information component of the first image more quickly and accurately.

[0125] In one case, the electronic device may directly use the first variance result as prior information.

[0126] In another case, the electronic device may also pre-set a variance threshold. The electronic device divides the pixel points whose variance is greater than or equal to the variance threshold into the first pixel area, and divides the pixel points whose variance is less than the variance threshold into the second pixel area. The first pixel area determined by the electronic device is a pixel area containing high-frequency information, and the second pixel area determined is a pixel area containing low-frequency information.

[0127] A convolutional neural network (CNN) is loaded in the electronic device, and the convolutional neural network has the ability to obtain high-frequency information according to the input first image. Specifically, Figure 6 As shown in FIG. 1 , the convolutional neural network mainly includes a convolutional layer and an activation layer, wherein the convolutional layer includes a first convolutional layer and a second convolutional layer in parallel, and the outputs of the first convolutional layer and the second convolutional layer are both connected to the input of the activation layer. In addition, the convolutional neural network also includes an input layer and an output layer, wherein the input layer is respectively connected to the first convolutional layer and the second convolutional layer, and the activation layer is connected to the output layer. In other words, the convolutional neural network includes an input layer, a first convolutional layer, a second convolutional layer, an activation layer, and an output layer in sequence.

[0128] The first convolution layer and the second convolution layer included in the convolutional neural network are different convolutional networks. The convolutional network includes one or more convolution kernels, and the parameters involved in the convolution operation of the convolution kernel may include convolution kernel specifications, weights, step lengths, etc.

[0129] The convolution kernel, also known as the convolution template or convolution window, is an operation matrix. The convolution kernel specification refers to the number of rows and columns of the operation matrix, the weight refers to the operation weight value of each element in the operation matrix, and the step size refers to the number of pixels moved during the convolution kernel movement operation. The convolution kernel specification determines the scope of the convolution operation, the weight determines the result of the convolution operation, and the step size determines the number of convolution operations.

[0130] The process of an electronic device using a convolutional network to perform a convolution operation on a first image refers to inputting the first image into the convolutional network, and the convolution window of the convolution kernel of the convolutional network slides on the first image in sequence, and a pixel area for convolution calculation is determined during each slide, and the pixel points in the pixel area correspond to the elements of the convolution kernel. The electronic device calculates the weighted sum according to the weights of each pixel point in the pixel area and the corresponding element to obtain a new grayscale value, and maps the new grayscale value as the grayscale value of the central pixel point of the pixel area to a new feature map. The electronic device uses the convolution kernel to slide and perform convolution operations on the first image in sequence until the convolution window of the convolution kernel covers all pixel areas of the first image, and the new feature map is the feature map obtained after the convolution operation of the first image.

[0131] In this embodiment, the convolutional network includes a first convolutional layer and a second convolutional layer, and the first convolutional layer and the second convolutional layer are different convolutional networks. The different convolutional networks mentioned here can be understood as the number of convolutional kernels, convolutional kernel specifications, weights or step sizes included in the first convolutional layer and the second convolutional layer are different or partially different, so that the first convolutional layer and the second convolutional layer perform different convolution operations on the same first input image, and the output feature maps are also different.

[0132] like Figure 7 As shown in FIG. 1 , it is a schematic diagram of an electronic device processing a first image using a convolutional neural network. The electronic device inputs the first image x and obtains the first variance result σ2(x) of the first image. The feature map output by the first convolutional layer is the first feature map F c1 (x), the feature map output by the second convolutional layer is F C2 (x).

[0133] In one example, in the convolutional neural network loaded by the electronic device, the convolution kernels of the first convolutional layer and the second convolutional layer are both standard convolution kernels, and the convolution operation principle is as follows: Figure 8 As shown in (1) in .

[0134] Figure 8 In (1), the convolution kernel size is 3*3, and the corresponding elements are a1, a2...a8 and a9. The pixel area size of the first image is 9*6, and the pixel points included in the first image are b11, b12...b68 and b69 in sequence.

[0135] Figure 8 In (1), the process of the electronic device performing the convolution operation includes: the electronic device performs the first convolution operation with the convolution kernel, and the corresponding pixel area includes the pixel points: b11, b12, b13, b21, b22, b23, b31, b32 and b33. The first convolution operation obtains the updated gray value corresponding to the central pixel b22 as c11, where:

[0136] C11=a1*b11+a2*b12+a3*b13+a4*b21+a5*b22+a6*b23+a7*b31+a8*b32+a9*b33.

[0137] Figure 8 In (2), the electronic device slides the convolution window of the convolution kernel to perform the next convolution operation, and the corresponding pixel area includes the following pixels: b12, b13, b14, b22, b23, b24, b32, b33 and b34. The updated gray value corresponding to the central pixel b23 obtained by this convolution operation is c12, where:

[0138] c12=a1*b12+a2*b13+a3*b14+a4*b22+a5*b23+a6*b24+a7*b32+a8*b33+a9*b34.

[0139] In another example, in a convolutional neural network loaded by an electronic device, the first convolutional layer and the second convolutional layer can also use the convolution kernel to perform a dilated convolution operation on the input first image. Dilated convolution is an algorithm for increasing the receptive field in a convolutional network. By introducing holes (or gaps) in the standard convolution kernel, the field of view of the convolutional network is expanded without increasing the number of parameters or using a pooling layer. Compared with standard convolution, dilated convolution adds a new parameter dilation rate (Dilation Rate). The dilation rate refers to the spacing between data objects, or the gap between pixels, when the convolution kernel processes data. In this way, when an electronic device uses dilated convolution for a convolution operation, the convolution kernel is not completely applied to the input first image, but is applied by skipping fixed pixel gaps.

[0140] The principle of hole convolution operation in electronic devices can be as follows: Figure 8 As shown in (3) and (4) in . In this example, the convolution kernel size is still 3*3, the pixel area size of the first image is still 9*6, and the hole rate of the convolution kernel is 1, that is, one pixel gap is skipped during the convolution operation.

[0141] Specifically, Figure 8 In (3), the process of the electronic device performing the first dilated convolution operation includes: the electronic device uses the pixel area corresponding to the convolution kernel to include the pixel points: b12, b14, b16, b32, b34, b36, b52, b54 and b56. The electronic device performs the first dilated convolution operation to obtain an updated gray value corresponding to the central pixel point b34, which is d11, where:

[0142] d11=a1*b12+a2*b14+a3*b16+a4*b32+a5*b34+a6*b36+a7*b52+a8*b54+a9*b56.

[0143] Figure 8 In (4), the electronic device slides the convolution window of the convolution kernel to perform the next dilated convolution operation, and the corresponding pixel area includes the following pixels: b13, b15, b17, b33, b35, b37, b53, b55 and b57. The updated grayscale value corresponding to the central pixel b35 obtained by this convolution operation is d12, where:

[0144] d12=a1*b13+a2*b15+a3*b17+a4*b33+a5*b35+a6*b37+a7*b53+a8*b55+a9*b57.

[0145] The parameters involved in the convolutional neural network used by electronic devices may include convolution kernel specifications, weights, step sizes, and void ratios. If one or more of these parameters are different, the convolution layer will perform different convolution operations on the input image, and the corresponding output feature maps will also be different.

[0146] In this embodiment, the neural network loaded by the electronic device sets the convolution kernel specifications and the hole rates of the first convolution layer and the second convolution layer to be different. The electronic device inputs the first image into the first convolution layer and the second convolution layer respectively, and the first convolution layer and the second convolution layer respectively perform different hole convolution operations on the first image, and the first feature map output by the first convolution layer is different from the feature map output by the second convolution layer. It should be noted that the first feature map obtained by the electronic device after performing a hole convolution operation on the first image using the first convolution layer and the second feature map obtained by performing a hole convolution operation on the second image using the second convolution layer have the same pixel size as the second feature map, so as to determine whether each pixel contains high-frequency features or low-frequency features pixel by pixel according to the first feature map and the second feature map, so as to obtain more accurate high-frequency information components.

[0147] In specific implementation, the convolution kernels and void rates of the first convolution layer and the second convolution layer are different, and the pixel sizes of the feature maps actually outputted may be different. The electronic device may also add a pixel padding operation to perform a zero padding operation on the feature maps outputted by the first convolution layer and the second convolution layer, so that the pixel sizes of the first feature map and the second feature map are the same. In other cases, the electronic device may also directly limit the pixel size of the input image and the pixel size of the output image of the first convolution layer and the second convolution layer, so that the pixel sizes of the input image and the output feature map of each convolution layer are consistent. Then, by inputting the same first image into the first convolution layer and the second convolution layer respectively, the first feature map and the second feature map with the same pixel size can be obtained.

[0148] The first image includes a high-frequency information component and a low-frequency information component. The electronic device performs different convolution operations on the first image. After the high-frequency information component undergoes different convolution operations, the difference will increase, and after the low-frequency information component undergoes different convolution operations, the difference will decrease. When the electronic device performs different convolution operations on the same first image, the high-frequency information component with a large pixel difference in the first image is effectively retained.

[0149] Based on this, the electronic device inputs the first feature map and the second feature map into the activation layer to perform contrast calculation pixel by pixel, thereby obtaining the high-frequency information component of the first image.

[0150] In one example, continue as Figure 7 As shown, the activation layer function of the convolutional neural network used by the electronic device can be a Sigmoid function. The function of the Sigmoid function is equivalent to compressing a real number to between 0 and 1. When z is a very large positive number, g(z) will approach 1, and when z is a very small negative number, g(z) will approach 0. The electronic device inputs the first feature map and the second feature map into the activation layer to perform contrast calculation pixel by pixel. The greater the pixel difference between the two feature maps, the greater the contrast and the more high-frequency information; the smaller the pixel difference, the smaller the contrast and the less high-frequency information. In this way, the electronic device can obtain the high-frequency information component F of the first image. H , where F H =Sigmoid[F c1 (x)-F C2 (x)].

[0151] When the electronic device uses the activation layer to obtain the high-frequency information component, it can set a contrast threshold, retain the pixel features in the first image whose pixel contrast is greater than the contrast threshold, and discard the pixel features whose pixel contrast is less than or equal to the contrast threshold, so as to obtain the high-frequency information component of the first image.

[0152] S413: The electronic device performs an inverse operation on the high-frequency information component to obtain an inverse mapping of the high-frequency information component, and acquires a low-frequency information component of the first image according to the first image and the inverse mapping of the high-frequency information component.

[0153] The pixel features included in the first image can be divided into a high-frequency information component and a low-frequency information component. After determining the high-frequency information component, the electronic device can reversely determine the low-frequency information component of the first image.

[0154] In one example, the electronic device detects the high frequency information component F H Perform the inverse operation to obtain the inverse mapping The inverse mapping Multiply it with the first image x to get the low-frequency information component F of the first image L ,

[0155] S42: The electronic device extracts a high-frequency feature map according to the high-frequency information component, and extracts a low-frequency feature map according to the low-frequency information component.

[0156] After the electronic device obtains the high-frequency information component and the low-frequency information component of the first image, it respectively performs operations of extracting a high-frequency feature map from the high-frequency information component and extracting a low-frequency feature map from the low-frequency information component. Specifically, the steps include:

[0157] S421: The electronic device inputs the high-frequency information component into the generative adversarial network to obtain a high-frequency feature map.

[0158] The electronic device loads a Generative Adversarial Networks (GAN) to re-extract high-frequency feature maps from high-frequency information components through the GAN. Specifically, Fig. 9 As shown in (1) in the figure, the generative adversarial network includes a generator (generative network or generative model, Generator) and a discriminator (discriminator network or discriminator model, Di scr iminator). The generator learns the characteristics of the training set data and, under the guidance of the discriminator, tries to fit the random noise distribution to the real distribution of the training data, thereby generating similar data with the characteristics of the training set. The discriminator is responsible for distinguishing whether the input data is real or fake data generated by the generator, and feeding it back to the generator. The two networks are trained alternately and their capabilities are improved synchronously until the data generated by the generator network can be indistinguishable from the real data and reaches a certain balance with the capabilities of the discriminator network.

[0159] The training process of the generative adversarial model usually includes: fixing the parameters of the generator and training the parameters of the discriminator. First, some vectors are randomly input, some random training values ​​are generated by the generator, and then some real values ​​are taken from the real database, and the training values ​​and real values ​​are used as the input of the discriminator to train the discriminator. The purpose of training is to enable the discriminator to distinguish between training values ​​and real values, and give a score as close to 0 as possible when receiving the training value, and a score as close to 1 as possible when receiving the real value. After that, the parameters of the discriminator are fixed and the parameters of the generator are adjusted. Specifically, a random input value is thrown into the generator to generate a picture, the picture is put into the discriminator, and a score is obtained, so that the score finally output by the discriminator is close to 1. This embodiment uses a generative adversarial network to extract high-frequency feature maps of high-frequency information components to obtain the effect of high-frequency feature enhancement.

[0160] In this embodiment, the generator as a whole is based on multi-scale wavelet transform (WT), which uses the characteristics of multi-scale wavelet transform to supplement the frequency and spatial position information of the feature map, and further enhances the high-frequency details.

[0161] Among them, the generative adversarial network includes a generator and a discriminator, the generator includes a multi-level processing network, each level of the processing network includes a wavelet transform network and an inverse wavelet transform network with the same scale and symmetrically arranged, the generator includes a plurality of wavelet transform networks with successively decreasing scales and inverse wavelet transform networks with successively increasing scales, the input of the first-level wavelet transform network is a high-frequency information component, the output of the first-level inverse wavelet transform network is a high-frequency feature map, the input of the wavelet transform networks of other levels is the first type of feature map output by the wavelet transform network of the previous adjacent level, and the input of each level of inverse wavelet transform is the first type of feature map output by the wavelet transform network of the same level and the second type of feature map output by the inverse wavelet transform network of the next level.

[0162] like Fig. 9 As shown in (2), the generator includes multiple pairs of networks, each pair of networks includes a wavelet transform network and an inverse wavelet transform network of the same scale, and the scales of adjacent wavelet transform networks are different. Each wavelet transform network includes a downsampling layer, a convolution layer, a normalization layer, and an activation layer. Correspondingly, each inverse wavelet transform network also includes an upsampling layer, a convolution layer, a normalization layer, and an activation layer.

[0163] The electronic device converts the high frequency information component (F H ) is input into the first-level wavelet transform network, down-sampled by the down-sampling layer, convolutional operation by the convolution layer, normalized by the normalization layer, and processed by the activation function of the activation layer to obtain the first type of feature map (T1).

[0164] The electronic device inputs the first type of feature map (T1) into the second level wavelet transform network with reduced scale and the first level inverse wavelet transform network with the same scale. The electronic device inputs the first feature map into the second level wavelet transform network, performs down sampling, convolution operation, normalization processing and activation function processing in sequence, and obtains the first type of feature map (T2).

[0165] The electronic device then inputs the first type of feature map (T2) into the third level wavelet transform network with reduced scale and the second level inverse wavelet transform network with the same scale. The electronic device inputs the second feature map into the third level wavelet transform network, performs down sampling, convolution operation, normalization processing and activation function processing in sequence, and obtains the first type of feature map (T3).

[0166] The electronic device inputs the third type of feature map (T3) into the fourth-level wavelet transform network with reduced scale and the third-level inverse wavelet transform network with the same scale, and so on, until the Nth-level wavelet transform network with the smallest scale outputs the first feature map (Tn).

[0167] The electronic device inputs the first feature map (Tn) into the Nth level inverse wavelet transform network with the same scale. The Nth level inverse wavelet transform network sequentially processes the Nth type of feature map as follows: upsampling by the upsampling layer, convolution operation by the convolution layer, normalization by the normalization layer, and activation function processing by the activation layer to obtain the second type of feature map (Tn).

[0168] The electronic device inputs the second type feature map to the (N-1)th level inverse wavelet transform network with increased value scale. The (N-1)th level inverse wavelet transform network combines the first type feature map (Tn-1) and the second type feature map (Tn) to perform upsampling, convolution operation, normalization processing and activation function processing respectively, and obtains the second type feature map (Tn-1).

[0169] The electronic device then transmits the second-type feature map (Tn-1) to the (N-2)-th level inverse wavelet transform network with the scale increased by the previous level. Similarly, the second-level inverse wavelet transform network obtains the first-type feature map (T2) and the second-type feature map (T3), and performs upsampling, convolution operation, normalization processing and activation function processing in sequence to obtain the second-type feature map (T2).

[0170] The electronic device transmits the second type of feature map (T2) to the first level inverse wavelet transform network. The first level inverse wavelet transform network combines the first type of feature map (T1) and the second type of feature map (T2), performs upsampling, convolution operation, normalization processing and activation function processing in sequence, and obtains the first and second type of feature maps. At this point, the electronic device can obtain the high-frequency feature map corresponding to the high-frequency information component of the first image.

[0171] The electronic device uses the multi-scale wavelet transform network included in the generator to extract high-frequency features, and also uses the discriminator to discriminate the output high-frequency features and feeds the discrimination results back to the generator to optimize the parameters of the generator. After multiple feedback trainings, the electronic device can obtain an enhanced high-frequency feature map.

[0172] In specific implementation, the electronic device can use the ESRGAN loss function as the loss function of the generative adversarial network during the training process, and the discriminator uses the Visual Geometry Group Network (VGGNet) to obtain a more accurate high-frequency feature map.

[0173] S422: The electronic device splits the low-frequency information component into a first low-frequency component and a second low-frequency component, extracts a third feature map from the first low-frequency component, and fuses the third feature map and the second low-frequency component to obtain a low-frequency feature map.

[0174] The electronic device re-extracts low-frequency features from the low-frequency information components and uses part of the low-frequency information components for calculation. Specifically, Fig.10 As shown, the electronic device performs channel separation on the low-frequency information component to obtain a first low-frequency component and a second low-frequency component. Among them, the first low-frequency component is used to extract low-frequency features, and the second low-frequency component is used for feature fusion. The electronic device can split the low-frequency information component in half, that is, the first low-frequency component and the second low-frequency component obtained by splitting are both half of all low-frequency information components. In other embodiments, the electronic device can also split the low-frequency information component in other proportions, and the splitting ratio can satisfy both the amount of calculation for low-frequency feature extraction and the accuracy of low-frequency features. For example, the electronic device can also split one-third of the low-frequency information component into the first low-frequency component, and the remaining two-thirds as the second low-frequency information component, so as to reduce the amount of calculation for low-frequency feature extraction. For another example, the electronic device can also split two-thirds of the low-frequency information component into the first low-frequency component, and the remaining one-third as the second low-frequency component, so as to increase the accuracy of low-frequency features.

[0175] The electronic device inputs the first low-frequency component into a convolutional neural network (CNN) to extract the low-frequency features of the first low-frequency component, which are recorded as the first features. In specific implementation, the electronic device uses the attention mechanism CNN to extract the low-frequency features. The electronic device superimposes the first feature and the second low-frequency component to obtain a feature map containing low-frequency information and low-frequency features, which is the low-frequency feature after low-frequency enhancement.

[0176] S43: The electronic device uses a feature fusion network to process the high-frequency feature map and the low-frequency feature map to generate a second image corresponding to the first image.

[0177] The electronic device loads a pre-trained feature fusion network, which has the ability to output a high-frequency enhanced image based on the high-frequency feature map and the low-frequency feature map. The electronic device inputs the high-frequency feature map and the low-frequency feature map of the first image obtained in the above step into the feature fusion network to obtain a new image, which is recorded as the second image. The second image obtained by the electronic device processing the first image has enhanced high-frequency features relative to the first image.

[0178] The electronic device applies the above-mentioned S41-S43 scheme to the above-mentioned Figure 3 The image captured by the image acquisition scene shown can perform image processing operations on the RAW images captured in real time by the camera of the electronic device in a dark light scene, enhance high-frequency features, obtain images with more detailed features, and improve the quality of images output by the electronic device.

[0179] The pre-trained feature fusion network used by electronic devices is essentially a network obtained by iteratively training a neural network using samples. A neural network refers to a network formed by connecting multiple single neural units together. Among them, the output of a neural unit can be the input of another or more neural units. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0180] The electronic devices used to train feature fusion networks usually include deep neural networks (DNN) or convolutional neural networks (CNN). Among them, deep neural networks are also called multi-layer neural networks, which can be understood as neural networks with multiple hidden layers. According to the positions of different layers, DNNs can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the multiple layers in the middle are hidden layers. The layers can be fully connected. In other words, any neuron in the i-th layer can be connected to any neuron in the i+1-th layer.

[0181] Convolutional neural network is a deep neural network with convolutional structure. Convolutional neural network contains a feature extractor composed of convolution layer and subsampling layer, which can be regarded as a filter. Convolutional layer refers to the neuron layer in convolutional neural network that performs convolution processing on the input signal. In the convolutional layer of convolutional neural network, a neuron can only be connected to some neurons in the adjacent layer. A convolutional layer usually contains several feature planes, each of which can be composed of some rectangularly arranged neural units.

[0182] Electronic devices usually require multiple groups of sample images for network training. Each group of sample images includes two types of images, one is the input sample image containing the features to be classified, and the other is the target sample image containing the target features. It should be noted that the features to be classified here refer to the features that need to be processed such as identification, classification, labeling, etc., such as features with characteristics such as defocus and motion blur. The target features are one or several types of features that need to be identified or processed, such as features with characteristics such as focus pixel area and clarity. The image content of the input sample image and the target sample image is associated.

[0183] Among them, the input sample image is used as the input value of the neural network, and the target sample image is used as the target value of the neural network. The input sample image contains various basic elements that may be included in the image to be processed by the feature fusion network. The target sample image can be an image obtained by processing the input sample image according to user needs, such as a defocused image and an image with motion blur. The target sample image can be an image obtained by processing the input sample image according to user needs. The target sample image may remove the unclear or motion-blurred elements that the user does not need based on the input sample image. In other words, the element content of the input sample image and the target sample image is basically the same, and the feature attributes may be different.

[0184] In this embodiment, the electronic device can train a neural network to obtain a feature fusion network, and then use the feature fusion network to perform image processing operations. The electronic device can also directly obtain a feature fusion network trained by other devices to perform the image processing method provided in this embodiment.

[0185] The process of an electronic device training a neural network to obtain a feature fusion network mainly includes: the electronic device obtains a sample image and a neural network without setting relevant parameters of a weight matrix, inputs the sample image into the neural network for iterative training, and gradually optimizes the weight matrix in the neural network according to the loss function until the loss function converges, so that the weight matrix in the neural network reaches the optimal value. At this time, the neural network can be used as a feature fusion network.

[0186] like Figure 5 As shown in (2), the electronic device uses a feature fusion network consisting of two convolutional layers to fuse high-frequency and low-frequency features, and the final output is calculated with the true value image by L1 loss function.

[0187] Specifically, the electronic device uses the loss function to calculate the difference between the predicted value and the true value, and then uses the difference to back propagate to modify the weight matrix; then the sample data for training is input into the neural network with the modified weight matrix, and multiple iterative training is performed until the difference is minimized, that is, the neural network converges.

[0188] When the neural network uses input values ​​and target values ​​for iterative training, the error back propagation algorithm is used to correct the parameters of the weight matrix in the initial neural network during the training process, so that the reconstruction error loss of the neural network becomes smaller and smaller. Specifically, the difference between the predicted value and the target value obtained in each iterative training is calculated, and then the difference is reversely transferred to the neural network to modify the weight matrix of the neural network, and the neural network after the modified weight matrix is ​​iteratively trained again, and the weight matrix of the neural network is continued to be updated with the difference obtained from the iterative training again. Repeat multiple iterative trainings and reversely modify the weight matrix with the difference of each iterative training until the difference between the predicted value and the target value obtained is within the allowable loss range of the loss function, that is, the loss function converges, so that the convergence of the neural network is achieved through multiple iterative training. The back propagation algorithm is a back propagation movement dominated by error loss, which is used to obtain the parameters of the optimal neural network model. Of course, iterative training can also be iteratively optimized using other similar algorithms without limitation.

[0189] The above embodiment explains the specific implementation process of the image processing method from the perspective of the electronic device. The following will explain the specific implementation process of the image processing method from the perspective of the internal software architecture of the electronic device.

[0190] like Fig.11 The figure shows the software and hardware architecture of the electronic device. The following will explain in detail the image processing process of the electronic device in combination with the internal architecture of the electronic device.

[0191] Specifically, the internal architecture of an electronic device can be divided into four layers, from top to bottom, namely, the application layer (APP), the framework layer (FWK), the hardware abstraction layer (HAL), and the kernel layer (or driver layer). It should be noted that in addition to these main functional layers, other functional modules may also be included without limitation.

[0192] The application layer may include a series of application packages, such as the camera application involved in this embodiment. In addition, the application layer also includes applications such as gallery, image processing applications with camera functions, etc. The application package may also include applications such as call, calendar, map, navigation, music, video, short message, etc.

[0193] The framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0194] The framework layer runs a camera service, which can be called by the camera application to implement functions related to shooting. In addition, the framework layer can also include a window manager, a content provider, a view system, a resource manager, a notification manager, etc. Among them, the window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc. The content provider is used to store and obtain data and make the data accessible to the application. The data can include video, images, audio, etc. The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The display interface can be composed of one or more views. The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, etc. The notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, notification information is used to inform the completion of downloads, message reminders, etc. Notification information can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog window. The notification information may also be, for example, a text message displayed in the status bar, a reminder sound, vibration of the power terminal, flashing indicator light, etc. It should be noted that the camera application may also call the content provider, resource manager, notification manager, window manager, view system, etc. according to actual business needs, and this embodiment does not impose any restrictions on this.

[0195] The kernel layer is a layer between hardware and software. The kernel layer includes at least a camera driver. The camera driver can be used to drive a hardware module with a shooting function, such as a camera sensor. In other words, the camera driver is responsible for data interaction with the camera sensor. The kernel layer may also include a display driver, an audio driver, a sensor driver, etc., which is not limited in this embodiment.

[0196] The hardware abstraction layer can encapsulate the drivers in the kernel layer and provide a calling interface to the framework layer, shielding the implementation details of the underlying hardware. Fig.11 As shown, the hardware abstraction layer may include a camera call processing module (Camera HAL), etc. The camera call processing module is the core software framework of the camera, and the camera call processing module includes an interface module, a sensor call module (Sensor Node), a high-frequency enhancement processing module, etc. The interface module, the sensor call module, and the high-frequency enhancement processing module are components in the image data and control instruction transmission pipeline in the camera call processing module.

[0197] Specifically, the sensor module may be a control node for the camera sensor, and the sensor module may control the camera sensor through the camera driver. The interface module may be a software interface for the application framework layer, and is used to interact with the application framework layer for data. Of course, the interface module may also interact with the high-frequency enhancement processing module in the camera call processing module for data.

[0198] like Fig.11 As shown, the camera application can receive the user's first operation, pass the camera shooting mode selected by the user, such as the night scene shooting mode, to the camera service of the framework layer, and then the camera service passes it to the sensor calling module through the interface module of the hardware abstraction layer. In addition, the camera application can also receive the user's second operation, generate a shooting instruction from the shooting operation triggered by the user and pass it to the camera service of the framework layer, and then the camera service passes it to the sensor calling module through the interface module of the hardware abstraction layer.

[0199] The camera application receives the second operation, and the camera service generates a photo request and sends it to the camera calling module, which is then sent to the sensor calling module via the interface module. The sensor calling module can call the camera sensor through the camera driver to collect the original image data, i.e., the first image, and transmit the first image to the high-frequency enhancement processing module for high-frequency enhancement processing to obtain the second image.

[0200] The high frequency enhancement processing module uploads the second image to the camera service for rendering processing for image display, and then transmits it to the camera application to display the second image.

[0201] Understandably, Fig.11 The layers in the software structure shown and the components contained in each layer do not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer layers than shown in the figure, and each layer may include more or fewer components, which is not limited in the present application.

[0202] In addition, it is understood that in order to implement the image processing method in this embodiment, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of this application.

[0203] The image processing method provided in the above embodiment uses an adaptive local difference description operator as prior information, inputs the first image into the frequency domain separation module to split the high-frequency information component, and inputs the split high-frequency information component into the high-frequency generation branch of the generative adversarial network based on the multi-scale wavelet transform to extract the high-frequency feature map. In addition, the electronic device performs an inverse operation on the high-frequency information component and multiplies it with the first image to obtain a low-frequency information component, and then splits the low-frequency information component into a part of the low-frequency information through channel separation for extracting the low-frequency feature map, which can reduce the amount of calculation and retain the low-frequency features. Afterwards, the electronic device inputs the extracted high-frequency feature map and the low-frequency feature map into the feature fusion network for fusion to obtain the second image corresponding to the first image. This scheme uses multi-scale wavelet transform downsampling to form UNet for high-frequency information feature extraction. The content of high-frequency features is greatly improved, and the clarity of images collected in dark light shooting scenes or night scene shooting modes is improved.

[0204] In addition, an embodiment of the present application further provides an electronic device, including a memory and a processor;

[0205] Memory stores computer-executable instructions;

[0206] The processor executes the computer-executable instructions stored in the memory, so that the electronic device executes the image processing method provided by the above embodiment. In addition to these main components, the electronic device also includes components for realizing basic functions. Fig.12 Provide specific instructions.

[0207] like Fig.12 The figure shows a schematic diagram of the structure of an electronic device 1200 provided in an embodiment of the present application. The electronic device 1200 may include a processor 1210, an external memory interface 1220, an internal memory 1221, a universal serial bus (USB) interface 1230, a charging management module 1240, a power management module 1241, a battery 1242, an antenna 1, an antenna 2, a mobile communication module 1250, a wireless communication module 1260, an audio module 1270, a speaker, a receiver, a microphone, an earphone interface, a sensor module 1280, a button 1290, a motor 1291, an indicator 1292, a camera 1293, a display screen 1294, and a SIM card module 1295. The sensor module 1280 may include a pressure sensor 1280A, a gyroscope sensor 1280B, an ambient light sensor 1280C, a touch sensor 1280D, and the like.

[0208] The structure shown in the embodiment of the present invention does not limit the electronic device 1200. It may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0209] The processor 1210 may include one or more processing units. For example, the processor 1210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processor (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0210] The controller can be a decision maker that directs the various components of the electronic device 1200 to work in coordination according to the instructions. It is the nerve center and command center of the electronic device 1200. The controller generates an operation control signal according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0211] The processor 1210 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 1210 is a high-speed cache memory, which can store instructions or data that the processor 1210 has just used or cyclically used. If the processor 1210 needs to use the instruction or data again, it can be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 1210, and thus improves the efficiency of the system.

[0212] In some embodiments, the processor 1210 may include an interface. The interface may include an Inter-Integrated Circuit (I2C) interface, an Inter-Integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver / Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a General-Purpose Input / Output (GPIO) interface, a SIM interface, and / or a USB interface, etc.

[0213] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (Serial Data Line, SDL) and a serial clock line (Serial Clock Line, SCL). In some embodiments, the processor 1210 may include multiple groups of I2C buses. The processor 1210 can be coupled to the touch sensor 1280D, the charger, the flash, the camera 1293, etc. through different I2C bus interfaces. For example: the processor 1210 can be coupled to the touch sensor 1280D through the I2C interface, so that the processor 1210 communicates with the touch sensor 1280D through the I2C bus interface, and the touch function of the electronic device 1200 is realized.

[0214] The I2S interface can be used for audio communication. In some embodiments, the processor 1210 can include multiple I2S buses. The processor 1210 can be coupled to the audio module 1270 via the I2S bus to achieve communication between the processor 1210 and the audio module 1270. In some embodiments, the audio module 1270 can transmit an audio signal to the wireless communication module 1260 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0215] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 1270 and the wireless communication module 1260 can be coupled via a PCM bus interface. In some embodiments, the audio module 1270 can also transmit audio signals to the wireless communication module 1260 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication, and the sampling rates of the two interfaces are different.

[0216] The UART interface is a universal serial data bus for asynchronous communication. The bus is a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 1210 and the wireless communication module 1260. For example, the processor 1210 communicates with the Bluetooth module through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 1270 can transmit audio signals to the wireless communication module 1260 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0217] The MIPI interface can be used to connect the processor 1210 with peripheral devices such as the display screen 1294 and the camera 1293. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 1210 and the camera 1293 communicate via the CSI interface to implement the shooting function of the electronic device 1200. The processor 1210 and the display screen 1294 communicate via the DSI interface to implement the display function of the electronic device 1200.

[0218] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 1210 with the camera 1293, the display 1294, the wireless communication module 1260, the audio module 1270, the sensor module 1280, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0219] The USB interface 1230 may be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 1230 may be used to connect a charger to charge the electronic device 1200, or may be used to transmit data between the electronic device 1200 and a peripheral device. The USB interface 1230 may also be used to connect headphones to play audio through the headphones. The USB interface 1230 may also be used to connect other electronic devices, such as AR devices, etc.

[0220] The interface connection relationship between the modules shown in the embodiment of the present invention is only for illustrative purposes and does not constitute a structural limitation on the electronic device 1200. The electronic device 1200 may use different interface connection methods in the embodiment of the present invention, or a combination of multiple interface connection methods.

[0221] The charging management module 1240 can be a rechargeable battery or a disposable battery. For a rechargeable battery, it can receive charging input through a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 1240 can receive charging input from a wired charger through the USB interface 1230. In some wireless charging embodiments, the charging management module 1240 can receive wireless charging input through a wireless charging coil of the electronic device 1200. While the charging management module 1240 is charging the battery 1242, it can also power the electronic device 1200 through the power management module 1241.

[0222] The power management module 1241 is used to connect the battery 1242, the charging management module 1240 and the processor 1210. The power management module 1241 receives input from the battery 1242 and / or the charging management module 1240, and provides power to the processor 1210, the memory 1221, the external memory interface 1220, the display screen 1294, the camera 1293, and the wireless communication module 1260. The power management module 1241 can also be used to monitor parameters such as the capacity of the charging management module, the number of cycles of the charging management module, and the health status (leakage, impedance) of the charging management module. In some embodiments, the power management module 1241 can also be set in the processor 1210. In some embodiments, the power management module 1241 and the battery 1242 can also be set in the same device.

[0223] The wireless communication function of the electronic device 1200 can be implemented through antenna 1, antenna 2, mobile communication module 1250, wireless communication module 1260, modem and baseband processor.

[0224] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 1200 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, a cellular network antenna can be reused as a wireless local area network diversity antenna. In some embodiments, the antenna can be used in combination with a tuning switch.

[0225] The mobile communication module 1250 may provide a communication processing module for wireless communication solutions including 2G / 3G / 4G / 5G, etc., applied to the electronic device 1200. The mobile communication module 1250 may include at least one filter, a switch, a power amplifier, a low noise amplifier (Low Noise Amplifier, LNA), etc. The mobile communication module 1250 receives electromagnetic waves from the antenna 1, and performs filtering, amplification, and other processing on the received electromagnetic waves, and transmits them to the modem for demodulation. The mobile communication module 1250 may also amplify the signal modulated by the modem, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 1250 may be arranged in the processor 1210. In some embodiments, at least some of the functional modules of the mobile communication module 1250 may be arranged in the same device as at least some of the modules of the processor 1210.

[0226] The modem may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker, a receiver, etc.), or displays an image or video through a display screen 1294. In some embodiments, the modem may be an independent device. In some embodiments, the modem may be independent of the processor 1210 and be set in the same device as the mobile communication module 1250 or other functional modules.

[0227] The wireless communication module 1260 can provide a communication processing module for wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc., which are applied to the electronic device 1200. The wireless communication module 1260 can be one or more devices integrating at least one communication processing module. The wireless communication module 1260 receives electromagnetic waves via the antenna 2, modulates the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 1210. The wireless communication module 1260 can also receive the signal to be sent from the processor 1210, modulate the frequency, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0228] In some embodiments, the antenna 1 of the electronic device 1200 is coupled to the mobile communication module 1250, and the antenna 2 is coupled to the wireless communication module 1260, so that the electronic device 1200 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include Global System for Mobile communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. GNSS may include Global Satellite Positioning System (SBAS), Global Navigation Satellite System (GLONASS), BeiDou navigation Satellite system (BDS), Quasi-Zenith Satellite System (QZSS) and / or Satellite Based Augmentation System (SBAS).

[0229] The electronic device 1200 implements the display function through a GPU, a display screen 1294, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 1294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 1210 may include one or more GPUs that execute program instructions to generate or change display information.

[0230] The display screen 1294 is used to display images, videos, etc. The display screen 1294 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light-emitting diode or an active matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini LED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 1200 may include 1 or N display screens 1294, where N is a positive integer greater than 1.

[0231] The electronic device 1200 can realize the shooting function through ISP, camera 1293, video codec, GPU, display screen and application processor.

[0232] The ISP is used to process the data fed back by the camera 1293. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and chromaticity of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 1293.

[0233] The camera 1293 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 1200 may include 1 or N cameras 1293, where N is a positive integer greater than 1.

[0234] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 1200 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0235] Video codecs are used to compress or decompress digital videos. The electronic device 1200 may support one or more video codecs. Thus, the electronic device 1200 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0236] NPU is a neural network (NN) computing processor, which can quickly process input information and continuously self-learn by drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain. NPU can realize applications such as intelligent cognition of electronic device 1200, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0237] The external memory interface 1220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 1200. The external memory card communicates with the processor 1210 through the external memory interface 1220 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0238] The internal memory 1221 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 1210 executes various functional applications and data processing of the electronic device 1200 by running the instructions stored in the internal memory 1221. The memory 1221 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 1200 (such as audio data, a phone book, etc.), etc. In addition, the memory 1221 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, other volatile solid-state storage devices, a universal flash memory (Universal Flash Storage, UFS), etc.

[0239] The electronic device 1200 can implement audio functions such as music playing and recording through the audio module 1270, speakers, receivers, microphones, headphone jacks, and application processors.

[0240] The audio module 1270 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 1270 can also be used to encode and decode audio signals. In some embodiments, the audio module 1270 can be arranged in the processor 1210, or some functional modules of the audio module 1270 can be arranged in the processor 1210.

[0241] The speaker, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 1200 can listen to music or listen to a hands-free call through the speaker.

[0242] The receiver, also called a "handset", is used to convert audio electrical signals into sound signals. When the electronic device 1200 receives a call or voice message, the voice can be received by placing the receiver close to the human ear.

[0243] A microphone, also called a "microphone" or "microphone", is used to convert sound signals into audio electrical signals. When making a call or sending a voice message, the user can put their mouth close to the microphone and speak to input the sound signal into the microphone. The electronic device 1200 can be provided with at least one microphone. In some embodiments, the electronic device 1200 can be provided with two microphones, which can not only collect sound signals but also realize noise reduction function. In some embodiments, the electronic device 1200 can also be provided with three, four or more microphones to realize the collection of sound signals, noise reduction, identification of sound sources, and realization of directional recording function, etc.

[0244] The headphone jack is used to connect a wired headphone and can be a USB interface, or a 3.5 mm Open Mobile Terminal Platform (OMTP) standard interface or a Cellular Telecommunications Industry Association of the USA (CTIA) standard interface.

[0245] The key 1290 includes a power button, a volume button, etc. The key 1290 may be a mechanical key or a touch key. The electronic device 1200 receives the key 1290 input and generates a key signal input related to the user settings and function control of the electronic device 1200.

[0246] Motor 1291 can generate vibration prompts. Motor 1291 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. Touch operations acting on different areas of the display screen 1294 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0247] Indicator 1292 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0248] The SIM card module 1295 is used to implement the communication function of the SIM card. The SIM card module 1295 may include a SIM card interface, a SIM card circuit, and related auxiliary devices. The SIM card can be connected to and separated from the electronic device 1200 by inserting it into the SIM card interface or pulling it out from the SIM card interface. The electronic device 1200 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card module 1295 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface at the same time. The types of the multiple cards can be the same or different. The SIM card interface can also be compatible with different types of SIM cards. The SIM card interface can also be compatible with external memory cards. The electronic device 1200 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 1200 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 1200 and cannot be separated from the electronic device 1200.

[0249] The image processing methods in the aforementioned embodiments can all be implemented in the electronic device 1200 having the aforementioned hardware structure.

[0250] On the basis of the above embodiments, an embodiment of the present application further provides an image processing device, the image processing device includes a processor, and the processor is used to execute the image processing method provided by the above embodiments.

[0251] The embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the image processing method provided in the above embodiment.

[0252] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the image processing method provided in the above embodiment.

[0253] The specific implementation methods of the electronic device, computer-readable storage medium, and computer program product containing instructions provided in the embodiments of the present application and the technical effects brought about by them can be found in the specific implementation process of the image processing method provided in the aforementioned embodiments and the technical effects brought about by them, which will not be repeated here.

[0254] In some embodiments, through the description of the above implementation methods, technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0255] Each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0256] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as flash memory, mobile hard disk, read-only memory, random access memory, disk or optical disk.

[0257] The above are only specific implementations of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto, and any changes or replacements within the technical scope disclosed in the embodiments of the present application should be included in the protection scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: Applied to electronic equipment, the image processing method comprises: In response to the first operation, displaying a shooting preview interface; wherein the shooting preview interface is a shooting preview interface of the electronic device in a low-light shooting mode or a night scene shooting mode, and the electronic device captures a first image in the low-light shooting mode or the night scene shooting mode; In response to the shooting operation of the shooting preview interface, the first image is subjected to high-frequency enhancement processing according to the high-frequency information component of the first image and the low-frequency information component obtained based on the high-frequency information component of the first image to obtain a second image; the clarity of the second image is higher than the clarity of the first image.

2. The image processing method according to claim 1, characterized in that: The step of performing high-frequency enhancement processing on the first image to obtain the second image according to the high-frequency information component of the first image and the low-frequency information component obtained based on the high-frequency information component of the first image comprises: Acquire the high-frequency information component of the first image using a convolutional neural network, and obtain the low-frequency information component based on the high-frequency information component of the first image; Extracting a high-frequency feature map according to the high-frequency information component, and extracting a low-frequency feature map according to the low-frequency information component; The high-frequency feature map and the low-frequency feature map are processed by a feature fusion network to generate the second image corresponding to the first image.

3. The image processing method according to claim 2, characterized in that: The step of acquiring the high-frequency information component of the first image by using a convolutional neural network comprises: Performing local variance statistics on the first image to obtain a first variance result; the first variance result includes a brightness difference between each pixel point and an adjacent pixel area in the first image; Using the first variance result as prior information, the first image is input into the convolutional neural network to obtain a high-frequency information component of the first image.

4. The image processing method according to claim 3, characterized in that: The convolutional neural network includes a first convolutional layer, a second convolutional layer and an activation layer; the step of inputting the first image into the convolutional neural network to obtain a high-frequency information component of the first image includes: Inputting the first image into the first convolution layer and the second convolution layer respectively to obtain a first feature map and a second feature map; the first convolution layer and the second convolution layer have different convolution kernel specifications and void ratios, and the first feature map and the second feature map have the same pixel size; The first feature map and the second feature map are input into the activation layer to obtain the high-frequency information component of the first image; the activation layer includes an activation function, and the activation function is used to compare the contrast of each pixel in the first feature map and the second feature map.

5. The image processing method according to claim 5 or 6, characterized in that: The step of obtaining the low-frequency information component based on the high-frequency information component of the first image comprises: Performing an inverse operation on the high-frequency information component to obtain an inverse mapping of the high-frequency information component; According to the inverse mapping of the first image and the high-frequency information component, the low-frequency information component of the first image is obtained.

6. The image processing method according to any one of claims 2 to 5, characterized in that: The step of extracting a high-frequency feature map according to the high-frequency information component comprises: Inputting the high-frequency information component into a generative adversarial network to obtain the high-frequency feature map; Among them, the generative adversarial network includes a generator and a discriminator, the generator includes a multi-level processing network, each level of the processing network includes a wavelet transform network and an inverse wavelet transform network with the same scale and symmetrically arranged, the generator includes a plurality of wavelet transform networks with successively decreasing scales and inverse wavelet transform networks with successively increasing scales, the input of the first-level wavelet transform network is the high-frequency information component, the output of the first-level inverse wavelet transform network is the high-frequency feature map, the input of the wavelet transform networks of other levels is the first type of feature map output by the wavelet transform network of the previous adjacent level, and the input of the inverse wavelet transform of each level is the first type of feature map output by the wavelet transform network of the same level and the second type of feature map output by the inverse wavelet transform network of the next level.

7. The image processing method according to any one of claims 2 to 6, characterized in that: The step of extracting a low-frequency feature map according to the low-frequency information component comprises: Splitting the low-frequency information component into a first low-frequency component and a second low-frequency component; extracting a third feature map from the first low-frequency component; The third feature map and the second low-frequency component are fused to obtain the low-frequency feature map.

8. The image processing method according to claim 7, characterized in that: The first low-frequency component and the second low-frequency component each include half of the low-frequency information component.

9. The image processing method according to any one of claims 1 to 8, characterized in that: The image processing method further comprises: In response to the first operation, calling the ambient light sensor of the electronic device to collect the ambient light brightness value; The step of displaying the shooting preview interface includes: If the ambient light brightness value is less than or equal to the preset low-light threshold, the shooting preview interface is displayed in the low-light shooting mode or the night scene shooting mode.

10. The image processing method according to any one of claims 1 to 9, characterized in that: The first operation is a click operation on a night scene shooting control.

11. An image processing method, characterized in that: Applied to electronic equipment, the image processing method comprises: displaying a first image; Acquire a high-frequency information component of the first image, and acquire a low-frequency information component of the first image according to the high-frequency information component of the first image; According to the high-frequency information component and the low-frequency information component of the first image, high-frequency enhancement processing is performed on the first image to generate a second image.

12. The image processing method according to claim 11, characterized in that: The step of performing high-frequency enhancement processing on the first image according to the high-frequency information component and the low-frequency information component of the first image to generate the second image comprises: Acquire the high-frequency information component of the first image using a convolutional neural network, and obtain the low-frequency information component based on the high-frequency information component of the first image; Extracting a high-frequency feature map according to the high-frequency information component, and extracting a low-frequency feature map according to the low-frequency information component; The high-frequency feature map and the low-frequency feature map are processed by a feature fusion network to generate the second image corresponding to the first image.

13. The image processing method according to claim 12, characterized in that: The convolutional neural network includes a first convolutional layer, a second convolutional layer and an activation layer; the step of acquiring the high-frequency information component of the first image by using the convolutional neural network includes: Performing local variance statistics on the first image to obtain a first variance result; Using the first variance result as prior information, the first image is input into the first convolution layer and the second convolution layer of the convolutional neural network respectively to obtain a first feature map and a second feature map; the convolution kernel specifications and the hole rates of the first convolution layer and the second convolution layer are different, and the pixel sizes of the first feature map and the second feature map are the same; Inputting the first feature map and the second feature map into the activation layer of the neural network to obtain the high-frequency information component of the first image; the activation layer includes an activation function, and the activation function is used to compare the contrast of each pixel point in the first feature map and the second feature map; and / or, The step of acquiring the low-frequency information component of the first image according to the high-frequency information component of the first image comprises: Performing an inverse operation on the high-frequency information component to obtain an inverse mapping of the high-frequency information component; The low-frequency information component of the first image is obtained according to inverse mapping of the first image and the high-frequency information component.

14. The image processing method according to claim 12 or 13, characterized in that: The step of extracting a high-frequency feature map according to the high-frequency information component comprises: Wherein, the generative adversarial network includes a generator and a discriminator, the generator includes a multi-level processing network, each level of the processing network includes a wavelet transform network and an inverse wavelet transform network of the same scale and symmetrically arranged, the generator includes a plurality of wavelet transform networks with successively decreasing scales and inverse wavelet transform networks with successively increasing scales, the input of the first-level wavelet transform network is the high-frequency information component, the output of the first-level inverse wavelet transform network is the high-frequency feature map, the input of the wavelet transform network of other levels is the first type of feature map output by the wavelet transform network of the previous adjacent level, and the input of each level of inverse wavelet transform is the first type of feature map output by the wavelet transform network of the same level and the second type of feature map output by the inverse wavelet transform network of the next level; and / or, The step of extracting a low-frequency feature map according to the low-frequency information component comprises: Splitting the low-frequency information component into a first low-frequency component and a second low-frequency component; extracting a first feature from the first low-frequency component; The first feature and the second low-frequency component are fused to obtain the low-frequency feature map.

15. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory is coupled to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the electronic device performs the image processing method according to any one of claims 1 to 14.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the image processing method according to any one of claims 1 to 14.

17. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the image processing method according to any one of claims 1 to 14.

Images