Image processing method, image processing model and electronic equipment

Abstract

The invention discloses an image processing method, an image processing model and electronic equipment, and relates to the technical field of image processing, and the method comprises the steps that the electronic equipment obtains an input image; the electronic equipment obtains a first image corresponding to the input image through a preset first super-division algorithm, and obtains a second image corresponding to the input image through a preset second super-division algorithm. And the electronic equipment performs image fusion processing on the first image and the second image based on the pseudo texture region to obtain a target image. A pseudo texture area exists in the first image, the quality of generated image details of the first image is higher than that of generated image details of the second image, and the image detail reduction degree of the second image is higher than that of the first image; the target image after fusion processing has high-reduction-degree original image details and high-quality generated image details, and the purpose of improving the overall quality of the high-resolution image is achieved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method, an image processing model, and an electronic device. Background Technology

[0002] With the development of image processing technology and the demand for high-resolution images in practical applications, image super-resolution technology has broad application prospects in many fields. Image super-resolution technology refers to generating a corresponding high-resolution image from a given low-resolution image using specific algorithms.

[0003] Image super-resolution technology has wide applications in many fields. In some practical applications, such as image compression, medical imaging, and remote sensing imaging, the images acquired directly may be low-resolution due to limitations of the acquisition equipment or the influence of the acquisition environment. In these applications, high-resolution images are required for display or subsequent image processing. In such cases, high-resolution images can be generated using super-resolution algorithms.

[0004] While existing super-resolution algorithms generate high-resolution images with new details, they also produce a lot of false textures, resulting in poor output high-resolution images. Summary of the Invention

[0005] This application provides an image processing method, an image processing model, and an electronic device. The electronic device can acquire two high-resolution images (a first image and a second image) with different performance effects through two different super-resolution algorithms. The second image has higher fidelity to the original image details, while the first image has higher quality generated image details. Based on the pseudo-texture regions in the first image, image fusion processing is performed on the first image and the second image to obtain a target image after eliminating the pseudo-texture regions. The target image after fusion processing combines the good performance effects of the first image and the second image (high fidelity to the original image details and high quality generated image details), achieving the goal of improving the overall quality of the high-resolution image.

[0006] To achieve the above objectives, the embodiments of this application adopt the following technical solutions.

[0007] Firstly, an image processing method is provided, the method comprising:

[0008] Electronic devices acquire input images.

[0009] The electronic device acquires a first image corresponding to the input image using a preset first super-resolution algorithm, and the electronic device acquires a second image corresponding to the input image using a preset second super-resolution algorithm.

[0010] The first and second images have higher resolutions than the input image, and the first image contains pseudo-texture regions. The quality of the generated image details from the first image is higher than that from the second image, and the image detail reproduction of the second image is higher than that of the first image.

[0011] The electronic device performs image fusion processing on the first image and the second image based on the pseudo-texture region to obtain the target image.

[0012] In this application, an electronic device can acquire two high-resolution images (a first image and a second image) with different performance effects through two different super-resolution algorithms. The second image has higher fidelity to the original image details, while the first image has higher quality generated image details. Based on the pseudo-texture regions in the first image, image fusion processing is performed on the first image and the second image to obtain a target image after eliminating the pseudo-texture regions. The target image after fusion processing combines the good performance effects of both the first image and the second image (high fidelity to the original image details and high quality generated image details), thereby improving the overall visual quality of the high-resolution image.

[0013] In one possible implementation of the first aspect, the output image of the preset first super-resolution algorithm has high-quality details, and the preset first super-resolution algorithm includes a super-resolution algorithm based on generative adversarial loss or a super-resolution algorithm based on a diffusion model; the output image of the preset second super-resolution algorithm has high detail with the original image, and the preset second super-resolution algorithm includes a super-resolution algorithm based on absolute value error loss or a super-resolution algorithm based on perceptual loss.

[0014] In this application, super-resolution algorithms based on generative adversarial loss or diffusion models can output high-resolution images with high-quality generated image details, but the output images may contain pseudo-texture regions. Super-resolution algorithms based on absolute value error loss or perceptual loss output high-resolution images without pseudo-texture regions, but they cannot generate high-quality image details. Electronic devices can acquire two high-resolution images with different performance effects through two different super-resolution algorithms. Based on the pseudo-texture regions in the first image, image fusion processing is performed on the first and second images. The target image after fusion processing eliminates pseudo-texture regions as much as possible, and combines high-fidelity original image details with high-quality generated image details, thereby improving the overall visual quality of the high-resolution image.

[0015] In another possible implementation of the first aspect, the method further includes:

[0016] The electronic device acquires the difference image between the input image and the first image, and determines the pseudo-texture region based on the degree of change in the grayscale values ​​of the pixels in the difference image.

[0017] In this application, the degree of change in the grayscale value of a pixel can, to some extent, reflect the texture features of the difference image. Therefore, electronic devices can determine the pseudo-texture region based on different grayscale value analysis methods. For example, the pseudo-texture region corresponding to the first image can be determined through the gray-level co-occurrence matrix (GLCM); or, the pseudo-texture region corresponding to the first image can be determined through local variance analysis. The pseudo-texture region determined based on the degree of change in the grayscale value of a pixel is relatively accurate and can provide effective data support for subsequent image fusion based on the pseudo-texture region.

[0018] In another possible implementation of the first aspect, the electronic device determines the pseudo-texture region based on the degree of change in the grayscale values ​​of pixels in the difference image, including:

[0019] The electronic device calculates the local variance of the difference image based on a preset window. The preset window is related to the size of the difference image, and the local variance is used to characterize the degree of change in the grayscale value of a pixel.

[0020] The electronic device acquires at least one high-frequency region with a local variance greater than a preset variance value, and the high-frequency region is a pseudo-texture region.

[0021] In this application, the high-frequency region is determined based on the local variance of the difference image, and the high-frequency region is used as the pseudo-texture region. This method of determining the pseudo-texture region is relatively simple, can save the computational load of electronic devices, and the obtained pseudo-texture region is also relatively accurate.

[0022] In another possible implementation of the first aspect, the electronic device performs image fusion processing on the first image and the second image based on a pseudo-texture region to obtain a target image, including:

[0023] The electronic device normalizes and stretches the local variance of the difference image to obtain a mask matrix.

[0024] The size of the mask matrix is ​​the same as that of the first image and the second image. The elements in the mask matrix represent the fusion weight of each pixel. The fusion weight of the pixels in the pseudo-texture region is greater than a preset value.

[0025] The electronic device performs weighted fusion of the first image and the second image based on a mask matrix to obtain the target image.

[0026] In this application, the electronic device can determine the corresponding mask matrix based on the pseudo-texture region. Then, weighted fusion is performed based on the mask matrix. For pseudo-texture regions, a larger proportion of pixel values ​​from the second image and a smaller proportion of pixel values ​​from the first image are fused to eliminate pseudo-texture regions in the first image as much as possible. For non-pseudo-texture regions, a larger proportion of pixel values ​​from the first image and a smaller proportion of pixel values ​​from the second image are fused to preserve high-quality generated image details in the non-pseudo-texture regions of the first image as much as possible, resulting in a target image with high visual quality after fusion processing.

[0027] In another possible implementation of the first aspect, the electronic device determines the pseudo-texture region based on the degree of change in the grayscale values ​​of pixels in the difference image, including:

[0028] The electronic device calculates the local variance of the difference image based on a preset window. The preset window is related to the size of the difference image, and the local variance is used to characterize the degree of change in the grayscale value of a pixel.

[0029] The electronic device normalizes and stretches the local variance of the difference image to obtain a mask matrix.

[0030] The size of the mask matrix is ​​the same as that of the first image and the second image. The elements in the mask matrix represent the fusion weight of each pixel. The regions in the mask matrix with fusion weights greater than a preset value are pseudo-texture regions.

[0031] Accordingly, the electronic device performs image fusion processing on the first image and the second image based on the pseudo-texture region to obtain the target image, including:

[0032] The electronic device performs weighted fusion of the first image and the second image based on a mask matrix to obtain the target image.

[0033] In this application, the electronic device can determine the mask matrix based on the local variance of the difference image. Then, weighted fusion is performed based on the mask matrix. For pseudo-texture regions, a larger proportion of pixel values ​​from the second image and a smaller proportion of pixel values ​​from the first image are fused to eliminate pseudo-texture regions in the first image as much as possible. For non-pseudo-texture regions, a larger proportion of pixel values ​​from the first image and a smaller proportion of pixel values ​​from the second image are fused to preserve high-quality generated image details in the non-pseudo-texture regions of the first image as much as possible, resulting in a target image with high visual quality after fusion processing.

[0034] In another possible implementation of the first aspect, the electronic device performs a weighted fusion of the first image and the second image based on a mask matrix to obtain a target image, including:

[0035] The electronic device uses the following expression to perform weighted fusion of the first image and the second image to obtain the target image;

[0036] The expression is represented as:

[0037] S = (1-M)*S1 + M*S2.

[0038] Where S is the target image, M is the mask matrix, S1 is the first image, and S2 is the second image.

[0039] In this application, the weighted fusion of the first image and the second image is performed by the electronic device mask matrix M, which can ensure the elimination of pseudo-texture regions and the preservation of non-pseudo-texture regions as much as possible. This allows the target image to maintain high-quality generated image details while eliminating pseudo-textures as much as possible, thus the target image has high visual quality.

[0040] In another possible implementation of the first aspect, the electronic device performs image fusion processing on the first image and the second image based on a pseudo-texture region to obtain a target image, including:

[0041] The electronic device obtains a first region corresponding to the pseudo-texture region in a first image based on the location of the pseudo-texture region, and the electronic device obtains a second region corresponding to the pseudo-texture region in a second image based on the location of the pseudo-texture region.

[0042] The electronic device replaces the first region in the first image with the second region to obtain the target image.

[0043] In this application, the electronic device directly replaces the first region (pseudo-texture region) of the first image with the second region based on the coordinate positions of each pixel in the pseudo-texture region, thereby reducing the pseudo-texture region in the first image. After the region replacement processing, the first image (target image) has reduced the pseudo-texture region while still maintaining high-quality generated image details; therefore, the target image has high visual quality.

[0044] In another possible implementation of the first aspect, the electronic device acquires a difference image between the input image and the first image, including:

[0045] The electronic device performs low-pass filtering on the first image to obtain the filtered first image. The processing kernel corresponding to the low-pass filtering is a preset value.

[0046] The electronic device calculates the difference between the grayscale values ​​of each pixel in the first image after filtering and the corresponding pixel in the input image, and obtains the difference image.

[0047] In this application, the first image obtained based on a preset first super-resolution algorithm may contain high-frequency noise details. Therefore, the electronic device can perform low-pass filtering on the first image. The purpose of low-pass filtering is to filter out details above a certain frequency. Some details above a certain frequency may be invalid high-frequency noise details. After low-pass filtering, the invalid high-frequency noise details are removed from the first image, making the information represented by the first image more accurate. Based on the low-pass filtered first image, the difference image between it and the input image can be calculated, which can improve the data validity of the difference image.

[0048] In another possible implementation of the first aspect, the method further includes:

[0049] The electronic device inputs the input image and the first image into a deep learning model to obtain pseudo-texture regions.

[0050] Although the method for determining pseudo-texture regions based on artificial intelligence deep learning models in this application involves model training, the pseudo-texture regions obtained in this way are relatively accurate and can provide effective data support in subsequent image fusion processing based on pseudo-texture regions.

[0051] Secondly, an image processing model is provided, which includes a first super-resolution module, a second super-resolution module, a pseudo-texture detection module, and a fusion module.

[0052] The first super-resolution module is used to obtain the first image corresponding to the input image through a preset first super-resolution algorithm.

[0053] The second super-resolution module is used to obtain the second image corresponding to the input image through a preset second super-resolution algorithm.

[0054] The pseudo-texture detection module is used to obtain the difference image between the input image and the first image, and to determine the pseudo-texture region based on the degree of change in the grayscale values ​​of the pixels in the difference image.

[0055] The fusion module is used to perform image fusion processing on the first image and the second image based on the pseudo-texture region to obtain the target image.

[0056] The first and second images have higher resolutions than the input image. The first image contains pseudo-texture regions. The quality of the generated image detail from the first image is higher than that from the second image, and the second image has a higher degree of detail reproduction than the first image.

[0057] In this application, an image processing method is implemented by constructing an image processing model. The first super-resolution module and the second super-resolution module in the image processing model can respectively use two different super-resolution algorithms to obtain two high-resolution images with different performance effects. Thus, the image fusion module can perform image fusion processing on the first image and the second image based on the pseudo-texture region in the first image. The target image after fusion processing eliminates the pseudo-texture region as much as possible, and has both high-fidelity original image details and high-quality generated image details, thereby achieving the effect of improving the overall visual quality of the high-resolution image.

[0058] In one possible implementation of the second aspect, the image processing model also includes an encoding module.

[0059] The encoding module is used to encode the features of the input image and obtain the image features corresponding to the input image.

[0060] The first super-resolution module is used to decode image features using a preset first super-resolution algorithm to obtain the first image.

[0061] The second super-resolution module is used to decode image features using a preset second super-resolution algorithm to obtain a second image.

[0062] In this application, the image processing model can also include an encoding module to process the input image to the image feature dimension, thereby making the image processing more accurate.

[0063] Thirdly, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in any of the first aspects above.

[0064] Fourthly, a computer-readable storage medium is provided that stores instructions which, when executed by a processor, implement the steps of the method described in any of the first aspects above.

[0065] Fifthly, a computer program product comprising instructions is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in any of the first aspects above.

[0066] In a sixth aspect, embodiments of this application provide a chip including a processor, the processor being configured to invoke a computer program in memory to perform the method as described in any one of the first aspects.

[0067] Understandably, the beneficial effects that the electronic device described in the third aspect, the computer-readable storage medium described in the fourth aspect, the computer program product described in the fifth aspect, and the chip described in the sixth aspect can achieve can be referred to the first aspect and any possible design mode thereof, or the beneficial effects of the first aspect and any possible design mode thereof, which will not be repeated here. Attached Figure Description

[0068] Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0069] Figure 2 A software structure block diagram of an electronic device provided in an embodiment of this application;

[0070] Figure 3 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0071] Figure 4 This is a schematic diagram of a process image in an image processing method provided in an embodiment of this application;

[0072] Figure 5 A schematic diagram of a difference image provided in an embodiment of this application;

[0073] Figure 6 A diagram illustrating a difference image and local variance provided in an embodiment of this application;

[0074] Figure 7 This is a schematic diagram of the structure of an image processing model provided in an embodiment of this application;

[0075] Figure 8 A schematic diagram illustrating a specific scenario of an image processing method provided in an embodiment of this application;

[0076] Figure 9 A schematic diagram illustrating a specific scenario of another image processing method provided in an embodiment of this application;

[0077] Figure 10 A schematic diagram illustrating a specific scenario of another image processing method provided in an embodiment of this application;

[0078] Figure 11 A possible structural schematic diagram of an electronic device provided in an embodiment of this application;

[0079] Figure 12 This is a schematic diagram of a chip system provided in an embodiment of this application. Detailed Implementation

[0080] In the description of the embodiments of this application, the terminology used in the following embodiments is for the purpose of describing specific embodiments only and is not intended to be a limitation of this application. As used in the specification and appended claims of this application, the singular expressions "a," "the," "the," "the," and "this" are intended to also include expressions such as "one or more," unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, "at least one" and "one or more" refer to one or more (including two). The term "and / or" is used to describe the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can indicate: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0081] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized. The term "connection" includes direct connections and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated.

[0082] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0083] With the development of image processing technology and the demand for high-resolution images in practical applications, image super-resolution technology has broad application prospects in many fields. Image super-resolution technology refers to generating a corresponding high-resolution image from a given low-resolution image using specific algorithms; this process can also be called super-resolution image reconstruction.

[0084] In some practical applications, due to limitations of the acquisition equipment or the influence of the acquisition environment, it may be impossible to directly acquire high-resolution images. In such cases, super-resolution image reconstruction based on low-resolution images is necessary to obtain higher-resolution images to meet the application requirements. For example, in applications with high real-time requirements such as video conferencing, images are often compressed to reduce storage space and transmission bandwidth needs. Compressed images are low-resolution images; super-resolution image reconstruction can be performed on these low-resolution images to recover high-resolution images for use, thus balancing the conflict between storage and transmission requirements and image quality. Similarly, some electronic devices with camera functions may have limitations in camera technology that prevent them from meeting users' needs for high-resolution images. These devices can acquire low-resolution images using cameras and then perform super-resolution image reconstruction to obtain higher-resolution images to satisfy users' demands for high-resolution images. Furthermore, in medical imaging, image acquisition equipment often has technological limitations, resulting in low-resolution medical images. Super-resolution image reconstruction of these low-resolution images yields high-resolution medical images, which helps to more accurately detect even minute lesions.

[0085] Super-resolution image reconstruction has advantages in various technical fields and specific application scenarios, leading to a proliferation of super-resolution image reconstruction algorithms (or super-resolution algorithms for short). Different super-resolution algorithms exhibit varying advantages and disadvantages.

[0086] For example, super-resolution algorithms can include those based on generative adversarial loss (GAR), those based on diffusion models, and so on. For instance, GAR or diffusion-based super-resolution algorithms output high-resolution images with high-quality image details, but they may also contain pseudo-texture regions. Specifically, GAR uses a discriminant network to differentiate between super-resolution and real images, diversifying the output of the solution model. However, in GAR, the training of the GAR network is unstable, often introducing unwanted pseudo-textures while generating details, resulting in pseudo-texture regions in the high-resolution image. Diffusion-based super-resolution algorithms include forward diffusion and backward diffusion. Forward diffusion starts with a real image and gradually adds noise. Backward diffusion trains a model to recover the image from noise. In the inference phase, given a low-resolution image, the diffusion model generates a high-resolution image through backward diffusion. Because the generated images are relatively random during diffusion, the details in some images differ significantly from the actual content of the low-resolution image, resulting in pseudo-texture regions in the high-resolution image. This results in poor quality output high-resolution images.

[0087] Super-resolution algorithms can also include those based on absolute value loss, perceptual loss, residual dense networks, and residual channel attention networks. For example, super-resolution algorithms based on absolute value loss or perceptual loss produce high-resolution images without pseudo-texture regions, but they also lack high-quality image details. Specifically, these algorithms use pixel-level loss functions. During high-resolution image generation, the network tends to find the pixel-level average value and reconstruct the high-resolution image based on this average. This results in an overly smooth high-resolution image, losing higher-quality image details. Perceptual loss-based super-resolution algorithms mitigate this problem to some extent, but these algorithms generally prioritize restoring original image details while outputting a high-resolution image, thus losing new, high-quality details. While absolute value loss-based super-resolution algorithms are simple and easy to use, they can also lead to loss of image details, resulting in poor-quality high-resolution images.

[0088] Moreover, although these super-resolution algorithms each have their own advantages, the overall image quality of the high-resolution images obtained by these algorithms is not good. The presence of pseudo-texture areas in the generated images or the inability to generate high-quality image details will affect the visual quality of the images.

[0089] This application provides an image processing method in its embodiments.

[0090] Electronic devices can acquire two high-resolution images (a first image and a second image) with different performance effects using two different super-resolution algorithms. Compared to the first image, the second image has higher fidelity to the original image details, while the first image has higher quality generated image details compared to the second image. Based on characterizing the pseudo-texture regions in the first image, image fusion processing is performed on the first and second images to obtain a target image after eliminating the pseudo-texture regions. Moreover, the fused target image combines the good performance effects of both the first and second images (high fidelity to the original image details and high quality generated image details), achieving the goal of improving the overall quality of the high-resolution image.

[0091] The image processing method provided in this application embodiment can be applied to electronic devices. Exemplarily, the electronic device can be a portable computer (such as a mobile phone), tablet computer, laptop computer, personal computer (PC), wearable electronic device (such as a smartwatch), in-vehicle computer, server (standalone server or server cluster), etc. The following embodiments do not impose any special limitations on the specific form of the electronic device.

[0092] Figure 1 A schematic diagram of the electronic device is shown.

[0093] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, sensor module 180, camera 193, and display screen 194, etc.

[0094] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0095] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0096] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0097] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0098] In this embodiment, multiple super-resolution algorithms can be deployed in the processor 110. The processor can call different super-resolution algorithms to obtain high-resolution images corresponding to low-resolution images. Different super-resolution algorithms have different algorithmic advantages, and the different high-resolution images output have different image quality and image effects. The processor 110 can perform image fusion processing based on at least two high-resolution images to obtain a target image with different image effects, thereby solving the contradiction between the inability to balance high-quality image details and the presence of false textures in the image.

[0099] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0100] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0101] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0102] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, external memory, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0103] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0104] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0105] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0106] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to a speaker, receiver, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0107] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR). The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0108] In this embodiment, the electronic device acts as an image processing server. It can receive low-resolution images sent by other devices, perform image processing on the low-resolution images to obtain high-resolution images, and return the high-resolution images to the corresponding devices. For example, in a specific application scenario, such as road monitoring and management, the central control server can communicate with multiple image acquisition devices installed on the road to acquire low-resolution images collected by these devices, perform image processing on the low-resolution images, and obtain high-resolution images for road monitoring and analysis.

[0109] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0110] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0111] In this embodiment, when the electronic device is a device with a display screen, the electronic device can display a high-resolution image after image processing on the display screen. For example, when the electronic device is a mobile phone, if the image captured by the mobile phone's camera is a low-resolution image, the mobile phone can perform image processing on the captured low-resolution image to obtain a corresponding high-resolution image. In one specific scenario, the mobile phone can perform image processing on a preview frame image and display the image-processed high-resolution preview frame image on the preview interface of the display screen; or, in another specific scenario, the mobile phone can respond to a shooting operation, acquire a low-resolution image captured by the camera, perform image processing on the low-resolution image to obtain a corresponding high-resolution image, and store it in the gallery. The mobile phone responds to the user's operation of launching the gallery and displays the gallery interface on the display screen. The gallery interface may include thumbnails of high-resolution images. In other scenarios, for example, in some applications / services used for image processing, after acquiring a low-resolution image, the application / service can respond to user-triggered image optimization / image enhancement operations, call the image processing method provided in this embodiment to acquire a high-resolution image, and then display the image-processed high-resolution image on the display interface.

[0112] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0113] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0114] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0115] In this embodiment, when the electronic device is equipped with a camera, it can perform image processing on the low-resolution image captured by the camera to output a high-resolution image. For example, when the electronic device takes pictures at night, due to the nighttime shooting environment, the image directly captured by the camera may be a low-resolution image. In this case, the electronic device can perform image processing on the low-resolution image to obtain a corresponding high-resolution image for display or other processing.

[0116] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP performs Fourier transforms on the frequency energy.

[0117] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0118] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0119] Internal memory 121 can be used to store computer executable program code, which includes instructions. Processor 110 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0120] Electronic device 100 can implement audio functions such as music playback and recording through audio module 170, speaker, receiver, microphone, headphone jack, and application processor.

[0121] Taking an electronic device as an example, which includes modules such as a camera, a display screen, and a processor, the electronic device can capture low-resolution images through the camera, process the low-resolution images through the processor to obtain high-resolution images corresponding to the low-resolution images, and display the high-resolution images through the display screen.

[0122] The software system of electronic devices can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This invention embodiment uses a layered architecture. Taking the system as an example, the software structure of electronic device 100 is illustrated.

[0123] Figure 2 This is a software structure block diagram of an electronic device according to an embodiment of the present invention.

[0124] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, [the following is omitted as the text is incomplete and likely refers to a specific implementation or feature]. The system is divided into four layers, from top to bottom: application layer, application framework layer, hardware abstraction layer (HAL), and hardware layer.

[0125] The application layer can include a series of application packages. For example... Figure 2 As shown, the application package can include camera apps or third-party apps with shooting capabilities, as well as gallery apps. Additionally, the application package may include applications such as calendar, maps, WLAN, and Bluetooth.

[0126] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example, ... Figure 2 As shown, the application framework layer may include a camera module interface. The camera or third-party applications in the application layer can send control commands to the HAL layer, or receive data returned by the HAL layer, through the camera module interface.

[0127] In addition, in some embodiments, the application framework layer may also include a window manager, a content provider, a view system, a resource manager, etc. The window manager manages window programs. It can obtain the screen size, determine if a status bar is present, lock the screen, and capture the screen. The content provider stores and retrieves data, making this data accessible to the application. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc. The view system includes visual controls, such as controls for displaying text and controls for displaying images. The view system can be used to build the application. The display interface can consist of one or more views. For example, a display interface including a text notification icon may include a view for displaying text and a view for displaying images. The resource manager provides the application with various resources, such as localized strings, icons, images, layout files, video files, etc.

[0128] The HAL layer includes the camera HAL. The camera HAL provides an interface for unified management of other underlying devices that can be accessed by the camera or camera module. An image processing module is also deployed in the HAL layer. This image processing module is used to execute the image processing methods provided in this embodiment.

[0129] The hardware layer includes the ISP, camera, and display screen.

[0130] The camera is used to acquire data upon receiving a shooting command. The data acquired by the camera is RAW data. The ISP converts the RAW data acquired by the camera into YUV data that can be processed by the HAL layer / CPU / GPU. The ISP can also perform pre-processing on the YUV data images, such as denoising, cropping, enhancement, and sharpening.

[0131] In some embodiments, a kernel layer is further included between the HAL layer and the hardware layer. The kernel layer includes drivers corresponding to various devices in the hardware layer, used to enable the devices and issue instructions and upload data corresponding to each device. For example, the kernel layer includes camera drivers, display drivers, sensor drivers, etc.

[0132] Taking the scenario where an electronic device captures a low-resolution image through a camera, processes the image, and outputs a high-resolution image as an example, this illustrates the workflow of the electronic device's software and hardware.

[0133] In response to a user's activation of the camera application, the electronic device displays a preview interface of the camera application on its screen. The electronic device can perform image processing on the preview frame image. For example, the RAW data of the initial preview frame image captured by the camera is processed by the ISP and outputs processable YUV data of the initial preview frame image to the camera HAL. The camera HAL transmits the YUV data of the initial preview frame image to the image processing module. The image processing module performs image processing on the initial preview frame image based on the acquired YUV data. The image processing module can obtain a first image corresponding to the initial preview frame image using a preset first super-resolution algorithm, determine the pseudo-texture region in the first image based on the first image and the initial preview frame image, and obtain a mask matrix based on the pseudo-texture region. The image processing module can obtain a second image corresponding to the initial preview frame image using a preset second super-resolution algorithm. Based on the calculated mask matrix, the image processing module can perform fusion processing on the first image and the second image to obtain a target image corresponding to the initial preview frame image, where the resolution of the target image is higher than that of the initial preview frame image. The processed target image can then be returned to the camera application via the camera HAL. The camera application controls the display screen to show the target image through the camera module interface and the camera HAL.

[0134] For example, when the camera application receives a user-triggered shooting operation, it generates an image acquisition command and controls the camera to acquire an initial image via the camera module interface and camera HAL. Similar to the processing flow of the preview frame image, the camera transmits the RAW data of the acquired initial image to the ISP for data format conversion and image pre-processing. The initial image after image pre-processing is then transmitted to the image processing module. The image processing module uses the image processing method provided in this embodiment to obtain the target image corresponding to the initial image, where the resolution is higher than that of the initial image. The target image after image processing can be returned to the camera application via the camera HAL. The camera application controls the display screen to display the target image through the camera module interface and camera HAL.

[0135] This embodiment provides an image processing method. After image processing, an initial input image can be obtained as a target image with high fidelity to the original image details and high-quality generated image details. The target image has a higher resolution than the input image, thus achieving the goal of improving the overall quality of a high-resolution image. Taking an electronic device as the execution subject as an example, exemplarily... Figure 3 A flowchart of an image processing method is given, combined with... Figure 4 A schematic diagram of the process image in the image processing method is given to illustrate the method. The image processing method includes:

[0136] S201, The electronic device acquires the input image.

[0137] The input image is a low-resolution image, with a resolution lower than that of the target image after image processing. The electronic device can either receive the input image sent by other devices, or it can control an image acquisition device to acquire the input image.

[0138] In some embodiments, after acquiring an input image, the electronic device can directly perform image processing on the input image to obtain a target image with a higher resolution than the input image.

[0139] Alternatively, in some other feasible embodiments, after acquiring the input image, the electronic device can make a preliminary judgment on the resolution of the input image. If it is determined that the resolution of the input image is relatively low, the image processing method provided in this embodiment is triggered to acquire the high-resolution image corresponding to the input image.

[0140] For example, the electronic device's preliminary determination of the input image's resolution may include invoking a preset detection plugin to determine the corresponding resolution of the input image. For instance, the preset tool could be a dots per inch (DPI) detection plugin; the higher the DPI value of the input image, the higher the image resolution generally is. When the DPI value of the input image is less than a preset DPI value, the input image can be determined to be a low-resolution image, triggering the electronic device to use the image processing method provided in this embodiment to obtain the corresponding high-resolution image. Alternatively, the electronic device can also input the input image into a preset algorithm model to obtain the corresponding resolution of the input image. For example, the preset algorithm model could be a Region Proposal Network (R-CNN) series model, which determines the image resolution by identifying and locating targets in the input image and analyzing the size and sharpness of the targets. When the resolution of the input image is less than a preset resolution value, the input image can be determined to be a low-resolution image, triggering the electronic device to use the image processing method provided in this embodiment to obtain the corresponding high-resolution image.

[0141] S202. The electronic device obtains the first image corresponding to the input image through a preset first super-resolution algorithm.

[0142] The resolution of the first image is higher than that of the input image.

[0143] In this embodiment, the first super-resolution algorithm is one that can generate high-quality image details. For example, the first super-resolution algorithm can be a super-resolution algorithm based on generative adversarial loss or a super-resolution algorithm based on a diffusion model.

[0144] Super-resolution with GAN loss is a method that uses generative adversarial networks (GANs) to improve image resolution. The core idea is to use a GAN generator to produce high-resolution images and then use a discriminator to evaluate the realism of the generated images, thereby guiding the generator to produce higher-quality super-resolution images.

[0145] Super-resolution algorithms based on diffusion models are methods that utilize a diffusion process to gradually improve image resolution. These algorithms typically involve an iterative process that transforms a low-resolution image into a high-resolution image through progressive denoising.

[0146] Both super-resolution algorithms based on generative adversarial loss and those based on diffusion models can output high-quality generated image details. However, the high-resolution images generated by these algorithms often contain pseudo-texture regions. In this embodiment, generated image details refer to new image details generated by the super-resolution algorithm that are distinct from the original image details in the input image.

[0147] For example, refer to Figure 4 The first image S1 shown corresponds to the input image obtained by the preset first super-resolution algorithm. Some grid edges in the first image S1 have jagged pseudo-textures, which form pseudo-texture regions.

[0148] In this embodiment, no internal improvements were made to the super-resolution algorithm based on the diffusion model or the super-resolution algorithm based on generative adversarial loss. The specific algorithm logic of the super-resolution algorithm based on generative adversarial loss and the super-resolution algorithm based on the diffusion model is not limited or elaborated.

[0149] S203. The electronic device determines the pseudo-texture region in the first image based on the first image and the input image.

[0150] In this embodiment, since the first image is a high-resolution image obtained through a preset first super-resolution algorithm, although the first image has high-quality generated image details, it contains pseudo-texture regions. To obtain a target image without pseudo-texture regions or with smaller pseudo-texture regions, this embodiment can determine the pseudo-texture regions based on the first image and the input image.

[0151] In some embodiments, the electronic device acquires pseudo-texture regions in various ways. In one feasible approach, the electronic device can input a first image and an input image into an artificial intelligence deep learning model to acquire pseudo-texture regions present in the first image compared to the input image. For example, the artificial intelligence deep learning model can be a multi-attention deep forgery detection model or an adversarial generative image detection model based on forgery traces in complex texture regions, etc. This embodiment does not improve the model structure of the artificial intelligence deep learning model, and the specific model structure is not limited or elaborated. Although the method for determining pseudo-texture regions based on the artificial intelligence deep learning model in this embodiment involves model training, the pseudo-texture regions obtained in this way are relatively accurate and can provide effective data support in subsequent image fusion processing based on the pseudo-texture regions.

[0152] In some embodiments, a pseudo-texture region is determined, and the remaining region in the first image is the non-pseudo-texture region. A mask image can be obtained based on the pseudo-texture region and the non-pseudo-texture region. Similar to... Figure 4The mask image shown is an example of a mask where highlighted areas represent pseudo-texture regions and shadowed areas represent non-pseudo-texture regions. In the mask image, the pixel values ​​in the non-pseudo-texture regions are preset values, which can be 0 or other specified values. The pixel values ​​in the pseudo-texture regions are the calculated values.

[0153] In another feasible approach, the electronic device can also determine the pseudo-texture regions in the first image based on a more typical computational method. Exemplarily, this includes:

[0154] S301, The electronic device acquires the difference image between the input image and the first image.

[0155] In this embodiment, the electronic device can obtain a difference image (S1-LR) by calculating the difference between the grayscale values ​​of each pixel in the first image S1 and the corresponding pixel in the input image LR. For example, Figure 5 A schematic diagram of a difference image is given. Electronic devices can calculate the difference between the gray values ​​of each pixel in image 1 and the corresponding pixel in image 2 to obtain the corresponding difference image 1.

[0156] Alternatively, in one implementation, the electronic device may first filter the first image, and then calculate the difference image between the filtered first image and the input image.

[0157] For example, in this embodiment, the first image obtained based on the preset first super-resolution algorithm may contain high-frequency noise details. Therefore, the electronic device can perform low-pass filtering on the first image. The purpose of low-pass filtering is to filter out details above a certain frequency. Some details above a certain frequency may be invalid high-frequency noise details. After low-pass filtering, the invalid high-frequency noise details are removed from the first image, making the information represented by the first image more accurate. Based on the low-pass filtered first image (S1-m), the difference image (S1-d) between it and the input image can be calculated, which can improve the data validity of the difference image.

[0158] The electronic device can perform low-pass filtering on the first image using various low-pass filters. For example, the low-pass filter can be an ideal low-pass filter, a Butterworth low-pass filter, a Gaussian low-pass filter, a mean filter, a median filter, etc. This embodiment does not improve the model structure of the low-pass filter, and the specific principles of the low-pass filter are not elaborated upon.

[0159] In this embodiment, the processing kernel of the low-pass filter can be set according to the image type of the input image being processed or the application scenario of the image processing method. That is, the processing kernel of the low-pass filter is a preset value.

[0160] S302. The electronic device determines the pseudo-texture region based on the degree of change in the grayscale value of the pixels in the difference image.

[0161] In this embodiment, the electronic device can analyze the degree of change in the grayscale values ​​of pixels in the difference image in various ways to determine the pseudo-texture region corresponding to the first image.

[0162] For example, an electronic device can determine the pseudo-texture region corresponding to the first image using a gray-level co-occurrence matrix (GLCM). Specifically, the electronic device can calculate the difference in gray values ​​between pixel pairs in different directions to obtain texture features of the difference image, such as energy, contrast, uniformity, and correlation. High-frequency regions, i.e., pseudo-texture regions, are determined based on these texture features. Alternatively, the electronic device can determine the pseudo-texture region corresponding to the first image using local binary patterns (LBP). Specifically, the electronic device can extract texture information by comparing the gray values ​​of the center pixel with its neighboring pixels to obtain local texture features, and determine high-frequency regions, i.e., pseudo-texture regions, based on these local texture features. Alternatively, the electronic device can determine the pseudo-texture region corresponding to the first image using image gradients. Specifically, the electronic device detects edges by calculating the rate of change of intensity between pixels in the difference image. In the difference image, regions with large gradients usually correspond to edges or areas with drastic texture changes; these regions may be pseudo-texture regions. Alternatively, the electronic device can determine the pseudo-texture region corresponding to the first image using local variance analysis. In the difference image, regions with large local variance (high-frequency regions) may correspond to pseudo-texture regions. Specifically, electronic devices can construct a local variance map by calculating the variance of the neighborhood of each pixel in an image, thereby identifying pseudo-texture regions.

[0163] Taking the example of how an electronic device can determine the pseudo-texture region corresponding to the first image through local variance analysis, the following is specifically included:

[0164] S3021. The electronic device calculates the local variance of the difference image according to the preset window.

[0165] In this embodiment, the preset window is a preset value. For example, the preset window can be a sliding window of different sizes, such as 2*2 or 3*3, and the preset window is related to the size of the difference image. When the size of the difference image is relatively large, the preset window can be designed to be slightly larger, such as 3*3 or 4*4, to improve the effect of local variance analysis. If the size of the difference image is relatively small, the preset window can be designed to be slightly smaller, such as 2*2 or 3*3, to improve the data validity and accuracy of the local variance analysis. In addition, the preset window can also be related to the overall numerical distribution of the difference image. For example, when the numerical distribution range is relatively large, the preset window can be set smaller to avoid data deviation; when the numerical distribution range is relatively small, the preset window can be set larger to improve computational efficiency.

[0166] Local variance characterizes the degree of change in the grayscale value of a pixel. In this embodiment, the local variance of the difference image is calculated. A greater change in grayscale value indicates a greater amount of variation between pixels in the difference image. Large changes in the difference image mean that the two images do not match; that is, the first image has generated new image features relative to the input image. These new image features refer to the generated pseudo-texture regions. Therefore, in this embodiment, the pseudo-texture regions can be determined based on the local variance of the difference image.

[0167] For example, Figure 6 A difference image and a schematic diagram of local variance are presented. (Reference) Figure 6 For the difference image, each pixel has a difference value. The local variance corresponding to each pixel in the difference image is calculated using a preset window to obtain the local variance corresponding to each pixel.

[0168] S3022. The electronic device acquires at least one high-frequency region where the local variance is greater than a preset variance value.

[0169] If the local variance of pixels in the high-frequency region M-loc is greater than a preset variance value, it means that the value difference between adjacent pixels in the high-frequency region is large, and the grayscale value change between pixels in the difference image is large. It also means that these pixels with large changes do not match in the two images corresponding to the difference image. That is, the first image generates new image features relative to the input image; these new image features refer to the generated pseudo-texture regions. Therefore, in this embodiment, high-frequency regions with local variance greater than the preset variance value can be considered pseudo-texture regions. For example, assuming the preset variance value is 4 / 81, then the regions marked by the dashed box in the image with local variance values ​​greater than 4 / 81 are all high-frequency regions. These high-frequency regions are pseudo-texture regions. In this embodiment, the preset variance value can be a fixed value, or it can be determined based on the numerical distribution of the local variance.

[0170] In this embodiment, the high-frequency region is determined based on the local variance of the difference image, and thus the high-frequency region is used as the pseudo-texture region. This method of determining the pseudo-texture region is relatively simple, can save the computational load of electronic devices, and the obtained pseudo-texture region is also relatively accurate.

[0171] In this embodiment, the pseudo-texture region can be further processed to obtain the mask matrix corresponding to the pseudo-texture region, providing data support for subsequent image fusion.

[0172] For example, in one implementation, the electronic device obtains the mask matrix corresponding to the pseudo-texture region by performing local normalization and affine transformation on the difference image. The affine transformation process can, for example, include stretching, scaling, rotation, and cropping. The elements in the mask matrix represent the fusion weights of each pixel. The fusion weights of pixels in non-pseudo-texture regions are preset values, or the fusion weights of pixels in non-pseudo-texture regions are less than a preset value. Generally, the fusion weights of pixels in non-pseudo-texture regions are close to or equal to 0. That is, the fusion weights of pixels in non-pseudo-texture regions in the mask matrix are much smaller than the fusion weights of pixels in pseudo-texture regions.

[0173] S204. The electronic device obtains the second image corresponding to the input image through a preset second super-resolution algorithm.

[0174] The second image has a higher resolution than the input image.

[0175] In this embodiment, the second super-resolution algorithm is one that can generate image details with high fidelity. For example, the second super-resolution algorithm can be a super-resolution algorithm based on absolute value error loss or a super-resolution algorithm based on perceptual loss.

[0176] Super-resolution algorithms based on absolute value error loss typically focus on minimizing the mean absolute error between the predicted image and the true high-resolution image. During the network model training process of these algorithms, absolute value error loss is used to optimize the network, ensuring that the error between the output high-resolution image and the real image is minimized, thus achieving the goal of outputting image details with high fidelity.

[0177] Super-resolution algorithms based on perceptual loss primarily utilize information from high-level features in deep learning models to train the network, thereby improving the visual quality of super-resolution images. By using perceptual loss, these algorithms make super-resolution images visually more realistic, better preserving image texture and details.

[0178] Both super-resolution algorithms based on absolute value error loss and those based on perceptual loss can output images with high fidelity to the original image details. However, while striving to restore as much detail as possible, these algorithms often fail to generate high-quality image details. For example, refer to... Figure 4 The second image S2 shown corresponds to the input image obtained by the preset second super-resolution algorithm. The image detail quality is low in some regions of the second image S2.

[0179] In this embodiment, no internal improvements were made to the super-resolution algorithm based on absolute value error loss or the super-resolution algorithm based on perceptual loss. The specific algorithm logic of the super-resolution algorithm based on absolute value error loss or the super-resolution algorithm based on perceptual loss is not limited or described in detail.

[0180] S205. The electronic device performs image fusion processing on the first image and the second image based on the pseudo-texture region to obtain the target image.

[0181] In this embodiment, after obtaining the pseudo-texture region, the electronic device can directly perform image fusion processing on the first image and the second image based on the pseudo-texture region to obtain the target image. Alternatively, the electronic device can determine a mask matrix based on the pseudo-texture region, and perform image fusion processing on the first image and the second image based on the mask matrix to obtain the target image. The elements in the mask matrix are used to represent the fusion weights corresponding to each pixel.

[0182] In one implementation, the electronic device can determine a first region in the first image formed by pixels corresponding to these coordinate positions, and a second region in the second image formed by pixels corresponding to these coordinate positions, based on the coordinate positions of the pseudo-texture regions in the difference image / local variance image. As seen in embodiments S202 and S204 above, the pseudo-texture problem is more severe in the first image than in the second image. Therefore, the electronic device can directly replace the first region (pseudo-texture region) of the first image with the second region to reduce the pseudo-texture region in the first image. After region replacement processing, the first image (target image) has fewer pseudo-texture regions while still maintaining high-quality generated image details, thus achieving a higher super-resolution effect.

[0183] In another implementation, the electronic device can further refine the granularity of the fusion of the first and second images.

[0184] In the above embodiments, after obtaining the pseudo-texture region, the electronic device can normalize and stretch the local variance of the difference image to obtain a mask matrix M (refer to embodiment S302 above). Then, based on the mask matrix M, the electronic device performs weighted fusion of the first image and the second image to obtain the target image.

[0185] Alternatively, in another feasible embodiment, the electronic device may acquire the mask matrix in a manner that includes:

[0186] After acquiring the difference image between the first image and the input image, the electronic device calculates the local variance of the difference image (refer to embodiment S3021 above), and directly performs normalization and stretching transformation on the local variance of the difference image to obtain the mask matrix M. Then, based on the mask matrix M, the electronic device performs weighted fusion of the first image and the second image to obtain the target image.

[0187] In this embodiment, based on the fact that the pixel values ​​corresponding to pseudo-texture regions in the mask matrix M are greater than the pixel values ​​corresponding to non-pseudo-texture regions, the key to weighted fusion of the first and second images is that, for pseudo-texture regions, a larger proportion of the pixel values ​​from the second image and a smaller proportion of the pixel values ​​from the first image are fused to eliminate pseudo-texture regions in the first image as much as possible. For non-pseudo-texture regions, a larger proportion of the pixel values ​​from the first image and a smaller proportion of the pixel values ​​from the second image are fused to preserve as much high-quality generated image detail as possible in the non-pseudo-texture regions of the first image.

[0188] Based on this fusion approach, the following expression can be used to perform weighted fusion of the first image S1 and the second image S2 to obtain the target image S.

[0189] The expression can be represented as:

[0190] S = (1-M)*S1 + M*S2.

[0191] In this embodiment, the weighted fusion of the first image and the second image is performed using the electronic device mask matrix M. This can ensure the elimination of pseudo-texture regions and the preservation of non-pseudo-texture regions as much as possible. This allows the target image to maintain high-quality generated image details while eliminating pseudo-textures as much as possible, thus achieving a high super-resolution effect.

[0192] In this embodiment, the electronic device can acquire two high-resolution images (a first image and a second image) with different performance effects through two different super-resolution algorithms. Compared with the first image, the second image has higher fidelity to the original image details, while the first image has higher quality generated image details compared with the second image. Based on characterizing the pseudo-texture regions in the first image, image fusion processing is performed on the first image and the second image to obtain the target image after eliminating the pseudo-texture regions. The target image after fusion processing combines the good performance effects of the first image and the second image (high fidelity to the original image details and high quality generated image details), achieving the goal of improving the overall quality of the high-resolution image.

[0193] In some embodiments, a network model can be constructed to implement the image processing method provided in the embodiments of this application. For example, Figure 7 A schematic diagram of an image processing model is given.

[0194] The image processing model includes a first super-resolution module, a second super-resolution module, a pseudo-texture detection module, and a fusion module. In some optional embodiments, the image processing model may also include an encoder module.

[0195] The encoding module encodes the input image, extracts its features, and outputs the features. The image features corresponding to the input image are transmitted to the first super-resolution module, which decodes them using a preset first super-resolution algorithm to obtain a high-resolution first image S1 (see S202 above). The image features corresponding to the input image are then transmitted to the second super-resolution module, which decodes them using a preset second super-resolution algorithm to obtain a high-resolution second image S2 (see S204 above). The pseudo-texture detection module detects pseudo-texture regions in the first image S1 based on the first image S1 and the input image (see S203 above). Furthermore, the pseudo-texture detection module can determine a mask matrix M based on the pseudo-texture regions (see S302 above). The fusion module performs fusion processing based on the mask matrix M, the first image S1, and the second image S2 to finally obtain the target image S (see S205 above).

[0196] In this embodiment, an image processing model can be constructed to implement the image processing method. The two super-resolution modules in the image processing model can obtain two high-resolution images (a first image and a second image) with different performance effects through two different super-resolution algorithms. Compared with the first image, the second image has higher fidelity to the original image details, while the first image has higher quality generated image details compared to the second image. The fusion module can perform image fusion processing on the first image and the second image based on the pseudo-texture regions in the first image, obtaining a target image after eliminating the pseudo-texture regions. The fused target image combines the good performance effects of both the first and second images (high fidelity to the original image details and high quality generated image details), achieving the goal of improving the overall quality of the high-resolution image.

[0197] The image processing method provided in the embodiments of this application is introduced in the context of specific practical application scenarios.

[0198] For example, in one scenario, the image processing method provided in this application embodiment is applied to a use case where a device only performs image processing. The electronic device can read the input image locally, or obtain the input image that needs to be super-resolution processed from a database. The electronic device performs image processing on the input image as described in S201-S205 to obtain a target image corresponding to the input image. The resolution of the target image is higher than the resolution of the input image.

[0199] This application scenario is specifically manifested in, for example, referencing Figure 8 , Figure 8 A schematic diagram of a specific scenario for an image processing method is provided. For example, when the electronic device is a mobile phone, the image processing method can be deployed in applications capable of image processing, such as a gallery app. The mobile phone displays the gallery interface of the gallery app, which can respond to image processing operations such as image enhancement (or image optimization) triggered by the user, perform image processing on the image selected by the user, and output the target image. During the image processing, a reminder message such as "Image enhancement in progress, please wait" can be displayed on the gallery interface. After obtaining the target image, for example, the target image can be displayed on the gallery interface, or the target image can be stored in the gallery app.

[0200] For example, in another scenario, the image processing method provided in this application is applied to a usage scenario where a device performs image acquisition and image processing. The electronic device includes an image acquisition device (such as a camera) and a processor. After the image acquisition device acquires an initial image, the electronic device performs image processing on the initial image as described in S201-S205 to obtain a target image corresponding to the input image. The resolution of the target image is higher than the resolution of the input image.

[0201] This application scenario is specifically manifested in, for example, referencing Figure 9 , Figure 9 A schematic diagram illustrating a specific scenario for another image processing method is provided. For example, when the electronic device is a mobile phone, the image processing method can be deployed in applications capable of image acquisition and processing, such as camera apps. The mobile phone displays the preview interface of the camera app, and the gallery app responds to the user's shooting action, acquiring and processing the captured image. During image processing, the electronic device can display the processing progress on the preview interface. After obtaining the target image, it can be stored in the gallery app, and the preview interface redisplays the preview image. When the electronic device receives a user action on the gallery button, it displays the gallery interface on the screen, which includes the target image after the last capture and image processing.

[0202] For example, in another scenario, the image processing method provided in this application embodiment is applied to an image processing system with at least two devices. The image acquisition device is used to acquire images, and the image processing device performs image processing. For example, the image acquisition device can be a device with a camera, such as a camera or surveillance device. The image processing device can be a computer, server, or other such device. The image processing device performs image processing as described in S201-S205 on the images acquired by the image acquisition device to obtain a corresponding target image. The resolution of the target image is higher than the resolution of the input image.

[0203] This application scenario is specifically manifested in, for example, referencing Figure 10 , Figure 10 A schematic diagram illustrating a specific scenario for another image processing method is provided. For example, when the image acquisition device is a surveillance camera and the image processing device is a computer, the image processing method can be deployed on the computer. The surveillance camera acquires an initial image and sends it to the computer via a communication connection. After acquiring the initial image, the computer uses image processing methods such as S201-S205 to obtain the target image corresponding to the initial image. In some embodiments, the computer can display the target image on a screen, or the computer can store the target image in a database.

[0204] In this embodiment, the image processing method can be applied to a variety of different scenarios to achieve super-resolution processing of images in different scenarios. This results in the target image having both high fidelity original image details and high-quality generated image details, thereby improving the overall quality of high-resolution images.

[0205] Figure 11 A possible structural schematic diagram of the electronic device involved in the above embodiments is shown. Figure 11 The electronic device 1100 shown includes a processing module 1101 and a storage module 1102.

[0206] The processing module 1101 may be a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The processor may include an application processor and a baseband processor. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0207] For example, processing module 1101 can be as follows: Figure 1 The processor 110 shown; the storage module 1102 can be as follows: Figure 1 The internal memory 121 shown. The electronic device provided in this application embodiment can be Figure 1 The electronic device 100 shown.

[0208] In some alternative embodiments, the electronic device 1100 may further include an image acquisition module 1103 and / or a display module 1104. The image acquisition module 1103 may be, for example: Figure 1 The camera 193 shown, and the display module 1104 are as follows: Figure 1 The display screen shown is 194.

[0209] This application also provides a chip system (e.g., a system-on-a-chip (SoC)). Figure 12 As shown, the chip system includes at least one processor 1201 and at least one interface circuit 1202. The processor 1201 and the interface circuit 1202 are interconnected via lines. For example, the interface circuit 1202 can be used to receive signals from other devices (e.g., the memory of an electronic device). As another example, the interface circuit 1202 can be used to send signals to other devices (e.g., the processor 1201 or the camera of an electronic device). Exemplarily, the interface circuit 1202 can read instructions stored in the memory and send those instructions to the processor 1201. When the instructions are executed by the processor 1201, the electronic device can perform the steps in the above embodiments. Of course, the chip system may also include other discrete components, which are not specifically limited in this application embodiment.

[0210] This application also provides a computer-readable storage medium including computer instructions that, when executed on the electronic device, cause the electronic device to perform various functions or steps performed by the electronic device 100 in the above method embodiment.

[0211] This application also provides a computer program product that, when run on a computer, causes the computer to perform the various functions or steps performed by the electronic device 100 in the above method embodiments. For example, the computer may be the aforementioned electronic device 100.

[0212] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0213] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0214] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0215] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0216] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0217] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, The method includes: Electronic devices acquire input images; The electronic device obtains the first image corresponding to the input image through a preset first super-resolution algorithm; The electronic device obtains the second image corresponding to the input image through a preset second super-resolution algorithm; Wherein, the resolution of the first image and the second image is higher than that of the input image, and there are pseudo-texture regions in the first image; the quality of the generated image details of the first image is higher than that of the generated image details of the second image, and the image detail reproduction degree of the second image is higher than that of the first image; The electronic device performs image fusion processing on the first image and the second image based on the pseudo-texture region to obtain the target image.

2. The method according to claim 1, characterized in that, The output image of the preset first super-resolution algorithm has high-quality details. The preset first super-resolution algorithm includes a super-resolution algorithm based on generative adversarial loss or a super-resolution algorithm based on a diffusion model. The output image of the preset second super-resolution algorithm has high detail with the original image. The preset second super-resolution algorithm includes a super-resolution algorithm based on absolute value error loss or a super-resolution algorithm based on perceptual loss.

3. The method according to claim 1 or 2, characterized in that, The method further includes: The electronic device acquires the difference image between the input image and the first image; The electronic device determines the pseudo-texture region based on the degree of change in the grayscale values ​​of pixels in the difference image.

4. The method according to claim 3, characterized in that, The electronic device determines the pseudo-texture region based on the degree of grayscale value change of pixels in the difference image, including: The electronic device calculates the local variance of the difference image according to a preset window; the preset window is related to the size of the difference image, and the local variance is used to characterize the degree of change in the gray value of the pixel. The electronic device acquires at least one high-frequency region with a local variance greater than a preset variance value, and the high-frequency region is the pseudo-texture region.

5. The method according to claim 4, characterized in that, The electronic device performs image fusion processing on the first image and the second image based on the pseudo-texture region to obtain a target image, including: The electronic device performs normalization and stretching transformation on the local variance of the difference image to obtain a mask matrix; the size of the mask matrix is ​​the same as the size of the first image and the second image, and the elements in the mask matrix represent the fusion weight of each pixel, and the fusion weight of the pixels in the pseudo-texture region is greater than a preset value. The electronic device performs weighted fusion of the first image and the second image based on the mask matrix to obtain the target image.

6. The method according to claim 3, characterized in that, The electronic device determines the pseudo-texture region based on the degree of grayscale value change of pixels in the difference image, including: The electronic device calculates the local variance of the difference image according to a preset window; the preset window is related to the size of the difference image, and the local variance is used to characterize the degree of change in the gray value of the pixel. The electronic device performs normalization and stretching transformation on the local variance of the difference image to obtain a mask matrix; the size of the mask matrix is ​​the same as the size of the first image and the second image, and the elements in the mask matrix represent the fusion weight of each pixel; the region in the mask matrix corresponding to the fusion weight greater than a preset value is the pseudo-texture region; The electronic device performs image fusion processing on the first image and the second image based on the pseudo-texture region to obtain a target image, including: The electronic device performs weighted fusion of the first image and the second image based on the mask matrix to obtain the target image.

7. The method according to claim 5 or 6, characterized in that, The electronic device performs weighted fusion of the first image and the second image based on the mask matrix to obtain the target image, including: The electronic device performs a weighted fusion of the first image and the second image using the following expression to obtain the target image; The expression is represented as: S = (1-M)*S1 + M*S2; Wherein, S is the target image, M is the mask matrix, S1 is the first image, and S2 is the second image.

8. The method according to any one of claims 1-4, characterized in that, The electronic device performs image fusion processing on the first image and the second image based on the pseudo-texture region to obtain a target image, including: The electronic device obtains the first region corresponding to the pseudo-texture region in the first image based on the position of the pseudo-texture region; The electronic device obtains the second region corresponding to the pseudo-texture region in the second image based on the position of the pseudo-texture region; The electronic device replaces the first region in the first image with the second region to obtain the target image.

9. The method according to claim 3, characterized in that, The electronic device acquires the difference image between the input image and the first image, including: The electronic device performs low-pass filtering on the first image to obtain the filtered first image; wherein, the processing kernel corresponding to the low-pass filtering is a preset value; The electronic device calculates the difference between the grayscale values ​​of each pixel in the filtered first image and the corresponding pixel in the input image, and obtains the difference image.

10. The method according to claim 1 or 2, characterized in that, The method further includes: The electronic device inputs the input image and the first image into a deep learning model to obtain the pseudo-texture region.

11. An image processing model, characterized in that, The image processing model includes a first super-resolution module, a second super-resolution module, a pseudo-texture detection module, and a fusion module; The first super-resolution module is used to obtain the first image corresponding to the input image through a preset first super-resolution algorithm; The second super-resolution module is used to obtain the second image corresponding to the input image through a preset second super-resolution algorithm; The pseudo-texture detection module is used to obtain the difference image between the input image and the first image, and to determine the pseudo-texture region based on the degree of change in the grayscale value of the pixels in the difference image; The fusion module is used to perform image fusion processing on the first image and the second image based on the pseudo-texture region to obtain the target image; The resolution of the first image and the second image is higher than that of the input image. The first image contains a pseudo-texture region. The quality of the generated image details of the first image is higher than that of the generated image details of the second image. The image detail reproduction of the second image is higher than that of the first image.

12. The model according to claim 11, characterized in that, The image processing model also includes an encoding module. The encoding module is used to perform feature encoding on the input image to obtain the image features corresponding to the input image; The first super-resolution module is used to decode the image features using a preset first super-resolution algorithm to obtain the first image; The second super-resolution module is used to decode the image features using a preset second super-resolution algorithm to obtain the second image.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-10.

14. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-10.

15. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-10.

Images