Image processing method, electronic device and computer readable storage medium
By encoding, denoising and stitching the image, the problem of difficult to restore high-frequency information in low-quality images is solved, and the effect of improving image clarity without increasing training costs is achieved.
Patent Information
- Application Number
- CN202410939827.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-07-15
AI Technical Summary
When the prior art performs super-resolution reconstruction of low-quality images, high-frequency information such as text is difficult to restore, and increasing the number of encoder output channels to solve this problem requires retraining the unet model, resulting in high training costs.
By acquiring the image to be processed, the first image encoder and the second image encoder respectively generate encoding results with different number of channels, determine the noise in the image and perform denoising processing, and splicing the denoised encoding results with the original encoding results , input the image decoder to generate enhanced images, avoiding high training costs.
Without increasing the training cost, high-frequency information such as text in the image is successfully restored, improving the clarity of the image.
Smart Images

Figure CN118469813B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, an electronic device, and a computer-readable storage medium. Background Art
[0002] Among the photos taken by mobile phones, there are inevitably some photos with insufficient clarity or loss of details, especially for telephoto shooting. In order to process low-quality images, the controlnet large model is usually used for super-resolution reconstruction to obtain high-quality images corresponding to low-quality images, including: 1. The original image (size is [3,512,512]) is compressed into a variable latent of size [4,64,64] by the encoder of the variational autoencoder VAE. 2. The latent of size [4,64,64] and the unet of input size 4 channels are used to train the controlnet with input size 4 channels. After training, the unet and controlnet models are used for inference to obtain the enhanced latent (size [4,64,64]). 3. The enhanced latent is decoded into a high-definition image (size [3,512,512]) by the decoder of the variational autoencoder VAE.
[0003] However, in the first step above, due to the high degree of compression of the encoder, a lot of image information is lost, resulting in the difficulty of restoring high-frequency information such as text when using ControlNet for super-resolution tasks. A common approach is to increase the number of output channels of the encoder, retrain a VAE model so that the VAE encoder output is [32,64,64], and the number of channels increases from 4 to 32, thereby increasing the amount of latent information, so that text information can be retained and reconstructed. However, this approach of increasing channels requires retraining the UNET model in the second step, and the training cost is very high. The specific training steps become: 1. The original image (size [3,512,512]) is compressed into a latent of size [32,64,64] by the VAE encoder. 2. First, use the latent of size [32,64,64] to train the main UNET model with an input size of 32 channels. 3. Then use the latent of [32,64,64] obtained in step 1 and the UNET model obtained in step 2 to train the ControlNet with an input size of 32 channels. After training, the enhanced latent (size of [32,64,64]) is inferred using the UNET and ControlNet models. 4. The enhanced latent is decoded into a high-definition image (size of [3,512,512]) by the VAE decoder. Since the main body of the UNET is a generative model, it is very difficult to train, and the data cost and server cost are very high. Summary of the invention
[0004] The purpose of the embodiments of the present application is to provide an image processing method, an electronic device, and a computer-readable storage medium to process images without requiring high training costs, thereby ensuring that high-frequency information such as text can be restored. The specific technical solution is as follows:
[0005] In a first aspect, in order to achieve the above-mentioned purpose, an embodiment of the present application provides an image processing method, the method comprising:
[0006] Get the image to be processed;
[0007] Inputting the image to be processed into a first image encoder to obtain a first image encoding result;
[0008] Inputting the image to be processed into a second image encoder to obtain a second image encoding result, wherein the number of channels of the first image encoding result is smaller than the number of channels of the second image encoding result;
[0009] Determining the noise contained in the image to be processed;
[0010] Performing denoising processing on the first image encoding result to obtain a denoised first image encoding result;
[0011] splicing the denoised first image encoding result and the second image encoding result to obtain a third image encoding result;
[0012] The third image encoding result is input into an image decoder to obtain an enhanced image to be processed.
[0013] The image processing method provided by the embodiment of the present application, after obtaining the image to be processed; input the image to be processed into the first image encoder to obtain the first image encoding result; input the image to be processed into the second image encoder to obtain the second image encoding result, the number of channels of the first image encoding result is less than the number of channels of the second image encoding result; determine the noise contained in the image to be processed; denoise the first image encoding result to obtain the denoised first image encoding result; splice the denoised first image encoding result and the second image encoding result to obtain the third image encoding result; input the third image encoding result into the image decoder to obtain the enhanced image to be processed. Thus, the image can be processed without requiring a high training cost, ensuring that high-frequency information such as text can be restored.
[0014] In a possible implementation manner, determining the noise contained in the image to be processed includes:
[0015] Inputting the first image encoding result into a semantic segmentation model to obtain noise contained in the image to be processed;
[0016] The semantic segmentation model is trained in the following way:
[0017] Acquire a first sample image, and add predetermined noise to the first sample image to obtain a noise sample image;
[0018] Inputting the noise sample image into the semantic segmentation model to be trained to obtain predicted noise;
[0019] Based on the difference between the predetermined noise and the predicted noise, the parameters of the semantic segmentation model to be trained are adjusted to obtain a trained semantic segmentation model.
[0020] In the embodiment of the present application, the noise contained in the image to be processed can be obtained through the semantic segmentation model for subsequent denoising. The semantic segmentation model is trained by noisy images and noise labels, and when faced with a noisy image as input, it can predict the noise contained in it.
[0021] In a possible implementation manner, the performing denoising on the first image encoding result to obtain the denoised first image encoding result includes:
[0022] Inputting the first image encoding result and the noise contained in the image to be processed into a control network to obtain a denoised first image encoding result;
[0023] In the embodiment of the present application, the control network can perform denoising on the first image encoding result according to the noise contained in the image to be processed and the first image encoding result to obtain the denoised first image encoding result.
[0024] The control network is trained in the following way:
[0025] Acquire a second sample image and its corresponding denoised sample image;
[0026] Inputting the second sample image into a semantic segmentation model to obtain noise contained in the second sample image;
[0027] Inputting the second sample image and the noise contained in the second sample image into the control network to be trained to obtain a denoised second sample image;
[0028] Based on the difference between the denoised second sample image and the denoised sample image, the parameters of the control network to be trained are adjusted to obtain a trained control network.
[0029] In an embodiment of the present application, a control network can be trained using a trained semantic segmentation model. The trained control network can be used for denoising processing. De-noising can be achieved through the control network to obtain a denoised first image encoding result.
[0030] In a possible implementation manner, inputting the first image encoding result and the noise contained in the image to be processed into a control network to obtain a denoised first image encoding result includes:
[0031] The first image coding result and the noise contained in the image to be processed are input into a control network, so that the control network subtracts the noise in the first image coding result from the noise contained in the image to be processed to obtain a denoised first image coding result.
[0032] In the embodiment of the present application, since the first image encoding result is obtained by encoding the image to be processed, the control network can denoise the first image encoding result according to the noise included in the image to be processed, and the first image encoding result after subtracting the noise included in the image to be processed from the first image encoding result is the denoised first image encoding result.
[0033] In a possible implementation manner, the step of splicing the denoised first image encoding result and the second image encoding result to obtain a third image encoding result includes:
[0034] According to the number of channels of the first image encoding result and the number of channels of the second image encoding result, the denoised first image encoding result and the second image encoding result are spliced to obtain a third image encoding result.
[0035] In an embodiment of the present application, the denoised first image encoding result and the second image encoding result are spliced according to the number of channels, so that two tensors with different numbers of channels are connected according to the channel dimension, so that a new input tensor can be constructed for subsequent model processing or training.
[0036] In a possible implementation manner, the second image encoder is trained in the following manner:
[0037] Acquire a sample image pair, wherein the sample image pair includes a first resolution image and a second resolution image, wherein the first resolution image and the second resolution image are images of the same object with different resolutions, and the first resolution is smaller than the second resolution;
[0038] Inputting the first resolution image into a second image encoder to be trained to obtain a first resolution image encoding result;
[0039] Obtaining a second-resolution image encoding result corresponding to the second-resolution image;
[0040] The second image encoder to be trained is adjusted based on the difference between the first resolution image encoding result and the second resolution image encoding result to obtain a trained second image encoder.
[0041] In the embodiment of the present application, the trained second image encoder has the ability to eliminate degradation, and when the input is a low-definition image, it can output the encoding result of a high-definition image corresponding to the low-definition image.
[0042] In a possible implementation, the method further includes:
[0043] When the image to be processed includes multiple degradation modes, for each degradation mode, a second image encoder is trained to obtain a second image encoder capable of eliminating a specific degradation mode;
[0044] The step of inputting the image to be processed into a second image encoder to obtain a second image encoding result includes:
[0045] Inputting the image to be processed into a plurality of second image encoders capable of eliminating a specific degradation mode to obtain a plurality of second image encoding sub-results;
[0046] The multiple second image encoding sub-results are concatenated to obtain a second image encoding result.
[0047] In an embodiment of the present application, when the image to be processed contains multiple degradation modes, the second image encoder can be trained specifically to increase its ability to eliminate the degradation information of the image to be processed, so that the image is cleaner before entering the large model.
[0048] In a possible implementation manner, acquiring the image to be processed includes:
[0049] Taking pictures in response to a shooting request to obtain an image to be processed;
[0050] In response to the selection operation on the image to be processed, the subsequent step of acquiring the image to be processed is performed.
[0051] In the embodiment of the present application, the image to be processed is obtained by responding to the shooting request, and only the selected image to be processed is processed, which improves the user experience and prevents the processing of some images that do not need to be processed, thereby causing a waste of resources.
[0052] In a second aspect, an embodiment of the present application further provides an electronic device, including:
[0053] one or more processors and memory;
[0054] The memory is coupled to the one or more processors, and the memory is used to store computer program code, wherein the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute any image processing method described in the first aspect.
[0055] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium, comprising a computer program, which, when executed on an electronic device, enables the electronic device to execute any image processing method described in the first aspect.
[0056] The image processing method provided by the embodiment of the present application, after obtaining the image to be processed; input the image to be processed into the first image encoder to obtain the first image encoding result; input the image to be processed into the second image encoder to obtain the second image encoding result, the number of channels of the first image encoding result is less than the number of channels of the second image encoding result; determine the noise contained in the image to be processed; denoise the first image encoding result to obtain the denoised first image encoding result; splice the denoised first image encoding result and the second image encoding result to obtain the third image encoding result; input the third image encoding result into the image decoder to obtain the enhanced image to be processed. Thus, the image can be processed without requiring a high training cost, ensuring that high-frequency information such as text can be restored. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0058] Figure 1 A hardware structure diagram of an electronic device provided in an embodiment of the present application;
[0059] Figure 2 A software structure block diagram of an electronic device provided in an embodiment of the present application;
[0060] Figure 3 A schematic diagram of a shooting interface;
[0061] Figure 4 A schematic diagram of a first flow chart of an image processing method provided in an embodiment of the present application;
[0062] Figure 5 A schematic diagram of the denoising network structure;
[0063] Figure 6 A second flow chart of the image processing method provided in the embodiment of the present application;
[0064] Figure 7 A schematic diagram of the gallery interface;
[0065] Figure 8 A schematic diagram of image processing in the gallery;
[0066] Fig. 9 A third flow chart of the image processing method provided in the embodiment of the present application;
[0067] Fig.10 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0068] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.
[0069] In order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first" and "second" are used to distinguish the same items or similar items with substantially the same functions and effects. For example, the first instruction and the second instruction are to distinguish different user instructions, and their order is not limited. Those skilled in the art can understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not necessarily limit them to be different.
[0070] It should be noted that, in this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplarily" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.
[0071] The following describes some concepts involved in the image processing method provided in the embodiments of the present application.
[0072] Super Resolution (SR): Super resolution is a low-level image processing task used to map a low-resolution image to a high-resolution image in order to enhance image details.
[0073] Among the photos taken by mobile phones, there are inevitably some photos with insufficient clarity or loss of details, especially for telephoto shooting. In order to process low-quality images, the controlnet large model is usually used for super-resolution reconstruction to obtain high-quality images corresponding to low-quality images, including: 1. The original image (size is [3,512,512]) is compressed into a latent of size [4,64,64] by the encoder of the variational autoencoder VAE. 2. The latent of size [4,64,64] and the unet of input size 4 channels are used to train the controlnet with input size 4 channels. After training, the unet and controlnet models are used for inference to obtain the enhanced latent (size [4,64,64]). 3. The enhanced latent is decoded into a high-definition image (size [3,512,512]) by the decoder of the variational autoencoder VAE.
[0074] However, in the first step mentioned above, due to the high degree of compression of the encoder, a lot of image information is lost, resulting in the difficulty of restoring high-frequency information such as text when using ControlNet for super-resolution tasks. A common approach is to increase the number of output channels of the encoder, retrain a VAE model so that the VAE encoder output is [32,64,64], and the number of channels increases from 4 to 32, thereby increasing the amount of latent information, so that text information can be retained and reconstructed. However, this approach of increasing channels requires retraining the UNET model in the second step, and the training cost is very high. The specific training steps are: 1. The original image (size [3,512,512]) is compressed into a latent of size [32,64,64] by the VAE encoder. 2. First, use the latent of size [32,64,64] to train the main UNET model with an input size of 32 channels. 3. Then use the latent of [32,64,64] obtained in step 1 and the UNET model obtained in step 2 to train the ControlNet with an input size of 32 channels. After training, the enhanced latent (size of [32,64,64]) is inferred using the UNET and ControlNet models. 4. The enhanced latent is decoded into a high-definition image (size of [3,512,512]) by the VAE decoder. Since the main body of the UNET is a generative model, the difficulty of training is very large, and the data cost and server cost are very high. In order to solve this problem, the image processing method provided in the embodiment of the present application is introduced below.
[0075] The embodiment of the present application provides an image processing method, which can be applied to electronic devices. The electronic device is an electronic device with a picture preview function, such as a smart phone, a tablet computer, a smart watch, etc. The scheme provided by the embodiment of the present application includes: obtaining an image to be processed; inputting the image to be processed into a first image encoder to obtain a first image encoding result; inputting the image to be processed into a second image encoder to obtain a second image encoding result, wherein the number of channels of the first image encoding result is less than the number of channels of the second image encoding result; determining the noise contained in the image to be processed; denoising the first image encoding result to obtain a denoised first image encoding result; splicing the denoised first image encoding result and the second image encoding result to obtain a third image encoding result; inputting the third image encoding result into an image decoder to obtain an enhanced image to be processed. Through the above scheme, the image can be processed without requiring a high training cost, ensuring that high-frequency information such as text can be restored.
[0076] The structure of the electronic device to which the above-mentioned image processing method is applied is introduced below.
[0077] For example, Figure 1 The structure diagram of the electronic device 100 is shown. The electronic device 100 may include a processor 110, a display screen 120, a camera 130, an internal memory 140, a subscriber identification module (Subscriber Identification Module, SIM) card interface 150, a universal serial bus (Universal Serial Bus, USB) interface 160, a charging management module 170, a battery management module 171, a battery 172 with a battery cell and a battery protection device, a sensor module 180, a mobile communication module 190, a wireless communication module 200, an antenna 1 and an antenna 2, etc. The sensor module 180 may include a pressure sensor 180A, a fingerprint sensor 180B, a touch sensor 180C, an ambient light sensor 180D, etc.
[0078] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0079] The processor 110 may include one or more processing units, for example: the processor 110 may include a CPU (Central Processing Unit), an AP (Application Processor), a modem processor, a GPU (graphics processing unit), an ISP (Image Signal Processor), a controller, a video codec, a DSP (Digital Signal Processor), a baseband processor, and / or an NPU (Neural-network Processing Unit), etc. Among them, different processing units may be independent components or integrated into one or more processors. In some embodiments, the electronic device 100 may also include one or more processors 110. Among them, the controller may generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions. In some other embodiments, a memory may also be provided in the processor 110 for storing instructions and data. Exemplarily, the memory in the processor 110 may be a cache memory. The memory may store instructions or data that the processor 110 has just used or circulated. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the electronic device 100 in processing data or executing instructions.
[0080] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an I2C (Inter-Integrated Circuit) interface, an I2S (Inter-Integrated Circuit Sound) interface, a PCM (Pulse Code Modulation) interface, a UART (Universal Asynchronous Receiver / Transmitter) interface, a MIPI interface (Mobile Industry Processor Interface), a GPIO (General-Purpose Input / Output) interface, a SIM card interface and / or a USB interface, etc. Among them, the USB interface 160 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 160 can be used to connect a charger to charge the electronic device 100, and can also be used to transmit data between the electronic device 100 and a peripheral device. The USB interface 160 can also be used to connect headphones to play audio through the headphones.
[0081] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is for illustrative purposes only and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.
[0082] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 190, the wireless communication module 200, the modem processor, the baseband processor, and the like.
[0083] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.
[0084] The electronic device 100 implements the display function through a GPU, a display screen 120, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 120 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.
[0085] The display screen 120 is used to display images, videos, etc. The display screen 120 includes a display panel. The display panel may be LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), active-matrix organic light-emitting diode or AMOLED (Active-Matrix Organic Light-Emitting Diode), FLED (Flex Light-Emitting Diode), MiniLed, MicroLed, Micro-oLed, QLED (Quantum Dot Light Emitting Diodes), etc. In some embodiments, the electronic device 100 may include one or more display screens 120.
[0086] In some embodiments of the present application, when the display panel is made of materials such as OLED, AMOLED, FLED, etc., the above Figure 1 The display screen 120 in the embodiment can be bent. Here, the display screen 120 can be bent means that the display screen can be bent to any angle at any position and can be kept at the angle, for example, the display screen 120 can be folded in half from the middle to the left or right. It can also be folded in half from the middle to the top or bottom.
[0087] The display screen 120 of the electronic device 100 can be a flexible screen. At present, flexible screens have attracted much attention for their unique characteristics and huge potential. Compared with traditional screens, flexible screens have the characteristics of strong flexibility and bendability, which can provide users with a new interaction method based on bendable characteristics, and can meet users' more needs for electronic devices. For electronic devices equipped with a foldable display, the foldable display on the electronic device can be switched between a small screen in a folded form and a large screen in an unfolded form at any time. Therefore, users use the split-screen function on electronic devices equipped with a foldable display more and more frequently.
[0088] The electronic device 100 can realize the shooting function through the ISP, the camera 130, the video codec, the GPU, the display screen 120 and the application processor, etc., wherein the camera 130 includes a front camera and a rear camera.
[0089] The ISP is used to process the data fed back by the camera 130. For example, when shooting, the shutter is opened, and light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can perform algorithm optimization on the noise, brightness and color of the image. The ISP can also optimize the exposure and color temperature of the shooting scene and other parameters. In some embodiments, the ISP can be set in the camera 130.
[0090] The camera 130 is used to take photos or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a CCD (Charge Coupled Cevice) or a CMOS (Complementary Metal-Oxide-Semiconductor) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB (Red Green Blue), YUV (a color encoding method) or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 130, where N is a positive integer greater than 1.
[0091] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.
[0092] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as MPEG (Moving Picture Experts Group) 1, MPEG2, MPEG3, and MPEG4.
[0093] NPU is a neural network computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.
[0094] The internal memory 140 can be used to store one or more computer programs, which include instructions. The processor 110 can execute the above instructions stored in the internal memory 140, so that the electronic device 100 performs the image processing method provided in some embodiments of the present application, as well as various applications and data processing. The internal memory 140 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system; the program storage area can also store one or more applications (such as a gallery, contacts, etc.). The data storage area can store data (such as photos, contacts, etc.) created during the use of the electronic device 100. In addition, the internal memory 140 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more disk storage components, flash memory components, general flash memory, etc. In some embodiments, the processor 110 can execute the image processing method provided in the embodiment of the present application, as well as other applications and data processing by running the instructions stored in the internal memory 140 and / or the instructions stored in the memory provided in the processor 110.
[0095] The internal memory 140 can be used to store the relevant programs of the image processing method provided in the embodiment of the present application, and the processor 110 can be used to call the relevant programs of the image processing method stored in the internal memory 140 when displaying information to execute the image processing method of the embodiment of the present application.
[0096] The sensor module 180 may include a pressure sensor 180A, a fingerprint sensor 180B, a touch sensor 180C, an ambient light sensor 180D, and the like.
[0097] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 120. There are many types of pressure sensors 180A, for example, they can be resistive pressure sensors, inductive pressure sensors or capacitive pressure sensors. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When the force acts on the pressure sensor 180A, the capacitance between the electrodes changes, and the electronic device 100 determines the intensity of the pressure according to the change in capacitance. When the touch operation acts on the display screen 120, the electronic device 100 detects the touch operation according to the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on a short message application icon, an instruction to view the short message is executed; when a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on a short message application icon, an instruction to create a new short message is executed.
[0098] The fingerprint sensor 180B is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to realize functions such as unlocking, accessing application locks, taking photos, and answering calls.
[0099] The touch sensor 180C is also called a touch control device. The touch sensor 180C can be arranged on the display screen 120, and the touch sensor 180C and the display screen 120 form a touch screen, which is also called a touch control screen. The touch sensor 180C is used to detect a touch operation acting on or near it. The touch sensor 180C can pass the detected touch operation to the application processor to determine the type of touch event. A visual output related to the touch operation can be provided through the display screen 120. In other embodiments, the touch sensor 180C can also be arranged on the surface of the electronic device 100, and is arranged at a different position from the display screen 120.
[0100] The ambient light sensor 180D is used to sense the brightness of the ambient light. The electronic device 100 can adaptively adjust the brightness of the display screen 120 according to the perceived brightness of the ambient light. The ambient light sensor 180D can also be used to automatically adjust the white balance when shooting. The ambient light sensor 180D can also transmit the environmental information of the device to the GPU.
[0101] The ambient light sensor 180D is also used to obtain the brightness, light ratio, color temperature, etc. of the environment in which the camera 130 captures images.
[0102] Figure 2A software structure block diagram of an electronic device applicable to an embodiment of the present application. The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. The layered architecture divides the software system of the electronic device into several layers, each layer has a clear role and division of labor, and the layers communicate with each other through software interfaces. In some embodiments, the software system can be divided into five layers, namely, the application layer, the application framework layer, the system operation layer, the hardware abstraction layer (HAL) and the kernel layer.
[0103] The application layer may include a camera, a gallery, and third-party applications with camera functions, etc. Exemplarily, the application layer may also include applications such as calls, calendars, maps, navigation, music, videos, and short messages.
[0104] The application framework layer provides application programming interface and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.
[0105] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a resource manager, a notification manager, a camera service, a semantic segmentation model, a control network, etc.
[0106] The window manager is used to manage window programs. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, take screenshots, etc.
[0107] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.
[0108] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.
[0109] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.
[0110] The notification manager allows applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, notification information is used to inform the completion of downloads, message reminders, etc. Notification information can also appear in the system top status bar in the form of icons or scroll bar text, such as notifications of applications running in the background, or in the form of dialog windows on the screen. Notification information can be, for example, a text message in the status bar, a reminder sound, vibration of electronic equipment, flashing indicator lights, etc.
[0111] The camera service is used to call the camera (including the front camera and / or the rear camera) in response to the application's request. Specifically in the embodiment of the present application, the camera service can call the camera to collect images in response to the photo command to obtain the corresponding image frame. For example, it is detected that the user clicks the camera's shooting control, determines that the photo command is received, and calls the camera to collect images.
[0112] For the captured images, they can be saved in the electronic device, and the user can view them by opening the gallery application. When the user operates on the low-quality photos, the semantic segmentation model and the control network of the application framework layer will perform specific processing on the photos to achieve super-resolution reconstruction of the photos, specifically, including: obtaining the image to be processed; inputting the image to be processed into the first image encoder to obtain a first image encoding result; inputting the image to be processed into the second image encoder to obtain a second image encoding result, and the number of channels of the first image encoding result is less than the number of channels of the second image encoding result; inputting the first image encoding result into the semantic segmentation model to obtain the noise contained in the image to be processed; inputting the first image encoding result and the noise contained in the image to be processed into the control network to obtain the denoised first image encoding result; splicing the denoised first image encoding result and the second image encoding result to obtain a third image encoding result; inputting the third image encoding result into the image decoder to obtain an enhanced image to be processed.
[0113] The system operation layer may include multiple functional modules, such as surface manager, media library, three-dimensional (3D) graphics processing library (such as OpenGL ES), two-dimensional (2D) graphics engine, etc.
[0114] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.
[0115] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264 (a new generation of digital video compression format), MP3 (Moving Picture Experts Group Audio Layer III, dynamic image expert compression standard audio layer 3), AAC (Advanced Audio Coding, advanced audio coding), AMR (Adaptive Multi-Rate, adaptive multi-bit rate), JPG (Joint Photographic Experts Group, an image file format), PNG (Portable Network Graphics, portable network graphics), etc.
[0116] The 3D graphics processing library is used to implement three-dimensional graphics drawing, image rendering, compositing, and layer processing, etc. The 2D graphics engine is a drawing engine for 2D drawing.
[0117] The hardware abstraction layer is located between the operating system kernel and the hardware circuit. It includes but is not limited to: an image processing module. Among them, the image processing module is used to process the image stream. In the embodiment of the present application, after the camera is started, the image processing module can receive the image stream from the camera and can process the image stream, such as noise reduction processing, brightness processing, etc.
[0118] Exemplarily, when the camera application is running, the user-triggered photo command can be sent to the camera service. On the one hand, the camera service can send the received photo command to the image processing module, so that the image processing module can call the camera driver in the kernel layer according to the received photo command, and the camera driver drives the camera and other hardware devices to respond to the photo command to collect image frames. For example, the camera can pass each frame of the collected image to the image processing module at a certain frame rate, and the image processing module will process it and send it to the display screen for display. On the other hand, after receiving the above-mentioned photo command, the camera service can determine the shooting strategy at this time according to the received photo command, and the shooting strategy sets the specific image processing tasks that need to be performed on the collected image data. For example, in the preview mode, the photo command can set the image processing task 1 in the shooting strategy to realize the face detection function. For another example, if the user turns on the beauty function in the preview mode, the camera service can also set the image processing task 2 in the shooting strategy to realize the beauty function. Then, the camera service can send the determined shooting strategy to the image processing module. When the image processing module receives each frame of image data collected by the camera, it can perform corresponding image processing tasks on the above image data according to the shooting strategy issued by the camera service to obtain each frame of the shooting picture after image processing.
[0119] Subsequently, the image processing module may report each frame of the captured image after image processing to the camera application through the camera service, and the camera application may display each frame of the captured image in the preview area of the display screen, or the camera application may save each frame of the captured image in the electronic device in the form of a photo or video.
[0120] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, image sensor driver, audio driver, etc. The hardware includes at least sensors, display screens, cameras, ISP, etc.
[0121] Understandably, Figure 2 The layers in the software structure shown and the components included in each layer do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer layers than shown, and each layer may include more or fewer components, which is not limited in the present application.
[0122] It is understandable that, in order to implement the image processing method provided by the embodiment of the present application, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiment disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.
[0123] The following describes an application scenario of the image processing method provided by the embodiment of the present application. In this scenario, the electronic device is a mobile phone as an example for explanation. For a distant scene, the user can click Figure 3 , and then shoot with telephoto. However, photos taken in telephoto mode may have problems such as insufficient clarity and loss of details. Therefore, such low-quality photos need to be processed to obtain high-quality photos corresponding to the low-quality photos.
[0124] The image processing method provided in the embodiment of the present application is described in detail below through specific examples.
[0125] See also Figure 4 , is an image processing method provided in an embodiment of the present application, the method comprising:
[0126] S201, obtaining an image to be processed.
[0127] The image to be processed here can be a photo taken by the camera application after the user clicks the "shoot" control, or a photo stored in the gallery application. After obtaining the image to be processed, the image to be processed can be subsequently processed.
[0128] S202: Input the image to be processed into a first image encoder to obtain a first image encoding result.
[0129] The image encoder is used to compress the image to be processed to reduce the amount of calculation. It can be an encoder in a variational autoencoder (VAE). The compression degree of the first image encoder is less than that of the second image encoder. A processed image of size [C, H, W] can be compressed by the first image encoder into a first image encoding result of size [2C, H / 8, W / 8].
[0130] S203: Input the image to be processed into a second image encoder to obtain a second image encoding result.
[0131] A to-be-processed image of size [C, H, W] can be compressed by a second image encoder into a second image encoding result of size [20C, H / 8, W / 8], wherein the number of channels of the first image encoding result is less than the number of channels of the second image encoding result. The number of channels of the second image encoding result can be 32, 64, 128, etc.
[0132] The second image encoder has the ability to eliminate degradation. Generally speaking, the degradation of an image mainly includes: noise, blur, distortion, geometric transformation, and illumination change. In the process of training the second image encoder, the input image is a low-definition image and the output image is a high-definition image. The trained second image encoder can eliminate degradation to a certain extent.
[0133] The second image encoder can be trained as follows:
[0134] Acquire a sample image pair, wherein the sample image pair includes a first resolution image and a second resolution image, wherein the first resolution image and the second resolution image are images of the same object with different resolutions, and the first resolution is smaller than the second resolution;
[0135] Inputting the first resolution image into a second image encoder to be trained to obtain a first resolution image encoding result;
[0136] Obtaining a second-resolution image encoding result corresponding to the second-resolution image;
[0137] The second image encoder to be trained is adjusted based on the difference between the first resolution image encoding result and the second resolution image encoding result to obtain a trained second image encoder.
[0138] In the embodiment of the present application, the trained second image encoder has the ability to eliminate degradation, and when the input is a low-definition image, it can output the encoding result of a high-definition image corresponding to the low-definition image.
[0139] S204: Determine the noise contained in the image to be processed.
[0140] The diffusion model is a model for image generation, and its main feature is to generate images through gradual diffusion and iteration. There are two processes: forward diffusion and backward diffusion.
[0141] The core part of the forward process of the diffusion model is to train the unet model, that is, to add noise to the original image, use the noise as a label, input a noisy image and time step to unet, let it predict the noise, and through continuous learning, reduce the loss value of predicted noise and actual noise, and finally get a trained unet model that can recognize noise. Figure 5 The purpose of this is to give the network a priori knowledge, and use the time of the corresponding stage as guidance information for different stages of training. Since the noise added during the denoising process is known, the network can be trained based on the known noise and the noisy training samples, and the network parameters can be adjusted based on the error between the noise predicted by the network and the known noise.
[0142] UNET is the core part of the diffusion model. It is a classic image segmentation network structure, which is often used in image segmentation tasks. Here it is mainly used for the prediction of noise maps. The UNET model consists of a symmetrical encoder and decoder. The middle part is a jump connection, which is used to fuse low-level and high-level features to improve segmentation accuracy. In the initialization function, the encoder, the middle part and the decoder, as well as the output layer, are defined. Both the encoder and the decoder consist of a series of residual blocks, in which the encoder converts the input image into a set of high-level feature maps through a series of convolution and downsampling operations, and gradually reduces the size of the feature maps through downsampling operations. The decoder restores these feature maps to the same size as the input image through upsampling and deconvolution operations. The middle part is some additional residual blocks.
[0143] In one example, the noise contained in the image to be processed can be obtained by inputting the first image encoding result into a semantic segmentation model;
[0144] The semantic segmentation model is trained in the following way:
[0145] Acquire a first sample image, and add predetermined noise to the first sample image to obtain a noise sample image;
[0146] Inputting the noise sample image into the semantic segmentation model to be trained to obtain predicted noise;
[0147] Based on the difference between the predetermined noise and the predicted noise, the parameters of the semantic segmentation model to be trained are adjusted to obtain a trained semantic segmentation model.
[0148] Among them, the semantic segmentation model can be the most common UNET model.
[0149] In the embodiment of the present application, the noise contained in the image to be processed can be obtained through the semantic segmentation model for subsequent denoising. The semantic segmentation model is trained by noisy images and noise labels, and when faced with a noisy image as input, it can predict the noise contained in it.
[0150] S205: Perform denoising processing on the first image encoding result to obtain a denoised first image encoding result.
[0151] In one example, the first image encoding result and the noise contained in the image to be processed can be input into a control network to obtain a denoised first image encoding result; wherein the control network is a controlnet model.
[0152] In the embodiment of the present application, the control network can perform denoising on the first image encoding result according to the noise contained in the image to be processed and the first image encoding result to obtain the denoised first image encoding result.
[0153] Specifically, the first image encoding result and the noise contained in the image to be processed may be input into a control network, so that the control network subtracts the noise in the first image encoding result from the noise contained in the image to be processed to obtain a denoised first image encoding result.
[0154] In the embodiment of the present application, since the first image encoding result is obtained by encoding the image to be processed, the control network can denoise the first image encoding result according to the noise included in the image to be processed, and the first image encoding result after subtracting the noise included in the image to be processed from the first image encoding result is the denoised first image encoding result.
[0155] The control network is trained in the following way:
[0156] Acquire a second sample image and its corresponding denoised sample image;
[0157] Inputting the second sample image into a semantic segmentation model to obtain noise contained in the second sample image;
[0158] Inputting the second sample image and the noise contained in the second sample image into the control network to be trained to obtain a denoised second sample image;
[0159] Based on the difference between the denoised second sample image and the denoised sample image, the parameters of the control network to be trained are adjusted to obtain a trained control network.
[0160] In an embodiment of the present application, a control network can be trained using a trained semantic segmentation model. The trained control network can be used for denoising processing. De-noising can be achieved through the control network to obtain a denoised first image encoding result.
[0161] S206: Concatenate the denoised first image encoding result and the second image encoding result to obtain a third image encoding result.
[0162] According to the number of channels of the first image encoding result and the number of channels of the second image encoding result, the denoised first image encoding result and the second image encoding result are spliced to obtain a third image encoding result.
[0163] In an embodiment of the present application, the denoised first image encoding result and the second image encoding result are spliced according to the number of channels, so that two tensors with different numbers of channels are connected according to the channel dimension, so that a new input tensor can be constructed for subsequent model processing or training.
[0164] In one example, it can be a concat operation along the channel dimension. The function of the concat layer is to concatenate two or more feature maps along the channel dimension. For example, if conv_9 and deconv_9 are concatenated along the channel dimension, first of all, except for the channel dimension, the other dimensions must be consistent (that is, H and W are consistent). At this time, the operation is only the channel k1 of conv_9 plus the channel k2 of deconv_9. The output of the concat layer can be expressed as: [(k1+k2), H, W].
[0165] In one example, a controllable feature wrapping module CFW (controllable feature wrapping) may be used to splice the denoised first image coding result and the second image coding result. The mechanism of CFW is: the number of common channels is determined according to the number of channels of the denoised first image coding result and the number of channels of the second image coding result, and the number of channels of the third image coding result obtained by splicing the denoised first image coding result and the second image coding result is the number of common channels.
[0166] S207: Input the third image encoding result into an image decoder to obtain an enhanced image to be processed.
[0167] The image decoder can be a decoder in a variational autoencoder (VAE). A variational autoencoder (VAE) is a neural network architecture for unsupervised learning that consists of an encoder and a decoder. The encoder receives the input (usually an image, text, or other structured data) and maps it to a latent space representation. This latent space is a low-dimensional continuous space where each point represents a distribution of possible outputs. In a VAE, the encoder outputs the parameters (mean and variance) of a Gaussian distribution to approximate the latent space. The decoder receives points from the latent space and maps them back to the original input space. It reconstructs the input data based on the latent representation generated by the encoder. The decoder is usually a neural network that learns to reconstruct the input data as accurately as possible.
[0168] The workflow of vae:
[0169] Encoding: The input data passes through the encoder network, which outputs the parameters (mean and variance) of the underlying Gaussian distribution.
[0170] Sampling: Using the mean and variance, sample a point from the underlying distribution.
[0171] Decoding: Points sampled from the latent space are fed into the decoder network to reconstruct the original input data.
[0172] The image processing method provided by the embodiment of the present application, after obtaining the image to be processed; input the image to be processed into the first image encoder to obtain the first image encoding result; input the image to be processed into the second image encoder to obtain the second image encoding result, the number of channels of the first image encoding result is less than the number of channels of the second image encoding result; determine the noise contained in the image to be processed; denoise the first image encoding result to obtain the denoised first image encoding result; splice the denoised first image encoding result and the second image encoding result to obtain the third image encoding result; input the third image encoding result into the image decoder to obtain the enhanced image to be processed. Thus, the image can be processed without requiring a high training cost, ensuring that high-frequency information such as text can be restored.
[0173] For an example, see Figure 6 , the method further comprises:
[0174] S301: When the image to be processed includes multiple degradation modes, a second image encoder is trained for each degradation mode to obtain a second image encoder capable of eliminating a specific degradation mode.
[0175] If the degradation mode of the image to be processed is fixed (such as raindrops, fog, dim light, etc. on the image), the second image encoder can be trained specifically so that it has the corresponding ability to eliminate the degradation.
[0176] S302: Input the image to be processed into a plurality of second image encoders capable of eliminating a specific degradation mode, to obtain a plurality of second image encoding sub-results.
[0177] When the image to be processed contains multiple degradation modes, the image to be processed can be input into the second image encoder capable of eliminating specific degradation modes. For the above-mentioned three situations of the image to be processed, namely, raindrops, fog, and dim light, the image to be processed needs to be encoded by the second image encoder capable of eliminating raindrops, the second image encoder capable of eliminating fog, and the second image encoder capable of eliminating dim light, respectively, to obtain three encoding sub-results.
[0178] S303: splice the multiple second image encoding sub-results to obtain a second image encoding result.
[0179] The multiple second image encoding sub-results may be concatenated through the above-mentioned cancat operation to obtain a second image encoding result.
[0180] In an embodiment of the present application, when the image to be processed contains multiple degradation modes, the second image encoder can be trained specifically to increase its ability to eliminate the degradation information of the image to be processed, so that the image is cleaner before entering the large model.
[0181] In one example, the method further comprises:
[0182] Taking pictures in response to a shooting request to obtain an image to be processed;
[0183] In response to the selection operation on the image to be processed, the subsequent step of acquiring the image to be processed is performed.
[0184] Users can click Figure 3 Click the shooting control 201 in the image processing window to trigger the camera to shoot and obtain the image to be processed; or Figure 3 Click on the taken photo 202 in the lower left corner, enter the gallery, and select the image to be processed from the gallery. Of course, the user can also directly click on the gallery application to enter the gallery and select the image to be processed.
[0185] After the image to be processed is obtained, there may be problematic images with poor clarity and unclear details, but there are also normal images; see Figure 7In the gallery application, you can long press any image from image 1 to image 5 to select it, or you can click the "Select All" button to select all five images. The following example uses the selection of image 5 as an example. The same applies to the case where all images are selected. Figure 8 , after long pressing image 5 to select image 5, a window will pop up, asking whether to optimize the current image. After clicking the "Yes" option, the execution of "inputting the image to be processed into the first image encoder to obtain a first image encoding result; inputting the image to be processed into the second image encoder to obtain a second image encoding result, the number of channels of the first image encoding result is less than the number of channels of the second image encoding result; determining the noise contained in the image to be processed; denoising the first image encoding result to obtain a denoised first image encoding result; splicing the denoised first image encoding result and the second image encoding result to obtain a third image encoding result; inputting the third image encoding result into the image decoder to obtain an enhanced image to be processed" will be started.
[0186] After obtaining the enhanced image to be processed, an image, namely, image 6, will be added to the gallery, arranged after image 5.
[0187] In the embodiment of the present application, the image to be processed is obtained by responding to the shooting request, and only the selected image to be processed is processed, which improves the user experience and prevents the processing of some images that do not need to be processed, thereby causing a waste of resources.
[0188] For an example, see Fig. 9 , is an image processing method provided by an embodiment of the present application, wherein the image to be processed passes through a 4-channel encoder and a 32-channel encoder respectively, and obtains an encoding result with a channel number of 4 and an encoding result with a channel number of 32. The 4-channel encoding result is input into a control network, and a latent space variable is output. The decoder receives the latent space variable obtained from the 4-channel encoding result and the 32-channel encoding result, and decodes to obtain a processing result of the image to be processed.
[0189] In the embodiment of the present application, the 32-channel encoder can be used without retraining UNET and ControlNet, and the trained 32-channel encoder is plug-and-play, which saves training costs to a certain extent.
[0190] On the other hand, see Fig.10 , the embodiment of the present application further provides an image processing device, including:
[0191] An acquisition module 401 is used to acquire an image to be processed;
[0192] A first encoding module 402, configured to input the image to be processed into a first image encoder to obtain a first image encoding result;
[0193] A second encoding module 403, configured to input the image to be processed into a second image encoder to obtain a second image encoding result, wherein the number of channels of the first image encoding result is smaller than the number of channels of the second image encoding result;
[0194] A noise determination module 404, used to determine the noise contained in the image to be processed;
[0195] A denoising module 405, configured to perform denoising processing on the first image encoding result to obtain a denoised first image encoding result;
[0196] A splicing module 406, configured to splice the denoised first image encoding result and the second image encoding result to obtain a third image encoding result;
[0197] The decoding module 407 is used to input the third image encoding result into an image decoder to obtain an enhanced image to be processed.
[0198] In one example, the noise determination module is specifically used to input the first image encoding result into a semantic segmentation model to obtain the noise contained in the image to be processed;
[0199] The semantic segmentation model is trained in the following way:
[0200] Acquire a first sample image, and add predetermined noise to the first sample image to obtain a noise sample image;
[0201] Inputting the noise sample image into the semantic segmentation model to be trained to obtain predicted noise;
[0202] Based on the difference between the predetermined noise and the predicted noise, the parameters of the semantic segmentation model to be trained are adjusted to obtain a trained semantic segmentation model.
[0203] In one example, the denoising module is specifically used to input the first image encoding result and the noise contained in the image to be processed into a control network to obtain a denoised first image encoding result;
[0204] The control network is trained in the following way:
[0205] Acquire a second sample image and its corresponding denoised sample image;
[0206] Inputting the second sample image into a semantic segmentation model to obtain noise contained in the second sample image;
[0207] Inputting the second sample image and the noise contained in the second sample image into the control network to be trained to obtain a denoised second sample image;
[0208] Based on the difference between the denoised second sample image and the denoised sample image, the parameters of the control network to be trained are adjusted to obtain a trained control network.
[0209] In one example, the denoising module is specifically used to input the first image encoding result and the noise contained in the image to be processed into a control network, so that the control network subtracts the noise in the first image encoding result from the noise contained in the image to be processed to obtain a denoised first image encoding result.
[0210] In one example, the splicing module is specifically used to splice the denoised first image encoding result and the second image encoding result according to the number of channels of the first image encoding result and the number of channels of the second image encoding result to obtain a third image encoding result.
[0211] In one example, the second image encoder is trained as follows:
[0212] Acquire a sample image pair, wherein the sample image pair includes a first resolution image and a second resolution image, wherein the first resolution image and the second resolution image are images of the same object with different resolutions, and the first resolution is smaller than the second resolution;
[0213] Inputting the first resolution image into a second image encoder to be trained to obtain a first resolution image encoding result;
[0214] Inputting the second resolution image into a second image encoder to be trained to obtain a second resolution image encoding result;
[0215] The second image encoder to be trained is adjusted based on the difference between the first resolution image encoding result and the second resolution image encoding result to obtain a trained second image encoder.
[0216] In one example, the device further comprises:
[0217] A training module, used for training the second image encoder for each degradation mode when the image to be processed contains multiple degradation modes, so as to obtain a second image encoder capable of eliminating a specific degradation mode;
[0218] The second encoding module is specifically used to input the image to be processed into multiple second image encoders capable of eliminating specific degradation modes to obtain multiple second image encoding sub-results; and splice the multiple second image encoding sub-results to obtain a second image encoding result.
[0219] In one example, the acquisition module is specifically used to shoot in response to a shooting request to obtain an image to be processed;
[0220] The selection module is used to execute the subsequent steps of acquiring the image to be processed in response to the selection operation on the image to be processed.
[0221] The present application also provides an electronic device, which includes one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute some or all of the steps in the above method embodiments.
[0222] The present application also provides a computer-readable storage medium, including a computer program, which, when executed on an electronic device, enables the electronic device to perform some or all of the steps in the above method embodiment. The above storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.
[0223] The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.
[0224] Program code may be applied to input instructions to perform the functions described herein and to generate output information. The output information may be applied to one or more output devices in a known manner. For purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.
[0225] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.
[0226] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed over a network or through other computer-readable media. Therefore, a machine-readable medium may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including but not limited to floppy disks, optical disks, optical disks, compact disc read-only memories (Compact Disc Read Only Memory, CD-ROMs), magneto-optical disks, read-only memories, random access memories, erasable programmable read-only memories (Erasable Programmable Read Only Memory, EPROM), electrically erasable programmable read-only memories (Electrically Erasable Programmable Read Only Memory, EEPROM), magnetic or optical cards, flash memory, or a tangible machine-readable memory for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in electrical, optical, acoustic, or other forms of propagation signals. Accordingly, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (eg, a computer).
[0227] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the accompanying drawings of the specification. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.
[0228] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.
[0229] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0230] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be apparent to those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present application.
Claims
1. An image processing method, It is characterized in that The method comprises: Get the image to be processed; Inputting the image to be processed into a first image encoder to obtain a first image encoding result; Inputting the image to be processed into a second image encoder to obtain a second image encoding result, wherein the number of channels of the first image encoding result is smaller than the number of channels of the second image encoding result; Determining the noise contained in the image to be processed includes: inputting the first image encoding result into a semantic segmentation model to obtain the noise contained in the image to be processed; Performing denoising processing on the first image coding result to obtain a denoised first image coding result, including: inputting the first image coding result and the noise contained in the image to be processed into a control network to obtain a denoised first image coding result; splicing the denoised first image encoding result and the second image encoding result to obtain a third image encoding result; The third image encoding result is input into an image decoder to obtain an enhanced image to be processed.
2. The method according to claim 1, It is characterized in that The semantic segmentation model is trained in the following way: Acquire a first sample image, and add predetermined noise to the first sample image to obtain a noise sample image; Inputting the noise sample image into the semantic segmentation model to be trained to obtain predicted noise; Based on the difference between the predetermined noise and the predicted noise, the parameters of the semantic segmentation model to be trained are adjusted to obtain a trained semantic segmentation model.
3. The method according to claim 1, It is characterized in that The control network is trained in the following way: Acquire a second sample image and its corresponding denoised sample image; Inputting the second sample image into a semantic segmentation model to obtain noise contained in the second sample image; Inputting the second sample image and the noise contained in the second sample image into the control network to be trained to obtain a denoised second sample image; Based on the difference between the denoised second sample image and the denoised sample image, the parameters of the control network to be trained are adjusted to obtain a trained control network.
4. The method according to claim 3, It is characterized in that The step of inputting the first image encoding result and the noise contained in the image to be processed into a control network to obtain a denoised first image encoding result includes: The first image coding result and the noise contained in the image to be processed are input into a control network, so that the control network subtracts the noise in the first image coding result from the noise contained in the image to be processed to obtain a denoised first image coding result.
5. The method according to claim 1, It is characterized in that The step of splicing the denoised first image encoding result and the second image encoding result to obtain a third image encoding result includes: According to the number of channels of the first image encoding result and the number of channels of the second image encoding result, the denoised first image encoding result and the second image encoding result are spliced to obtain a third image encoding result.
6. The method according to claim 1, It is characterized in that The second image encoder is trained in the following manner: Acquire a sample image pair, wherein the sample image pair includes a first resolution image and a second resolution image, wherein the first resolution image and the second resolution image are images of the same object with different resolutions, and the first resolution is smaller than the second resolution; Inputting the first resolution image into a second image encoder to be trained to obtain a first resolution image encoding result; Obtaining a second-resolution image encoding result corresponding to the second-resolution image; The second image encoder to be trained is adjusted based on the difference between the first resolution image encoding result and the second resolution image encoding result to obtain a trained second image encoder.
7. The method according to claim 1, It is characterized in that The method further comprises: When the image to be processed includes multiple degradation modes, for each degradation mode, a second image encoder is trained to obtain a second image encoder capable of eliminating a specific degradation mode; The step of inputting the image to be processed into a second image encoder to obtain a second image encoding result includes: Inputting the image to be processed into a plurality of second image encoders capable of eliminating a specific degradation mode to obtain a plurality of second image encoding sub-results; The multiple second image encoding sub-results are concatenated to obtain a second image encoding result.
8. The method according to claim 1, It is characterized in that The obtaining of the image to be processed comprises: Taking pictures in response to a shooting request to obtain an image to be processed; In response to the selection operation on the image to be processed, the subsequent step of acquiring the image to be processed is performed.
9. An electronic device, It is characterized in that include: one or more processors and memory; The memory is coupled to the one or more processors, and the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method according to any one of claims 1 to 8.
10. A computer-readable storage medium, It is characterized in that The method comprises a computer program, which, when executed on an electronic device, enables the electronic device to execute the method according to any one of claims 1 to 8.
