Image processing method, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an image processing method, electronic equipment and a storage medium, and the method comprises the steps: obtaining a first image in response to a moire elimination operation; inputting the first image into a target image processing model to perform moire elimination operation to obtain a second image; wherein the target image processing model is a coding and decoding network model trained based on a multi-scale training method. The image processing model obtained by training through the multi-scale training method is adopted to carry out moire elimination processing on the first image with moire, the moire in the image can be effectively eliminated, the definition of the output image can be obviously improved, and the quality of the image is improved.

Description

Technical Field

[0001] This application relates to the technical field of terminals, and in particular, to an image processing method, an electronic device, and a storage medium. Background Art

[0002] When a user uses an electronic device such as a mobile phone or a camera, the user may use the camera of the electronic device to take a picture of the content displayed on another display device. Since the sampling frequency of the camera of the electronic device such as a mobile phone or a camera is inconsistent with the display frequency of the display device, the high-frequency information in the two frequencies interferes with each other, and irregular color stripes often appear in the captured screen content. Such color stripes are moiré patterns (exemplarily, as Figure 1 shown). The moiré pattern will affect the display quality of the image. When the user takes a picture of the content displayed on the display device, it is mostly to record the content that the user is interested in. The appearance of the moiré pattern will also significantly reduce the perception and readability of the captured content. Summary of the Invention

[0003] Embodiments of this application provide an image processing method, an electronic device, and a storage medium, which can eliminate moiré patterns in an image, improve the clarity of the image, and improve the image quality.

[0004] In a first aspect, the image processing method provided by the embodiments of this application includes: in response to a moiré pattern elimination operation, obtaining a first image; inputting the first image into a target image processing model for moiré pattern elimination operation to obtain a second image; where the target image processing model is an encoder-decoder network model trained based on a multi-scale training method.

[0005] Herein, the above moiré pattern elimination operation may be a click operation corresponding to a moiré pattern elimination control clicked by the user, or a moiré pattern elimination operation automatically triggered when the electronic device automatically recognizes that there is a moiré pattern in the image. The above first image is specifically an image with moiré patterns, which may be an image with moiré patterns generated when the user uses the camera of the electronic device to take a picture of the content displayed on the display device, or an image with moiré patterns stored in the electronic device.

[0006] The above multi-scale training method may specifically be a training method in which image patches in a moiré pattern image are randomly intercepted at multiple scales during training, and after the image patches of different sizes are scaled to a preset size, the image patches of the preset size are input into the image processing model for training.

[0007] In the embodiments of this application, the size of the image patch refers to the length and width of the image patch, and the length and width are in units of the number of pixels included. A size of 256*256 means that there are 256 pixels in the length direction and 256 pixels in the width direction of the image patch.

[0008] It should also be noted that the above preset sizes can be set according to the actual application and actual training requirements. For example, they can also be set to 256*256, 256*512, 512*512, etc.

[0009] Compared with the single-scale training with fixed sizes (that is, only one size of image patches is intercepted each time when intercepting image patches, and no scaling operation is required), the multi-scale training method provided by the embodiments of the present application can improve the moiré elimination scenarios at different scales, and enhance the moiré elimination ability for fine moirés and the ability to retain the details of the original image; compared with the non-fixed-size multi-scale training without scaling (that is, the sizes of the intercepted image patches are different, but no scaling is performed, and the sizes of the image patches input into the image processing model are also different), due to the performance optimization of the open-source deep learning framework for the fixed-size training framework in the embodiments of the present application, no additional training time cost will be incurred. That is, the multi-scale training method proposed in the embodiments of the present application can enhance the moiré elimination ability, better retain the details of the original image, and effectively improve the image quality.

[0010] In a possible implementation manner, the target image processing model includes a number of encoding sub-modules and a number of decoding sub-modules; the encoding sub-modules and the decoding sub-modules are symmetrically arranged: both the encoding sub-modules and the decoding sub-modules include a feature adjustment module, and the feature adjustment module is used to average the brightness of each feature channel.

[0011] Here, the above target processing model may include five encoding sub-modules and five decoding sub-modules. Taking the input image traversal order as an example, the first encoding sub-module includes a 3*3 convolutional layer, a residual block, and a feature adjustment module; the second encoding sub-module includes a downsampling layer with a 2*2 convolution with a stride of 2, a residual block, and a feature adjustment module; the third encoding sub-module also includes a downsampling layer with a 2*2 convolution with a stride of 2, a residual block, and a feature adjustment module; the fourth encoding sub-module also includes a downsampling layer with a 2*2 convolution with a stride of 2, a residual block, and a feature adjustment module; the fifth encoding sub-module also includes a downsampling layer with a 2*2 convolution with a stride of 2, a residual block, and a feature adjustment module. Symmetrically, the first decoding sub-module includes a residual block, a feature adjustment module, and an upsampling convolutional module; the second decoding sub-module includes a residual block, a feature adjustment module, and an upsampling convolutional module; the third decoding sub-module includes a residual block, a feature adjustment module, and an upsampling convolutional module; the fourth decoding sub-module includes a residual block, a feature adjustment module, and an upsampling convolutional module; the fifth decoding sub-module includes a residual block, a feature adjustment module, and a 3*3 convolutional layer.

[0012] Here, the brightness of the image after removing moiré patterns obtained by processing with the image processing model provided by the embodiments of the present application, which is added with a feature adjustment module, is relatively average, reducing the risk of abnormal bright spots and improving the quality of the output image.

[0013] In a possible implementation manner, the feature adjustment module includes a mean calculation module and a multi-layer perceptron;

[0014] The mean calculation module is used to perform global averaging on the feature tensors of each feature channel to obtain the average value of each feature channel;

[0015] The multi-layer perceptron is used to learn the weights of each feature channel based on the average value of each feature channel;

[0016] The feature adjustment module is specifically used to perform feature enhancement or feature suppression channel by channel according to the feature tensors of each feature channel and the weights of each feature channel.

[0017] In a possible implementation manner, the decoding sub-module further includes an upsampling convolution module; the upsampling convolution module includes an upsampling interpolation module and a 3*3 convolution layer with a stride of 1.

[0018] The upsampling convolution module provided by the embodiments of the present application first performs interpolation through the upsampling interpolation module to magnify the features by 2*2 interpolation, and then perceives the neighborhood information of pixels through the 3*3 convolution layer with a stride of 1, so as to restore the difference of adjacent pixels, thereby reducing the generation of periodic pseudo-textures.

[0019] The above upsampling interpolation module can use bilinear interpolation for upsampling interpolation. Of course, the above upsampling interpolation module can also use other interpolation methods for upsampling interpolation, such as nearest neighbor interpolation, etc.

[0020] In a possible implementation manner, before obtaining the first image in response to the moiré pattern elimination operation, it further includes:

[0021] Obtaining training data;

[0022] Performing multi-scale iterative training on the initial image processing model according to the training data to obtain the target image processing model.

[0023] The training data used in the embodiments of the present application can be selected from open-source moiré pattern datasets to obtain training data. The above open-source moiré pattern datasets can be the utra-high-definition demoireing dataset (UHDM) dataset. Of course, the above open-source moiré pattern datasets can also include, but are not limited to, the open-sourced TIP2018, LCDMoire, and FHDMi and other moiré pattern datasets.

[0024] Randomly obtain training data from the open-source moiré dataset, and divide the training data into a training set and a test set. Exemplarily, randomly obtain 4,500 training data pairs from the open-source moiré dataset as the training set and 500 test data pairs as the test set. Each data pair includes a moiré image and a clean original image corresponding to the moiré image, and the sizes of these two images can be the same. In a specific application, the above images can be 4K images, where a 4K image refers to an image with the number of pixel values per row in the horizontal direction reaching or approaching 4,096 pixels.

[0025] It should be noted that the number of data pairs in the above training dataset and test dataset can be selected according to the application situation. Of course, a validation dataset can also be set, and the number of data pairs in the training dataset, validation dataset, and test dataset can be 6:2:2, etc., or 7:2:1, etc.

[0026] In a possible implementation manner of the first aspect, the training data includes training data pairs, and the training data pairs include moiré images and clean original images corresponding to the moiré images. Performing multi-scale iterative training on the initial image processing model according to the training data pairs to obtain the target image processing model includes:

[0027] Randomly shuffle the order of the training data pairs in the training data;

[0028] Successively extract training data pairs, randomly intercept image patches of different sizes in the moiré images in the training data pairs, and intercept corresponding image patches from the clean original images according to the positions of the image patches as label images;

[0029] After scaling the image patches intercepted from the moiré images to a preset size, input the scaled image patches into the initial image processing model for moiré elimination operation to obtain an output image;

[0030] Calculate the loss function according to the output image and the label image, and backpropagate to update the network parameters of the image processing model, and repeat the above operations until the loss function converges.

[0031] Since eliminating moiré is essentially removing abnormal textures (moiré) that do not exist in the original image, therefore, the image processing model in the embodiments of the present application is similar to an image denoising algorithm, and there is also a risk of affecting the clarity of the original image. Based on this, in the embodiments of the present application, a multi-scale training method is adopted to train the image processing model to significantly improve the clarity of the output image.

[0032] In a possible implementation manner of the first aspect, the loss function includes a perceptual loss function and an absolute value loss function.

[0033] Here, the above loss function may include a perceptual loss function and an L1 loss function (absolute value loss function). The perceptual loss function can utilize the semantic understanding information of the image by the image classification network. The image output by the image processing model and the label image are input into the pre-trained image classification network, and then the intermediate features of the image at different stages of the image classification network are obtained. Then, the L1 distance between each pixel of the intermediate features corresponding to the two images at the corresponding stage is calculated, and the final perceptual loss is obtained by adding the L1 distances calculated for all pixels and each stage. The L1 loss function calculates the sum of the L1 distances of all pixels in the output image of the image classification network and the label image. Adding the perceptual loss function and the L1 loss function gives the finally calculated loss. Using the perceptual loss function and the L1 loss function together to determine the loss of the network can further improve the robustness of the image processing model after training.

[0034] The above image classification network can use the pre-trained VGG16 network. Of course, the above image classification network can also use other pre-trained image classification networks, such as the GoogLeNet network, the AlexNet network, etc.

[0035] In a second aspect, an embodiment of the present application provides an electronic device, one or more processors, and a memory. The memory is coupled to the one or more processors. The memory is used to store computer program code, and the computer program code includes computer instructions. The one or more processors call the computer instructions to cause the electronic device to execute the method according to any one of the above first aspects.

[0036] In a third aspect, an embodiment of the present application provides a computer-readable storage medium. The computer-readable storage medium includes instructions that, when run on an electronic device, cause the electronic device to execute the method according to any one of the above first aspects.

[0037] In a fourth aspect, an embodiment of the present application provides a chip system. The chip system is applied to an electronic device. The chip system includes one or more processors, and the one or more processors are used to call computer instructions to cause the electronic device to execute the method according to any one of the first aspects.

[0038] In a fifth aspect, an embodiment of the present application provides a computer program product that, when run on an electronic device, causes the electronic device to execute the method according to any one of the above first aspects.

[0039] It can be understood that the beneficial effects of the above second aspect to the fifth aspect can refer to the relevant descriptions in the above first aspect, and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Schematic diagram of an image with moiré in the embodiments of the present application;

[0041] Figure 2 Schematic diagram of the hardware structure of the electronic device 100 provided in the embodiments of the present application;

[0042] Figure 3 Schematic diagram of the software architecture of the electronic device 100 provided in the embodiments of the present application;

[0043] Figure 4 Schematic diagram of an application scenario of an image processing method provided in the embodiments of the present application;

[0044] Figure 5 Schematic diagram of an application scenario of another image processing method provided in the embodiments of the present application;

[0045] Figure 6 Schematic diagram of the structure of an object image processing model provided in the embodiments of the present application;

[0046] Figure 7 Schematic diagram of the processing processes of an upsampling convolution module and a traditional upsampling convolution module provided in the embodiments of the present application;

[0047] Figure 8 Comparison diagram of the images processed by the upsampling convolution module and the traditional upsampling convolution module provided in the embodiments of the present application;

[0048] Figure 9 Schematic diagram of the structure of a feature adjustment module provided in the embodiments of the present application;

[0049] Figure 10 Schematic diagram of the effect of a feature adjustment module provided in the embodiments of the present application;

[0050] Figure 11 Schematic diagram of the effect of the multi-scale training method provided in the embodiments of the present application;

[0051] Figure 12 Schematic diagram of the implementation process of an image processing method provided in the embodiments of the present application

[0052] Figure 13 Schematic diagram of the structure of a chip provided in the embodiments of the present application. Detailed implementation manners

[0053] In the embodiments of the present application, terms such as "first" and "second" are used to distinguish identical or similar items with basically the same functions and effects. For example, the first chip and the second chip are only used to distinguish different chips, and do not limit their sequence. Those skilled in the art can understand that terms such as "first" and "second" do not limit the quantity and execution order, and terms such as "first" and "second" do not necessarily mean different.

[0054] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific manner.

[0055] In the embodiments of the present application, "at least one" means one or more, and "a plurality" means two or more. "And / or" describes the association relationship of associated objects and indicates that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone, where A and B may be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one (item)" or its similar expression refers to any combination of these items, including any combination of single item (item) or plural items (items). For example, at least one (item) of a, b, or c may represent: a, b, c, a - b, a - c, b - c, or a - b - c, where a, b, and c may be single or multiple.

[0056] In some application scenarios, when a user uses an electronic device such as a mobile phone or a camera, the user may use the camera of the electronic device to take pictures of the content displayed on other display devices. Since the sampling frequency of the camera of the electronic device such as a mobile phone or a camera is inconsistent with the display frequency of the display device, the high-frequency information in the two frequencies interferes with each other, and irregular color stripes often appear in the captured screen content. Such color stripes are moiré patterns (exemplary, as Figure 1 shown). Moiré patterns will affect the display quality of the image. When the user takes pictures of the content displayed on the display device, it is mostly to record the content that the user is interested in. The appearance of moiré patterns will also significantly reduce the visual perception and readability of the captured content. Therefore, it is necessary to eliminate the moiré patterns in the image to improve the display quality of the image.

[0057] Based on this, an embodiment of the present application provides an image processing method. When it is necessary to eliminate moiré patterns, an image with moiré patterns is input into an image processing module. The image processing module includes a pre-trained image processing model, and the image processing model is used to perform moiré pattern elimination processing on the image with moiré patterns. The moiré pattern elimination processing includes,

[0058] The solution provided by the embodiment of the present application can be applied to any electronic device with a camera. The above-mentioned electronic device can also be referred to as a terminal, a user equipment (UE), a mobile station (MS), a mobile terminal (MT), etc. The electronic device can be a mobile phone, a smart TV, a wearable device, a tablet computer (Pad), a computer with wireless transceiver function, a virtual reality (VR) electronic device, an augmented reality (AR) electronic device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in transportation safety, a wireless terminal in smart city, and so on. The embodiments of the present application do not limit the specific technologies and specific device forms adopted by the electronic device.

[0059] In order to better understand the embodiment of the present application, the structure of the electronic device in the embodiment of the present application will be introduced below:

[0060] Figure 2The structural schematic diagram of the electronic device 100 is shown. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0061] It can be understood that the structure schematically shown in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than those shown in the figure, or combine certain components, or split certain components, or have different component arrangements. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0062] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0063] The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0064] In some embodiments, the above-mentioned processor 110 is implemented by a system on chip (SOC), that is, in some embodiments of the present application, the processor 110 specifically refers to an SOC chip.

[0065] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can save the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be called from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0066] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0067] It can be understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is for illustrative purposes only and does not constitute a structural limitation on the electronic device 100. In some other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods or a combination of multiple interface connection methods in the above embodiments.

[0068] In some embodiments, antenna 1 of the electronic device is coupled to the mobile communication module 150, and antenna 2 is coupled to the wireless communication module 160, enabling the electronic device to communicate with a network and other devices via wireless communication technologies. The wireless communication technologies may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), Bluetooth (BT), Global Navigation Satellite System (GNSS), Wireless Local Area Network (WLAN), Near Field Communication (NFC), and / or Infrared (IR) technology, etc. The GNSS may include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), BeiDou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0069] The electronic device 100 implements audio functions through the speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor, etc. For example, music playback, audio acquisition, recording, etc.

[0070] The electronic device 100 implements the display function through the GPU, the display screen 194, the application processor, etc. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0071] In some embodiments, the GPU can also be used to perform operations corresponding to the target image processing model, that is, to input the image with moiré (the first image) into the target image processing model for moiré elimination operation, so as to obtain the image with moiré eliminated (i.e., the second image).

[0072] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0073] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 194, the electronic device detects the touch operation intensity according to the pressure sensor 180A. The electronic device can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0074] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0075] It can be understood that the above-mentioned electronic device 100 may include one or more cameras 193. When the electronic device 100 captures the content displayed on the display screen through the camera 193, the image captured by the camera can be processed by the ISP to obtain an RGB image, and then the RGB image is input as an image that needs to perform moiré removal into a target image processing model deployed in the electronic device to perform moiré removal on the RGB image, thereby obtaining an RGB image with moiré removed.

[0076] In other embodiments, the image processed by ISP may also be an image in other formats, such as an image in YUV format, an image in BGR format, an image in YCbCr format, an image in CMYK format, an image in CIE L*a*b* format, etc. For images in different formats, an image processing model corresponding to the image in that format may be trained specifically, and then the target image processing model corresponding to the image in that format may be used to eliminate the moiré in the image in that format. For example, for an image in YUV format, an image processing model for processing a YUV image and eliminating moiré from the YUV image may be trained, and then after obtaining the YUV image, the YUV image may be input into the target image processing model to eliminate the moiré, thereby obtaining a YUV image with the moiré eliminated.

[0077] For the electronic device 100 with the above hardware structure, the software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture, etc. In the embodiments of this application, taking the Android system with a layered architecture as an example, the software structure of the electronic device 100 will be exemplarily described.

[0078] Figure 3 This is the software structure block diagram of the electronic device 100 in the embodiments of this application.

[0079] The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely the application layer, the application framework layer, the system layer, and the kernel layer.

[0080] The application layer may include a series of application packages. Such as Figure 3 shown, the application packages may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, video, instant messaging, reading, etc.

[0081] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions.

[0082] The system layer includes system libraries, Android runtime, and core libraries.

[0083] Android runtime includes core libraries and a virtual machine. Android runtime is responsible for the scheduling and management of the Android system.

[0084] The core libraries contain two parts: one part is the functional functions that need to be called by the Java language, and the other part is the core libraries of Android.

[0085] The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files in the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0086] The kernel layer is the layer between the hardware and the software. The kernel layer includes at least a display driver, a camera driver, an audio driver, and a sensor driver.

[0087] The following will exemplarily describe the working processes of the software and hardware of the above-mentioned electronic device 100 in a camera scenario:

[0088] When the touch sensor 180K receives a touch operation, the touch sensor 180k generates a corresponding hardware interrupt and sends it to the kernel layer. The kernel layer processes the touch operation into a raw input event (including information such as touch coordinates and the timestamp of the touch operation). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the raw input event. Taking the touch operation as a click operation and the control corresponding to the click operation being the control of the camera application icon as an example, the camera application calls the interface of the application framework layer to start the camera application, then calls the kernel layer to start the camera driver, and captures a static image or video through the camera 193.

[0089] After introducing the execution entities involved in the embodiments of the present application, the following briefly introduces the possible application scenarios applicable to the embodiments of the present application. Here, it is illustrated by taking the electronic device as a mobile phone as an example.

[0090] Please refer to Figure 4 , Figure 4 which is a schematic diagram of an application scenario shown in an exemplary embodiment. In a possible scenario, when the user is using the mobile phone and takes a photo of the content displayed on the display screen, for example, takes a photo of the text displayed on the display screen. In this case, please refer to Figure 4 in (a). The user can start the camera of the mobile phone through the camera application to collect the content displayed on the display screen. Since the sampling frequency of the mobile phone camera is inconsistent with the display frequency of the display screen, the high-frequency information in the two frequencies interferes with each other. Then the image captured by the camera is as shown in Figure 4 in (b). In order to eliminate the moiré pattern in the captured image, the user can click the "Eliminate Moiré Pattern" button displayed on the current interface; in response to the click operation on the "Eliminate Moiré Pattern" button, the mobile phone uses the target image processing model pre-trained and deployed in the mobile phone to eliminate the moiré pattern in the captured image, and displays the captured image with the moiré pattern eliminated as shown in Figure 4 in (c).

[0091] In another possible scenario, after the mobile phone obtains the captured image with the moiré pattern eliminated, it can store both the captured image with the moiré pattern and the captured image with the moiré pattern eliminated.

[0092] In another possible scenario, when the mobile phone takes a photo, it can also automatically identify whether there is a moiré pattern in the captured image. In the case where it is identified that there is a moiré pattern in the captured image, without the user manually triggering, the mobile phone can automatically use the target image processing model pre-trained and deployed in the mobile phone to eliminate the moiré pattern in the captured image to obtain a captured image with the moiré pattern eliminated.

[0093] In another possible scenario, the user can perform moiré removal processing on one or more moiré - containing images stored in the photo album. Exemplarily, please refer to Figure 5 , Figure 5 , which shows a schematic diagram of an application scenario shown in another exemplary embodiment. As shown in (a) of Figure 5 , the mobile phone displays an image selection interface, and the user can select the image for which moiré needs to be removed; in response to the user's selection operation, referring to Figure 5 , in (b), the selected identifier corresponding to the image selected by the user can be highlighted (in the drawings of the embodiments of the present application, the selected identifier is a circle, and the circle being blackened indicates the highlighting of the selected identifier). After the user has made a selection, the user can click the "Remove Moiré" button; in response to the click operation on the "Remove Moiré" button, the mobile phone can sequentially remove the moiré in the selected images through the target image processing model. After the moiré is removed, the images displayed on the mobile phone can be as shown in (c) of Figure 5 , showing multiple images with moiré removed.

[0094] It should be noted that in the embodiments of the present application, only the above - mentioned Figure 4 or Figure 5 shown application scenarios are used as examples for illustration, and do not constitute a limitation on the embodiments of the present application.

[0095] The model architecture of the image processing model is described in detail as follows:

[0096] In the embodiments of the present application, the above - mentioned image processing model is an encoding - decoding network model, and the above - mentioned image processing model may include a number of symmetrically arranged encoding sub - modules and a number of decoding sub - modules. A network model with an encoding - decoding architecture can effectively reduce the complexity of the image processing model and also reduce the resource consumption of the image processing model. Feature adjustment modules are provided in both the above - mentioned encoding sub - modules and decoding sub - modules. The feature adjustment module is used to average the brightness of each feature channel. For a detailed description of the feature adjustment model, please refer to the descriptions in Figure 9 and Figure 10 of this application, and this application will not repeat it here.

[0097] Exemplarily, please refer to Figure 6 , Figure 6 , which shows a schematic diagram of the model architecture of the image processing model provided by the embodiments of the present application. As shown in Figure 6As shown in the figure, the above image processing model may include five encoding sub - modules and five decoding sub - modules. Taking the input image traversal order as an example, the first encoding sub - module includes a 3×3 convolutional layer, a residual block, and a feature adjustment module; the second encoding sub - module includes a downsampling layer with a 2×2 convolutional layer with a stride of 2, a residual block, and a feature adjustment module; the third encoding sub - module also includes a downsampling layer with a 2×2 convolutional layer with a stride of 2, a residual block, and a feature adjustment module; the fourth encoding sub - module also includes a downsampling layer with a 2×2 convolutional layer with a stride of 2, a residual block, and a feature adjustment module; the fifth encoding sub - module also includes a downsampling layer with a 2×2 convolutional layer with a stride of 2, a residual block, and a feature adjustment module. Symmetrically, the first decoding sub - module includes a residual block, a feature adjustment module, and an upsampling convolutional module; the second decoding sub - module includes a residual block, a feature adjustment module, and an upsampling convolutional module; the third decoding sub - module includes a residual block, a feature adjustment module, and an upsampling convolutional module; the fourth decoding sub - module includes a residual block, a feature adjustment module, and an upsampling convolutional module; the fifth decoding sub - module includes a residual block, a feature adjustment module, and a 3×3 convolutional layer.

[0098] Among them, the residual block is composed of two 3×3 convolutional layers; the input end and the output end of the above image processing model are connected by a short connection for fusing the input features and the output features; the output features of the first three encoding sub - modules are copied and then sequentially concatenated with the input features of the last three decoding sub - modules, that is, the features output by the first encoding sub - module are copied and input into the fifth decoding sub - module, the features output by the second encoding sub - module are copied and input into the fourth decoding sub - module, and the features output by the third encoding sub - module are copied and input into the third decoding sub - module. The processing by the third decoding sub - module, the fourth decoding sub - module, and the fifth decoding sub - module is used to increase the details of the output image.

[0099] It should be noted that the above 3×3 convolutional layer, residual block, and downsampling layer with a 2×2 convolutional layer with a stride of 2 are all common modules in the art. The related processing procedures and principles can be found in relevant technical literature, and the present application will not elaborate on them.

[0100] It should also be noted that the above Figure 6 is only an exemplary structural schematic diagram of the image processing model provided by the embodiments of the present application. The image processing model provided by the embodiments of the present application may also include more or fewer encoding sub - modules and decoding sub - modules.

[0101] In an embodiment of the present application, the above upsampling convolutional module includes an upsampling interpolation module and a 3×3 convolutional layer.

[0102] Traditional upsampling convolution modules use a 2×2 transposed convolution layer for upsampling. Refer to Figure 7 , Figure 7 . Figure (a) in Figure 7 shows the schematic diagram of the processing procedure of upsampling by a 2×2 transposed convolution layer. As shown in Figure (a) in Figure 8 , when a 2×2 transposed convolution layer performs upsampling, a corresponding 2×2-sized feature pixel block can be restored from one feature pixel value. However, the obtained upsampling result cannot effectively obtain neighborhood information and there is a risk of homogenization. The upsampling feature of the 2×2 transposed convolution layer is a 2×2 pixel block restored from the same pixel and has a color similar to the source pixel value. After continuous upsampling, the convergence of adjacent pixels is reflected in the output image, generating periodic pseudo-textures that affect the visual perception (as shown in Figure (a) in

[0103] Figure 7 Figure 8 However, the upsampling convolution module provided by the embodiment of the present application first performs interpolation through an upsampling interpolation module to interpolate and magnify the feature by 2×2, and then perceives the neighborhood information of pixels through a 3×3 convolution layer with a stride of 1, so as to restore the difference between adjacent pixels. As shown in Figure (b) in Figure 8 , the upsampling module that first performs upsampling interpolation and then performs 3×3 convolution can perceive and restore the color difference between adjacent pixels, thereby reducing the generation of periodic pseudo-textures (the processed image effect is as shown in Figure (b) in

[0104] The above upsampling interpolation module can use bilinear interpolation for upsampling interpolation. Of course, the above upsampling interpolation module can also use other interpolation methods for upsampling interpolation, such as nearest neighbor interpolation, etc.

[0105] In an embodiment of the present application, the above feature adjustment module includes a mean calculation module and a multi-layer perceptron. The mean calculation module is used to perform global averaging on the feature tensor (H×W×C feature map) of each feature channel to obtain the average value of each feature channel. Then, the average value of each feature channel is input into the multi-layer perceptron (MLP) to calculate the weight value (1×C) of each feature channel, and then the weight value of each feature channel is used to enhance or suppress the channel feature (as shown in Figure 9 ), thereby reducing the risk of uneven brightness between feature channels and eliminating abnormal bright spots in the output image.

[0106] Since there is a difference in the environment when the moiré pattern image and the label image in the training data are captured, there is a brightness difference between the two. Without adding the above feature adjustment module, there will be abnormal bright spots in the output image after eliminating the moiré pattern. Exemplarily, as shown in Figure 10 , Figure 10The image in (a) is the input image (with moiré pattern in the image). Figure 10 The image in (b) is the output image processed by the image processing model without the feature adjustment module (the image with abnormal bright spots). Figure 10 The image in (c) is the output image processed by the image processing model with the feature adjustment module provided in the embodiments of the present application. From Figure 10 It can be seen that the brightness of the image after removing the moiré pattern obtained by processing with the image processing model with the feature adjustment module provided in the embodiments of the present application is relatively average, reducing the risk of abnormal bright spots and improving the quality of the output image.

[0107] The above is the description of the model architecture of the image processing model provided in the embodiments of the present application. The following is the description of the training process of this image processing model as follows:

[0108] The training data used in the embodiments of the present application can be selected from the open-source moiré dataset to obtain the training data. The above open-source moiré dataset can be the utra-high-definition demoireing dataset (UHDM) dataset. Of course, the above open-source moiré dataset can also include, but is not limited to, the open-source moiré datasets such as TIP2018, LCDMoire, and FHDMi.

[0109] Randomly obtain training data from the open-source moiré dataset, and divide the training data into a training set and a test set. Exemplarily, randomly obtain 4500 training data pairs from the open-source moiré dataset as the training set and 500 test data pairs as the test set. Each data pair includes a moiré image and a clean original image corresponding to the moiré image, and the sizes of these two images can be the same. In a specific application, the above image can be a 4K image, where a 4K image refers to an image with the number of pixel values per row in the horizontal direction reaching or approaching 4096 pixels.

[0110] It should be noted that the number of data pairs in the above training dataset and test dataset can be selected according to the application situation. Of course, a validation dataset can also be set. The number of data pairs in the training dataset, validation dataset, and test dataset can be 6:2:2, etc., or 7:2:1, etc. The present application does not make specific limitations on this.

[0111] During the training process, first, according to the above Figure 6Construct the model structure of the image processing model shown. Then randomly shuffle the order of the training data pairs in the training set, and sequentially extract one data pair. Randomly intercept image patches of the same position and size from the moiré image and the clean original image. The image patch intercepted from the moiré image is the input in the initial image processing model, and the image patch intercepted from the clean original image is the label image patch. Input the moiré image patch into the initial image processing model for image processing to obtain an output image patch. Calculate the loss function for the output image patch and the label image patch, and then update the network parameters in the image processing model through backpropagation. Repeat the above operations until all the training data pairs in the training set are traversed and the loss function converges (i.e., the loss function remains stable and no longer decreases significantly). Then, the trained image processing model can be tested using the test data pairs in the test set. Passing the test means obtaining the trained target image processing model. If the test fails, the above training process can be continued until the trained image processing model passes the test.

[0112] In the embodiment of the present application, the above loss function may include a perceptual loss function and an L1 loss function (absolute value loss function). The perceptual loss function can utilize the semantic understanding information of the image by the image classification network. Input the image output by the image processing model and the label image into the pre-trained image classification network, then obtain the intermediate features of the image at different stages of the image classification network, and then calculate the L1 distance between each pixel of the intermediate features corresponding to the two images at each stage. Then add up all the pixels and the L1 distances calculated at each stage to obtain the final perceptual loss. The L1 loss function calculates the sum of the L1 distances of all pixels in the output image and the label image of the image classification network. Adding the perceptual loss function and the L1 loss function gives the finally calculated loss. Using the perceptual loss function and the L1 loss function together to determine the loss of the network can further improve the robustness of the trained image processing model.

[0113] In the embodiment of the present application, the above image classification network can use the pre-trained VGG16 network. Of course, the above image classification network can also use other pre-trained image classification networks, such as the GoogLeNet network, the AlexNet network, etc.

[0114] Since eliminating moiré is essentially removing abnormal textures (moiré) that do not exist in the original image, therefore, the image processing model in the embodiment of the present application is similar to the image denoising algorithm and there is also a risk of affecting the clarity of the original image. Based on this, in the embodiment of the present application, a multi-scale training method is adopted to train the image processing model to significantly improve the clarity of the output image.

[0115] The above multi-scale training method is specifically as follows: randomly intercept image patches of any size at any position in the moiré image, and then scale the size of the intercepted image patches to the preset size and input them into the above-mentioned initial image processing model for training. For example, if the preset size is 256*256, then the intercepted image patches are scaled to image patches with a size of 256*256. That is, the sizes of the image patches intercepted from the moiré image each time are different, and then the image patches of different sizes are all scaled to the image patches of the preset size. The image patches of the preset size are the inputs of the image processing model.

[0116] Compared with the single-scale training with a fixed size (that is, only intercepting image patches of one size each time when intercepting image patches and no scaling operation is required), the multi-scale training method provided by the embodiments of the present application can improve the moiré elimination scenarios at different scales, and enhance the moiré elimination ability for fine moirés and the ability to retain the details of the original image; compared with the non-fixed-size multi-scale training without scaling (that is, the sizes of the intercepted image patches are different, but no scaling is performed, and the sizes of the image patches input into the image processing model are also different), due to the performance optimization of the open-source deep learning framework for the fixed-size training framework in the embodiments of the present application, it will not additionally increase the training time cost. That is, the multi-scale training method proposed in the embodiments of the present application can not only improve the moiré elimination ability and better retain the details of the original image, but also will not additionally increase the training time cost.

[0117] Exemplarily, please refer to Figure 11 , Figure 11 is a schematic diagram of the effects of the output images of the existing single-scale training method and the multi-scale training method provided by the embodiments of the present application. Among them, Figure 11 in (a) is the input image with moiré, Figure 11 in (b) is the image output by the image processing model after being trained by the single-scale training method, Figure 11 in (c) is the image output by the image processing model after being trained by the multi-scale training method provided by the embodiments of the present application. As can be seen from Figure 11 , the multi-scale training method provided by the embodiments of the present application can effectively retain more image details.

[0118] It should be noted that the size of the image patch in the embodiments of the present application refers to the length and width of the image patch, and the length and width are in units of the number of pixels included. The size of 256*256 means that there are 256 pixels in the length direction and 256 pixels in the width direction of the image patch.

[0119] It should also be noted that the above preset size can be set according to actual applications and actual training requirements. For example, it can also be set to 256*512, 512*512, etc. The present application does not limit this.

[0120] It should also be noted that the value of the size of the randomly intercepted image block described above can be 0.5 times to 2 times the preset size. Exemplarily, assuming the preset size is 256*256, the size of the intercepted image block can be 512*256, 128*256, 256*256, 256*512, etc.

[0121] After the image processing model is trained, it can be converted into a storage type adapted to the electronic device and deployed to the electronic device. When the user captures the display content on a display device such as a display screen through the electronic device, after obtaining the output RGB image through a series of algorithm paths, the image (i.e., the image that needs to eliminate moiré) can be input into the target image processing model deployed in the electronic device, so as to obtain the output image with moiré eliminated.

[0122] In a specific application, the above image processing model can be specifically deployed in the GPU of the electronic device, and the GPU calls the image processing model to process the input image, so as to obtain the output image with moiré eliminated.

[0123] It should be noted that the above image processing model can also be deployed in other positions of the electronic device, such as deployed in the CPU, etc., and the present application does not make specific limitations on this.

[0124] Based on the execution subject and application scenario provided in the above embodiments, next, the image processing method provided in the embodiments of the present application will be introduced. Please refer to Figure 12 , Figure 12 is a schematic flowchart of the image processing method shown according to an exemplary embodiment. By way of example and not limitation, the execution subject of the image processing method provided in the embodiments of the present application can be the above-mentioned electronic device 100. Taking a mobile phone as an example for illustration, in addition to including modules such as Figure 2 and Figure 3 shown, the mobile phone can also include other modules. For example, it can include a moiré elimination module. The moiré elimination module can be set in the GPU of the mobile phone. The above-mentioned trained image processing model is deployed in the moiré elimination module. Taking the interaction implementation among the camera, gallery, and moiré elimination module of the mobile phone as an example for illustration, as Figure 12 shown, the above image processing method can specifically include the following steps:

[0125] S1201: The camera receives a shooting operation triggered by the user.

[0126] During the process of using the mobile phone, the user can use the camera of the mobile phone to capture the content displayed on the display device. For example, Figure 4The scenario shown in [description], in this case, the user can start the camera application to trigger the shooting operation, that is, the user can click the shooting control of the camera application on the mobile phone to trigger the shooting operation of the camera, so that the camera can receive the shooting operation triggered by the user.

[0127] Of course, the user may also use the camera during the process of using the social application. In the case where the user makes a video call or shoots a video / image, the camera can also receive the shooting operation triggered by the user.

[0128] S1202: In response to the above shooting operation, the camera captures the first image.

[0129] S1203: The camera sends the captured first image to the gallery.

[0130] In a specific application, the camera sending the captured first image to the gallery may output the first image in RGB format to the gallery after a series of image output operations. The gallery can be used to store and display the first image.

[0131] S1204: The gallery receives and displays the first image.

[0132] S1205: The gallery receives the moiré pattern elimination operation triggered by the user.

[0133] S1206: In response to the moiré pattern elimination operation, the gallery sends the first image to the moiré pattern elimination module.

[0134] Since the camera shoots the content displayed on other display devices, the captured first image is likely to have moiré patterns. If the first image has moiré patterns, the user can trigger the moiré pattern elimination operation (such as Figure 4 the click operation of the user clicking the "Eliminate Moiré Pattern" button in (b) in [description]), so that the gallery can receive the moiré pattern elimination operation.

[0135] S1207: The moiré pattern elimination module receives the first image and inputs the first image into the target image processing model for moiré pattern elimination processing, and outputs the second image.

[0136] In the embodiment of the present application, the above target image processing model is a trained image processing model based on a multi-scale training method. Among the image processing models, the one that can effectively eliminate moiré patterns, so the output second image is an image that has eliminated moiré patterns.

[0137] S1208: The moiré pattern elimination module sends the second image to the gallery.

[0138] S1209: The gallery receives the second image and displays the second image.

[0139] In the embodiments of the present application, by using the image processing model provided by the embodiments of the present application, moiré patterns in the first image can be effectively eliminated, thereby effectively improving the quality of the image.

[0140] It can be understood that Figure 12 The interactive schematic diagram shown is only an example. After the mobile phone captures the first image, it can also automatically identify whether there are moiré patterns in the first image. When it is recognized that there are moiré patterns in the first image, the first image is automatically sent to the moiré pattern elimination module for moiré pattern elimination.

[0141] The above mainly introduces the solutions provided by the embodiments of the present application from the perspectives of the structure of the image processing model, the training method, and the image processing method. To implement the above functions, it includes the corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should easily realize that, in combination with the method steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0142] The embodiments of the present application can divide the device for the audio channel switching method into functional modules according to the above method examples. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiments of the present application is illustrative, only a logical function division, and there may be other division methods in actual implementation.

[0143] Figure 13 It is a schematic structural diagram of a chip provided by the embodiments of the present application. As Figure 13 shown, the chip 1300 includes one or more than two (including two) processors 1301, communication lines 1302, communication interfaces 1303, and a memory 1304.

[0144] In some embodiments, the memory 1304 stores the following elements: executable modules or data structures, or subsets thereof, or extended sets thereof.

[0145] The method described in the embodiments of the present application above can be applied to, or implemented by, the processor 1301. The processor 1301 may be an integrated circuit chip with signal processing capabilities. In the implementation process, the steps of the method executed by the above-mentioned first device or second device can be completed by the integrated logic circuit in hardware or instructions in software form in the processor 1301. The above-mentioned processor 1301 may be a general-purpose processor (for example, a microprocessor or a conventional processor), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate, transistor logic devices, or discrete hardware components. The processor 1301 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application.

[0146] The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. Among them, the software module can be located in a mature storage medium in the art such as a random access memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable read-only memory (EEPROM). This storage medium is located in the memory 1304, and the processor 1301 reads the information in the memory 1304 and combines its hardware to complete the steps of the above method.

[0147] Communication can be carried out between the processor 1301, the memory 1304, and the communication interface 1303 through the communication line 1302.

[0148] In the above embodiments, the instructions stored in the memory for the processor to execute can be implemented in the form of a computer program product. Among them, the computer program product can be pre-written in the memory in advance, or downloaded and installed in the memory in software form.

[0149] Embodiments of the present application also provide a computer program product including one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from a website, a computer, a server, or a data center to another website, a computer, a server, or a data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be stored by a computer or a data storage device such as a server or a data center integrating one or more available media. For example, the available medium may include magnetic media (such as floppy disks, hard disks, or magnetic tapes), optical media (such as digital versatile discs (DVDs)), or semiconductor media (such as solid state disks (SSDs)).

[0150] Embodiments of the present application also provide a computer-readable storage medium. The methods described in the above embodiments may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. The computer-readable medium may include computer storage media and communication media, and may also include any medium that can transfer a computer program from one place to another. The storage medium may be any target medium accessible by a computer.

[0151] As a possible design, the computer-readable medium may include a compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM, or other optical disc memories; the computer-readable medium may include disk memories or other disk storage devices. Moreover, any connection line may also be appropriately referred to as a computer-readable medium. For example, if software is transmitted from a website, a server, or other remote sources using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies (such as infrared, radio, and microwave), then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. As used herein, disks and optical discs include optical discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks generally reproduce data magnetically, while optical discs use lasers to optically reproduce data.

[0152] The above combinations should also be included within the scope of the computer-readable medium. The above is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. An image processing method, characterized in that Including: In response to a moiré elimination operation, obtain a first image; Input the first image into a target image processing model for moiré elimination operation to obtain a second image; wherein, the target image processing model is an encoder-decoder network model trained based on a multi-scale training method.

2. The image processing method according to claim 1, wherein The target image processing model includes a plurality of encoding sub-modules and a plurality of decoding sub-modules; The encoding sub-modules and the decoding sub-modules are symmetrically arranged: Both the encoding sub-modules and the decoding sub-modules include a feature adjustment module, and the feature adjustment module is used to average the brightness of each feature channel.

3. The image processing method according to claim 2, characterized in that, The feature adjustment module includes a mean calculation module and a multi-layer perceptron; The mean calculation module is used to perform global averaging on the feature tensor of each feature channel to obtain the average value of each feature channel; The multi-layer perceptron is used to learn the weight of each feature channel based on the average value of each feature channel; The feature adjustment module is specifically used to perform feature enhancement or feature suppression channel by channel according to the feature tensor of each feature channel and the weight of each feature channel.

4. The image processing method according to claim 2 or 3, wherein The decoding sub-module further includes an upsampling convolution module; The upsampling convolution module includes an upsampling interpolation module and a 3*3 convolution layer with a stride of 1.

5. The image processing method according to any one of claims 1 to 4, characterized in that, Before the step of obtaining the first image in response to the moiré elimination operation, it further includes: Obtain training data; Perform multi-scale iterative training on the initial image processing model according to the training data to obtain the target image processing model.

6. The image processing method according to claim 5, wherein The training data includes training data pairs, and each training data pair includes a moiré image and a clean original image corresponding to the moiré image. The step of performing multi-scale iterative training on the initial image processing model according to the training data to obtain the target image processing model includes: Randomly shuffle the order of the training data pairs in the training data; Successively extract training data pairs, randomly intercept image blocks of different sizes in the moiré images in the training data pairs, and intercept corresponding image blocks from the clean original images according to the positions of the image blocks as label images; After scaling the image blocks intercepted from the moiré images to a preset size, input the scaled image blocks into the initial image processing model for moiré elimination operation to obtain an output image; Calculate a loss function according to the output image and the label image, and backpropagate to update the network parameters of the image processing model. Repeat the above operations until the loss function converges.

7. The image processing method according to claim 6, wherein The loss function includes a perceptual loss function and an absolute value loss function.

8. An electronic device, characterized in that, Including a processor and a memory, the processor and the memory are coupled, and the memory is used to store a computer program. When the processor executes the computer program, the electronic device executes the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a computer program. When the computer program runs on an electronic device, the electronic device executes the steps of the method according to any one of claims 1 to 7.

10. A chip system, characterized in that, The chip system is applied to an electronic device. The chip system includes one or more processors, the processors are coupled to a memory, the memory is used for storing computer program instructions, and the one or more processors are used for calling the computer instructions to cause the electronic device to execute the steps of the method according to any one of claims 1 to 7.

Images