Image processing method and device, electronic equipment, storage medium and program product

Abstract

The application provides an image processing method and device based on artificial intelligence, electronic equipment, computer readable storage medium and computer program product; the method comprises the following steps: acquiring a noise image and a noise gradient image corresponding to the noise image; performing pixel feature sampling processing on the noise image based on a complementary attention mechanism to obtain an attention noise feature; performing gradient residual processing on the noise gradient image based on the complementary attention mechanism based on the attention noise feature to obtain an attention gradient feature; performing pixel conversion processing on the attention gradient feature and the attention noise feature based on the complementary attention mechanism to obtain a noise reduction feature image; and performing pixel superposition processing on the noise reduction feature image and the noise image to obtain a noise reduction image corresponding to the noise image. Through the application, the texture details can be preserved while accurately reducing noise, thereby optimizing the noise reduction effect.

Description

Technical Field

[0001] This application relates to image processing technology, and more particularly to an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product based on artificial intelligence. Background Technology

[0002] Digital images in reality are often affected by noise interference from imaging equipment and the external environment during digitization and transmission, resulting in noisy images. Noise is a significant cause of image interference, and images in practical applications may contain various types of noise, which may be generated during transmission or during processing such as quantization.

[0003] In related technologies, it is difficult to preserve the texture details of noisy images when performing noise reduction. The images become too smooth after noise reduction, which reduces the amount of information in the images and causes information loss. Summary of the Invention

[0004] This application provides an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product based on artificial intelligence, which can accurately reduce noise while preserving texture details, thereby optimizing the noise reduction effect.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides an image processing method based on artificial intelligence, including:

[0007] Acquire a noisy image and acquire a noise gradient image corresponding to the noisy image;

[0008] The noisy image is subjected to pixel feature sampling processing based on a complementary attention mechanism to obtain attention noise features;

[0009] Based on the attention noise features, the noise gradient image is subjected to gradient residual processing based on the complementary attention mechanism to obtain attention gradient features;

[0010] The attention gradient features and the attention noise features are subjected to pixel transformation processing based on the complementary attention mechanism to obtain a denoised feature image;

[0011] The denoised feature image and the noise image are pixel-overlayed to obtain a denoised image corresponding to the noise image.

[0012] This application provides an image processing device based on artificial intelligence, including:

[0013] An acquisition module is used to acquire a noise image and acquire a noise gradient image corresponding to the noise image;

[0014] The sampling module is used to perform pixel feature sampling processing on the noisy image based on the complementary attention mechanism to obtain attention noise features;

[0015] The residual module is used to perform gradient residual processing on the noise gradient image based on the complementary attention mechanism, based on the attention noise features, to obtain attention gradient features;

[0016] The mapping module is used to perform pixel transformation processing on the attention gradient features and the attention noise features based on the complementary attention mechanism to obtain a noise-reduced feature image;

[0017] The overlay module is used to perform pixel overlay processing on the denoised feature image and the noise image to obtain a denoised image corresponding to the noise image.

[0018] In some embodiments, the pixel feature sampling process is performed through a first feature network, which includes a first convolutional network and a symmetric network; the sampling module is further configured to: perform a first convolutional process on the noisy image through the first convolutional network to obtain a first initial feature corresponding to the noisy image; and perform sampling processing on the first initial feature based on the complementary attention mechanism through the symmetric network to obtain the attention noise feature.

[0019] In some embodiments, the symmetric network includes N downsampling networks, a connection network, and M upsampling networks, where N is an integer greater than or equal to 1, and M is equal to N; the sampling module is further configured to: perform downsampling processing on the first initial feature based on the complementary attention mechanism through the N downsampling networks to obtain downsampling results corresponding to the N downsampling networks; perform a first feature mapping processing on the downsampling results through the connection network to obtain connection features corresponding to the noisy image; and perform upsampling processing on the connection features based on the complementary attention mechanism through the M upsampling networks to obtain the attention noise features.

[0020] In some embodiments, when N is an integer greater than or equal to 2, the sampling module is further configured to: perform complementary attention processing on the input of the nth downsampling network through the nth downsampling network among the N downsampling networks to obtain the nth downsampling result corresponding to the N downsampling networks, and transmit the nth downsampling result to the (n+1)th downsampling network for further complementary attention processing to obtain the (n+1)th downsampling result; wherein, n is an integer variable whose value increments from 1, and the value range of n is... n When n is 1, the input to the nth downsampling network is the first initial feature; when n is 2... n When n is n-1, the input of the nth downsampling network is the (n-1)th downsampling result output by the (n-1)th downsampling network. When n takes the value N-1, the (n+1)th downsampling result is the downsampling result of the corresponding N downsampling networks.

[0021] In some embodiments, the sampling module is further configured to: perform a second convolution process on the input of the m-th upsampling network through the m-th upsampling network among the M upsampling networks, and perform complementary attention processing on the obtained convolution result to obtain the m-th upsampling result; transmit the m-th upsampling result output by the m-th upsampling network to the (m+1)-th upsampling network for further second convolution processing, and perform complementary attention processing on the obtained convolution result to obtain the (m+1)-th upsampling result; when m takes the value of At that time, the (m+1)th upsampling result is added to the first initial feature to obtain the attention noise feature; where m is an integer variable that increments from 1, and the value range of m is... m When m is 1, the input of the m-th sampling network is the connection feature; when m is 2... At that time, the input of the m-th upsampling network is the (m-1)-th upsampling result output by the (m-1)-th upsampling network.

[0022] In some embodiments, the sampling module is further configured to: perform upsampling processing on the connection features based on the complementary attention mechanism through the M upsampling networks to obtain the attention noise features, before m takes the value The output of the Mth downsampling network is concatenated with the output of the connection network, and the concatenated result is used as the input of the mth upsampling network, instead of using the output of the connection network as the input of the mth upsampling network; when m takes the value When the output of the (M+1-m)th downsampling network is concatenated with the output of the (m-1)th upsampling network, the concatenation result is used as the input of the mth upsampling network, instead of using the output of the (m-1)th upsampling network as the input of the mth upsampling network.

[0023] In some embodiments, the gradient residual processing is performed through a second feature network, which includes a second convolutional network and a residual network. The residual module is further configured to: perform a first convolutional processing on the noisy gradient image through the second convolutional network to obtain a second initial feature corresponding to the noisy gradient image; perform residual iterative processing on the second initial feature and the attention noise feature based on the complementary attention mechanism through the residual network to obtain attention gradient iterative features; and perform feature addition processing on the attention gradient iterative features and the second initial feature through the residual network to obtain the attention gradient features.

[0024] In some embodiments, when the residual network comprises T iterative units, and T is an integer greater than or equal to 2, the residual module is further configured to: perform complementary attention processing on the input of the t-th iterative unit through the t-th iterative unit among the T iterative units, and perform a first convolution processing on the obtained complementary attention result to obtain the t-th iterative result; perform feature addition processing on the t-th iterative result and any intermediate noise feature, and transmit the obtained addition result to the (t+1)-th iterative unit to continue complementary attention processing, and perform a first convolution processing on the obtained complementary attention result to obtain the (t+1)-th iterative result; wherein, t is an integer variable whose value increments from 1, and the value range of t is... When t is 1, the input of the t-th iteration unit is the sum of the second initial feature and the first initial feature, where the first initial feature is obtained by performing a first convolution on the noisy image; when t is 2... When t is -1, the intermediate noise feature is obtained by downsampling the first initial feature. In this case, any one of the intermediate noise features is the attention noise feature.

[0025] In some embodiments, the mapping module is further configured to: perform a first convolution processing on the attention gradient features to obtain noise gradient features of the noise gradient image; perform feature merging processing on the attention noise features and the noise gradient features to obtain a first merged feature; perform complementary attention processing on the first merged feature, and perform the first convolution processing on the obtained complementary attention result to obtain a first convolution feature; perform feature addition processing on the first convolution feature and the first initial feature, and perform a third convolution processing on the obtained addition result to obtain the denoised feature image.

[0026] In some embodiments, the mapping module is further configured to: perform channel-based partitioning on the first merged feature to obtain a first channel feature and a second channel feature; perform spatial attention processing on the first channel feature to obtain spatial attention features, and perform channel attention processing on the first channel feature to obtain channel attention features; perform a second feature mapping processing on the spatial attention features and the channel attention features to obtain dense features corresponding to the first merged feature; perform a third feature mapping processing on the second channel feature to obtain sparse features corresponding to the first merged feature; and perform a fourth feature mapping processing on the dense features and the sparse features based on the first merged feature to obtain the complementary attention result.

[0027] In some embodiments, the mapping module is further configured to: perform the first convolution processing on the first channel features to obtain a second convolution feature; perform a first activation processing on the second convolution feature to obtain a first activation result; perform the first convolution processing on the first activation result to obtain a third convolution feature; perform max pooling processing on the third convolution feature to obtain a first pooling result; and perform average pooling processing on the third convolution feature to obtain a second pooling result; perform feature merging processing on the first pooling result and the second pooling result to obtain a second merged feature; perform the first convolution processing on the second merged feature to obtain a fourth convolution feature; perform a second activation processing on the fourth convolution feature to obtain a second activation result; and perform feature multiplication processing on the second activation result and the third convolution feature to obtain the spatial attention feature.

[0028] In some embodiments, the mapping module is further configured to: perform the first convolution processing on the first channel feature to obtain a second convolution feature; perform a first activation processing on the second convolution feature to obtain a first activation result; perform the first convolution processing on the first activation result to obtain a third convolution feature; perform average pooling processing on the third convolution feature to obtain a second pooling result; perform the first convolution processing on the second pooling result to obtain a fifth convolution feature; perform a first activation processing on the fifth convolution feature to obtain a third activation result; and perform the second convolution processing on the third activation result to obtain a sixth convolution feature; perform a second activation processing on the sixth convolution feature to obtain a fourth activation result; and perform feature multiplication processing on the fourth activation result and the third convolution feature to obtain the channel attention feature.

[0029] In some embodiments, the mapping module is further configured to: perform feature merging processing on the spatial attention features and the channel attention features to obtain a third merged feature; perform the first convolution processing on the third merged feature to obtain a seventh convolutional feature; and perform feature addition processing on the seventh convolutional feature and the first channel feature to obtain the dense feature.

[0030] In some embodiments, the mapping module is further configured to: perform the second convolution processing on the second channel features to obtain an eighth convolution feature; perform batch normalization processing on the eighth convolution feature to obtain a normalization result; perform the first activation processing on the normalization result to obtain a ninth convolution feature corresponding to each channel; perform inexpensive linear mapping processing on the ninth convolution feature; and perform feature merging processing on the inexpensive linear mapping result corresponding to the ninth convolution feature and the ninth convolution feature to obtain the sparse feature.

[0031] In some embodiments, the mapping module is further configured to: perform feature merging processing on the dense features and the sparse features to obtain a fourth merged feature; and perform feature addition processing on the fourth merged feature and the first merged feature to obtain the complementary attention result.

[0032] In some embodiments, the image processing method is implemented by invoking the prediction network of a lightweight denoising model. The apparatus further includes a training module, which is configured to: acquire noisy image samples, labeled denoised images corresponding to the noisy image samples, gradient images corresponding to the labeled denoised images, and noise gradient image samples corresponding to the noisy image samples; forward propagate the noisy image samples and the noise gradient image samples in the prediction network of the lightweight denoising model to obtain predicted denoised images corresponding to the noisy image samples and sample attention gradient features corresponding to the noise gradient image samples; and apply the sample attention gradient features to the gradient map generation network of the lightweight denoising model. The gradient features are subjected to a second convolution process to obtain a predicted gradient image corresponding to the noisy image sample; pixel loss is determined based on the error between the labeled denoised image and the predicted denoised image; the gradient image corresponding to the predicted denoised image is obtained, and pixel gradient loss is determined based on the error between the gradient image corresponding to the predicted denoised image and the gradient image corresponding to the labeled denoised image; gradient loss is determined based on the error between the gradient image corresponding to the labeled denoised image and the predicted gradient image; the pixel loss, the pixel gradient loss, and the gradient loss are fused to obtain a comprehensive loss, and the parameters of the lightweight denoising model are updated based on the comprehensive loss.

[0033] This application provides an electronic device, including:

[0034] Memory is used to store executable instructions for a computer;

[0035] The processor, when executing computer-executable instructions stored in the memory, implements the artificial intelligence-based image processing method provided in the embodiments of this application.

[0036] This application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the artificial intelligence-based image processing method provided in this application.

[0037] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the artificial intelligence-based image processing method provided in this application.

[0038] The embodiments of this application have the following beneficial effects:

[0039] This application embodiment performs pixel feature sampling processing on a noisy image based on a complementary attention mechanism to obtain attention noise features. Based on the attention noise features, gradient residual processing is performed on a noisy gradient image based on a complementary attention mechanism to obtain attention gradient features. By processing the noisy gradient image, structure preservation capability is introduced, which helps to fully preserve the original details and texture information of the image while completing noise reduction. Pixel transformation processing based on a complementary attention mechanism is performed on the attention gradient features and attention noise features to obtain a denoised feature image. Through the dual-branch design of pixel feature sampling on the noisy image and gradient residual processing on the noisy gradient image, the entire processing flow is lightweight while improving the denoising performance of the noisy image. Therefore, subsequent pixel superposition processing of the denoised feature image and the noisy image results in a denoised image with excellent denoising effect. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the structure of an artificial intelligence-based image processing system provided in an embodiment of this application;

[0041] Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0042] Figures 3A-3C This is a flowchart illustrating the image processing method based on artificial intelligence provided in an embodiment of this application;

[0043] Figures 4A-4B The image is a denoised image obtained by applying a denoising scheme based on relevant technologies;

[0044] Figure 4CThe image is a denoised image obtained by applying the lightweight denoising model provided in the embodiments of this application;

[0045] Figure 5 This is a framework diagram of the lightweight noise reduction model of the image processing method based on artificial intelligence provided in the embodiments of this application;

[0046] Figure 6 This is a schematic diagram of the attention mechanism of the image processing method based on artificial intelligence provided in the embodiments of this application;

[0047] Figure 7 This is a schematic diagram of the data flow of the image processing method based on artificial intelligence provided in the embodiments of this application;

[0048] Figure 8 This is a flowchart illustrating the training process of an artificial intelligence-based image processing method provided in an embodiment of this application.

[0049] Figure 9 This is an application flowchart of the image processing method based on artificial intelligence provided in the embodiments of this application;

[0050] Figure 10 This is a noise reduction effect diagram of the image processing method based on artificial intelligence provided in the embodiments of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0053] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0055] In the implementation of this application, the collection and processing of relevant data should be strictly in accordance with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0056] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0057] 1) Model inference: This is the image processing process in deep learning to obtain the model's prediction results.

[0058] 2) Convolutional Neural Networks (CNNs) are a type of feedforward neural network that includes convolutional computation and has a deep structure. Artificial neurons can respond to surrounding units. CNNs include convolutional layers and pooling layers.

[0059] 3) U-network is a U-shaped network structure that is a symmetric time structure containing downsampling and upsampling parts.

[0060] 4) Adam is an optimizer for neural network training iteration. It replaces the traditional stochastic gradient descent process and performs first-order gradient optimization on a stochastic objective function. It can iteratively update the neural network weights based on the training data.

[0061] 5) Image noise refers to unnecessary or redundant interference information in image data. Any factors in an image that hinder people's acceptance of its information can be called image noise.

[0062] 6) Video transcoding refers to converting an already compressed and encoded video stream into another video stream to adapt to different network bandwidths, different terminal processing capabilities, and different user needs. Transcoding is essentially a process of first decoding and then encoding.

[0063] 7) Blind denoising: A network is specifically trained to denoise images with a particular level of noise variance. This network will not work well at a given noise level. Using a network model to process images with unknown noise levels is called blind denoising.

[0064] The following denoising schemes exist in related technologies: 1) Non-AI denoising schemes: First, similar image blocks in the global region are calculated, and then wavelet transform domain denoising is performed. Two-dimensional image blocks that are close to the reference block are determined by similarity judgment, and similar image blocks are combined into three-dimensional groups. Collaborative filtering is performed on the three-dimensional groups, and the processing results are aggregated to the position of the original image block; 2) Neural network-based denoising schemes: Neural networks are used to simulate the forward and reverse processes of the camera ISP module to obtain more realistic synthetic data and train a denoising network; 3) Attention network-based denoising schemes: Dynamic non-local similarity at the block level is studied. Graph convolution processing can be performed. Each node has a dynamic and adaptive number of neighboring nodes, and the image reconstruction process is adaptively adjusted through its connected neighboring nodes.

[0065] The drawbacks of non-AI denoising solutions are: poor performance and inadequate denoising effect of blind denoising; and because they require filtering in 3D space and then inversely transforming and fusing the results into 2D, the overall processing time is long, the denoising effect is poor, the generalization ability is poor, and failures are prone to occur, making it difficult to achieve good results in real-world business scenarios. The drawbacks of neural network-based denoising solutions are: the model structure stacks many feature extraction models, resulting in excessively high computational complexity and long model inference time, leading to low transcoding efficiency. The drawbacks of attention network-based denoising solutions are: the dynamic graph attention mechanism causes excessively high model complexity and long processing time; incomplete preservation of image texture details; and overly smooth images after denoising, resulting in image quality loss.

[0066] This application provides an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product based on artificial intelligence, which can accurately reduce noise while preserving texture details, thereby optimizing the noise reduction effect.

[0067] See Figure 1 , Figure 1 This is a schematic diagram of the structure of an artificial intelligence-based image processing system provided in an embodiment of this application. The terminal 400 is connected to the server 200 through a network 300. The network 300 can be a wide area network or a local area network, or a combination of both.

[0068] In some embodiments, the AI-based image processing method provided in this application can be implemented collaboratively by a terminal and a server. The terminal 400 receives a noisy image (e.g., a video frame in a video) uploaded by a user. The terminal 400 sends the noisy image to the server 200. The server 200 obtains the noise gradient image of the corresponding noisy image and calls a lightweight denoising model to perform pixel feature sampling processing on the noisy image based on a complementary attention mechanism to obtain attention noise features. Based on the attention noise features, the server performs gradient residual processing on the noise gradient image based on a complementary attention mechanism to obtain attention gradient features. The server performs pixel transformation processing on the attention gradient features and the attention noise features based on a complementary attention mechanism to obtain a denoised feature image. The server performs pixel overlay processing on the denoised feature image and the noisy image to obtain a denoised image of the corresponding noisy image. The server 200 returns the denoised image to the terminal 400, and the denoised image (the denoised video frame) is displayed on the terminal 400.

[0069] In some embodiments, the AI-based image processing method provided in this application can also be implemented by a terminal or a server alone. Taking the implementation by a terminal alone as an example, the terminal 400 receives a noisy image (e.g., a video frame in a video) uploaded by a user. The terminal 400 performs pixel feature sampling processing on the noisy image based on a complementary attention mechanism to obtain attention noise features. Based on the attention noise features, the terminal performs gradient residual processing on the noisy gradient image based on a complementary attention mechanism to obtain attention gradient features. The terminal performs pixel transformation processing on the attention gradient features and the attention noise features based on a complementary attention mechanism to obtain a denoised feature image. The terminal performs pixel superposition processing on the denoised feature image and the noisy image to obtain a denoised image of the corresponding noisy image, and presents the denoised image (the denoised video frame) on the terminal 400.

[0070] In some embodiments, server 200 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminal 400 may be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, smart voice interaction device, smart home appliance, vehicle terminal, aircraft, etc., but is not limited thereto. Terminals and servers can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0071] In some embodiments, the terminal or server can implement the AI-based image processing method provided in this application by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run, such as a video APP or a live streaming APP; it can also be a mini-program, i.e., a program that only needs to be downloaded to a browser environment to run; or it can be a mini-program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module, or plugin.

[0072] See Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may be a terminal or a server. In this embodiment of the application, the electronic device is used as a terminal for illustration. Figure 2 The terminal 400 shown includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the terminal 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 440.

[0073] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0074] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0075] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0076] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0077] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0078] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0079] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0080] Presentation module 453 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 associated with user interface 430 (e.g., a display screen, a speaker, etc.).

[0081] The input processing module 454 is used to detect and translate one or more user inputs or interactions from one or more input devices 432.

[0082] In some embodiments, the artificial intelligence-based image processing apparatus provided in this application can be implemented in software. Figure 2 An AI-based image processing device 455, stored in memory 450, is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: an acquisition module 4551, a sampling module 4552, a residual module 4553, a mapping module 4554, a superposition module 4555, and a training module 4556. These modules are logically connected and can therefore be arbitrarily combined or further divided according to their implemented functions. The functions of each module will be described below.

[0083] See Figure 3A , Figure 3AThis is a flowchart illustrating the artificial intelligence-based image processing method provided in the embodiments of this application, which will be combined with... Figure 3A The steps shown are explained below. Figure 3A The steps shown can be performed by an electronic device.

[0084] In step 101, a noise image is acquired, and a noise gradient image corresponding to the noise image is acquired.

[0085] As an example, the gradient map of image I can be referenced from formulas (1) to (4):

[0086] (1);

[0087] (2);

[0088] (3);

[0089] (4);

[0090] Where i and j are the positions of pixel x in image I. It is the difference between the left and right adjacent pixels of pixel x. It is the difference between the upper and lower adjacent pixels of pixel x. It is a two-dimensional vector formed by two differences. It is the image gradient corresponding to pixel x.

[0091] In step 102, pixel feature sampling processing based on complementary attention mechanism is performed on the noisy image to obtain attention noise features.

[0092] As an example, pixel feature sampling processing is performed through a first feature network, which includes a first convolutional network and a symmetric network.

[0093] In some embodiments, see Figure 3B In step 102, pixel feature sampling processing based on complementary attention mechanism is performed on the noisy image to obtain attention noise features, which can be achieved through steps 1021 to 1022.

[0094] In step 1021, the noisy image is processed by the first convolutional network to obtain the first initial features of the corresponding noisy image.

[0095] As an example, the first convolutional processing is performed using a 3*3 convolutional network (Conv 3*3), and the first initial feature is the convolution result obtained by performing the first convolutional processing on the noisy image X using a 3*3 convolutional network (Conv 3*3).

[0096] In step 1022, the first initial features are sampled using a complementary attention mechanism through a symmetric network to obtain attention noise features.

[0097] In some embodiments, the sampling processing of the first initial features based on the complementary attention mechanism using a symmetric network in step 1022 to obtain attention noise features can be achieved through the following technical solution: The first initial features are downsampled using N downsampling networks based on the complementary attention mechanism to obtain downsampling results corresponding to the N downsampling networks; the downsampling results are then processed by a connection network using a first feature mapping to obtain connection features corresponding to the noise image. The connection network is located between the N downsampling networks and M upsampling networks. This connection network can be a single convolutional layer or multiple convolutional layers, and it can also be a complementary attention network used to perform complementary attention processing. The connection features are then upsampled using M upsampling networks based on the complementary attention mechanism to obtain attention noise features. The design of the U-shaped network structure allows for downsampling results at different scales, thereby improving subsequent denoising performance.

[0098] As an example, the symmetric network includes N downsampling networks, a connection network, and M upsampling networks, where N is an integer greater than or equal to 1, and M is equal to N. The connection network is cascaded with the N downsampling networks and the M upsampling networks, meaning the connection network is positioned between the downsampling networks and the upsampling networks. When both N and M are 1, the first initial feature is downsampled using a complementary attention mechanism through one downsampling network to obtain the intermediate noise feature output by the downsampling network, which serves as the downsampling result. The downsampling result is then processed by the connection network using a first feature mapping mechanism to obtain the connection features of the corresponding noise image. This first feature mapping process can be either complementary attention processing or convolutional processing. Finally, the connection features are upsampled using a complementary attention mechanism through one upsampling network to obtain the attention noise features.

[0099] In some embodiments, when N is an integer greater than or equal to 2, the above-mentioned downsampling processing of the first initial feature based on complementary attention mechanism through N downsampling networks to obtain the downsampling results corresponding to the N downsampling networks can be achieved through the following technical solution: Complementary attention processing is performed on the input of the nth downsampling network among the N downsampling networks to obtain the nth downsampling result (specifically, complementary attention processing is implemented through a complementary attention network; downsampling processing is performed on the output of the complementary attention network to obtain the nth downsampling result), and the nth downsampling result is transmitted to the (n+1)th downsampling network for further complementary attention processing to obtain the (n+1)th downsampling result; where n is an integer variable whose value increments from 1, and the value range of n is... n When n is 1, the input to the nth downsampling network is the first initial feature; when n is 2... n At this point, the input to the nth downsampling network is the (n-1)th downsampling result output by the (n-1)th downsampling network. By using cascaded downsampling networks, feature information at multiple scales can be extracted, thereby improving subsequent denoising performance.

[0100] As an example, suppose N is 3, meaning there are three cascaded downsampling networks. The input (first initial feature) of the first downsampling network is downsampled using complementary attention. The first downsampling result (first intermediate noise feature) output by the first downsampling network is transmitted to the second downsampling network for further downsampling, resulting in the second downsampling result (second intermediate noise feature). The second downsampling result is then transmitted to the third downsampling network for further downsampling, resulting in the third downsampling result (third intermediate noise feature). The third downsampling network is connected to the connection network, so the third downsampling result is used as the downsampling result. The downsampling result output by each downsampling network is an intermediate noise feature, and the intermediate noise features output by different downsampling networks have different scale information, thus allowing for the acquisition of multi-scale features.

[0101] In some embodiments, the above-mentioned upsampling of connection features using M upsampling networks based on complementary attention mechanism to obtain attention noise features can be achieved through the following technical solution: The input of the m-th upsampling network is subjected to a second convolution process, and the resulting convolution is subjected to complementary attention processing to obtain the m-th upsampling result (specifically, complementary attention processing is implemented through a complementary attention network; upsampling the output of the complementary attention network yields the m-th upsampling result); the m-th upsampling result output from the m-th upsampling network is transmitted to the (m+1)-th upsampling network for further second convolution processing, and the resulting convolution is subjected to complementary attention processing to obtain the (m+1)-th upsampling result; when m takes the value... At this point, the (m+1)th upsampling result is added to the first initial feature to obtain the attention noise feature; where m is an integer variable starting from 1 and increasing in value, and the range of m is... m When m is 1, the input of the m-th sampling network is the connection feature; when m is 2... At this point, the input to the m-th upsampling network is the (m-1)-th upsampling result output by the (m-1)-th upsampling network. By cascading upsampling networks, feature information at multiple scales is extracted, thereby improving subsequent noise reduction performance.

[0102] As an example, when M is also 3, the input of the first upsampling network in the three cascaded upsampling networks is upsampled to obtain the first upsampling result corresponding to the first upsampling network. (The input of the first upsampling network is then subjected to a second convolution, and complementary attention is applied to the resulting convolution to obtain the first upsampling result. The second convolution is performed through a 1*1 convolutional network.) (1*1) Convolution processing is performed. The input of the first sampling network is the connection feature output by the connection network. The first upsampling result corresponding to the first upsampling network is transmitted to the second upsampling network for further upsampling processing to obtain the second upsampling result corresponding to the second upsampling network (the second convolution processing is performed on the first upsampling result, and complementary attention processing is performed on the obtained convolution result to obtain the second upsampling result). The second upsampling result corresponding to the second upsampling network is transmitted to the third upsampling network for further upsampling processing to obtain the third upsampling result corresponding to the third upsampling network (the second convolution processing is performed on the second upsampling result, and complementary attention processing is performed on the obtained convolution result to obtain the third upsampling result). The first initial feature of the third upsampling result is subjected to feature addition processing to obtain the attention noise feature. The first initial feature is the convolution result obtained by performing the first convolution processing on the noisy image, and is also used as the intermediate noise feature.

[0103] In some embodiments, before upsampling the connection features using a complementary attention mechanism through M upsampling networks to obtain attention noise features, the following processing is performed: when m takes the value... The output of the Mth downsampling network is concatenated with the output of the connection network, and the concatenated result is used as the input of the mth upsampling network, instead of using the output of the connection network as the input of the mth upsampling network; when m takes the value of In this process, the output of the (M+1-m)th downsampling network is concatenated with the output of the (m-1)th upsampling network, and the concatenated result is used as the input of the m-th upsampling network, instead of using the output of the corresponding (m-1)th upsampling network as the input of the m-th upsampling network. This cross-layer connection approach enhances information representation capabilities, thereby improving subsequent noise reduction performance.

[0104] As an example, a symmetric network can be improved upon the structure of a cascaded downsampling network, convolutional network, and upsampling network. Specifically, downsampling and upsampling networks can be connected across layers. For instance, a symmetric network might consist of three downsampling networks and three upsampling networks. The output of the third downsampling network is connected to the input of the first upsampling network. Therefore, the output of the third downsampling network is concatenated with the output of the connection network, and the concatenated result is used as the input of the first upsampling network, instead of directly using the connection features output by the connection network as the input of the first upsampling network. Similarly, the output of the second downsampling network (the second downsampling result) is concatenated with the output of the first upsampling network. The first upsampling result is stitched together and used as the input to the second upsampling network, instead of using the first upsampling result corresponding to the first upsampling network as the input to the second upsampling network. The output of the first downsampling network (the first downsampling result) and the output of the second upsampling network (the second upsampling result) are stitched together and used as the input to the third upsampling network, instead of using the second upsampling result corresponding to the second upsampling network as the input to the third upsampling network. Extensive experiments have shown that cross-layer connection structures that connect downsampling networks and upsampling networks can help output denoised images with higher realism and credibility.

[0105] In step 103, based on the attention noise features, gradient residual processing based on complementary attention mechanism is performed on the noise gradient image to obtain attention gradient features.

[0106] As an example, gradient residual processing is performed through a second feature network, which includes a second convolutional network and a residual network.

[0107] In some embodiments, step 103, based on the attention noise features, performs gradient residual processing on the noisy gradient image using a complementary attention mechanism to obtain attention gradient features. This can be achieved through the following technical solution: The noisy gradient image is processed by a first convolution through a second convolutional network to obtain a second initial feature corresponding to the noisy gradient image; the second initial feature and the attention noise features are processed by a residual network using a complementary attention mechanism for residual iterative processing to obtain attention gradient iterative features; and the attention gradient iterative features and the second initial feature are summed using a residual network to obtain the attention gradient features. By constructing gradient branches to introduce structure preservation capabilities, the original detailed texture information of the image is fully preserved while denoising is completed, solving the problem of excessive smoothness and image quality loss in denoised images.

[0108] As an example, the first convolutional processing is performed using a 3x3 convolutional network (Conv 3x3). The second initial feature is the result of the first convolutional processing on the noise gradient image of the noisy image X using a 3x3 convolutional network (Conv 3x3). When the residual network includes one iterative unit, the second initial feature and the attention noise feature are summed. The summed result is then subjected to complementary attention processing. The complementary attention result is then subjected to the first convolutional processing to obtain the attention gradient iterative feature. Finally, the attention gradient iterative feature and the second initial feature are summed to obtain the attention gradient feature.

[0109] In some embodiments, when the residual network includes T iteration units, and T is an integer greater than or equal to 2, the above-mentioned residual iterative processing based on complementary attention mechanism on the second initial feature and attention noise feature through the residual network to obtain attention gradient iterative features can be implemented through the following technical solution: Complementary attention processing is performed on the input of the t-th iteration unit in the T iteration units, and the obtained complementary attention result is subjected to a first convolution processing to obtain the t-th iteration result; the t-th iteration result is added to any intermediate noise feature, and the added result is transmitted to the (t+1)-th iteration unit for further complementary attention processing, and the obtained complementary attention result is subjected to a first convolution processing to obtain the (t+1)-th iteration result. Residual processing can improve the feature representation ability of the gradient branch, thereby improving the structure preservation ability and helping to retain effective detailed texture information.

[0110] As an example, t is an integer variable whose value starts from 1 and increments, and the range of t is... When t is 1, the input of the t-th iteration unit is the sum of the second initial feature and the first initial feature, where the first initial feature is obtained by performing a first convolution on the noisy image; when t is 2... When t is -1, the intermediate noise feature is obtained by downsampling the first initial feature at least once. When t takes the value of At that time, any of the above intermediate noise features is an attention noise feature, and the result of the (t+1)th iteration is the attention gradient iterative feature.

[0111] As an example, assuming T is 3, meaning there are three cascaded iterative networks, the first iterative unit in the three cascaded iterative units performs complementary attention processing on its input, and then performs a first convolution on the resulting complementary attention, yielding the first iteration result. The input of the first iterative unit is the sum of the second initial feature and the first initial feature, which is obtained by performing a first convolution on the noisy image. The first iteration result output by the first iterative unit, plus the sum of any intermediate noise feature, is transmitted to the second iterative unit for further complementary attention processing, and the resulting complementary attention is then subjected to a first convolution, yielding the second iteration result. The second iteration result, plus the sum of the attention noise feature, is transmitted to the third iterative unit for further complementary attention processing, and the resulting complementary attention is then subjected to a first convolution, yielding the third iteration result. Finally, the third iteration result is used to apply attention gradient iterative features.

[0112] In step 104, pixel transformation processing based on complementary attention mechanism is performed on the attention gradient features and attention noise features to obtain a denoised feature image.

[0113] In some embodiments, see Figure 3C In step 104, pixel transformation processing based on a complementary attention mechanism is performed on the attention gradient features and attention noise features to obtain a denoised feature image, which can be obtained through... Figure 3C Steps 1041 to 1044 are implemented.

[0114] In step 1041, the attention gradient features are subjected to a first convolution process to obtain the noise gradient features of the noise gradient image.

[0115] As an example, the first convolutional processing is performed by a 3*3 convolutional network (Conv 3*3), and the noise gradient feature is the result of the first convolutional processing of the attention gradient feature by a 3*3 convolutional network (Conv 3*3).

[0116] In step 1042, the attention noise features and noise gradient features are processed by feature merging to obtain the first merged feature.

[0117] In step 1043, complementary attention processing is performed on the first merged feature, and the obtained complementary attention result is subjected to first convolution processing to obtain the first convolution feature.

[0118] In some embodiments, the complementary attention processing of the first merged feature in step 1043 can be implemented through the following technical solution: The first merged feature is partitioned based on channels to obtain a first channel feature and a second channel feature. The size of the first merged feature is H*w*c, the size of the first channel feature is H*w*c / 2, and the size of the second channel feature is H*w*c / 2, where H is the feature height, w is the feature width, and c is the number of channels in the first merged feature. Spatial attention processing is performed on the first channel feature to obtain spatial attention features, and channel attention processing is performed on the first channel feature to obtain channel attention features. A second feature mapping processing is performed on the spatial attention features and channel attention features to obtain dense features corresponding to the first merged feature. A third feature mapping processing is performed on the second channel feature to obtain sparse features corresponding to the first merged feature. A fourth feature mapping processing is performed on the dense features and sparse features based on the first merged feature to obtain the complementary attention result. All complementary attention processing involved in this application can be implemented through the above technical solutions, only requiring the first merged feature to be replaced with the feature data that needs to undergo complementary attention processing. By leveraging a complementary attention mechanism, both dense and sparse features are simultaneously mined to construct an efficient and lightweight architecture, addressing the issues of overly heavy models with long inference times and low preprocessing transcoding efficiency.

[0119] Complementary attention processing can be divided into two parts: a dense branch (processing the features of the first channel) and a sparse branch (processing the features of the second channel), aiming to explore the complementarity between dense and sparse features. The dense branch suppresses useless feature information in terms of space and channels, allowing the lightweight denoising model to focus more on key feature information. The sparse branch uses different inexpensive linear operators to replace ordinary convolution operators, performing these operations on each channel of the second channel feature to generate sparse features with low computational complexity. The complementarity between dense and sparse features enhances the representational ability of the lightweight denoising model and significantly reduces the amount of computation.

[0120] In some embodiments, the spatial attention processing of the first channel features to obtain spatial attention features can be implemented through the following technical solution: A first convolution processing is performed on the first channel features to obtain a second convolution feature. The first convolution processing is performed using a 3*3 convolutional network (Conv 3*3). The second convolution feature is the convolution result obtained by performing the first convolution processing on the first channel features using a 3*3 convolutional network (Conv 3*3). A first activation processing is performed on the second convolution feature to obtain a first activation result. The first activation processing is ReLU function activation processing. A first convolution processing is performed on the first activation result to obtain a third convolution feature. The first convolution processing is performed using a 3*3 convolutional network (Conv 3*3). The third convolution feature is obtained by performing the first convolution processing on a 3*3 convolutional network (Conv 3*3). The convolution result is obtained by performing a first convolution on the first activation result (3x3). Max pooling is then performed on the third convolution feature to obtain the first pooling result, followed by average pooling to obtain the second pooling result. The first and second pooling results are then combined to obtain the second combined feature. The second combined feature is then subjected to a first convolution to obtain the fourth convolution feature. The first convolution is performed using a 3x3 convolutional network (Conv 3x3). The fourth convolution feature is obtained by performing a first convolution on the second combined feature using a 3x3 convolutional network (Conv 3x3). The fourth convolution feature is then subjected to a second activation (sigmoid function activation) to obtain the second activation result. The third convolution feature is input into the spatial attention branch, and after max pooling and average pooling, a 3x3 convolution and sigmoid function activation are performed to obtain the spatial weights (second activation result) corresponding to the third convolution feature. The second activation result is then multiplied with the third convolution feature to obtain the spatial attention feature.

[0121] In some embodiments, the channel attention processing of the first channel features to obtain channel attention features can be implemented through the following technical solution: Perform a first convolution on the first channel features to obtain a second convolution feature; perform a first activation on the second convolution feature to obtain a first activation result; perform a first convolution on the first activation result to obtain a third convolution feature; perform max pooling on the third convolution feature to obtain a first pooling result; perform a first convolution on the second pooling result to obtain a fifth convolution feature; perform a first activation on the fifth convolution feature to obtain a third activation result; perform a second convolution on the third activation result to obtain a sixth convolution feature; perform a second activation on the sixth convolution feature to obtain a fourth activation result; input the third convolution feature to the channel attention branch, perform average pooling, then perform 3x3 convolution, ReLU activation, and 1x1 convolution (equivalent to performing compression operations for encoding global information); finally, perform sigmoid activation to obtain channel weights, which can completely capture the relationships between channels; multiply the fourth activation result with the third convolution feature to obtain the channel attention feature.

[0122] As an example, the first channel feature is subjected to a first convolutional process to obtain a second convolutional feature. The first convolutional process is performed using a 3x3 convolutional network (Conv 3x3). The second convolutional feature is the result of the first convolutional process performed on the first channel feature using a 3x3 convolutional network (Conv 3x3). The second convolutional feature is then subjected to a first activation process to obtain a first activation result. The first activation process is ReLU activation. The first activation result is then subjected to a first convolutional process to obtain a third convolutional feature. The first convolutional process is performed using a 3x3 convolutional network (Conv 3x3). The third convolutional feature is the result of the first convolutional process performed on the first activation result using a 3x3 convolutional network (Conv 3x3). The third convolutional feature is then subjected to average pooling to obtain a second pooling result. The second pooling result is then subjected to a first convolutional process to obtain a fifth convolutional feature. The first convolutional process is performed using a 3x3 convolutional network (Conv 3x3). The convolution process is performed using a 3x3 network. The fifth convolutional feature is obtained by performing a first convolution on the second pooling result using a 3x3 convolutional network (Conv 3x3). The fifth convolutional feature is then subjected to a first activation process to obtain a third activation result. The first activation process is the ReLU function activation process. The third activation result is then subjected to a second convolution process to obtain a sixth convolutional feature. The second convolution process is performed using a 1x1 convolutional network (Conv 1x1). The sixth convolutional feature is obtained by performing a second convolution on the third activation result using a 1x1 convolutional network (Conv 1x1). The sixth convolutional feature is then subjected to a second activation process (sigmoid function activation process) to obtain a fourth activation result. The fourth activation result is multiplied with the third convolutional feature to obtain the channel attention feature.

[0123] In some embodiments, the above-mentioned second feature mapping processing of spatial attention features and channel attention features to obtain dense features corresponding to the first merged features can be implemented by the following technical solution: performing feature merging processing on spatial attention features and channel attention features to obtain third merged features; performing first convolution processing on the third merged features to obtain seventh convolution features, wherein the first convolution processing is performed by a 3*3 convolutional network (Conv 3*3), and the seventh convolutional feature is the convolution result obtained by performing the first convolution processing on the third merged features by a 3*3 convolutional network (Conv 3*3); performing feature addition processing on the seventh convolutional features and the first channel features to obtain dense features, wherein the size of the dense features is H*w*c / 2.

[0124] In some embodiments, the above-mentioned third feature mapping processing on the second channel features to obtain sparse features corresponding to the first merged features can be implemented through the following technical solution: Performing a second convolution processing on the second channel features to obtain an eighth convolution feature, wherein the second convolution processing is performed using a 1*1 convolutional network (Conv 1*1), and the eighth convolution feature is the convolution result obtained by performing the second convolution processing on the second channel features using a 1*1 convolutional network (Conv 1*1); performing batch normalization processing on the eighth convolution feature to obtain a normalized result; performing a first activation processing on the normalized result to obtain a ninth convolution feature, wherein the first activation processing is ReLU function activation processing; performing a cheap linear mapping processing on the ninth convolution feature, and merging the cheap linear mapping result corresponding to the ninth convolution feature with the ninth convolution feature to obtain sparse features, wherein the size of the sparse features is H*w*c / 2. Cheap linear mapping can reduce computational complexity and improve processing efficiency.

[0125] As an example, the second channel feature F2 is input into the sparse branch for inexpensive computation. Specifically, the second channel feature F2 undergoes 1x1 convolution, batch normalization, and ReLU activation to obtain the ninth convolution feature. The transformation process can be found in formula (5).

[0126] (5);

[0127] Where X is the second channel feature ， Y is the ninth convolution feature. It includes mapping processing that includes 1x1 convolution processing, batch normalization processing, and ReLU activation processing.

[0128] Perform a cheap linear operation on each ninth convolutional feature to obtain multiple feature maps. The proportion of addition operations in the cheap linear operation is greater than the proportion threshold. See formula (6) for the cheap linear operation:

[0129] (6);

[0130] Where i and j represent the input and output channel numbers, respectively. For example, the input channel is channel i and the output channel is channel j. Represents the ninth convolution feature After cheap calculation The calculated j-th feature, It is the ninth convolutional feature of the i-th channel. It corresponds to the j-th cheap operation of the i-th channel.

[0131] In some embodiments, the above-mentioned fourth feature mapping processing based on the first merged feature on dense and sparse features to obtain complementary attention results can be implemented through the following technical solution: performing feature merging processing on dense and sparse features to obtain a fourth merged feature, the size of which is H*w*c; and adding the fourth merged feature to the first merged feature to obtain complementary attention results, the size of which is also H*w*c. Through the complementary attention mechanism, both dense and sparse features are simultaneously mined to construct an efficient lightweight architecture, solving the problems of excessively heavy models with long inference times and low preprocessing transcoding efficiency.

[0132] In step 1044, the first convolutional feature and the first initial feature are added together, and the result of the addition is subjected to a third convolution to obtain a denoised feature image.

[0133] As an example, the first initial feature is the convolution result obtained by performing a first convolution process on the noisy image. The first convolution process is performed by a 3*3 convolutional network (Conv 3*3), and the third convolution process is performed by performing two 3*3 convolutional networks (Conv 3*3) sequentially to obtain the color value of each pixel in the denoised feature image.

[0134] In step 105, the denoised feature image and the noise image are pixel-overlayed to obtain the denoised image of the corresponding noise image.

[0135] The color values ​​of pixels at the same position in the denoised feature image and the noise image are added together to obtain a new color value for each pixel position. Based on the new color value of each pixel position, the denoised image of the corresponding noise image is obtained.

[0136] In some embodiments, the image processing method is implemented by calling the prediction network of a lightweight denoising model. The lightweight denoising model further includes a gradient map generation network. Before executing step 102, the lightweight denoising model needs to be trained. The training process is as follows: Noise image samples, labeled denoised images corresponding to the noise image samples, gradient images corresponding to the labeled denoised images, and noise gradient image samples corresponding to the noise image samples are obtained. The noise image samples and noise gradient image samples are forward propagated through the prediction network of the lightweight denoising model. The prediction network is used to implement all steps 102 to 105, the only difference being that the processing objects are the noise image samples and noise gradient image samples used for training, rather than the noise images and noise gradient images. The predicted denoised images and sample attention gradient features corresponding to the noise image samples are obtained. The sample attention gradient features are then subjected to a second convolution processing (Conv) through the gradient map generation network of the lightweight denoising model. 1*1), obtain the predicted gradient image of the corresponding noisy image sample; determine the pixel loss based on the error between the labeled denoised image and the predicted denoised image; obtain the gradient image of the corresponding predicted denoised image, and determine the pixel gradient loss based on the error between the gradient image of the corresponding predicted denoised image and the gradient image of the corresponding labeled denoised image; determine the gradient loss based on the error between the gradient image of the corresponding labeled denoised image and the predicted gradient image; fuse the pixel loss, pixel gradient loss and gradient loss to obtain the comprehensive loss, and update the parameters of the lightweight denoising model based on the comprehensive loss.

[0137] In some embodiments, the prediction network can also be used to implement all steps 101 to 105. Before step 101, the lightweight denoising model is trained to obtain noisy image samples, labeled denoised images of the corresponding noisy image samples, and gradient images of the corresponding labeled denoised images. The noisy image samples are forward propagated in the lightweight denoising model to obtain the predicted denoised images of the corresponding noisy image samples and sample attention gradient features, which are used as the forward propagation results. Subsequently, the parameters are updated based on the comprehensive loss corresponding to the forward propagation results.

[0138] As an example, during the training phase, this embodiment of the application employs a supervised training method, performing iterative optimization on noisy-noise-free image pairs. See [link to relevant documentation]. Figure 7First, training data is prepared. For the same scene, images with ISO below the ISO threshold are captured as labeled denoised images, or multiple frames are captured and weighted averaged to obtain labeled denoised images. Images with ISO above the ISO threshold are considered noisy images. Camera parameters such as exposure time are adjusted to make the brightness of the two images the same, thus obtaining noisy-noise image sample pairs. The images are then cut into 256x256 image blocks and data augmentation processing (random rotation, vertical flip, mirror flip, etc.) is performed to construct the dataset.

[0139] The first optimization objective is pixel-based loss optimization. This objective is based on optimizing the pixel differences between the denoised image and the labeled image. This metric loss can reduce the average pixel difference between the restored image and the real image and is widely used to improve model fitting and denoising performance. We use an improved L1-based function as the pixel loss, see Equation (7):

[0140] (7);

[0141] in, It's a pixel loss. It is a predicted denoised image. It is a labeled, denoised image.

[0142] The second optimization objective is gradient-based loss optimization. The goal of gradient-based loss optimization is to reduce the difference between the output noisy gradient map and the gradient map extracted from the corresponding labeled image. Under the supervision of pixel control and gradient space, the lightweight denoising model ensures denoising while avoiding the loss of high-frequency details. Therefore, two gradient loss terms can be designed, including: pixel gradient loss and gradient loss, where the pixel gradient loss is given by formula (8):

[0143] (8);

[0144] in, It is pixel gradient loss. It predicts the gradient map corresponding to the denoised image. It is the gradient map corresponding to the denoised image. Pixel gradient loss is used to reduce the difference between the gradient map of the denoised image and the gradient map of the denoised image.

[0145] The gradient loss is given by formula (9):

[0146] (9);

[0147] in, It is gradient loss. It is the denoised gradient map (predicted gradient image) output by the lightweight denoising model. It is the gradient map corresponding to the denoised image. The gradient loss is used to reduce the difference between the denoised gradient map output by the gradient branch and the gradient map corresponding to the denoised image. A high-quality gradient map is reconstructed by minimizing the gradient loss.

[0148] Random initialization Figure 5 The lightweight denoising model is used as the initial lightweight denoising model. The Adam optimization algorithm is adopted, and the comprehensive loss is used as the optimization objective. The lightweight denoising model is trained until convergence. The comprehensive loss is shown in formula (10).

[0149] (10);

[0150] in, It is a comprehensive loss. It is gradient loss. It is pixel gradient loss. It's a pixel loss.

[0151] This application embodiment performs pixel feature sampling processing on a noisy image based on a complementary attention mechanism to obtain attention noise features. Based on the attention noise features, gradient residual processing is performed on a noisy gradient image based on a complementary attention mechanism to obtain attention gradient features. By processing the noisy gradient image, structure preservation capability is introduced, which helps to fully preserve the original details and texture information of the image while completing noise reduction. Pixel transformation processing based on a complementary attention mechanism is performed on the attention gradient features and attention noise features to obtain a denoised feature image. Through the dual-branch design of pixel feature sampling on the noisy image and gradient residual processing on the noisy gradient image, the entire processing flow is lightweight while improving the denoising performance of the noisy image. Therefore, subsequent pixel superposition processing of the denoised feature image and the noisy image results in a denoised image with excellent denoising effect.

[0152] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0153] In some embodiments, the terminal receives a video uploaded by a user. The video frames may contain noise. The video frames are treated as noise images, and the terminal sends the noise images to a video server. The video server obtains the noise gradient image corresponding to the noise image, performs pixel feature sampling processing based on a complementary attention mechanism on the noise image to obtain attention noise features. Based on the attention noise features, gradient residual processing based on a complementary attention mechanism is performed on the noise gradient image to obtain attention gradient features. Pixel transformation processing based on a complementary attention mechanism is performed on the attention gradient features and attention noise features to obtain a denoising feature image. Pixel superposition processing is performed on the denoising feature image and the noise image to obtain a denoised image corresponding to the noise image. The denoised image is returned to the terminal, and the denoised video is played on the terminal. The image processing method provided in this application embodiment can denoise video clips with excessively noisy scenes, or perform pre-denoising processing for old film restoration, and perform pre-processing for standard dynamic range / high dynamic range enhancement of video to improve the success rate of enhancement processing.

[0154] This application proposes a lightweight denoising model to address the image quality degradation caused by noise in video restoration, such as in video super-resolution and old film restoration scenarios. This application improves the model structure of related technologies by employing complementary attention mechanisms and structure preservation, thereby enhancing both image denoising performance and model inference speed. This accelerates the transcoding efficiency of video restoration tasks while more effectively preserving the texture details of the denoised image, thus improving video image quality.

[0155] In some embodiments, see Figures 4A-4C , Figure 4A and Figure 4B The image shown is a denoised image obtained by applying a denoising scheme based on relevant techniques. Figure 4C The image shown is a denoised image obtained by applying the image processing method provided in the embodiments of this application. Figure 4C The denoised image shown demonstrates better denoising performance, with a signal-to-noise ratio of 36.8 dB, which is higher than... Figure 4A and Figure 4B The noise-reduced image shown. Figure 4C The denoised image shown retains more texture details. This application provides an efficient and lightweight denoising model for blind denoising of video images. By leveraging the complementarity of dense and sparse features and the structure preservation capability brought by gradient information, it improves image denoising performance, preserves texture and contour details of the image, and maintains the model's lightweight nature to reduce computational complexity and improve the transcoding efficiency of video preprocessing.

[0156] In some embodiments, see Figure 5 , Figure 5The framework of the lightweight denoising model is shown. The lightweight denoising model includes a gradient branch and a pixel branch. The gradient branch takes a noise gradient map as input and has a cascaded residual structure. The pixel branch uses a U-shaped network structure. This design aims to allow the lightweight denoising model to learn more gradient feature information, thus obtaining a denoised image that retains more texture details. The gradient feature information introduced by the gradient branch contains rich image contour edge information, providing the lightweight denoising model with structural priors, thereby generating an approximate gradient map of the denoised image. The pixel branch can improve feature extraction capabilities based on prior knowledge, guiding the lightweight denoising model to avoid over-smoothing during the learning process and making it easier to extract geometric features in the gradient space to preserve the structural contours of the image.

[0157] In some embodiments, the gradient map of image I can be referenced from formulas (11)-(14):

[0158] (11);

[0159] (12);

[0160] (13);

[0161] (14);

[0162] Where i and j are the positions of pixel x in image I. It is the difference between the left and right adjacent pixels of pixel x. It is the difference between the upper and lower adjacent pixels of pixel x. It is a two-dimensional vector formed by two differences. It is the image gradient corresponding to pixel x.

[0163] The gradient-based branching approach employs a residual cascade structure to transform the input noisy gradient map into a corresponding denoised gradient map, and utilizes image-to-image conversion techniques to learn the mapping between the two modes. The specific process is as follows: The input noisy gradient map first passes through a 3x3 convolutional network (first convolution processing) to extract the first initial features; the first initial features pass through a residual network composed of three complementary attention networks to obtain the first merged features. Before being input into each complementary attention network, feature information from the pixel branch is introduced and superimposed with the features of that branch. The first merged features obtained through the residual network are then passed through a 3x3 convolutional network and a 1x1 convolutional network to output the denoised gradient map.

[0164] The pixel branch adopts a U-shaped network structure design, which includes downsampling and upsampling structures. It can capture features at different levels and make full use of multi-scale feature information through cross-layer connections, converting the input noisy image X into the corresponding denoised image. The specific process is as follows: The input noisy image first passes through a 3x3 convolutional network to extract initial features; the initial features (H*w*c) pass through a U-shaped network constructed from 4 downsampling networks, 4 upsampling networks, and 1 connection network to obtain the first multi-scale merged feature (H*w*c). Each downsampling network includes a complementary attention model (CAM), and each upsampling network includes a CAM and a 1x1 convolutional network. The connection network is a CAM. The feature dimension changes according to the data flow order of the U-shaped network as follows: the feature dimension output by the first downsampling network is 0.5H*0.5w*2c, the feature dimension output by the second downsampling network is 0.25H*0.25w*4c, the feature dimension output by the third downsampling network is 0.125H*0.125w*8c, and the feature dimension output by the fourth downsampling network is 0.0625H*0. The feature dimension of the first upsampling network is 0.125H*0.125w*8c, the feature dimension of the second upsampling network is 0.25H*0.25w*4c, the feature dimension of the third upsampling network is 0.5H*0.5w*2c, the feature dimension of the fourth upsampling network is H*w*c, and the feature dimension of the fifth upsampling network is H*w*c. The first merged feature of the multi-scale is merged with the gradient feature extracted by the gradient branch, and then passed through a complementary attention network and a 3x3 convolutional network to obtain the first merged feature of the pixel. After the first merged feature of the pixel is superimposed on the initial feature, it is passed through two more 3x3 convolutional networks to obtain the denoised image.

[0165] CAM is used to implement complementary attention processing and is applied throughout the lightweight noise reduction model as a basic module. See [link / reference]. Figure 6CAM can be divided into two parts: dense branches and sparse branches, aiming to explore the complementarity of dense and sparse features. The dense branch suppresses useless features in terms of space and channels, allowing the lightweight denoising model to focus more on key feature information. The sparse branch uses different inexpensive linear operators to replace ordinary convolution operators, performing these operations on each channel to generate sparse features with low computational complexity and explore the complementarity of sparse features, thereby improving the representation ability of the lightweight denoising model and significantly reducing the amount of computation. The specific process of CAM is as follows: Input feature x (feature dimension is H*w*c) is divided into two parts according to channels: first channel feature F1 (feature dimension is H*w*0.5c) and second channel feature F2 (feature dimension is H*w*0.5c). These are then processed in dense and sparse branches. The first channel feature F1 is fed into the dense module for feature weighting in spatial and channel dimensions. Specifically, the first channel feature F1 undergoes 3x3 convolution processing, ReLU activation processing, and 3x3 convolution processing to obtain the third convolution feature. The third convolution feature is input into the spatial attention branch, undergoes max pooling and average pooling processing, followed by 3x3 convolution processing and sigmoid activation processing to obtain the spatial weights of the corresponding third convolution feature. The initial feature is multiplied by the spatial weights to obtain the spatial attention feature F. SA Additionally, the third convolutional feature is input into the channel attention branch, processed by average pooling, followed by 3x3 convolution, ReLU activation, and 1x1 convolution (equivalent to compression for encoding global information). Finally, sigmoid activation is performed to obtain channel weights, which fully capture the relationships between channels. The initial features are multiplied by the channel weights to obtain the channel attention feature F. CA Spatial attention features F SA and channel attention features F CA Feature merging is performed, followed by 3x3 convolution, and finally the first channel feature F1 is superimposed to output dense feature F. dense (Feature dimension is H*w*0.5c); The second channel feature F2 is input into the sparse branch for inexpensive computation. Specifically, the second channel feature F2 undergoes 1x1 convolution, batch normalization, and ReLU activation to obtain the intermediate feature (ninth convolution feature). The transformation process can be found in formula (15):

[0166] (15);

[0167] Where X is the second channel feature F2 ， Y is an intermediate feature. It includes mapping processing that includes 1x1 convolution processing, batch normalization processing, and ReLU activation processing.

[0168] Perform a cheap linear operation on each intermediate feature to obtain multiple feature maps. See formula (16) for the linear operation:

[0169] (16);

[0170] Where i and j represent the input and output channel numbers, respectively. For example, the input channel is channel i and the output channel is channel j. Representing intermediate features After cheap calculation The calculated j-th feature, It is the intermediate feature of the i-th channel. It corresponds to the j-th cheap operation of the i-th channel.

[0171] The intermediate features and low-cost computation features obtained from the above two steps are combined to obtain sparse features. (Feature dimension is H*w*0.5c), dense features and sparsity features Merge the features to obtain the output feature y of the entire complementary attention network (feature dimension is H*w*c).

[0172] exist Figure 6 In the dense branches shown, spatial attention and channel attention mechanisms can be processed in parallel and then the feature maps are merged. Alternatively, a cascaded serial spatial attention and channel attention mechanism processing method can be considered.

[0173] In this application embodiment, the parameters of a lightweight denoising model are trained using existing data during the training phase, and the trained lightweight denoising model is used for online services during the application phase. During the training phase, this application embodiment employs a supervised training method, iteratively optimizing the training on noisy-noise-free image pairs. See [link to relevant documentation]. Figure 7 First, training data is prepared. For the same scene, images with ISO below the ISO threshold are captured as labeled denoised images, or multiple frames are captured and weighted averaged to obtain labeled denoised images. Images with ISO above the ISO threshold are considered noisy images. Camera parameters such as exposure time are adjusted to make the brightness of the two images the same, thus obtaining noisy-noise image sample pairs. The images are then cut into 256x256 image blocks and data augmentation processing (random rotation, vertical flip, mirror flip, etc.) is performed to construct the dataset.

[0174] The first optimization objective is pixel-based loss optimization. The loss function used in the first optimization objective is the Charbonnier loss function. The principle of this objective is to optimize by comparing the pixel differences between the denoised image and the labeled denoised image. After the noisy image is input into the lightweight denoising model, the pixel branch output of the model is the denoised image. The noisy image and the labeled denoised image come from the noisy-noise image pair dataset. This metric loss can reduce the average pixel difference between the restored image and the real image and is widely used to improve model fitting and denoising performance. We use an improved function based on L1 as the pixel loss, see formula (7):

[0175] (17);

[0176] in, It's a pixel loss. It is a predicted denoised image. It is a labeled, denoised image.

[0177] The second optimization objective is gradient-based loss optimization. The goal of gradient-based loss optimization is to reduce the difference between the gradient map corresponding to the predicted denoised image and the gradient map corresponding to the corresponding labeled noisy image. Under the supervision of pixel control and gradient space, the lightweight denoising model ensures denoising while avoiding the loss of high-frequency details. Therefore, two gradient loss terms can be designed, including: pixel gradient loss and gradient loss, where the pixel gradient loss is given by formula (18):

[0178] (18);

[0179] in, It is pixel gradient loss. It predicts the gradient map corresponding to the denoised image. It is the gradient map corresponding to the labeled denoised image. Pixel gradient loss is used to reduce the difference between the gradient map of the predicted denoised image and the gradient map of the labeled denoised image.

[0180] The gradient loss is given by formula (19):

[0181] (19);

[0182] in, It is gradient loss. It is the predicted gradient image output by the lightweight noise reduction model. It is the gradient map corresponding to the labeled denoised image. The gradient loss is used to reduce the difference between the predicted gradient map output by the gradient branch and the gradient map corresponding to the labeled denoised image. A high-quality gradient map is reconstructed by minimizing the gradient loss.

[0183] Random initialization Figure 5 The lightweight denoising model is used as the initial lightweight denoising model. The Adam optimization algorithm is adopted, and the comprehensive loss is used as the optimization objective. The lightweight denoising model is trained until convergence. The comprehensive loss is shown in formula (20).

[0184] (20);

[0185] in, It is a comprehensive loss. It is gradient loss. It is pixel gradient loss. It's a pixel loss.

[0186] Regarding pixel loss, embodiments of this application may also employ L1 loss or introduce structural loss to form a joint pixel loss.

[0187] In some embodiments, see Figure 8 , Figure 8 Steps 801 to 806 shown are all executed by electronic devices. In step 801, a training dataset consisting of noisy and noiseless images is acquired. In step 802, the images are cut into 256*256 image blocks and data augmentation operations such as affine transformation are performed. In step 803, each noisy image is input into the lightweight denoising model and forward propagation inference is performed to obtain the restored denoised image. In step 804, the reconstructed denoised image, the denoised gradient image, and the corresponding label image are compared to calculate the corresponding pixel loss and gradient loss. In step 805, backpropagation processing is performed to update and obtain the optimal convolution weights and bias parameters. In step 806, the lightweight denoising model is validated using a validation set, and the optimal lightweight denoising model is saved.

[0188] In some embodiments, during the application phase, see Figure 9 The noisy video is broken down into multiple single-frame noisy images. These frames are then sequentially fed into a lightweight denoising model to obtain denoised individual frames. These denoised frames are then combined to form a complete denoised video. Once the lightweight denoising model is trained, it can be offered online. The specific process is as follows:

[0189] In some embodiments, to verify the effectiveness of the embodiments of this application, experiments were conducted using open-source datasets and compared with related technologies. The comparison results are shown in Table 1 and... Figure 10As shown, Peak Signal to Noise Ratio (PSNR) and Structural Similarity Index (SSIM) are used to measure the denoising performance of the image, and Floating Point Operations Per Second (FLOPS) is used to measure the computational complexity of the model. The lightweight denoising model provided in this application achieves better denoising performance with less computational complexity.

[0190] Table 1 Comparison of indicators between the embodiments of this application and the prior art.

[0191]

[0192] The lightweight denoising model provided in this application benefits from an efficient U-shaped structure design and a complementary attention mechanism, achieving both lightweight performance and improved denoising capabilities. The lightweight denoising model provided in this application improves the PSNR of related lightweight denoising models on open-source datasets by 0.09 dB, while its computational complexity is only one-third of that of related technologies. This application introduces structure preservation capabilities by constructing gradient-based branches, effectively preserving fine texture and contour details of the image during denoising. Furthermore, the lightweight denoising model provided in this application can be used for blind denoising tasks, exhibiting stronger generalization performance, being more lightweight, and more suitable for complex denoising tasks in scenes such as TV series and movies.

[0193] The following continues to describe the exemplary structure of the artificial intelligence-based image processing device 455 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the AI-based image processing device 455 in the memory 450 may include: an acquisition module 4551, used to acquire a noise gradient image corresponding to a noisy image; a sampling module 4552, used to perform pixel feature sampling processing on the noisy image based on a complementary attention mechanism to obtain attention noise features; a residual module 4553, used to perform gradient residual processing on the noise gradient image based on a complementary attention mechanism based on the attention noise features to obtain attention gradient features; a mapping module 4554, used to perform pixel conversion processing on the attention gradient features and attention noise features based on a complementary attention mechanism to obtain a denoised feature image; and an overlay module 4555, used to perform pixel overlay processing on the denoised feature image and the noisy image to obtain a denoised image corresponding to the noisy image.

[0194] In some embodiments, pixel feature sampling processing is performed through a first feature network, which includes a first convolutional network and a symmetric network; the sampling module 4552 is further configured to: perform a first convolutional processing on the noisy image through the first convolutional network to obtain a first initial feature of the corresponding noisy image, and use the first initial feature as an intermediate noise feature; and perform sampling processing on the first initial feature based on a complementary attention mechanism through the symmetric network to obtain attention noise features.

[0195] In some embodiments, the symmetric network includes N downsampling networks, a connection network, and M upsampling networks, where N is an integer greater than or equal to 1, and M is equal to N; the sampling module 4552 is further configured to: perform downsampling processing on the first initial feature based on a complementary attention mechanism through the N downsampling networks to obtain downsampling results corresponding to the N downsampling networks; perform a first feature mapping processing on the downsampling results through the connection network to obtain connection features corresponding to the noise image; and perform upsampling processing on the connection features based on a complementary attention mechanism through the M upsampling networks to obtain attention noise features.

[0196] In some embodiments, when N is an integer greater than or equal to 2, the sampling module 4552 is further configured to: perform complementary attention processing on the input of the nth downsampling network through the nth downsampling network among the N downsampling networks to obtain the nth downsampling result, and transmit the nth downsampling result to the (n+1)th downsampling network for further complementary attention processing to obtain the (n+1)th downsampling result; wherein, n is an integer variable whose value starts from 1 and increments, and the value range of n is... n When n is 1, the input to the nth downsampling network is the first initial feature; when n is 2... n When the input of the nth downsampling network is the (n-1)th downsampling result output by the (n-1)th downsampling network, the input of the nth downsampling network is the (n-1)th downsampling result.

[0197] In some embodiments, the sampling module 4552 is further configured to: perform a second convolution process on the input of the m-th upsampling network through the m-th upsampling network among the M upsampling networks, and perform complementary attention processing on the obtained convolution result to obtain the m-th upsampling result; transmit the m-th upsampling result output by the m-th upsampling network to the (m+1)-th upsampling network for further second convolution processing, and perform complementary attention processing on the obtained convolution result to obtain the (m+1)-th upsampling result; when m takes the value of At this point, the (m+1)th upsampling result is added to the first initial feature to obtain the attention noise feature; where m is an integer variable starting from 1 and increasing in value, and the range of m is... m When m is 1, the input of the m-th sampling network is the connection feature; when m is 2... At that time, the input of the m-th upsampling network is the m-1-th upsampling result output by the (m-1)-th upsampling network.

[0198] In some embodiments, the sampling module 4552 is further configured to: perform upsampling processing on the connection features based on a complementary attention mechanism through M upsampling networks, and before obtaining the attention noise features, when m takes the value of The output of the Mth downsampling network is concatenated with the output of the connection network, and the concatenated result is used as the input of the mth upsampling network, instead of using the output of the connection network as the input of the mth upsampling network; when m takes the value of When the output of the (M+1-m)th downsampling network is concatenated with the output of the (m-1)th upsampling network, the concatenation result is used as the input of the mth upsampling network, instead of using the output of the corresponding (m-1)th upsampling network as the input of the mth upsampling network.

[0199] In some embodiments, gradient residual processing is performed through a second feature network, which includes a second convolutional network and a residual network. The residual module 4553 is further configured to: perform a first convolutional processing on the noisy gradient image through the second convolutional network to obtain a second initial feature corresponding to the noisy gradient image; perform residual iterative processing on the second initial feature and the attention noise feature based on a complementary attention mechanism through the residual network to obtain attention gradient iterative features; and perform feature addition processing on the attention gradient iterative features and the second initial feature through the residual network to obtain attention gradient features.

[0200] In some embodiments, when the residual network includes T iterative units, and T is an integer greater than or equal to 2, the residual module 4553 is further configured to: perform complementary attention processing on the input of the t-th iterative unit through the t-th iterative unit among the T iterative units, and perform a first convolution processing on the obtained complementary attention result to obtain the t-th iterative result; perform feature addition processing on the t-th iterative result with any intermediate noise feature, and transmit the obtained addition result to the (t+1)-th iterative unit to continue complementary attention processing, and perform a first convolution processing on the obtained complementary attention result to obtain the (t+1)-th iterative result; wherein, t is an integer variable whose value starts from 1 and increments, and the value range of t is... When t is 1, the input of the t-th iteration unit is the sum of the second initial feature and the first initial feature, where the first initial feature is obtained by performing a first convolution on the noisy image; when t is 2... When t is -1, the intermediate noise feature is obtained by downsampling the first initial feature. When t takes the value of At that time, any intermediate noise feature is an attention noise feature.

[0201] In some embodiments, the mapping module 4554 is further configured to: perform a first convolution processing on the attention gradient features to obtain noise gradient features of the noise gradient image; perform feature merging processing on the attention noise features and the noise gradient features to obtain a first merged feature; perform complementary attention processing on the first merged feature, and perform a first convolution processing on the obtained complementary attention result to obtain a first convolution feature; perform feature addition processing on the first convolution feature and the first initial feature, and perform a third convolution processing on the obtained addition result to obtain a denoised feature image.

[0202] In some embodiments, the mapping module 4554 is further configured to: perform channel-based partitioning on the first merged feature to obtain a first channel feature and a second channel feature; perform spatial attention processing on the first channel feature to obtain spatial attention features, and perform channel attention processing on the first channel feature to obtain channel attention features; perform a second feature mapping processing on the spatial attention features and channel attention features to obtain dense features corresponding to the first merged feature; perform a third feature mapping processing on the second channel feature to obtain sparse features corresponding to the first merged feature; and perform a fourth feature mapping processing on the dense features and sparse features based on the first merged feature to obtain complementary attention results.

[0203] In some embodiments, the mapping module 4554 is further configured to: perform a first convolution process on the first channel features to obtain a second convolution feature; perform a first activation process on the second convolution feature to obtain a first activation result; perform a first convolution process on the first activation result to obtain a third convolution feature; perform max pooling on the third convolution feature to obtain a first pooling result; and perform average pooling on the third convolution feature to obtain a second pooling result; perform feature merging on the first pooling result and the second pooling result to obtain a second merged feature; perform a first convolution process on the second merged feature to obtain a fourth convolution feature; perform a second activation process on the fourth convolution feature to obtain a second activation result; and perform feature multiplication on the second activation result and the third convolution feature to obtain a spatial attention feature.

[0204] In some embodiments, the mapping module 4554 is further configured to: perform a first convolution process on the first channel features to obtain a second convolution feature; perform a first activation process on the second convolution feature to obtain a first activation result; perform a first convolution process on the first activation result to obtain a third convolution feature; perform average pooling on the third convolution feature to obtain a second pooling result; perform a first convolution process on the second pooling result to obtain a fifth convolution feature; perform a first activation process on the fifth convolution feature to obtain a third activation result; and perform a second convolution process on the third activation result to obtain a sixth convolution feature; perform a second activation process on the sixth convolution feature to obtain a fourth activation result; and perform feature multiplication between the fourth activation result and the third convolution feature to obtain a channel attention feature.

[0205] In some embodiments, the mapping module 4554 is further configured to: perform feature merging processing on the spatial attention features and the channel attention features to obtain a third merged feature; perform a first convolution processing on the third merged feature to obtain a seventh convolutional feature; and perform feature addition processing on the seventh convolutional feature and the first channel feature to obtain a dense feature.

[0206] In some embodiments, the mapping module 4554 is further configured to: perform a second convolution process on the second channel features to obtain an eighth convolution feature; perform batch normalization processing on the eighth convolution feature to obtain a normalization result; perform a first activation process on the normalization result to obtain a ninth convolution feature; perform a cheap linear mapping process on the ninth convolution feature; and perform feature merging processing on the cheap linear mapping result corresponding to the ninth convolution feature and the ninth convolution feature to obtain sparse features.

[0207] In some embodiments, the mapping module 4554 is further configured to: perform feature merging processing on dense features and sparse features to obtain a fourth merged feature; and perform feature addition processing on the fourth merged feature and the first merged feature to obtain a complementary attention result.

[0208] In some embodiments, the image processing method is implemented by calling the prediction network of a lightweight denoising model. The apparatus further includes a training module 4556, which is configured to: acquire noisy image samples, labeled denoised images corresponding to the noisy image samples, and gradient images of the labeled denoised images; propagate the noisy image samples forward through the prediction network of the lightweight denoising model to obtain predicted denoised images of the corresponding noisy image samples and sample attention gradient features of the corresponding noisy image samples; perform a second convolution on the sample attention gradient features through the gradient map generation network of the lightweight denoising model to obtain predicted gradient images of the corresponding noisy image samples; determine pixel loss based on the error between the labeled denoised images and the predicted denoised images; acquire the gradient image of the corresponding predicted denoised image, and determine pixel gradient loss based on the error between the gradient image of the corresponding predicted denoised image and the gradient image of the corresponding labeled denoised image; determine gradient loss based on the error between the gradient image of the corresponding labeled denoised image and the predicted gradient image; fuse the pixel loss, pixel gradient loss, and gradient loss to obtain a comprehensive loss, and update the parameters of the lightweight denoising model based on the comprehensive loss.

[0209] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the artificial intelligence-based image processing method described above in this application.

[0210] This application provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are executed by a processor, they cause the processor to execute the artificial intelligence-based image processing method provided in this application. For example... Figures 3A-3C The image processing method based on artificial intelligence is shown.

[0211] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0212] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0213] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0214] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0215] In summary, this application's embodiments perform pixel feature sampling processing on noisy images based on a complementary attention mechanism to obtain attention noise features; based on these attention noise features, gradient residual processing is performed on noisy gradient images based on a complementary attention mechanism to obtain attention gradient features. By processing the noisy gradient images, structure preservation capability is introduced, which helps to fully preserve the original details and texture information of the image while completing noise reduction. Pixel transformation processing based on a complementary attention mechanism is then performed on the attention gradient features and attention noise features to obtain a denoised feature image. Through a dual-branch design of pixel feature sampling on the noisy image and gradient residual processing on the noisy gradient image, the entire processing flow is lightweight while improving the denoising performance for noisy images. Consequently, subsequent pixel overlay processing of the denoised feature image and the noisy image yields a denoised image with excellent denoising effect.

[0216] The above are merely embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. An image processing method based on artificial intelligence, characterized in that, The method includes: A noisy image is acquired, and a noise gradient image corresponding to the noisy image is acquired, wherein the noise gradient image includes the image gradient of each pixel in the noisy image; The noisy image is subjected to pixel feature sampling processing based on a complementary attention mechanism to obtain attention noise features; Based on the attention noise features, the noise gradient image is subjected to gradient residual processing based on the complementary attention mechanism to obtain attention gradient features; A first merged feature is obtained based on the attention gradient feature and the attention noise feature; The first merged feature is subjected to channel-based partitioning to obtain the first channel feature and the second channel feature; Spatial attention processing is performed on the first channel features to obtain spatial attention features, and channel attention processing is performed on the first channel features to obtain channel attention features; The spatial attention features and the channel attention features are subjected to a second feature mapping process to obtain dense features corresponding to the first merged features; The second channel features are subjected to a third feature mapping process to obtain sparse features corresponding to the first merged features; Based on the first merged feature, a fourth feature mapping process is performed on the dense feature and the sparse feature to obtain complementary attention results; A denoised feature image is obtained based on the complementary attention results; The denoised feature image and the noise image are pixel-overlayed to obtain a denoised image corresponding to the noise image.

2. The method according to claim 1, characterized in that, The pixel feature sampling process is performed through a first feature network, which includes a first convolutional network and a symmetric network. The step of performing pixel feature sampling processing on the noisy image based on a complementary attention mechanism to obtain attention noise features includes: The first convolutional network is used to perform a first convolutional process on the noisy image to obtain a first initial feature corresponding to the noisy image. The first initial feature is sampled using the symmetric network based on the complementary attention mechanism to obtain the attention noise feature.

3. The method according to claim 2, characterized in that, The symmetric network includes N downsampling networks, a connection network, and M upsampling networks. The connection network is cascaded with the N downsampling networks and the M upsampling networks. N is an integer greater than or equal to 1, and M is equal to N. The step of sampling the first initial feature using the symmetric network based on the complementary attention mechanism to obtain the attention noise feature includes: The first initial feature is downsampled using the N downsampling networks based on the complementary attention mechanism to obtain the downsampling results corresponding to the N downsampling networks; The downsampling result is processed by the connection network to obtain the connection features corresponding to the noisy image; The attention noise features are obtained by upsampling the connection features using the M upsampling networks based on the complementary attention mechanism.

4. The method according to claim 3, characterized in that, When N is an integer greater than or equal to 2, the step of performing downsampling processing on the first initial feature based on the complementary attention mechanism through the N downsampling networks to obtain the downsampling results corresponding to the N downsampling networks includes: The input of the nth downsampling network is processed by complementary attention to obtain the nth downsampling result. The nth downsampling result is then transmitted to the (n+1)th downsampling network for further complementary attention processing to obtain the (n+1)th downsampling result. Where n is an integer variable that increments from 1, and the range of n is... n When n is 1, the input of the nth downsampling network is the first initial feature; when n is 2... n When n is n-1, the input of the nth downsampling network is the (n-1)th downsampling result output by the (n-1)th downsampling network. When n takes the value N-1, the (n-1)th downsampling result is the downsampling result corresponding to the N downsampling networks.

5. The method according to claim 3, characterized in that, When M is an integer greater than or equal to 2, the upsampling process performed on the connection features through the M upsampling networks based on the complementary attention mechanism to obtain the attention noise features includes: The input of the m-th upsampling network is subjected to a second convolution process through the m-th upsampling network among the M upsampling networks, and the resulting convolution result is subjected to complementary attention processing to obtain the m-th upsampling result; The m-th upsampling result output by the m-th upsampling network is transmitted to the (m+1)-th upsampling network to continue the second convolution processing, and complementary attention processing is performed on the obtained convolution result to obtain the (m+1)-th upsampling result. When m takes the value At that time, the (m+1)th upsampling result is added to the first initial feature to obtain the attention noise feature; Where m is an integer variable whose value starts from 1 and increments, and the range of m is... m When m is 1, the input of the m-th upsampling network is the connection feature; when m is 2... At that time, the input of the m-th upsampling network is the (m-1)-th upsampling result output by the (m-1)-th upsampling network.

6. The method according to claim 5, characterized in that, Before obtaining the attention noise features by upsampling the connection features using the M upsampling networks based on the complementary attention mechanism, the method further includes: When m takes the value The output of the Mth downsampling network is concatenated with the output of the connection network, and the concatenation result is used as the input of the mth upsampling network, instead of using the output of the connection network as the input of the mth upsampling network. When m takes the value When the output of the (M+1-m)th downsampling network is concatenated with the output of the (m-1)th upsampling network, the concatenation result is used as the input of the mth upsampling network, instead of using the output of the (m-1)th upsampling network as the input of the mth upsampling network.

7. The method according to claim 1, characterized in that, The gradient residual processing is performed through a second feature network, which includes a second convolutional network and a residual network. The step of performing gradient residual processing on the noise gradient image based on the complementary attention mechanism, based on the attention noise features, to obtain attention gradient features includes: The noise gradient image is processed by the second convolutional network to obtain the second initial feature corresponding to the noise gradient image; The second initial feature and the attention noise feature are subjected to residual iterative processing based on the complementary attention mechanism through the residual network to obtain the attention gradient iterative feature; The attention gradient features are obtained by adding the attention gradient iterative features and the second initial features through the residual network.

8. The method according to claim 7, characterized in that, The residual network comprises T iteration units, where T is an integer greater than or equal to 2. The residual network is used to perform residual iterative processing on the second initial feature and the attention noise feature based on the complementary attention mechanism to obtain attention gradient iterative features, including: Through the t-th iteration unit among the T iteration units, complementary attention processing is performed on the input of the t-th iteration unit, and the first convolution processing is performed on the obtained complementary attention result to obtain the t-th iteration result; The t-th iteration result is added to any intermediate noise feature, and the result is transmitted to the (t+1)-th iteration unit to continue complementary attention processing. The first convolution processing is then performed on the complementary attention result to obtain the (t+1)-th iteration result. Where t is an integer variable whose value starts from 1 and increments, and the range of t is... When t is 1, the input of the t-th iteration unit is the sum of the second initial feature and the first initial feature, where the first initial feature is obtained by performing the first convolution process on the noisy image; when t is 2... When t is -1, the intermediate noise feature is obtained by downsampling the first initial feature at least once. When any intermediate noise feature is the attention noise feature, the (t+1)th iteration result is the attention gradient iteration feature.

9. The method according to claim 1, characterized in that, The first merged feature obtained based on the attention gradient feature and the attention noise feature includes: The attention gradient features are subjected to a first convolution process to obtain the noise gradient features of the noise gradient image; The attention noise features and the noise gradient features are combined to obtain the first combined feature; The process of obtaining the denoised feature image based on the complementary attention result includes: The complementary attention result is subjected to the first convolutional processing to obtain the first convolutional feature; The first convolutional feature and the first initial feature are added together, and the result of the addition is subjected to a third convolution to obtain the noise reduction feature image. The first initial feature is obtained by performing the first convolution on the noise image.

10. The method according to claim 1, characterized in that, The step of performing spatial attention processing on the first channel features to obtain spatial attention features includes: The first channel feature is subjected to a first convolution process to obtain a second convolution feature. The second convolution feature is subjected to a first activation process to obtain a first activation result. The first activation result is subjected to the first convolution process to obtain a third convolution feature. Max pooling is performed on the third convolutional feature to obtain the first pooling result, and average pooling is performed on the third convolutional feature to obtain the second pooling result. The first pooling result and the second pooling result are subjected to feature merging processing to obtain the second merged feature; The second merged feature is subjected to the first convolution process to obtain the fourth convolution feature. The fourth convolution feature is subjected to the second activation process to obtain the second activation result. The second activation result is multiplied with the third convolution feature to obtain the spatial attention feature.

11. The method according to claim 1, characterized in that, The process of performing channel attention processing on the first channel features to obtain channel attention features includes: The first channel feature is subjected to a first convolution process to obtain a second convolution feature. The second convolution feature is subjected to a first activation process to obtain a first activation result. The first activation result is subjected to the first convolution process to obtain a third convolution feature. The third convolutional feature is subjected to average pooling to obtain the second pooling result; The second pooling result is subjected to the first convolution process to obtain the fifth convolution feature. The fifth convolution feature is subjected to the first activation process to obtain the third activation result. The third activation result is subjected to the second convolution process to obtain the sixth convolution feature. The sixth convolutional feature is subjected to a second activation process to obtain a fourth activation result. The fourth activation result is then multiplied with the third convolutional feature to obtain the channel attention feature.

12. The method according to claim 1, characterized in that, The second feature mapping process, which involves processing the spatial attention features and the channel attention features to obtain dense features corresponding to the first merged features, includes: The spatial attention features and the channel attention features are combined to obtain a third combined feature; The third merged feature is subjected to a first convolution process to obtain the seventh convolution feature; The seventh convolutional feature is added to the first channel feature to obtain the dense feature.

13. The method according to claim 1, characterized in that, The step of performing a third feature mapping process on the second channel features to obtain sparse features corresponding to the first merged features includes: The second channel feature is subjected to a second convolution process to obtain an eighth convolution feature. The eighth convolution feature is subjected to batch normalization to obtain a normalization result. The normalization result is subjected to a first activation process to obtain a ninth convolution feature. The ninth convolutional feature is subjected to a cheap linear mapping process, and the cheap linear mapping result corresponding to the ninth convolutional feature is merged with the ninth convolutional feature to obtain the sparse feature.

14. The method according to claim 1, characterized in that, The step of performing a fourth feature mapping process on the dense features and the sparse features based on the first merged feature to obtain complementary attention results includes: The dense features and the sparse features are combined to obtain a fourth merged feature. The fourth merged feature is added to the first merged feature to obtain the complementary attention result.

15. The method according to claim 1, characterized in that, The image processing method is implemented by calling the prediction network of a lightweight denoising model, which further includes a gradient map generation network. The method also includes: Acquire a noisy image sample, a label-denoised image corresponding to the noisy image sample, a gradient image corresponding to the label-denoised image, and a noise gradient image sample corresponding to the noisy image sample; The noisy image samples and the noise gradient image samples are forward propagated in the prediction network of the lightweight denoising model to obtain the predicted denoised image corresponding to the noisy image sample and the sample attention gradient feature corresponding to the noise gradient image sample. The gradient graph generation network of the lightweight noise reduction model performs a second convolution on the attention gradient features of the sample to obtain the predicted gradient image corresponding to the noisy image sample. Pixel loss is determined based on the error between the labeled denoised image and the predicted denoised image; Obtain the gradient image corresponding to the predicted denoised image, and determine the pixel gradient loss based on the error between the gradient image corresponding to the predicted denoised image and the gradient image corresponding to the label denoised image; The gradient loss is determined based on the error between the gradient image of the corresponding denoised image and the predicted gradient image. The pixel loss, pixel gradient loss, and gradient loss are fused to obtain a comprehensive loss, and the parameters of the lightweight noise reduction model are updated based on the comprehensive loss.

16. An image processing device based on artificial intelligence, characterized in that, The device includes: An acquisition module is used to acquire a noisy image and acquire a noise gradient image corresponding to the noisy image, wherein the noise gradient image includes the image gradient of each pixel in the noisy image; The sampling module is used to perform pixel feature sampling processing on the noisy image based on the complementary attention mechanism to obtain attention noise features; The residual module is used to perform gradient residual processing on the noise gradient image based on the complementary attention mechanism, based on the attention noise features, to obtain attention gradient features; A mapping module is used to obtain a first merged feature based on the attention gradient features and the attention noise features; perform channel-based partitioning on the first merged feature to obtain a first channel feature and a second channel feature; perform spatial attention processing on the first channel feature to obtain spatial attention features, and perform channel attention processing on the first channel feature to obtain channel attention features; perform a second feature mapping processing on the spatial attention features and the channel attention features to obtain dense features corresponding to the first merged feature; perform a third feature mapping processing on the second channel feature to obtain sparse features corresponding to the first merged feature; perform a fourth feature mapping processing on the dense features and the sparse features based on the first merged feature to obtain a complementary attention result; and obtain a denoised feature image based on the complementary attention result. The overlay module is used to perform pixel overlay processing on the denoised feature image and the noise image to obtain a denoised image corresponding to the noise image.

17. The apparatus according to claim 16, characterized in that, The pixel feature sampling process is performed through a first feature network, which includes a first convolutional network and a symmetric network; the sampling module is further configured to: The first convolutional network is used to perform a first convolutional process on the noisy image to obtain a first initial feature corresponding to the noisy image. The first initial feature is sampled using the symmetric network based on the complementary attention mechanism to obtain the attention noise feature.

18. The apparatus according to claim 17, characterized in that, The symmetric network includes N downsampling networks, a connection network, and M upsampling networks. The connection network is cascaded with the N downsampling networks and the M upsampling networks, where N is an integer greater than or equal to 1, and M is equal to N. The sampling module is further used for: The first initial feature is downsampled using the N downsampling networks based on the complementary attention mechanism to obtain the downsampling results corresponding to the N downsampling networks; The downsampling result is processed by the connection network to obtain the connection features corresponding to the noisy image; The attention noise features are obtained by upsampling the connection features using the M upsampling networks based on the complementary attention mechanism.

19. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions for a computer; A processor, when executing computer-executable instructions stored in the memory, implements the artificial intelligence-based image processing method according to any one of claims 1 to 15.

20. A computer-readable storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed by the processor, they implement the artificial intelligence-based image processing method according to any one of claims 1 to 15.

21. A computer program product, comprising a computer program or computer-executable instructions, characterized in that, When the computer program or computer-executable instructions are executed by a processor, they implement the artificial intelligence-based image processing method according to any one of claims 1 to 15.

Images