Image processing method and related device

Abstract

Disclosed are an image processing method and a related device, which are used for improving the quality of an image that has been processed by means of a quantized image processing model. The method comprises: acquiring a first image; and on the basis of the first image, obtaining a second image by means of a quantized image processing model, wherein the quantized image processing model is a quantized model for a full-precision image processing model, the full-precision image processing model comprises a target network layer and a data distribution optimization layer and / or a data distribution compression layer, the data distribution optimization layer and / or the data distribution compression layer are located at an input side of the target network layer, the data distribution optimization layer is used for optimizing the data distribution of activation values that are input into the target network layer, the data distribution compression layer is used for compressing the data distribution of the activation values that are input into the target network layer, the quantized model comprises a quantized network layer corresponding to the target network layer, and the quantized network layer is used for processing the first image or a feature value corresponding to the first image.

Description

Image processing method and related equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on September 19, 2023, with application number 202311214537.3 and application name “Image Processing Methods and Related Equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of image processing technology, and in particular to an image processing method and related equipment. Background Art

[0003] With the continuous development of artificial intelligence, neural networks have demonstrated powerful computing capabilities in multiple fields, including image, video, and voice. However, since neural network algorithms themselves are computationally intensive and require a large number of parameters, deploying neural network models on edge devices poses significant challenges.

[0004] To enable deployment of neural network models on end devices, model quantization can be used to reduce the model size. Specifically, model quantization converts the original parameters of a neural network model from floating-point storage to integer storage to obtain a quantized model. Because each original parameter in floating-point storage occupies 32 bits (binary digits, bits) of storage space, while integer storage can reduce the number of bits corresponding to each parameter, model quantization significantly reduces the model's storage space, making the resulting quantized model easier to deploy. Using a quantized model for inference can accelerate the inference process.

[0005] In the underlying visual tasks, the activation value distribution of the full-precision image processing model is complex, showing high dynamic range, irregularity, and asymmetric characteristics, which causes the performance of the quantized model obtained after quantization of the full-precision image processing model to degrade seriously, resulting in poor image quality obtained by the quantized model.

[0006] Summary of the Invention

[0007] The present application provides an image processing method and related equipment to solve the problem that the quantization model of the current full-precision image processing model has poor performance in processing underlying visual tasks.

[0008] The first aspect provides an image processing method. The method can be applied to a terminal device or a server. The method includes: obtaining a first image. The first image is an image to be processed. Then, based on the first image, a second image is obtained by a quantized image processing model. The quantized image processing model is a quantized model of a full-precision image processing model. The full-precision image processing model includes a target network layer, and a data distribution optimization layer and / or a data distribution compression layer. The data distribution optimization layer and / or the data distribution compression layer are located on the input side of the target network layer. The data distribution optimization layer is used to optimize the data distribution of the activation value input to the target network layer. The data distribution compression layer is used to compress the data distribution of the activation value input to the target network layer. The quantized model includes a quantized network layer corresponding to the target network layer. The quantized network layer is used to process the first image or the eigenvalue corresponding to the first image.

[0009] Both the full-precision image processing model and the quantized image processing model are used to process low-level visual tasks, such as noise reduction, super-resolution, and deblurring.

[0010] In order to ensure the performance of the underlying visual processing model, the underlying visual processing model does not include a batch normalization layer. In this application, a data distribution optimization layer and / or a data distribution compression layer are provided in the full-precision image processing model. The data distribution optimization layer scales and translates the data distribution of the activation values ​​to make the data distribution of the activation values ​​of the input target network layer more regular. The data distribution compression layer truncates the data distribution of the activation values ​​to make the data distribution of the activation values ​​of the input target network layer more compact. Both can simplify the data distribution, thereby reducing the quantization complexity, reducing the performance loss of the quantized image processing model corresponding to the full-precision image processing model, and improving the performance of the quantized image processing model. As a result, the second image obtained by performing underlying visual processing on the first image based on the quantized image processing model has higher image quality.

[0011] In one possible implementation, the full-precision image processing model includes a data distribution optimization layer and a data distribution compression layer. The data distribution compression layer is located between the data distribution optimization layer and the target network layer. The data distribution compression layer is used to compress the data distribution of activation values ​​output by the data distribution optimization layer before outputting it to the target network layer. Optimizing the data distribution of activation values ​​followed by compression can further simplify the data distribution, making the data distribution of activation values ​​input to the target network layer more compact and regular. This can reduce the learning difficulty of the quantization algorithm and increase the upper limit of the quantization algorithm's capabilities, significantly improving the performance of the final quantized image processing model and, in turn, improving the quality of the second image processed by the quantized image processing model.

[0012] In one possible implementation, the full-precision image processing model includes a data distribution optimization layer and a data distribution compression layer. The data distribution optimization layer is located between the data distribution compression layer and the target network layer. The data distribution layer is used to optimize the data distribution of activation values ​​output by the data distribution compression layer before outputting them to the target network layer. Optimizing the data distribution after compressing it also simplifies the data distribution of activation values, making the data distribution of activation values ​​input to the target network layer more compact and regular. This reduces the learning difficulty of the quantization algorithm and increases the upper limit of the quantization algorithm's capabilities, significantly improving the performance of the final quantized image processing model and, in turn, improving the quality of the second image processed by the quantized image processing model.

[0013] In one possible implementation, the full-precision image processing model includes a first data distribution optimization layer, a second data distribution optimization layer, a first target network layer, a second target network layer and an activation layer. The input side of the first target network layer does not include the activation layer, and the input side of the second target network layer includes the activation layer. The activation layer is used to perform nonlinear mapping on the output of the previous network layer and then input it into the second target network layer. The first data distribution optimization layer is located on the input side of the first target network layer, and the second data distribution optimization layer is located on the input side of the activation layer or on the input side of the second target network layer.

[0014] In one possible implementation, the full-precision image processing model includes a first data distribution optimization layer, a second data distribution optimization layer, a data distribution compression layer, a first target network layer, a second target network layer and an activation layer. The input side of the first target network layer does not include the activation layer, and the input side of the second target network layer includes the activation layer. The activation layer is used to perform nonlinear mapping on the output of the previous network layer and then input it into the second target network layer. The first data distribution optimization layer and the data distribution compression layer are located on the input side of the first target network layer, and the second data distribution optimization layer is located on the input side of the activation layer.

[0015] In one possible implementation, the full-precision image processing model includes a data distribution compression layer, a first target network layer, a second target network layer and an activation layer. The input side of the first target network layer does not include the activation layer, and the input side of the second target network layer includes the activation layer. The activation layer is used to perform nonlinear mapping on the output of the previous network layer and then input it into the second target network layer. The data distribution compression layer is located on the input side of the first target network layer.

[0016] In one possible implementation, the target network layer is a convolutional layer, deconvolutional layer, attention layer, or linear layer. These layers include matrix multiplication operations, which require significant computing power at full precision. Quantizing the target network layer can accelerate matrix multiplication operations, thereby accelerating the inference speed of the quantized image model. It can also reduce the model size, making the quantized model easier to deploy.

[0017] In one possible implementation, the data distribution compression layer is a ReLU layer, a LeakyReLU layer, or a HardSwish layer.

[0018] In one possible implementation, the data distribution optimization layer is a batch normalization layer, an instance normalization layer, or a layer normalization layer.

[0019] A second aspect provides an image processing device, which includes a processing module configured to implement the image processing method in the first aspect or any possible implementation of the first aspect.

[0020] A third aspect provides a device, comprising a processor and a memory, wherein the processor is coupled to the memory and configured to execute the image processing method of the first aspect or any possible implementation of the first aspect based on instructions stored in the memory.

[0021] A fourth aspect provides a computer-readable storage medium comprising instructions, which, when the computer-readable storage medium is run on a computer, enables the computer to execute the image processing method in the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] FIG1 is a schematic diagram of a system architecture provided by this application;

[0023] FIG2 a is a schematic diagram of an optional hardware structure of the terminal 100 in FIG1 ;

[0024] FIG2 b is a schematic diagram of an optional hardware structure of the server 200 in FIG1 ;

[0025] FIG3 is a schematic diagram of another system framework provided by this application;

[0026] FIG4 is a flow chart of an image processing method provided by the present application;

[0027] Figure 5 shows the statistical information of the activation distribution of different layers in the underlying visual processing model after removing the BN layer;

[0028] FIG6 a is a schematic diagram of a partial structure of an underlying visual processing model provided by this application;

[0029] FIG6 b is a schematic diagram of the partial structure of another underlying visual processing model provided by the present application;

[0030] FIG6 c is a schematic diagram of the partial structure of another underlying visual processing model provided by this application;

[0031] FIG6 d is a schematic diagram of the partial structure of another underlying visual processing model provided by the present application;

[0032] FIG6e is a schematic diagram of the partial structure of another underlying visual processing model provided by this application;

[0033] FIG6f is a schematic diagram of the partial structure of another underlying visual processing model provided by the present application;

[0034] FIG6g is a schematic diagram of the partial structure of another underlying visual processing model provided by this application;

[0035] FIG7 shows the statistical information of data distribution after the data distribution optimization layer and data distribution compression layer proposed in this application are added to the underlying visual processing model;

[0036] FIG8 shows the statistical information of the average quantization error of different network layers after the data distribution optimization layer and data distribution compression layer proposed in this application are added to the underlying visual processing model;

[0037] FIG9 is a schematic structural diagram of an image processing device provided in this application. DETAILED DESCRIPTION

[0038] The present application provides an image processing method and related equipment to improve the image quality of images processed by a quantization model in low visual task scenarios.

[0039] The technical solutions in this application will be described in detail below in conjunction with the drawings in this application. Obviously, the described embodiments are only part of the embodiments of this application, rather than all the embodiments.

[0040] The terms "first," "second," "third," "fourth," and the like (if any) in the specification and claims of this application and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequential sequence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions, e.g., a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0041] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.

[0042] (1) Neural Network

[0043] A neural network can be composed of neural units. A neural unit can refer to an operation unit that takes xs (i.e., input data) and intercept 1 as input. The output of the operation unit can be:

[0044] Where s = 1, 2, ... n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal of the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid, Relu or tanh function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0045] (2) Deep Neural Networks

[0046] Deep Neural Network (DNN), also known as multi-layer neural network, can be understood as a neural network with many hidden layers. There is no special metric for "many" here. Based on the position of different layers in DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since there are many DNN layers, the coefficient W and the offset vector The definition of these parameters in DNN is as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscript corresponds to the output of the third layer index 2 and the input of the second layer index 4. In summary, the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as It's important to note that the input layer has no W parameter. In deep neural networks, more hidden layers allow the network to better capture complex real-world situations. Theoretically, a model with more parameters has higher complexity and greater "capacity," meaning it can handle more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrices for all layers of a trained deep neural network (a weight matrix formed by the vectors W across many layers).

[0047] (3) Neural Network Quantization

[0048] Quantization refers to the process of transforming data so that the converted data is smaller than the storage space occupied by the original data. For example, the process of converting a floating-point number that occupies 32 bits of storage space into an 8-bit, 4-bit, or 2-bit integer, or converting an integer that occupies 32 bits of storage space into a 16-bit integer, can be called a quantization process. Neural network quantization is a model compression technology that converts floating-point storage (operations) into integer storage (operations). For example, the weights and activation values ​​of a model are originally represented by float32 (32-bit floating point). After quantization, the weights and activation values ​​of the model are represented by int8 (8-bit fixed point). Through the quantization operation of the model, the size of the model is reduced and the model inference speed is improved at the cost of a small loss of accuracy. In this way, the quantized model can be deployed on the end-side device according to the quantization parameters corresponding to each operation parameter, meeting the end-side device's deployment requirements for the neural network and facilitating the subsequent use of the quantized model.

[0049] Neural network quantization includes two quantization methods: Quantization-Aware Training (QAT) and Post-training quantization (PTQ). QAT can automatically adjust network parameters during training to adapt to the quantized model. The main idea of ​​QAT is to incorporate quantization error into the training process to maximize the performance of the quantized model while minimizing the loss function. PTQ is a method that directly converts a pre-trained FP32 (32-bit floating point, also known as full precision) network into a fixed-point network.

[0050] (4) Batch Normalization (BN)

[0051] Batch normalization is a neural network training technique that automatically adjusts network parameters to accommodate changes in batch data during training, thereby improving the model's convergence speed and generalization capabilities. The basic principle of batch normalization is to normalize the input of each neuron so that the mean of the input data is 0 and the variance is 1. This prevents problems such as vanishing or exploding gradients and improves model training efficiency and performance.

[0052] (5) Low-level visual tasks

[0053] Low-level visual tasks include image and video denoising, deblurring, defogging, deraining, colorization, enhancement, restoration, compression, and super-resolution (SR). These tasks typically process low-level visual features such as color, brightness, orientation, and texture. Low-level visual tasks maximize image quality without changing the image's semantic information.

[0054] In underlying visual image processing models, such as super-resolution models and dehazing, rain removal, and denoising models, batch normalization (BN) layers are removed by default. This is because BN layers can cause network activations to lose range flexibility and reduce image scale information, leading to performance degradation in the underlying visual image processing models. BN accelerates and stabilizes deep neural network training by normalizing and scaling the layer inputs.

[0055] Removing the BN layer can improve the performance of the underlying visual image processing model and save computational effort. The quantization model of the underlying visual image processing model also follows this setting, that is, there is no normalization layer in the visual image processing quantization model. In the process of quantizing the full-precision underlying visual image processing model, due to the absence of the BN layer, the distribution of activation values ​​becomes extremely complex, showing high dynamic range, irregularity, and asymmetric characteristics, making quantization complex and difficult, the quantization error is large, and the quantization model is difficult to converge. When quantizing at extremely low bits (2 bits, 4 bits, etc.), the performance degradation of the quantization model is more serious, reducing the quality of image processing.

[0056] Therefore, in order to solve the problem that the poor performance of the quantization model of the current underlying visual image processing model affects the image quality, the present application provides the following embodiments.

[0057] Referring to Figure 1 , a schematic diagram of a system architecture is shown. The system may include a terminal 100 and a server 200. Server 200 may include one or more servers (Figure 1 illustrates one server as an example). Server 200 may provide one or more terminals with a full-precision quantization model corresponding to the underlying visual processing model.

[0058] Specifically, the server 200 can train a full-precision image processing model for implementing the underlying visual processing task. The input side of the target network layer in the full-precision image processing model is provided with a data distribution optimization module and / or a data distribution compression module. The trained full-precision image processing model is then quantized to obtain a corresponding quantized image processing model. The server 200 then sends the quantized image processing model to the terminal 100 so that the terminal 100 deploys the quantized image processing model.

[0059] Among them, a quantized image processing model for performing underlying visual processing on images is deployed on the terminal 100, and the terminal 100 uses the quantized model to process images.

[0060] It should be understood that in some optional implementations, the server 200 may also use a quantization model to process images, which is not limited in the embodiments of the present application.

[0061] Next, the product form of the terminal 100 in FIG1 is described;

[0062] The terminal 100 in the embodiment of the present application can be a mobile phone, a tablet computer, a wearable device, an in-vehicle device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any restrictions on this.

[0063] FIG. 2 a shows a schematic diagram of an optional hardware structure of the terminal 100 .

[0064] 2a , the terminal 100 may include components such as a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), a processor 170, an external interface 180, and a power supply 190. Those skilled in the art will appreciate that FIG2a is merely an example of a terminal or multi-function device and does not limit the terminal or multi-function device. The terminal or multi-function device may include more or fewer components than shown, or may combine certain components or have different components.

[0065] The input unit 130 can be used to receive input digital or character information and generate key signal input related to user settings and function control of the portable multifunction device. Specifically, the input unit 130 may include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can detect user touch operations on or near it (for example, operations performed on or near the touch screen using a finger, joint, stylus, or any other suitable object) and drive corresponding connected devices according to pre-set programs. The touch screen can detect user touch actions on the touch screen, convert the touch actions into touch signals and transmit them to the processor 170. It can also receive and execute commands sent by the processor 170; the touch signals include at least touch point coordinate information. The touch screen 131 provides an input interface and an output interface between the terminal 100 and the user. Touch screens can be implemented using various types, including resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch screen 131, the input unit 130 may also include other input devices. Specifically, the other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0066] The display unit 140 may be used to display information input by the user or provided to the user, various menus of the terminal 100, interactive interfaces, file display, and / or playback of any multimedia file. In the embodiment of the present application, the display unit 140 may be used to display an interface for controlling image processing, image processing results, etc.

[0067] Memory 120 can be used to store instructions and data. It primarily includes an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files like images and videos, as well as text. The instruction storage area can store software units such as the operating system, applications, and instructions required for at least one function, or subsets or extensions thereof. It may also include non-volatile random access memory (RAM). It provides processor 170 with management functions for the hardware, software, and data resources within the computing and processing device, supporting control software and applications. It is also used to store multimedia files and running programs and applications.

[0068] The processor 170 is the control center of the terminal 100. It connects all components of the terminal 100 using various interfaces and circuits. By executing instructions stored in the memory 120 and accessing data stored therein, it executes various functions of the terminal 100 and processes data, thereby providing overall control of the terminal device. Optionally, the processor 170 may include one or more processing units. Preferably, the processor 170 may integrate an application processor and a modem processor, with the application processor primarily processing the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into the processor 170. In some embodiments, the processor and memory may be implemented on a single chip; in other embodiments, they may be implemented on separate chips. The processor 170 may also generate corresponding operational control signals and send them to the corresponding components of the computing and processing device. It may also read and process data in the software, particularly the data and programs in the memory 120, to enable the various functional modules therein to perform their corresponding functions, thereby controlling the corresponding components to operate as instructed.

[0069] Among them, the memory 120 can be used to store software codes and quantization models related to the image method, the processor 170 can execute the steps of the chip's image processing method, and can also schedule other units to implement corresponding functions, such as scheduling the camera 150 to capture images, or reading images stored in the memory 120.

[0070] The RF unit 110 (optional) can be used to send and receive information or receive and send signals during a call. For example, after receiving downlink information from the base station, it is passed to the processor 170 for processing; in addition, it sends the designed uplink data to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF unit 110 can also communicate with network devices and other devices via wireless communication. This wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0071] In this embodiment of the present application, the radio frequency unit 110 can receive the quantization model sent to the server 200 and deploy it in the terminal 100.

[0072] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

[0073] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, thereby managing functions such as charging, discharging, and power consumption through the power management system.

[0074] The terminal 100 further includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal 100 to other devices for communication, or to connect a charger to charge the terminal 100 .

[0075] Although not shown, the terminal 100 may also include a flash, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which are not described in detail here. Some or all of the methods described below may be applied to the terminal 100 shown in FIG2a .

[0076] Next, the product form of the server 200 in FIG1 is described;

[0077] FIG2b provides a schematic structural diagram of a server 200. As shown in FIG2b, the server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other via the bus 201.

[0078] Bus 201 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. Buses can be categorized as address buses, data buses, and control buses. For ease of illustration, FIG2 b uses only one thick line, but this does not imply that there is only one bus or only one type of bus.

[0079] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0080] The memory 204 may include volatile memory, such as random access memory (RAM). The memory 204 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard drive (HDD), or solid state drive (SSD).

[0081] The memory 204 may be used for software codes related to the image processing method, and the processor 202 may execute the steps of the image processing method of the chip, and may also schedule other units to implement corresponding functions.

[0082] It should be understood that the above-mentioned terminal 100 and server 200 can be centralized or distributed devices, and the processors in the above-mentioned terminal 100 and server 200 (such as processor 170 and processor 202) can be hardware circuits (such as application specific integrated circuit (ASIC), field-programmable gate array (FPGA), general-purpose processor, digital signal processor (DSP), microprocessor or microcontroller, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0083] It should be understood that the image processing methods in the embodiments of this application involve AI-related operations. When performing AI operations, the instruction execution architecture of the terminal device and server is not limited to the processor-memory architecture shown in Figures 2a and 2b. The system architecture provided in the embodiments of this application is described in detail below with reference to Figure 3.

[0084] FIG3 is a schematic diagram of the system architecture provided by an embodiment of the present application. As shown in FIG3 , the system architecture 500 includes an execution device 510 , a training device 520 , a database 530 , a client device 540 , a data storage system 550 , and a data acquisition system 560 .

[0085] The execution device 510 includes a calculation module 511, an I / O interface 512, a pre-processing module 513, and a post-processing module 514. The calculation module 511 may include the target model / rule 501, and the pre-processing module 513 and the post-processing module 514 are optional.

[0086] The execution device 510 may be a terminal device or a server that runs the above-mentioned image processing method.

[0087] The data acquisition device 560 is used to collect training samples. The training samples can be images, etc. After collecting the training samples, the data acquisition device 560 stores these training samples in the database 530.

[0088] The training device 520 can train the neural network to be trained (e.g., the full-precision image processing model before training in the embodiment of the present application) based on the training samples maintained in the database 530. The training device 520 can also quantize the trained full-precision image processing model to obtain the target model / rule 501 (quantized image processing model).

[0089] It should be noted that, in actual applications, the training samples maintained in the database 530 may not all be collected by the data acquisition device 560, but may also be received from other devices. It should also be noted that the training device 520 may not train the target model / rule 501 entirely based on the training samples maintained in the database 530, but may also obtain training samples from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.

[0090] The target model / rule 501 obtained through training with the training device 520 can be applied to different systems or devices, such as the execution device 510 shown in FIG3 . The execution device 510 can be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) / virtual reality (VR) device, an in-vehicle terminal, etc., or a server, etc.

[0091] Specifically, the training device 520 may transmit the quantization model to the execution device 510. The execution device 510 processes the first image to be processed using the quantization image processing model to obtain a second image.

[0092] In FIG3 , the execution device 510 is configured with an input / output (I / O) interface 512 for data interaction with an external device. A user can input data (eg, a first image, etc.) to the I / O interface 512 through a client device 540 .

[0093] Preprocessing module 513 and preprocessing module 514 are used to preprocess the input data received by I / O interface 512. It should be understood that preprocessing module 513 and preprocessing module 514 may be absent or only one preprocessing module may be present. If preprocessing module 513 and preprocessing module 514 are absent, computing module 511 may be used directly to process the input data.

[0094] When the execution device 510 preprocesses the input data, or when the computing module 511 of the execution device 510 performs calculations and other related processing, the execution device 510 can call the data, code, etc. in the data storage system 550 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing in the data storage system 550.

[0095] Finally, the I / O interface 512 provides the processing result (eg, the second image in the embodiment of the present application) to the client device 540 , and thus to the user.

[0096] In the scenario shown in FIG3 , the user can manually input data, and this "manual input data" can be operated through the interface provided by I / O interface 512. In another scenario, client device 540 can automatically send input data to I / O interface 512. If user authorization is required for client device 540 to automatically send input data, the user can set the corresponding permissions in client device 540. The user can view the results output by execution device 510 on client device 540, and the specific presentation form can be a display, sound, action, or other specific method. Client device 540 can also serve as a data acquisition terminal, collecting input data input into I / O interface 512 and output results output from I / O interface 512 as new sample data, and storing them in database 530. Of course, collection can also be performed without client device 540, and instead the I / O interface 512 directly stores the input data input into I / O interface 512 and output results output from I / O interface 512 as new sample data in database 530.

[0097] It is worth noting that FIG3 is merely a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationships between the devices, components, modules, etc. shown in the figure do not constitute any limitation. For example, in FIG3 , the data storage system 550 is an external memory relative to the execution device 510. In other cases, the data storage system 550 can also be placed in the execution device 510. It should be understood that the execution device 510 can be deployed in the client device 540.

[0098] From the inference side of the model:

[0099] In the embodiment of the present application, the calculation module 511 of the above-mentioned execution device 510 can obtain the code stored in the data storage system 550 to implement the image processing method in the embodiment of the present application.

[0100] In an embodiment of the present application, the computing module 511 of the execution device 510 may include a hardware circuit (such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, etc.), or a combination of these hardware circuits. For example, the training device 520 may be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0101] Specifically, the computing module 511 of the execution device 510 can be a hardware system with the function of executing instructions, and the image processing method provided in the embodiment of the present application can be a software code stored in the memory. The computing module 511 of the execution device 510 can obtain the software code from the memory and execute the obtained software code to implement the image processing method provided in the embodiment of the present application.

[0102] It should be understood that the computing module 511 of the execution device 510 can be a combination of a hardware system that does not have the function of executing instructions and a hardware system that has the function of executing instructions. Some steps of the image processing method provided in the embodiment of the present application can also be implemented by the hardware system that does not have the function of executing instructions in the computing module 511 of the execution device 510, which is not limited here.

[0103] From the training side of the model:

[0104] In an embodiment of the present application, the above-mentioned training device 520 can obtain the code stored in the memory (not shown in Figure 3, which can be integrated into the training device 520 or deployed separately from the training device 520) to implement the steps related to model training and quantization in the embodiment of the present application.

[0105] In an embodiment of the present application, the training device 520 may include a hardware circuit (such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, etc.), or a combination of these hardware circuits. For example, the training device 520 may be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0106] It should be understood that the training device 520 can be a combination of a hardware system that does not have the function of executing instructions and a hardware system that has the function of executing instructions. Some of the steps related to model training provided in the embodiments of the present application can also be implemented by the hardware system in the training device 520 that does not have the function of executing instructions, which is not limited here.

[0107] As shown in FIG4 , FIG4 is a flow chart of an image processing method provided by the present application. This embodiment can be executed by the above-mentioned terminal / execution device or by the above-mentioned server, which are collectively referred to as devices in this embodiment. This embodiment includes the following steps:

[0108] S401: Acquire a first image.

[0109] The first image is the image to be processed. The first image can be an image input by the user, an image read from a memory of the device, or an image captured in real time by the device's camera. It should be noted that the image can also refer to a frame in a video.

[0110] S402: Based on the first image, a second image is obtained through a quantized image processing model, where the quantized image processing model is a quantized model of a full-precision image processing model, and the full-precision image processing model includes a target network layer, and a data distribution optimization layer and / or a data distribution compression layer. The data distribution optimization layer and / or the data distribution compression layer are located on the input side of the target network layer, and the quantized model includes a quantized network layer corresponding to the target network layer.

[0111] In this embodiment, the quantized image processing model is a model for processing underlying visual tasks, including at least one of denoising, deblurring, defogging, deraining, coloring, enhancement, restoration, compression, and SR.

[0112] A quantized image processing model is a quantized version of a full-precision image processing model. A full-precision image processing model is a low-level visual processing model used to process low-level visual tasks, such as denoising models, deblurring models, defogging models, rain removal models, coloring models, enhancement models, restoration models, compression models, and SR models, or an image processing model capable of performing multiple of these low-level tasks.

[0113] Low-level vision is the computational foundation for mid- and high-level vision. Furthermore, the low-level visual processing model processes low-level image information, such as pixel-level features. This requires the low-level visual processing model to have high performance to ensure it outputs high-quality images after processing. To ensure the performance of the low-level visual processing model, it is widely accepted in the industry to remove the batch normalization layer from the low-level visual processing model. This is because the batch normalization layer causes activations in the network to lose range flexibility, reduces the scale information of the image, and reduces edge differentiation in the image. In other words, the batch normalization layer causes the performance of the low-level visual processing model to degrade.

[0114] As shown in Figure 5, due to the absence of the BN layer, the activation distribution of the underlying visual processing model becomes extremely complex, showing high dynamic range, irregular, and asymmetric characteristics (see the first row in Figure 5 (activation distribution of different layers)), and the maximum and minimum activation values ​​of different input samples are very different (see the second row in Figure 5 (distribution of maximum and minimum values ​​of all training samples)). These characteristics are very unfavorable for model quantization, greatly increasing the difficulty of optimizing the quantization algorithm. When quantizing at extremely low bits (such as 2 bits, 3 bits, or 4 bits), the performance degradation of the quantized underlying visual processing model is even more serious. Regarding how to improve the performance of the quantized underlying visual processing model, the current mainstream direction is to improve the quantization algorithm to cope with the highly complex activation distribution and achieve complexity. This approach can improve the performance of the quantized model to a certain extent, but for low-bit quantization models, the performance loss caused by quantization is still very large.

[0115] Different from the current consensus of removing the BN layer in the underlying visual processing model, the full-precision image processing model (full-precision underlying visual processing model) in this embodiment includes a data distribution optimization layer and / or a data distribution compression layer on the input side of the target network layer. Among them, the target network layer is a network layer that needs to be quantized, and the target network layer contains matrix multiplication operations. The target network layer can be a network layer for extracting features, such as a convolution layer, a deconvolution layer, and an attention layer. The target network layer can also be a layer for classification, such as a linear layer. These network layers include matrix multiplication operations, and the matrix multiplication operations of full precision (FP32) consume more computing power, and the corresponding matrix will take up a larger storage space, so these network layers are generally network layers that need to be quantized. The full-precision image processing model may include one target network layer or multiple target network layers, which is not limited here. In this application, "multiple" refers to two or more.

[0116] In this embodiment, the data distribution optimization layer is used to optimize the data distribution of the activation values ​​of the input value target network layer. The data distribution optimization layer can simplify the data distribution, reduce the learning difficulty of the quantization algorithm, and make the quantization model easier to optimize. The data distribution optimization layer is, for example, a batch normalization layer, an instance normalization (IN) layer, or a layer normalization (LN) layer. Of course, the data distribution optimization layer can also be customized to achieve data distribution optimization modules.

[0117] The data distribution compression layer compresses the data distribution of the activation values ​​of the input target network layer. This layer truncates the data distribution range, thereby compressing the data distribution, reducing the granularity of the quantization representation, and improving the upper limit of the quantization algorithm. Specifically, the data distribution compression module can be an activation function layer such as the ReLU, LeakyReLU, or HardSwish layer. Of course, other modules that truncate the data distribution range and compress the data distribution are also possible.

[0118] In one possible implementation, a full-precision image processing model can be constructed and trained to process underlying visual tasks. When constructing the full-precision image processing model, a data distribution optimization layer and / or a data distribution compression layer are added to the input side of each target network layer. The input side of the target network layer can include only the data distribution optimization layer or only the data distribution compression layer. The data distribution optimization layer can make the activation values ​​input to the target network layer more regular, while the data distribution compression layer can reduce the data distribution range of the activation values ​​input to the target network layer. Both layers can reduce the complexity of the activation value distribution of the input target network layer, thereby reducing the performance loss of the quantized image processing model when quantizing the target network layer. Of course, the input side of the target network layer can include both the data distribution optimization layer and the data distribution compression layer. The data distribution optimization layer can be located on the input side of the data distribution compression layer. That is, the data distribution optimization layer outputs the optimized activation values ​​to the data distribution compression layer, which then truncates the data distribution of the activation values ​​before outputting them to the target network layer. This preserves the data distribution as much as possible and ensures the performance of the quantized image processing model. Alternatively, the data distribution compression layer is located on the input side of the data distribution optimization layer, that is, the data distribution compression layer compresses the data distribution of the activation value and outputs it to the data distribution optimization layer. The data distribution optimization layer then optimizes the data distribution of the activation value input to the data distribution compression layer, and then outputs it to the target network layer. It can also compress and optimize the data distribution, making the distribution of the activation value input to the target network layer more regular and compact, and reducing the difficulty of quantization of the target network layer. It can be understood that being located on the input side of the target network layer does not mean that the data distribution optimization layer / data distribution compression layer can be located at any position upstream of the target network layer, but means that the distribution optimization layer / data distribution compression layer is located adjacent to the target network layer and can affect the data distribution of the activation value input to the target network layer.

[0119] In another possible implementation, the full-precision image processing model can be obtained by inserting a data distribution optimization layer and / or a data distribution compression layer into the input side of a target network layer in an existing underlying visual processing model. The original underlying visual processing model can be a pre-trained underlying visual processing model or a constructed model framework for processing underlying visual tasks. The original underlying visual processing model can be traversed to determine the target network layer, and then the data distribution optimization layer and / or the data distribution compression layer can be inserted into the input side of the target network layer. For example, when the target network layer includes a convolutional layer, the convolutional layer is searched in the original underlying visual model, and the data distribution optimization layer and / or the data distribution compression layer is inserted into the input side of the convolutional layer. When the target network layer includes an attention layer, the attention layer is searched in the original underlying visual model, and the data distribution optimization layer and / or the data distribution compression layer is inserted into the input side of the attention layer. The target network layer can include at least two different types of network layers, for example, the target network layer includes at least two of a convolutional layer, an attention layer, a deconvolution layer, and a linear layer. There are multiple possible scenarios for inserting a data distribution optimization layer and / or a data distribution compression layer into the original underlying visual model, which are described below.

[0120] 1. Insert the data distribution optimization layer only on the input side of the target network layer in the original underlying vision model

[0121] This situation can be divided into two cases, the input side of the target network layer includes an activation layer (hereinafter, the target network layer whose input side includes the activation layer is sometimes referred to as the second target network layer), and the input side of the target network layer does not include an activation layer (hereinafter, the target network layer whose input side does not include the activation layer is sometimes referred to as the first target network layer).

[0122] 1.1 The input side of the target network layer does not include the activation layer

[0123] Exemplarily, as shown in Figure 6a, Figure 6a is a schematic diagram of the local structure of a bottom-level visual processing model provided by the present application. When the original bottom-level visual model includes multiple first target network layers (target network layer 1 and target network layer 2), a data distribution optimization layer is inserted at the input side of each first target network layer. For example, data distribution optimization layer 1 is inserted at the input side of target network layer 1, and data distribution optimization layer 2 is inserted at the input side of target network layer 2. It can be understood that Figure 6a only takes the example of the original bottom-level visual processing model including two target network layers 1 and target network layer 2. The number of target network layers can be 1 or more, and there is no limitation here.

[0124] 1.2 The input side of the target network layer includes the activation layer

[0125] Exemplarily, as shown in Figure 6b, Figure 6b is a schematic diagram of the local structure of another underlying visual processing model provided by the present application. When there is a second target network layer (target network layer 2) in the original underlying visual processing model, the data distribution optimization layer 2 can be inserted into the input side of the activation layer, that is, the data distribution optimization layer 2 optimizes the data distribution of the activation value to be input into the target network layer 2, and then outputs it to the activation layer. The activation layer activates the activation value input into the data distribution optimization layer 2 and then outputs it to the target network layer 2. There may also be a first target network layer (target network layer 1) in the original underlying visual model. For example, there is often no activation layer on the input side of the target network layer in the first hidden layer. For the target network layer 1, it is sufficient to insert the data distribution optimization layer 1 on the input side of the target network layer 1.

[0126] 2. Insert a data distribution compression layer only on the input side of the target network layer in the original underlying visual model

[0127] 2.1 The input side of the target network layer does not include the activation layer

[0128] For example, as shown in FIG6c, FIG6c is a schematic diagram of the partial structure of another underlying visual processing model provided by the present application. When the original underlying visual model includes multiple first target network layers (target network layer 1 and target network layer 2), a data distribution compression layer is inserted at the input side of each first target network layer. For example, data distribution compression layer 1 is inserted at the input side of target network layer 1, and data distribution compression layer 2 is inserted at the input side of target network layer 2.

[0129] 2.2 The input side of the target network layer includes the activation layer

[0130] The data distribution compression layer in this embodiment may include activation functions such as ReLU, LeakyReLU or HardSwish, and ReLU, LeakyReLU or HardSwish themselves may also be activation functions in the activation layer. In this application, the activation function layer that already exists in the original underlying visual processing model is called the activation layer, and the activation function layer such as ReLU, LeakyReLU or HardSwish that is subsequently inserted into the original underlying visual processing model is called the data distribution compression layer. The activation layer is generally located on the output side of the convolution layer, etc., and is used to activate the output of the convolution layer, etc., while the data distribution compression layer of this application is located on the input side of the target network layer such as the convolution layer, and is used to compress the distribution range of the activation value input to the target network layer. The activation layer located between the two network layers processes the output of the previous network layer and then inputs the value to the next network layer. When the activation function in the activation layer meets the requirements, the activation layer in the original underlying visual processing model can be retained without replacing, so that the activation layer also plays the role of the data distribution compression layer. When the activation function in the activation layer does not meet the requirements, the activation layer is replaced with the data distribution compression layer. The activation function meets the requirements if it has the function of truncation and compression. For example, if the activation function in the activation layer is ReLU, LeakyReLU, or HardSwish, it is considered to meet the requirements. If the activation function in the activation layer is tanh, sigmoid, or other activation functions that do not have truncation and compression functions, the activation layer needs to be replaced with a data distribution compression layer. Of course, when the activation function in the activation layer is inconsistent with the activation function in the data distribution compression layer, the activation layer can also be replaced with a data distribution compression layer. When the original underlying visual processing model includes the first target network layer, the data distribution compression layer needs to be inserted on the input side of the first target network layer.

[0131] Exemplarily, as shown in Figure 6d, Figure 6d is a schematic diagram of the local structure of another underlying visual processing model provided by the present application. When there is a second target network layer (target network layer 2) in the original underlying visual processing model, and the activation layer on the output side of the second target network layer meets the requirements, the activation layer on the input side of the second target network layer can be retained, and the data distribution compression layer is not inserted into the input side of the target network layer. When there is a second target network layer in the original underlying visual processing model, and the activation layer on the output side of the second target network layer does not meet the requirements, the activation layer is replaced with the data distribution compression layer 2. The first target network layer may also exist in the original underlying visual model. For example, the input side of the target network layer in the first hidden layer often does not have an activation layer. For the first target network layer (target network layer 1), it is sufficient to insert the data distribution compression layer 1 on the input side of the first target network layer.

[0132] 3. Insert the data distribution optimization layer and data distribution compression layer into the input side of the target network layer in the original underlying visual processing model

[0133] The following describes two scenarios: the input side of the target network layer includes an activation layer and the input side does not include an activation layer.

[0134] 3.1 The input side of the target network layer does not include the activation layer

[0135] When the input side of the target network layer in the original underlying visual processing model does not include an activation layer, a data distribution optimization layer and a data distribution compression layer can be inserted into the input side of the target network layer. As shown in Figure 6e, data distribution optimization layer 1 and data distribution compression layer 1 are inserted into the input side of target network layer 1, and data distribution optimization layer 2 and data distribution compression layer 2 are inserted into the input side of target network layer 2. In one possible implementation, the data distribution optimization layer can be located on the input side of the data distribution compression layer, and the data distribution compression layer is located between the data distribution optimization layer and the target network layer. In another implementation, the data distribution compression layer can be located on the input side of the data distribution optimization layer, and the data distribution optimization layer is located between the data distribution compression layer and the target network layer.

[0136] 3.2 The input side of the target network layer includes the activation layer

[0137] As shown in Figure 6f, when the input side of the target network layer in the original underlying visual processing model includes an activation layer, that is, when the original underlying visual processing model has a second target network layer (target network layer 2), when the activation layer meets the requirements, the activation layer on the input side of the second target network layer is retained, and the data distribution compression layer is not inserted on the input side of the second target network layer. Only the second data distribution optimization layer (data distribution optimization layer 2) is inserted on the input side of the second target network layer. In this case, the data distribution optimization layer can be located on the input side of the activation layer, that is, the activation layer is located between the data distribution optimization layer and the second target network layer, or it can be located on the output side of the activation layer, that is, the data distribution optimization layer is located between the activation layer and the second target network layer. For the first target network layer (target network layer 1), the first data distribution optimization layer (data distribution optimization layer 1) and the data distribution compression layer are inserted on its input side. The first data distribution optimization layer can be located between the data distribution optimization layer and the second target network layer, or the data distribution compression layer can be located between the first data distribution optimization layer and the second target network layer.

[0138] As shown in Figure 6g, when the activation layer does not meet the requirements, the activation layer is removed, and the second data distribution optimization layer (data distribution optimization layer 2) and the second data distribution compression layer (data distribution compression layer 2) are inserted on the input side of the second target network layer (target network layer 2). The second data distribution optimization layer can be located between the second data distribution optimization layer and the second target network layer, or the second data distribution compression layer can be located between the second data distribution optimization layer and the second target network layer. For the first target network layer (target network layer 1), the first data distribution optimization layer (data distribution optimization layer 1) and the first data distribution compression layer (data distribution compression layer 1) are inserted on its input side. The first data distribution optimization layer can be located between the first data distribution compression layer and the second target network layer, or the first data distribution compression layer can be located between the first data distribution optimization layer and the second target network layer.

[0139] As shown in Figure 7, Figure 7 shows the data distribution after adding the data distribution optimization layer and data distribution compression layer proposed in this application. Compared with Figure 5, it can be seen that the optimized data distribution becomes compact and regular, which greatly reduces the learning difficulty of the quantization algorithm and improves the ability upper limit of the quantization algorithm, achieving a significant improvement in the performance of the final quantized image processing model.

[0140] As shown in Figure 8, Figure 8 shows the average quantization error of different network layers. It can be seen that after introducing the data distribution optimization layer and data distribution compression layer proposed in this application, the quantization error can be reduced by 6 to 100 times. In addition, as the number of network layers increases, the quantization error will gradually accumulate. After introducing the data distribution optimization layer and data distribution compression layer proposed in this application, the error expansion range is reduced from 100 times to 5 times. These further demonstrate the effectiveness and superiority of the solution provided by this application.

[0141] The model obtained by inserting the data distribution optimization layer and / or the data distribution compression layer into the original underlying visual processing model is trained to obtain a full-precision image processing model. In the full-precision image processing model, the target network layer, the data distribution optimization layer, and the data distribution compression layer can be independent network layers. In another possible implementation, the target network layer can be folded with the data distribution optimization layer connected to the input side of the target network layer, thereby accelerating the inference operation. For example, the convolution layer and the BN layer on the output side of the convolution layer can be folded into a convolution BN layer. For the data distribution optimization layer on the input side of the first target network layer, it is independent of the first target network layer.

[0142] After obtaining the full-precision image processing model, the full-precision image processing model is quantized using a quantization algorithm to obtain a corresponding quantized image processing model. The quantization algorithm can be a quantization-aware quantization algorithm or a post-training quantization algorithm, and there is no limitation here. The quantized image processing model is used to implement the underlying image processing task. Because the input side of the target network layer in the full-precision image processing model includes a data distribution optimization layer and / or a data distribution compression layer, the distribution of the activation values ​​input to the target network layer is regular and compact, which is conducive to reducing the complexity of the quantization algorithm and reducing the performance loss of the quantized image processing model, ensuring the performance of the quantized image processing model, and thus improving the quality of the image processed by the quantized image processing model. The quantization model includes a quantization network layer corresponding to the target network layer, and the quantization network layer is used to process the first image or the eigenvalues ​​corresponding to the first image. For example, when the quantization network layer is a network layer in the first hidden layer, the quantization network layer processes the first image, such as performing convolution on the first image. When the quantization network layer is a network layer or a linear layer in the attention layer or other hidden layers, the quantization network layer processes the eigenvalues ​​of the first image. The function of the quantized network layer is similar to that of the corresponding target network layer. For example, if the target network layer is a convolutional layer, the quantized network layer is also a convolutional layer, which performs convolution on the image to extract features in the image. The difference is that the quantized network layer performs computational reasoning based on the quantized weights and activation values, resulting in more efficient reasoning efficiency.

[0143] In this embodiment, by setting a data distribution optimization layer and / or a data distribution compression layer on the input side of the target network layer in the full-precision image processing model used to process the underlying visual tasks, the data distribution of the activation values ​​of the input target network layer can be improved, making the data distribution of the activation values ​​of the input target network layer more compact and regular, thereby reducing the learning difficulty of the quantization algorithm and improving the upper limit of the ability of the quantization algorithm, thereby achieving an improvement in the performance of the final quantized image processing model and ensuring the quality of the image processed by the quantized image processing model. Although setting a data distribution optimization layer and / or a data distribution compression layer in the full-precision image processing model will cause performance loss of the full-precision image processing model, it will instead improve the performance of the quantized image processing model corresponding to the full-precision image processing model, which is beneficial to the quantized image processing model. When the quantized image processing model is used to perform the underlying visual image processing tasks, higher quality images can be obtained.

[0144] Unlike the current practice of adding a BN layer on the output side of the convolutional layer in the neural network model to accelerate model training, in this embodiment, a data distribution optimization layer and / or a data distribution compression layer is set on the input side of the target network layer, which can simplify the data distribution of the activation value and reduce the quantization error. This not only simplifies the quantization algorithm, but also improves the performance of the quantized visual image processing model, thereby ensuring the quality of the image processed by the quantized visual image processing model.

[0145] The two modules proposed in this application greatly simplify data distribution, especially when quantizing at very low bits. They make the previously highly complex and irregular data distribution compact and regular, significantly reducing the learning difficulty of the quantization algorithm and significantly improving the final quantization performance. This benefit is not only reflected in quantization-aware training tasks, but also brings significant benefits to post-training quantization tasks.

[0146] To demonstrate the benefits of the proposed solution, we conducted a super-resolution (Super-Resolution) experiment using a classic EDSR network on the DIV2K dataset. We used the post-training quantization method DFSQ (Zhong, Yunshan, Mingbao Lin, Jingjing Xie, Yuxin Zhang, Fei Chao, and Rongrong Ji. "Distribution-Flexible Subset Quantization for Post-Quantizing Super-Resolution Networks." arXiv preprint arXiv:2305.05888.2023) as a baseline algorithm, and performed post-training quantization experiments on Enhanced Deep Residual Networks (EDSR) using 2, 3, and 4 bits, respectively. BNQ adopts the method proposed in this application, that is, the EDSR network (a super-resolution network, the original underlying visual processing model) is trained after inserting the data distribution optimization layer and the data distribution compression layer proposed in this application to obtain a full-precision EDSR network (full-precision image processing model), and then quantizes it using the post-training quantization method DFSQ to obtain a quantized image processing model, and uses the quantized image processing model to perform quantization experiments in DIV2K. The benchmark is to not insert the data distribution optimization layer and the data distribution compression layer into the EDSR network, and use the post-training quantization method DFSQ to quantize the EDSR network to obtain the corresponding quantization model, and use the quantization model to perform quantization experiments in DIV2K. As can be seen from Table 1, by adding the data distribution optimization layer and the data distribution compression layer proposed in this application, there is a very obvious performance improvement in extremely low-bit quantization, which further illustrates the effectiveness of the solution provided by this application. Peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) are used here to evaluate the quality of the output image of the quantization model.

[0147] Table 1. Performance comparison of this application and the post-training quantization method DFSQ based on the super-resolution network EDSR on the DIV2K dataset

[0148] Furthermore, we verified the performance improvements of setting only the data distribution optimization layer (taking the BN layer as an example) on the input side of the target network layer, setting only the data distribution compression layer (taking the ReLU layer as an example), and setting both the data distribution optimization layer and the data distribution compression layer at the same time. The results are shown in Table 2. It can be seen that the performance improvement of the quantized image processing model is most obvious when both the data distribution optimization layer and the data distribution compression layer are set, and the performance of the quantized image processing model when only the data distribution optimization layer or the data distribution compression layer is set is significantly improved compared to the baseline.

[0149] Table 2. Ablation experiments of data distribution optimization module and data distribution compression module

[0150] Tables 3 and 4 show the performance comparison of this application and other quantization-aware training methods based on super-resolution networks EDSR and RDN on the DIV2K dataset. It can be seen that compared with other currently advanced research methods (sota), this application achieves the best quantization performance, and the advantage is even more pronounced at very low bits, further demonstrating the effectiveness of the technical solution provided by this application.

[0151] Table 3 Performance comparison of this application and other QAT quantization methods based on super-resolution network EDSR on the DIV2K dataset

[0152] Table 4. Performance comparison of this application and other QAT quantization methods based on super-resolution network RDN on the DIV2K dataset

[0153] Based on the same inventive concept, the present application also provides an image processing device. As shown in Figure 9, the device 900 includes a processing module 901. The processing module 901 is used to obtain a first image. The processing module 901 is also used to obtain a second image based on the first image through a quantized image processing model. The quantized image processing model is a quantized model of a full-precision image processing model. The full-precision image processing model includes a target network layer, and a data distribution optimization layer and / or a data distribution compression layer. The data distribution optimization layer and / or the data distribution compression layer are located on the input side of the target network layer. The data distribution optimization layer is used to optimize the data distribution of the activation value input to the target network layer. The data distribution compression layer is used to compress the data distribution of the activation value input to the target network layer. The quantized model includes a quantized network layer corresponding to the target network layer. The quantized network layer is used to process the first image or the eigenvalues ​​corresponding to the first image.

[0154] In one possible implementation, the full-precision image processing model includes a data distribution optimization layer and a data distribution compression layer. The data distribution compression layer is located between the data distribution optimization layer and the target network layer. The data distribution compression layer is used to compress the data distribution of the activation value output by the data distribution optimization layer and then output it to the target network layer.

[0155] In one possible implementation, the full-precision image processing model includes a data distribution optimization layer and a data distribution compression layer. The data distribution optimization layer is located between the data distribution compression layer and the target network layer. The data distribution layer is used to optimize the data distribution of the activation value output by the data distribution compression layer and then output it to the target network layer.

[0156] In one possible implementation, the full-precision image processing model includes a first data distribution optimization layer, a second data distribution optimization layer, a first target network layer, a second target network layer and an activation layer. The input side of the first target network layer does not include the activation layer, and the input side of the second target network layer includes the activation layer. The activation layer is used to perform nonlinear mapping on the output of the previous network layer and then input it into the second target network layer. The first data distribution optimization layer is located on the input side of the first target network layer, and the second data distribution optimization layer is located on the input side of the activation layer or on the input side of the second target network layer.

[0157] In one possible implementation, the full-precision image processing model includes a first data distribution optimization layer, a second data distribution optimization layer, a data distribution compression layer, a first target network layer, a second target network layer and an activation layer. The input side of the first target network layer does not include the activation layer, and the input side of the second target network layer includes the activation layer. The activation layer is used to perform nonlinear mapping on the output of the previous network layer and then input it into the second target network layer. The first data distribution optimization layer and the data distribution compression layer are located on the input side of the first target network layer, and the second data distribution optimization layer is located on the input side of the activation layer.

[0158] In one possible implementation, the full-precision image processing model includes a data distribution compression layer, a first target network layer, a second target network layer and an activation layer. The input side of the first target network layer does not include the activation layer, and the input side of the second target network layer includes the activation layer. The activation layer is used to perform nonlinear mapping on the output of the previous network layer and then input it into the second target network layer. The data distribution compression layer is located on the input side of the first target network layer.

[0159] In one possible implementation, the target network layer is a convolutional layer, a deconvolutional layer, an attention layer, or a linear layer.

[0160] In one possible implementation, the data distribution compression layer is a ReLU layer, a LeakyReLU layer, or a HardSwish layer.

[0161] In one possible implementation, the data distribution optimization layer is a batch normalization layer, an instance normalization layer, or a layer normalization layer.

[0162] An embodiment of the present application also provides a chip or chip system, which includes at least one processor and a communication interface, the communication interface and the at least one processor being interconnected by lines, and the at least one processor being used to run a computer program or instruction to perform the image processing method described in any one of the possible implementation methods of the embodiment shown in Figure 4; wherein the communication interface in the chip can be an input / output interface, a pin or a circuit, etc.

[0163] In one possible implementation, the chip or chip system described above in this application further includes at least one memory, in which instructions are stored. The memory may be a storage unit within the chip, such as a register, a cache, etc., or a storage unit of the chip (e.g., a read-only memory, a random access memory, etc.).

[0164] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer, implements the image processing method flow of any of the above method embodiments.

[0165] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0166] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical or other forms.

[0167] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0168] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0169] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the technical solution of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

Claims

1. An image processing method, characterized in that: The method comprises: acquiring a first image; According to the first image, a second image is obtained by a quantized image processing model, wherein the quantized image processing model is a quantized model of a full-precision image processing model, wherein the full-precision image processing model includes a target network layer, and a data distribution optimization layer and / or a data distribution compression layer, wherein the data distribution optimization layer and / or the data distribution compression layer are located at the input side of the target network layer, wherein the data distribution optimization layer is used to optimize the data distribution of activation values ​​input to the target network layer, wherein the data distribution compression layer is used to compress the data distribution of activation values ​​input to the target network layer, wherein the quantized model includes a quantized network layer corresponding to the target network layer, wherein the quantized network layer is used to process the first image or a eigenvalue corresponding to the first image.

2. The method according to claim 1, characterized in that The full-precision image processing model includes the data distribution optimization layer and the data distribution compression layer, wherein the data distribution compression layer is located between the data distribution optimization layer and the target network layer, and the data distribution compression layer is used to compress the data distribution of the activation value output by the data distribution optimization layer and then output it to the target network layer.

3. The method according to claim 1, characterized in that The full-precision image processing model includes the data distribution optimization layer and the data distribution compression layer. The data distribution optimization layer is located between the data distribution compression layer and the target network layer. The data distribution layer is used to optimize the data distribution of the activation value output by the data distribution compression layer and then output it to the target network layer.

4. The method according to claim 1, characterized in that: The full-precision image processing model includes a first data distribution optimization layer, a second data distribution optimization layer, a first target network layer, a second target network layer and an activation layer. The input side of the first target network layer does not include the activation layer, and the input side of the second target network layer includes the activation layer. The activation layer is used to perform nonlinear mapping on the output of the previous network layer and then input it into the second target network layer. The first data distribution optimization layer is located at the input side of the first target network layer, and the second data distribution optimization layer is located at the input side of the activation layer or at the input side of the second target network layer.

5. The method according to claim 1, characterized in that: The full-precision image processing model includes a first data distribution optimization layer, a second data distribution optimization layer, the data distribution compression layer, a first target network layer, a second target network layer and an activation layer. The input side of the first target network layer does not include the activation layer, and the input side of the second target network layer includes the activation layer. The activation layer is used to perform nonlinear mapping on the output of the previous network layer and then input it into the second target network layer. The first data distribution optimization layer and the data distribution compression layer are located on the input side of the first target network layer, and the second data distribution optimization layer is located on the input side of the activation layer.

6. The method according to claim 1, characterized in that The full-precision image processing model includes a data distribution compression layer, a first target network layer, a second target network layer and an activation layer. The input side of the first target network layer does not include the activation layer, and the input side of the second target network layer includes the activation layer. The activation layer is used to perform nonlinear mapping on the output of the previous network layer and then input it into the second target network layer. The data distribution compression layer is located on the input side of the first target network layer.

7. The method according to any one of claims 1 to 6, characterized in that The target network layer is a convolutional layer, a deconvolutional layer, an attention layer or a linear layer.

8. The method according to any one of claims 1 to 7, characterized in that The data distribution compression layer is a ReLU layer, a LeakyReLU layer or a HardSwish layer.

9. The method according to any one of claims 1 to 8, characterized in that The data distribution optimization layer is a batch normalization layer, an instance normalization layer or a layer normalization layer.

10. An image processing device, characterized in that: The device comprises a processing module, and the processing module is used to execute the image processing method according to any one of claims 1 to 9.

11. A device, characterized in that: The device comprises a processor and a memory, wherein the processor is coupled to the memory, and the processor is configured to execute the image processing method according to any one of claims 1 to 9 based on instructions stored in the memory.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises instructions, which, when the computer-readable storage medium is run on a computer, causes the computer to execute the image processing method according to any one of claims 1 to 9.

Images