Image processing method and related equipment

Abstract

The invention discloses an image processing method and related equipment so as to improve the quality of an image processed by a quantized image processing model. The method comprises the steps of obtaining a first image; according to the first image, a second image is obtained through a quantization image processing model, the quantization image processing model is a quantization model of a full-precision image processing model, and the full-precision image processing model comprises a target network layer and a data distribution optimization layer and / or a data distribution compression layer; the data distribution optimization layer and / or the data distribution compression layer are / is located on the input side of the target network layer, the data distribution optimization layer is used for optimizing data distribution of activation values input into the target network layer, and the data distribution compression layer is used for compressing the data distribution of the activation values input into the target network layer. The quantization model comprises a quantization network layer corresponding to the target network layer, and the quantization network layer is used for processing the first image or a feature value corresponding to the first image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method and related equipment. Background Art

[0002] With the continuous development of artificial intelligence, neural networks have demonstrated super computing power in many fields such as images, videos, and voice. Since the neural network algorithm itself has a large amount of computing and parameters, this will bring great challenges to the deployment of neural network models on terminal devices.

[0003] In order to enable the neural network model to be deployed on the end-side device, the scale of the neural network model can be reduced by model quantization. Specifically, model quantization refers to the process of converting the original parameters of the neural network model from floating-point storage to integer storage to obtain a quantized model. Since each original parameter will occupy 32 bits (binary digit, bit) of storage space in the floating-point storage method, and the integer storage method can reduce the bits corresponding to each parameter, model quantization will greatly reduce the storage space of the model, and the obtained quantized model will also be easier to deploy. Using the quantized model for reasoning can accelerate the reasoning process.

[0004] In the underlying visual tasks, the activation value distribution of the full-precision image processing model is complex, showing high dynamic range, irregular, and asymmetric characteristics, which causes the performance of the quantized model obtained after quantization of the full-precision image processing model to degrade seriously, resulting in poor image quality obtained by the quantized model. Summary of the invention

[0005] The present application provides an image processing method and related devices to solve the problem that the quantization model of the current full-precision image processing model has poor performance in processing underlying visual tasks.

[0006] The first aspect provides an image processing method. The method can be applied to a terminal device or a server. The method includes: obtaining a first image. The first image is an image to be processed. Then, according to the first image, a second image is obtained by a quantized image processing model, the quantized image processing model is a quantized model of a full-precision image processing model, the full-precision image processing model includes a target network layer, and a data distribution optimization layer and / or a data distribution compression layer, the data distribution optimization layer and / or the data distribution compression layer is located at the input side of the target network layer, the data distribution optimization layer is used to optimize the data distribution of the activation value input to the target network layer, the data distribution compression layer is used to compress the data distribution of the activation value input to the target network layer, the quantized model includes a quantized network layer corresponding to the target network layer, and the quantized network layer is used to process the first image or the eigenvalue corresponding to the first image.

[0007] The full-precision image processing model and the quantized image processing model are both models used to process low-level visual tasks, such as noise reduction, super-resolution, and deblurring.

[0008] In order to ensure the performance of the underlying visual processing model, the underlying visual processing model does not include a batch normalization layer. In this application, a data distribution optimization layer and / or a data distribution compression layer are provided in the full-precision image processing model. The data distribution optimization layer scales and translates the data distribution of the activation values ​​to make the data distribution of the activation values ​​of the input target network layer more regular. The data distribution compression layer truncates the data distribution of the activation values ​​to make the data distribution of the activation values ​​of the input target network layer more compact. Both can simplify the data distribution, thereby reducing the quantization complexity, reducing the performance loss of the quantized image processing model corresponding to the full-precision image processing model, and improving the performance of the quantized image processing model. Thus, the second image obtained by performing underlying visual processing on the first image based on the quantized image processing model has higher image quality.

[0009] In a possible implementation, the full-precision image processing model includes a data distribution optimization layer and a data distribution compression layer, the data distribution compression layer is located between the data distribution optimization layer and the target network layer, and the data distribution compression layer is used to compress the data distribution of the activation value output by the data distribution optimization layer and then output it to the target network layer. Optimizing the data distribution of the activation value and then compressing it can further simplify the data distribution, making the data distribution of the activation value input to the target network layer more compact and regular, reducing the learning difficulty of the quantization algorithm, and improving the upper limit of the quantization algorithm, achieving a significant improvement in the performance of the final quantized image processing model, and thereby improving the quality of the second image processed by the quantized image processing model.

[0010] In a possible implementation, the full-precision image processing model includes a data distribution optimization layer and a data distribution compression layer, the data distribution optimization layer is located between the data distribution compression layer and the target network layer, and the data distribution layer is used to optimize the data distribution of the activation values ​​output by the data distribution compression layer and then output it to the target network layer. Optimizing the data distribution after compressing the data distribution can also simplify the data distribution of the activation values, making the data distribution of the activation values ​​input to the target network layer more compact and regular, reducing the learning difficulty of the quantization algorithm, and improving the upper limit of the quantization algorithm, achieving a significant improvement in the performance of the final quantized image processing model, and thereby improving the quality of the second image processed by the quantized image processing model.

[0011] In one possible implementation, the full-precision image processing model includes a first data distribution optimization layer, a second data distribution optimization layer, a first target network layer, a second target network layer and an activation layer. The input side of the first target network layer does not include the activation layer, the input side of the second target network layer includes the activation layer, the activation layer is used to perform nonlinear mapping on the output of the previous network layer and then input it into the second target network layer. The first data distribution optimization layer is located at the input side of the first target network layer, and the second data distribution optimization layer is located at the input side of the activation layer or at the input side of the second target network layer.

[0012] In one possible implementation, the full-precision image processing model includes a first data distribution optimization layer, a second data distribution optimization layer, a data distribution compression layer, a first target network layer, a second target network layer and an activation layer. The input side of the first target network layer does not include the activation layer, and the input side of the second target network layer includes the activation layer. The activation layer is used to perform nonlinear mapping on the output of the previous network layer and then input it into the second target network layer. The first data distribution optimization layer and the data distribution compression layer are located on the input side of the first target network layer, and the second data distribution optimization layer is located on the input side of the activation layer.

[0013] In one possible implementation, the full-precision image processing model includes a data distribution compression layer, a first target network layer, a second target network layer and an activation layer. The input side of the first target network layer does not include the activation layer, and the input side of the second target network layer includes the activation layer. The activation layer is used to perform nonlinear mapping on the output of the previous network layer and then input it into the second target network layer. The data distribution compression layer is located on the input side of the first target network layer.

[0014] In one possible implementation, the target network layer is a convolutional layer, a deconvolutional layer, an attention layer, or a linear layer. These network layers include matrix multiplication operations. Full-precision matrix multiplication operations require more computing power. Quantizing the target network layer can accelerate matrix multiplication operations, thereby accelerating the inference speed of the quantized image model, and can also reduce the size of the model, making the quantized model easier to deploy.

[0015] In a possible implementation, the data distribution compression layer is a ReLU layer, a LeakyReLU layer, or a HardSwish layer.

[0016] In a possible implementation, the data distribution optimization layer is a batch normalization layer, an instance normalization layer, or a layer normalization layer.

[0017] A second aspect provides an image processing device, which includes a processing module, and the processing module is used to implement the image processing method in the first aspect or any possible implementation of the first aspect.

[0018] A third aspect provides a device, comprising a processor and a memory, wherein the processor is coupled to the memory, and the processor is configured to execute the image processing method in the first aspect or any possible implementation of the first aspect based on instructions stored in the memory.

[0019] A fourth aspect provides a computer-readable storage medium, comprising instructions, which, when the computer-readable storage medium is run on a computer, enables the computer to execute the image processing method in the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of a system architecture provided for this application;

[0021] Figure 2a for Figure 1 An optional hardware structure diagram of the terminal 100 in FIG. 1 ;

[0022] Figure 2b for Figure 1 An optional hardware structure diagram of the server 200;

[0023] Figure 3 Another system framework schematic diagram provided for this application;

[0024] Figure 4 A flowchart of an image processing method provided in this application;

[0025] Figure 5 Statistics of activation distributions of different layers after removing BN layers from the underlying visual processing model;

[0026] Figure 6a A schematic diagram of the local structure of an underlying visual processing model provided in this application;

[0027] Figure 6b A schematic diagram of the local structure of another underlying visual processing model provided for this application;

[0028] Figure 6c A schematic diagram of the local structure of another underlying visual processing model provided for this application;

[0029] Figure 6d A schematic diagram of the local structure of another underlying visual processing model provided for this application;

[0030] Figure 6e A schematic diagram of the local structure of another underlying visual processing model provided for this application;

[0031] Figure 6f A schematic diagram of the local structure of another underlying visual processing model provided for this application;

[0032] Figure 6g A schematic diagram of the local structure of another underlying visual processing model provided for this application;

[0033] Figure 7 The statistical information of data distribution after adding the data distribution optimization layer and data distribution compression layer proposed in this application to the underlying visual processing model;

[0034] Figure 8 The statistical information of the average quantization error of different network layers after adding the data distribution optimization layer and data distribution compression layer proposed in this application to the underlying visual processing model;

[0035] Fig. 9 A schematic diagram of the structure of an image processing device provided in this application. DETAILED DESCRIPTION

[0036] The present application provides an image processing method and related equipment to improve the image quality of images processed by a quantization model in low visual task scenarios.

[0037] The technical solution in the present application will be described in detail below in conjunction with the drawings in the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0038] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0039] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.

[0040] (1) Neural Network

[0041] A neural network may be composed of neural units, and a neural unit may refer to an operation unit that takes xs (i.e., input data) and intercept 1 as input, and the output of the operation unit may be:

[0042]

[0043] Where s=1, 2, ...n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolution layer. The activation function can be a sigmoid, Relu or tanh function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0044] (2) Deep Neural Networks

[0045] Deep Neural Network (DNN), also known as multi-layer neural network, can be understood as a neural network with many hidden layers. There is no special metric for "many" here. From the position of different layers of DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The definition of these parameters in DNN is as follows: Take coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by many layers of vector W).

[0046] (3) Neural Network Quantization

[0047] Quantization refers to the conversion of data so that the converted data is smaller than the storage space occupied by the original data; for example, the process of converting a floating point number occupying 32 bits of storage space into an 8-bit, 4-bit or 2-bit integer, or converting an integer occupying 32 bits of storage space into a 16-bit integer, can be called a quantization process. Neural network quantization is a model compression technology that converts floating point storage (operation) into integer storage (operation). For example, the weights and activation values ​​of a model are originally represented by float32 (32-bit floating point). After quantization, the weights and activation values ​​of the model are represented by int8 (8-bit fixed point). Through the quantization operation of the model, the size of the model is reduced and the model inference speed is improved at the cost of a small loss of accuracy. In this way, the quantized model can be deployed on the end-side device according to the quantization parameters corresponding to each operation parameter, meeting the end-side device's deployment requirements for the neural network, so as to facilitate the subsequent use of the quantized model.

[0048] Neural network quantization includes two quantization methods, one is quantization-aware training (QAT) and post-training quantization (PTQ). QAT can automatically adjust network parameters during training to adapt to the quantized model. The main idea of ​​QAT is to add quantization error during training so as to maximize the performance of the quantized model while minimizing the loss function. PTQ is a method to directly convert a pre-trained FP32 (32-bit floating point, also known as full precision) network into a fixed-point network.

[0049] (4) Batch Normalization (BN)

[0050] It is a neural network training technique that can automatically adjust network parameters during training to adapt to changes in batch data, thereby improving the convergence speed and generalization ability of the model. The basic principle of batch normalization is to normalize the input of each neuron so that the mean of the input data is 0 and the variance is 1. This can avoid problems such as gradient disappearance or explosion, and improve the training efficiency and performance of the model.

[0051] (5) Low-level visual tasks

[0052] The underlying visual tasks include denoising, deblurring, defogging, deraining, coloring, enhancement, restoration, compression, and super-resolution (SR) of images and videos. These tasks usually deal with the underlying visual features, such as color, brightness, direction, and texture. The underlying visual tasks improve the image quality as much as possible without changing the semantic information of the image.

[0053] In the underlying visual image processing models, such as super-resolution models, defogging, deraining / noise reduction models, etc., the batch normalization (BN) layer is removed by default, because the BN layer will make the activation in the network lose range flexibility and reduce the scale information of the image, thus causing the performance degradation of the underlying visual image processing model. BN can accelerate and stabilize the training of deep neural networks by normalizing and scaling the input of the layer.

[0054] Removing the BN layer can improve the performance of the underlying visual image processing model and save computational effort. The quantization model of the underlying visual image processing model also follows this setting, that is, there is no normalization layer in the visual image processing quantization model. In the process of quantizing the full-precision underlying visual image processing model, due to the absence of the BN layer, the distribution of activation values ​​becomes extremely complex, showing high dynamic range, irregular, and asymmetric characteristics, making quantization complex and difficult, the quantization error is large, and the quantization model is difficult to converge. When quantizing at extremely low bits (2 bits, 4 bits, etc.), the performance degradation of the quantization model is more serious, reducing the image processing quality.

[0055] Therefore, in order to solve the problem that the poor performance of the quantization model of the current underlying visual image processing model affects the image quality, the present application provides the following embodiments.

[0056] See also Figure 1 , Figure 1 A schematic diagram of a system architecture is shown. The system may include a terminal 100 and a server 200. The server 200 may include one or more servers ( Figure 1 In the example of including one server, the server 200 can provide a quantization model corresponding to the full-precision underlying visual processing model for one or more terminals.

[0057] Specifically, the server 200 can train a full-precision image processing model for implementing the underlying visual processing task. The input side of the target network layer in the full-precision image processing model is provided with a data distribution optimization module and / or a data distribution compression module. Then, the trained full-precision image processing model is quantized to obtain a corresponding quantized image processing model. Then, the server 200 sends the quantized image processing model to the terminal 100 so that the terminal 100 deploys the quantized image processing model.

[0058] Among them, a quantized image processing model for performing underlying visual processing on images is deployed on the terminal 100, and the terminal 100 uses the quantized model to process images.

[0059] It should be understood that in some optional implementations, the server 200 may also use a quantization model to process images, which is not limited in the embodiments of the present application.

[0060] Next describe Figure 1 The product form of the mid-terminal 100;

[0061] The terminal 100 in the embodiment of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any limitation on this.

[0062] Figure 2a An optional hardware structure diagram of the terminal 100 is shown.

[0063] refer to Figure 2a As shown, the terminal 100 may include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), a processor 170, an external interface 180, a power supply 190 and other components. Those skilled in the art will appreciate that Figure 2a These are merely examples of terminals or multi-function devices and do not constitute limitations on the terminals or multi-function devices, which may include more or fewer components than those shown in the figures, or combinations of certain components, or different components.

[0064] The input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the portable multifunctional device. Specifically, the input unit 130 may include a touch screen 131 (optional) and / or other input devices 132. The touch screen 131 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal at least includes the touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can also include other input devices. Specifically, other input devices 132 may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0065] The display unit 140 may be used to display information input by the user or provided to the user, various menus of the terminal 100, interactive interfaces, file display, and / or playback of any multimedia file. In the embodiment of the present application, the display unit 140 may be used to display an interface for controlling image processing, image processing results, etc.

[0066] The memory 120 can be used to store instructions and data. The memory 120 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files such as images and videos, texts, etc.; the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, as well as the storage of running programs and applications.

[0067] The processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire terminal 100. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal 100 and processes data, thereby controlling the terminal device as a whole. Optionally, the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0068] Among them, the memory 120 can be used to store software codes and quantization models related to the image method, the processor 170 can execute the steps of the image processing method of the chip, and can also schedule other units to implement corresponding functions, such as scheduling the camera 150 to capture images, or reading images stored in the memory 120.

[0069] The radio frequency unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0070] In this embodiment of the present application, the RF unit 110 can receive the quantization model sent by the server 200 and deploy it in the terminal 100.

[0071] It should be understood that the radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.

[0072] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0073] The terminal 100 further includes an external interface 180 , which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal 100 to communicate with other devices, or to connect a charger to charge the terminal 100 .

[0074] Although not shown, the terminal 100 may also include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which are not described in detail here. Some or all of the methods described below may be applied in the following embodiments. Figure 2a In the terminal 100 shown.

[0075] Next describe Figure 1 The product form of the server 200;

[0076] Figure 2b A structural diagram of a server 200 is provided, such as Figure 2b As shown, the server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate with each other via the bus 201.

[0077] The bus 201 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 2b Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0078] The processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0079] The memory 204 may include a volatile memory, such as a random access memory (RAM). The memory 204 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard drive (HDD), or a solid state drive (SSD).

[0080] The memory 204 may be used for software codes related to the image processing method, and the processor 202 may execute the steps of the image processing method of the chip, and may also schedule other units to implement corresponding functions.

[0081] It should be understood that the above-mentioned terminal 100 and server 200 can be centralized or distributed devices, and the processors in the above-mentioned terminal 100 and server 200 (such as processor 170 and processor 202) can be hardware circuits (such as application specific integrated circuit (ASIC), field-programmable gate array (FPGA), general-purpose processor, digital signal processor (DSP), microprocessor or microcontroller, etc.), or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0082] It should be understood that the image processing method in the embodiment of the present application involves AI-related operations. When performing AI operations, the instruction execution architecture of the terminal device and the server is not limited to Figure 2a as well as Figure 2b The processor combined with the memory architecture shown below Figure 3 The system architecture provided in the embodiments of the present application is introduced in detail.

[0083] Figure 3 This is a schematic diagram of the system architecture provided in the embodiment of the present application. Figure 3 As shown, the system architecture 500 includes an execution device 510 , a training device 520 , a database 530 , a client device 540 , a data storage system 550 , and a data collection system 560 .

[0084] The execution device 510 includes a calculation module 511, an I / O interface 512, a preprocessing module 513 and a preprocessing module 514. The calculation module 511 may include a target model / rule 501, and the preprocessing module 513 and the preprocessing module 514 are optional.

[0085] The execution device 510 may be a terminal device or a server for executing the above-mentioned image processing method.

[0086] The data acquisition device 560 is used to collect training samples. The training samples can be images, etc. After collecting the training samples, the data acquisition device 560 stores these training samples in the database 530.

[0087] The training device 520 can train the neural network to be trained (such as the full-precision image processing model before training in the embodiment of the present application) based on the training samples maintained in the database 530. The training device 520 can also quantize the trained full-precision image processing model to obtain the target model / rule 501 (quantized image processing model).

[0088] It should be noted that, in actual applications, the training samples maintained in the database 530 may not all come from the data acquisition device 560, but may also be received from other devices. It should also be noted that the training device 520 may not train the target model / rule 501 entirely based on the training samples maintained in the database 530, but may also obtain training samples from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.

[0089] The target model / rule 501 obtained by training the training device 520 can be applied to different systems or devices, such as Figure 3 The execution device 510 shown may be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) / virtual reality (VR) device, a vehicle terminal, etc., or a server, etc.

[0090] Specifically, the training device 520 may transfer the quantization model to the execution device 510. The execution device 510 processes the first image to be processed using the quantization image processing model to obtain the second image.

[0091] exist Figure 3 In the embodiment, the execution device 510 is configured with an input / output (I / O) interface 512 for data interaction with an external device. A user can input data (eg, a first image, etc.) to the I / O interface 512 through a client device 540 .

[0092] The preprocessing module 513 and the preprocessing module 514 are used to preprocess the input data received by the I / O interface 512. It should be understood that there may be no preprocessing module 513 and the preprocessing module 514 or only one preprocessing module. When there is no preprocessing module 513 and the preprocessing module 514, the computing module 511 may be directly used to process the input data.

[0093] When the execution device 510 preprocesses the input data, or when the computing module 511 of the execution device 510 performs calculations and other related processing, the execution device 510 can call the data, code, etc. in the data storage system 550 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing into the data storage system 550.

[0094] Finally, the I / O interface 512 provides the processing result (eg, the second image in the embodiment of the present application) to the client device 540 , and thus to the user.

[0095] exist Figure 3 In the case shown, the user can manually give input data, and the "manually given input data" can be operated through the interface provided by the I / O interface 512. In another case, the client device 540 can automatically send input data to the I / O interface 512. If the client device 540 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 540. The user can view the results output by the execution device 510 on the client device 540, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 540 can also be used as a data acquisition terminal to collect the input data of the input I / O interface 512 and the output results of the output I / O interface 512 as shown in the figure as new sample data, and store them in the database 530. Of course, it is also possible not to collect through the client device 540, but the I / O interface 512 directly stores the input data of the input I / O interface 512 and the output results of the output I / O interface 512 as new sample data in the database 530.

[0096] It is worth noting that Figure 3 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 3 In the embodiment, the data storage system 550 is an external memory relative to the execution device 510. In other cases, the data storage system 550 may also be placed in the execution device 510. It should be understood that the execution device 510 may be deployed in the client device 540.

[0097] From the inference side of the model:

[0098] In the embodiment of the present application, the calculation module 511 of the above-mentioned execution device 520 can obtain the code stored in the data storage system 550 to implement the image processing method in the embodiment of the present application.

[0099] In an embodiment of the present application, the computing module 511 of the execution device 520 may include a hardware circuit (such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, etc.), or a combination of these hardware circuits. For example, the training device 520 may be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0100] Specifically, the computing module 511 of the execution device 520 can be a hardware system with an execution instruction function, and the image processing method provided in the embodiment of the present application can be a software code stored in a memory. The computing module 511 of the execution device 520 can obtain the software code from the memory and execute the obtained software code to implement the image processing method provided in the embodiment of the present application.

[0101] It should be understood that the computing module 511 of the execution device 520 can be a combination of a hardware system that does not have the function of executing instructions and a hardware system that has the function of executing instructions. Some steps of the image processing method provided in the embodiment of the present application can also be implemented by the hardware system that does not have the function of executing instructions in the computing module 511 of the execution device 520, which is not limited here.

[0102] From the training side of the model:

[0103] In the embodiment of the present application, the training device 520 can obtain the memory ( Figure 3 Not shown in the figure, the code stored in the training device 520 (or deployed separately from the training device 520) can be integrated into the training device 520 to implement the steps related to model training and quantization in the embodiments of the present application.

[0104] In an embodiment of the present application, the training device 520 may include a hardware circuit (such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, etc.), or a combination of these hardware circuits. For example, the training device 520 may be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0105] It should be understood that the training device 520 can be a combination of a hardware system that does not have the function of executing instructions and a hardware system that has the function of executing instructions. Some of the steps related to model training provided in the embodiments of the present application can also be implemented by the hardware system that does not have the function of executing instructions in the training device 520, which is not limited here.

[0106] like Figure 4 As shown, Figure 4 This is a flowchart of an image processing method provided by the present application. This embodiment can be executed by the above-mentioned terminal / execution device, or by the above-mentioned server, which is collectively referred to as a device in this embodiment. This embodiment includes the following steps:

[0107] S401: Acquire a first image.

[0108] The first image is an image to be processed. The first image may be an image input by a user, an image read from a memory by a device, or an image captured in real time by a camera of the device. It should be noted that the image may also refer to a frame in a video.

[0109] S402: According to the first image, a second image is obtained through a quantized image processing model, where the quantized image processing model is a quantized model of a full-precision image processing model, where the full-precision image processing model includes a target network layer, and a data distribution optimization layer and / or a data distribution compression layer, where the data distribution optimization layer and / or the data distribution compression layer is located at an input side of the target network layer, and the quantized model includes a quantized network layer corresponding to the target network layer.

[0110] In this embodiment, the quantized image processing model is a model for processing underlying visual tasks, which include at least one of denoising, deblurring, defogging, deraining, coloring, enhancement, restoration, compression, and SR.

[0111] The quantized image processing model is a quantized model of the full-precision image processing model. The full-precision image processing model is an underlying visual processing model for processing underlying visual tasks, such as a denoising model, a deblurring model, a defogging model, a deraining model, a coloring model, an enhancement model, a restoration model, a compression model, and an SR model, or an image processing model that can implement a plurality of the above underlying tasks.

[0112] The underlying vision is the computational basis of mid- and high-level vision, and the underlying visual processing model processes low-level information of the image, such as pixel-level features. This requires the underlying visual processing model to have high performance to ensure that it outputs high-quality images after processing the image. In order to ensure the performance of the underlying visual processing model, it is a widely accepted consensus in the industry to remove the BN layer from the underlying visual processing model. This is because the BN layer will cause the activation in the network to lose range flexibility, reduce the scale information of the image, and reduce the edge distinction in the image. That is, the BN layer causes the performance of the underlying visual processing model to degrade.

[0113] like Figure 5 As shown in Figure 2, due to the absence of the BN layer, the activation distribution of the underlying visual processing model becomes extremely complex, showing high dynamic range, irregular, and asymmetric characteristics (see Figure 5 The first row in (distribution of activations at different layers) shows that the maximum and minimum activations of different input samples vary greatly (see Figure 5 The second row in (the maximum and minimum distribution of all training samples)). These characteristics are very unfavorable for model quantization, greatly increasing the optimization difficulty of the quantization algorithm. When quantizing at very low bits (such as 2 bits, 3 bits or 4 bits, etc.), the performance degradation of the quantized underlying visual processing model is more serious. Regarding how to improve the performance of the quantized underlying visual processing model, the current mainstream direction is to improve the quantization algorithm to cope with highly complex activation distributions and achieve complexity. This approach can improve the performance of the quantized model to a certain extent, but for low-bit quantized models, the performance loss caused by quantization is still very large.

[0114] Different from the current consensus of removing the BN layer in the underlying visual processing model, the full-precision image processing model (full-precision underlying visual processing model) in this embodiment includes a data distribution optimization layer and / or a data distribution compression layer on the input side of the target network layer. Among them, the target network layer is a network layer that needs to be quantized, and the target network layer contains matrix multiplication operations. The target network layer can be a network layer for extracting features, such as a convolution layer, a deconvolution layer, and an attention layer. The target network layer can also be a layer for classification, such as a linear layer. These network layers include matrix multiplication operations, and the matrix multiplication operations of full precision (FP32) consume more computing power, and the corresponding matrix will take up a large storage space, so these network layers are generally network layers that need to be quantized. The full-precision image processing model may include one target network layer, or may include multiple target network layers, which is not limited here. In this application, "multiple" refers to two or more.

[0115] In this embodiment, the data distribution optimization layer is used to optimize the data distribution of the activation value of the input value target network layer. The data distribution optimization layer can simplify the data distribution, reduce the learning difficulty of the quantization algorithm, and make the quantization model easier to optimize. The data distribution optimization layer is, for example, a BN layer, an instance normalization (IN) layer, or a layer normalization (LN) layer. Of course, the data distribution optimization layer can also customize a module that can achieve data distribution optimization.

[0116] The data distribution compression layer is used to compress the data distribution of the activation value of the input value target network layer. The data distribution compression layer can truncate the data distribution range, thereby compressing the data distribution, reducing the quantization representation granularity, and improving the upper limit of the quantization algorithm capability. The data distribution compression module can specifically be an activation function layer such as ReLU, LeakyReLU or HardSwish layer. Of course, other modules that can truncate the data distribution range and compress the data distribution can also be used.

[0117] In a possible implementation, the full-precision image processing model can be constructed and trained to obtain a model for processing underlying visual tasks. When constructing the full-precision image processing model, a data distribution optimization layer and / or a data distribution compression layer are added to the input side of each target network layer. The input side of the target network layer may include only a data distribution optimization layer or only a data distribution compression layer. The data distribution optimization layer can make the activation value of the input target network layer more regular, and the data distribution compression layer can reduce the data distribution range of the activation value of the input target network layer, and both can reduce the complexity of the distribution of the activation value of the input target network layer, so that when the target network layer is quantized, the performance loss of the quantized image processing model can be reduced. Of course, the input side of the target network layer may include a data distribution optimization layer and a data distribution compression layer. The data distribution optimization layer may be located on the input side of the data distribution compression layer, that is, the data distribution optimization layer outputs the optimized activation value to the data distribution compression layer, and the data distribution compression layer then truncates the data distribution of the activation value and then outputs it to the target network layer, so that the data distribution can be retained as much as possible to ensure the performance of the quantized image processing model. Alternatively, the data distribution compression layer is located on the input side of the data distribution optimization layer, that is, the data distribution compression layer compresses the data distribution of the activation value and outputs it to the data distribution optimization layer, and the data distribution optimization layer then optimizes the data distribution of the activation value input to the data distribution compression layer, and then outputs it to the target network layer. It is also possible to compress and optimize the data distribution, making the distribution of the activation value input to the target network layer more regular and compact, and reducing the difficulty of quantization of the target network layer. It can be understood that being located on the input side of the target network layer does not mean that the data distribution optimization layer / data distribution compression layer can be located at any position upstream of the target network layer, but means that the distribution optimization layer / data distribution compression layer is located adjacent to the target network layer and can affect the data distribution of the activation value input to the target network layer.

[0118] In another possible implementation, the full-precision image processing model can be obtained by inserting a data distribution optimization layer and / or a data distribution compression layer on the input side of the target network layer in the existing original underlying visual processing model. The original underlying visual processing model can be a pre-trained underlying visual processing model or a constructed model framework for processing underlying visual tasks. The original underlying visual processing model can be traversed to determine the target network layer therein, and then the data distribution optimization layer and / or the data distribution compression layer can be inserted on the input side of the target network layer. For example, when the target network layer includes a convolutional layer, the convolutional layer is searched in the original underlying visual model, and the data distribution optimization layer and / or the data distribution compression layer is inserted on the input side of the convolutional layer. When the target network layer includes an attention layer, the attention layer is searched in the original underlying visual model, and the data distribution optimization layer and / or the data distribution compression layer is inserted on the input side of the attention layer. The target network layer can include at least two different types of network layers, for example, the target network layer includes at least two of a convolutional layer, an attention layer, a deconvolution layer, and a linear layer. There are multiple possible scenarios for inserting a data distribution optimization layer and / or a data distribution compression layer into the original underlying visual model, which are described below.

[0119] 1. Insert the data distribution optimization layer only on the input side of the target network layer in the original underlying vision model

[0120] This situation can be divided into two cases, the input side of the target network layer includes an activation layer (hereinafter, the target network layer whose input side includes an activation layer is sometimes referred to as the second target network layer), and the input side of the target network layer does not include an activation layer (hereinafter, the target network layer whose input side does not include an activation layer is sometimes referred to as the first target network layer).

[0121] 1.1 The input side of the target network layer does not include the activation layer

[0122] For example, Figure 6a As shown, Figure 6a A schematic diagram of the local structure of a bottom-level visual processing model provided in this application. When the original bottom-level visual model includes multiple first target network layers (target network layer 1 and target network layer 2), a data distribution optimization layer is inserted at the input side of each first target network layer. For example, data distribution optimization layer 1 is inserted at the input side of target network layer 1, and data distribution optimization layer 2 is inserted at the input side of target network layer 2. It can be understood that Figure 6a In the example, only the original underlying visual processing model including two target network layers 1 and target network layer 2 is taken. The number of target network layers can be 1 or more, and there is no limitation here.

[0123] 1.2 There is an input side of the target network layer including an activation layer

[0124] For example, Figure 6b As shown, Figure 6b A schematic diagram of the local structure of another underlying visual processing model provided for the present application. When there is a second target network layer (target network layer 2) in the original underlying visual processing model, the data distribution optimization layer 2 can be inserted into the input side of the activation layer, that is, the data distribution optimization layer 2 optimizes the data distribution of the activation value to be input into the target network layer 2, and then outputs it to the activation layer. The activation layer activates the activation value input into the data distribution optimization layer 2, and then outputs it to the target network layer 2. The first target network layer (target network layer 1) may also exist in the original underlying visual model. For example, there is often no activation layer on the input side of the target network layer in the first hidden layer. For the target network layer 1, it is sufficient to insert the data distribution optimization layer 1 on the input side of the target network layer 1.

[0125] 2. Insert a data distribution compression layer only on the input side of the target network layer in the original underlying vision model

[0126] 2.1 The input side of the target network layer does not include the activation layer

[0127] For example, Figure 6c As shown, Figure 6c A schematic diagram of the partial structure of another underlying visual processing model provided by the present application. When the original underlying visual model includes multiple first target network layers (target network layer 1 and target network layer 2), a data distribution compression layer is inserted at the input side of each first target network layer. For example, data distribution compression layer 1 is inserted at the input side of target network layer 1, and data distribution compression layer 2 is inserted at the input side of target network layer 2.

[0128] 2.2 The input side of the target network layer includes the activation layer

[0129] The data distribution compression layer in this embodiment may include activation functions such as ReLU, LeakyReLU or HardSwish, and ReLU, LeakyReLU or HardSwish themselves may also be activation functions in the activation layer. In this application, the activation function layer that already exists in the original underlying visual processing model is called the activation layer, and the activation function layer such as ReLU, LeakyReLU or HardSwish that is subsequently inserted into the original underlying visual processing model is called the data distribution compression layer. The activation layer is generally located on the output side of the convolutional layer, etc., and is used to activate the output of the convolutional layer, etc., while the data distribution compression layer of this application is located on the input side of the target network layer such as the convolutional layer, and is used to compress the distribution range of the activation value of the input target network layer. The activation layer located between the two network layers processes the output of the previous network layer and then inputs the value to the next network layer. When the activation function in the activation layer meets the requirements, the activation layer in the original underlying visual processing model can be retained without replacing it, so that the activation layer also plays the role of the data distribution compression layer. When the activation function in the activation layer does not meet the requirements, the activation layer is replaced with the data distribution compression layer. The activation function meets the requirements if it has the function of truncation and compression. For example, if the activation function in the activation layer is ReLU, LeakyReLU or HardSwish, it is considered to meet the requirements. If the activation function in the activation layer is tanh, sigmoid or other activation functions that do not have the function of truncation and compression, then the activation layer needs to be replaced with a data distribution compression layer. Of course, when the activation function in the activation layer is inconsistent with the activation function in the data distribution compression layer, the activation layer can also be replaced with a data distribution compression layer. When the original underlying visual processing model includes the first target network layer, it is necessary to insert a data distribution compression layer on the input side of the first target network layer.

[0130] For example, Figure 6d As shown, Figure 6d A schematic diagram of the local structure of another underlying visual processing model provided for the present application. When there is a second target network layer (target network layer 2) in the original underlying visual processing model, and the activation layer on the output side of the second target network layer meets the requirements, the activation layer on the input side of the second target network layer can be retained, and the data distribution compression layer is not inserted into the input side of the target network layer. When there is a second target network layer in the original underlying visual processing model, and the activation layer on the output side of the second target network layer does not meet the requirements, the activation layer is replaced with the data distribution compression layer 2. The first target network layer may also exist in the original underlying visual model. For example, the input side of the target network layer in the first hidden layer often does not have an activation layer. For the first target network layer (target network layer 1), it is sufficient to insert the data distribution compression layer 1 on the input side of the first target network layer.

[0131] 3. Insert the data distribution optimization layer and data distribution compression layer on the input side of the target network layer in the original underlying visual processing model

[0132] The following describes two scenarios: the input side of the target network layer includes an activation layer and the input side does not include an activation layer.

[0133] 3.1 The input side of the target network layer does not include the activation layer

[0134] When the input side of the target network layer in the original underlying visual processing model does not include an activation layer, a data distribution optimization layer and a data distribution compression layer can be inserted into the input side of the target network layer. Figure 6e As shown, data distribution optimization layer 1 and data distribution compression layer 1 are inserted at the input side of target network layer 1, and data distribution optimization layer 2 and data distribution compression layer 2 are inserted at the input side of target network layer 2. In one possible implementation, the data distribution optimization layer can be located at the input side of the data distribution compression layer, and the data distribution compression layer is located between the data distribution optimization layer and the target network layer. In another implementation, the data distribution compression layer can be located at the input side of the data distribution optimization layer, and the data distribution optimization layer is located between the data distribution compression layer and the target network layer.

[0135] 3.2 The input side of the target network layer includes the activation layer

[0136] like Figure 6f As shown, when the input side of the target network layer in the original underlying visual processing model includes an activation layer, that is, when the original underlying visual processing model has a second target network layer (target network layer 2), when the activation layer meets the requirements, the activation layer on the input side of the second target network layer is retained, and the data distribution compression layer is not inserted on the input side of the second target network layer, and only the second data distribution optimization layer (data distribution optimization layer 2) is inserted on the input side of the second target network layer. In this case, the data distribution optimization layer can be located on the input side of the activation layer, that is, the activation layer is located between the data distribution optimization layer and the second target network layer, or it can be located on the output side of the activation layer, that is, the data distribution optimization layer is located between the activation layer and the second target network layer. For the first target network layer (target network layer 1), the first data distribution optimization layer (data distribution optimization layer 1) and the data distribution compression layer are inserted on its input side, and the first data distribution optimization layer can be located between the data distribution optimization layer and the second target network layer, or the data distribution compression layer can be located between the first data distribution optimization layer and the second target network layer.

[0137] like Figure 6gAs shown, when the activation layer does not meet the requirements, the activation layer is removed, and the second data distribution optimization layer (data distribution optimization layer 2) and the second data distribution compression layer (data distribution compression layer 2) are inserted on the input side of the second target network layer (target network layer 2). The second data distribution optimization layer can be located between the second data distribution optimization layer and the second target network layer, or the second data distribution compression layer can be located between the second data distribution optimization layer and the second target network layer. For the first target network layer (target network layer 1), the first data distribution optimization layer (data distribution optimization layer 1) and the first data distribution compression layer (data distribution compression layer 1) are inserted on its input side. The first data distribution optimization layer can be located between the first data distribution compression layer and the second target network layer, or the first data distribution compression layer can be located between the first data distribution optimization layer and the second target network layer.

[0138] like Figure 7 As shown, Figure 7 The data distribution after adding the data distribution optimization layer and data distribution compression layer proposed in this application is shown. Figure 5 In comparison, we can see that the optimized data distribution becomes compact and regular, which greatly reduces the learning difficulty of the quantization algorithm and improves the upper limit of the quantization algorithm's capabilities, achieving a significant improvement in the performance of the final quantized image processing model.

[0139] like Figure 8 As shown, Figure 8 The average quantization error of different network layers is shown. It can be seen that after the introduction of the data distribution optimization layer and the data distribution compression layer proposed in this application, the quantization error can be reduced by 6 to 100 times. In addition, as the number of network layers increases, the quantization error will gradually accumulate. After the introduction of the data distribution optimization layer and the data distribution compression layer proposed in this application, the error expansion range is reduced from 100 times to 5 times. These further demonstrate the effectiveness and superiority of the solution provided by this application.

[0140] The model obtained by inserting the data distribution optimization layer and / or the data distribution compression layer into the original underlying visual processing model is trained to obtain a full-precision image processing model. In the full-precision image processing model, the target network layer, the data distribution optimization layer, and the data distribution compression layer can be independent network layers. In another possible implementation, the target network layer can be folded with the data distribution optimization layer connected to the input side of the target network layer, thereby accelerating the reasoning operation. For example, the convolutional layer and the BN layer on the output side of the convolutional layer can be folded into a convolutional BN layer. For the data distribution optimization layer on the input side of the first target network layer, it is independent of the first target network layer.

[0141] After obtaining the full-precision image processing model, the full-precision image processing model is quantized using a quantization algorithm to obtain a corresponding quantized image processing model. The quantization algorithm may be a quantization-aware quantization algorithm or a post-training quantization algorithm, which is not limited here. The quantized image processing model is used to implement the underlying image processing task. Since the input side of the target network layer in the full-precision image processing model includes a data distribution optimization layer and / or a data distribution compression layer, the distribution of the activation value of the input target network layer is regular and compact, which is conducive to reducing the complexity of the quantization algorithm and reducing the performance loss of the quantized image processing model, thereby ensuring the performance of the quantized image processing model, and thus improving the quality of the image processed by the quantized image processing model. The quantization model includes a quantized network layer corresponding to the target network layer, and the quantized network layer is used to process the first image or the eigenvalue corresponding to the first image. For example, when the quantized network layer is a network layer in the first hidden layer, the quantized network layer processes the first image, such as convolving the first image. When the quantized network layer is a network layer or a linear layer in an attention layer or other hidden layers, the eigenvalue of the first image processed by the quantized network layer is processed. The function of the quantized network layer is similar to that of the corresponding target network layer. For example, if the target network layer is a convolutional layer, the quantized network layer is also a convolutional layer, which performs convolution on the image to extract features in the image. The difference is that the quantized network layer performs computational reasoning based on the quantized weights and activation values, thus having more efficient reasoning efficiency.

[0142] In this embodiment, by setting a data distribution optimization layer and / or a data distribution compression layer on the input side of the target network layer in the full-precision image processing model used to process the underlying visual task, the data distribution of the activation value of the input target network layer can be improved, making the data distribution of the activation value of the input target network layer more compact and regular, thereby reducing the learning difficulty of the quantization algorithm, and improving the upper limit of the ability of the quantization algorithm, achieving the improvement of the performance of the final quantized image processing model, and ensuring the quality of the image processed by the quantized image processing model. Although setting a data distribution optimization layer and / or a data distribution compression layer in the full-precision image processing model will cause performance loss of the full-precision image processing model, it will improve the performance of the quantized image processing model corresponding to the full-precision image processing model, which is beneficial to the quantized image processing model. When the quantized image processing model is used to perform the underlying visual image processing task, a higher quality image can be obtained.

[0143] Different from the current practice of adding a BN layer on the output side of the convolutional layer in the neural network model to accelerate model training, in this embodiment, a data distribution optimization layer and / or a data distribution compression layer is set on the input side of the target network layer, which can simplify the data distribution of the activation value and reduce the quantization error. This not only simplifies the quantization algorithm, but also improves the performance of the quantized visual image processing model, thereby ensuring the quality of the image processed by the quantized visual image processing model.

[0144] The two modules proposed in this application greatly simplify the data distribution, especially in the case of extremely low bit quantization, making the originally highly complex and irregular data distribution compact and regular, which greatly reduces the learning difficulty of the quantization algorithm and significantly improves the final quantization performance. This benefit is not only reflected in the quantization perception training task, but also brings obvious benefits to the post-training quantization task.

[0145] In order to demonstrate the beneficial effects of the solution provided in this application, the classic EDSR network was used for super-resolution tasks, and the data set used was DIV2K. The post-training quantization method DFSQ (Zhong, Yunshan, Mingbao Lin, Jingjing Xie, Yuxin Zhang, Fei Chao, and Rongrong Ji. "Distribution-FlexibleSubset Quantization for Post-Quantizing Super-Resolution Networks." arXivpreprint arXiv:2305.05888.2023) was used as the benchmark algorithm, and 2, 3, and 4-bit post-training quantization experiments were performed on the Enhanced Deep Residual Networks (EDSR) network. BNQ adopts the method proposed in this application, that is, the EDSR network (a super-resolution network, the original underlying visual processing model) is trained after inserting the data distribution optimization layer and the data distribution compression layer proposed in this application to obtain a full-precision EDSR network (full-precision image processing model), and then quantizes it using the post-training quantization method DFSQ to obtain a quantized image processing model, and uses the quantized image processing model to perform quantization experiments in DIV2K. The benchmark is to not insert the data distribution optimization layer and the data distribution compression layer into the EDSR network, use the post-training quantization method DFSQ to quantize the EDSR network to obtain the corresponding quantization model, and use the quantization model to perform quantization experiments in DIV2K. As can be seen from Table 1, by adding the data distribution optimization layer and the data distribution compression layer proposed in this application, there is a very obvious performance improvement at extremely low bit quantization, which further illustrates the effectiveness of the solution provided in this application. Peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are used here to evaluate the quality of the output image of the quantization model.

[0146] Table 1. Performance comparison of this application and the post-training quantization method DFSQ based on the super-resolution network EDSR on the DIV2K dataset

[0147]

[0148] Furthermore, the performance improvement of only setting the data distribution optimization layer (taking the BN layer as an example) on the input side of the target network layer, only setting the data distribution compression layer (taking the ReLU as an example), and setting both the data distribution optimization layer and the data distribution compression layer are verified, and the results are shown in Table 2. It can be seen that the performance improvement of the quantized image processing model is most obvious when both the data distribution optimization layer and the data distribution compression layer are set, and the performance of the quantized image processing model when only the data distribution optimization layer or the data distribution compression layer is set is greatly improved compared with the baseline.

[0149] Table 2. Ablation experiments of data distribution optimization module and data distribution compression module

[0150]

[0151] Tables 3 and 4 show the performance comparison of this application and other quantization-aware training methods based on super-resolution networks EDSR and RDN on the DIV2K dataset. It can be seen that compared with other currently advanced research methods (sota), the quantization performance corresponding to this application is the best, and the advantage is more obvious at extremely low bits, which further proves the effectiveness of the technical solution provided by this application.

[0152] Table 3 Performance comparison of this application and other QAT quantization methods based on super-resolution network EDSR on DIV2K dataset

[0153]

[0154] Table 4. Performance comparison of this application and other QAT quantization methods based on super-resolution network RDN on the DIV2K dataset

[0155]

[0156] Based on the same inventive concept, the present application also provides an image processing device. Fig. 9As shown, the device 900 includes a processing module 901. The processing module 901 is used to obtain a first image. The processing module 901 is also used to obtain a second image according to the first image through a quantized image processing model, the quantized image processing model is a quantized model of a full-precision image processing model, the full-precision image processing model includes a target network layer, and a data distribution optimization layer and / or a data distribution compression layer, the data distribution optimization layer and / or the data distribution compression layer are located at the input side of the target network layer, the data distribution optimization layer is used to optimize the data distribution of the activation value input to the target network layer, the data distribution compression layer is used to compress the data distribution of the activation value input to the target network layer, the quantized model includes a quantized network layer corresponding to the target network layer, and the quantized network layer is used to process the first image or the eigenvalue corresponding to the first image.

[0157] In one possible implementation, the full-precision image processing model includes a data distribution optimization layer and a data distribution compression layer. The data distribution compression layer is located between the data distribution optimization layer and the target network layer. The data distribution compression layer is used to compress the data distribution of the activation value output by the data distribution optimization layer and then output it to the target network layer.

[0158] In one possible implementation, the full-precision image processing model includes a data distribution optimization layer and a data distribution compression layer. The data distribution optimization layer is located between the data distribution compression layer and the target network layer. The data distribution layer is used to optimize the data distribution of the activation value output by the data distribution compression layer and then output it to the target network layer.

[0159] In one possible implementation, the full-precision image processing model includes a first data distribution optimization layer, a second data distribution optimization layer, a first target network layer, a second target network layer and an activation layer. The input side of the first target network layer does not include the activation layer, the input side of the second target network layer includes the activation layer, the activation layer is used to perform nonlinear mapping on the output of the previous network layer and then input it into the second target network layer. The first data distribution optimization layer is located at the input side of the first target network layer, and the second data distribution optimization layer is located at the input side of the activation layer or at the input side of the second target network layer.

[0160] In one possible implementation, the full-precision image processing model includes a first data distribution optimization layer, a second data distribution optimization layer, a data distribution compression layer, a first target network layer, a second target network layer and an activation layer. The input side of the first target network layer does not include the activation layer, and the input side of the second target network layer includes the activation layer. The activation layer is used to perform nonlinear mapping on the output of the previous network layer and then input it into the second target network layer. The first data distribution optimization layer and the data distribution compression layer are located on the input side of the first target network layer, and the second data distribution optimization layer is located on the input side of the activation layer.

[0161] In one possible implementation, the full-precision image processing model includes a data distribution compression layer, a first target network layer, a second target network layer and an activation layer. The input side of the first target network layer does not include the activation layer, and the input side of the second target network layer includes the activation layer. The activation layer is used to perform nonlinear mapping on the output of the previous network layer and then input it into the second target network layer. The data distribution compression layer is located on the input side of the first target network layer.

[0162] In one possible implementation, the target network layer is a convolutional layer, a deconvolutional layer, an attention layer, or a linear layer.

[0163] In a possible implementation, the data distribution compression layer is a ReLU layer, a LeakyReLU layer, or a HardSwish layer.

[0164] In a possible implementation, the data distribution optimization layer is a batch normalization layer, an instance normalization layer, or a layer normalization layer.

[0165] The embodiment of the present application also provides a chip or a chip system, which includes at least one processor and a communication interface, the communication interface and the at least one processor are interconnected through a line, and the at least one processor is used to run a computer program or instruction to perform Figure 4 The image processing method described in any one of any possible implementations of the illustrated embodiments; wherein the communication interface in the chip may be an input / output interface, a pin or a circuit, etc.

[0166] In a possible implementation, the chip or chip system described above in the present application further includes at least one memory, in which instructions are stored. The memory may be a storage unit inside the chip, such as a register, a cache, etc., or a storage unit of the chip (e.g., a read-only memory, a random access memory, etc.).

[0167] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a computer, the image processing method flow of any of the above method embodiments is implemented.

[0168] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0169] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical or other forms.

[0170] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0171] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0172] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the technical solution of the present application can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. An image processing method, characterized in that: The method comprises: acquiring a first image; According to the first image, a second image is obtained by a quantized image processing model, wherein the quantized image processing model is a quantized model of a full-precision image processing model, wherein the full-precision image processing model includes a target network layer, and a data distribution optimization layer and / or a data distribution compression layer, wherein the data distribution optimization layer and / or the data distribution compression layer are located at the input side of the target network layer, wherein the data distribution optimization layer is used to optimize the data distribution of activation values ​​input to the target network layer, wherein the data distribution compression layer is used to compress the data distribution of activation values ​​input to the target network layer, wherein the quantized model includes a quantized network layer corresponding to the target network layer, wherein the quantized network layer is used to process the first image or a eigenvalue corresponding to the first image.

2. The method according to claim 1, characterized in that The full-precision image processing model includes the data distribution optimization layer and the data distribution compression layer, wherein the data distribution compression layer is located between the data distribution optimization layer and the target network layer, and the data distribution compression layer is used to compress the data distribution of the activation value output by the data distribution optimization layer and then output it to the target network layer.

3. The method according to claim 1, characterized in that The full-precision image processing model includes the data distribution optimization layer and the data distribution compression layer. The data distribution optimization layer is located between the data distribution compression layer and the target network layer. The data distribution layer is used to optimize the data distribution of the activation value output by the data distribution compression layer and then output it to the target network layer.

4. The method according to claim 1, characterized in that: The full-precision image processing model includes a first data distribution optimization layer, a second data distribution optimization layer, a first target network layer, a second target network layer and an activation layer. The input side of the first target network layer does not include the activation layer, and the input side of the second target network layer includes the activation layer. The activation layer is used to perform nonlinear mapping on the output of the previous network layer and then input it into the second target network layer. The first data distribution optimization layer is located at the input side of the first target network layer, and the second data distribution optimization layer is located at the input side of the activation layer or at the input side of the second target network layer.

5. The method according to claim 1, characterized in that: The full-precision image processing model includes a first data distribution optimization layer, a second data distribution optimization layer, the data distribution compression layer, a first target network layer, a second target network layer and an activation layer. The input side of the first target network layer does not include the activation layer, and the input side of the second target network layer includes the activation layer. The activation layer is used to perform nonlinear mapping on the output of the previous network layer and then input it into the second target network layer. The first data distribution optimization layer and the data distribution compression layer are located on the input side of the first target network layer, and the second data distribution optimization layer is located on the input side of the activation layer.

6. The method according to claim 1, characterized in that The full-precision image processing model includes a data distribution compression layer, a first target network layer, a second target network layer and an activation layer. The input side of the first target network layer does not include the activation layer, and the input side of the second target network layer includes the activation layer. The activation layer is used to perform nonlinear mapping on the output of the previous network layer and then input it into the second target network layer. The data distribution compression layer is located on the input side of the first target network layer.

7. The method according to any one of claims 1 to 6, characterized in that The target network layer is a convolutional layer, a deconvolutional layer, an attention layer or a linear layer.

8. The method according to any one of claims 1 to 7, characterized in that The data distribution compression layer is a ReLU layer, a LeakyReLU layer or a HardSwish layer.

9. The method according to any one of claims 1 to 8, characterized in that The data distribution optimization layer is a batch normalization layer, an instance normalization layer or a layer normalization layer.

10. An image processing device, characterized in that: The device comprises a processing module, and the processing module is used to execute the image processing method according to any one of claims 1 to 9.

11. A device, characterized in that: The device comprises a processor and a memory, wherein the processor is coupled to the memory, and the processor is configured to execute the image processing method according to any one of claims 1 to 9 based on instructions stored in the memory.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises instructions, which, when the computer-readable storage medium is run on a computer, causes the computer to execute the image processing method according to any one of claims 1 to 9.

Images