Diffusion model extremely low bit quantization method and device based on mixed precision quantization

Abstract

The invention provides a diffusion model extremely low bit quantization method and device based on mixed precision quantization. The method comprises the following steps: acquiring a diffusion model to be subjected to extremely low bit quantization, acquiring initial quantization precision of the extremely low bit quantization, increasing the initial quantization precision to obtain high bit quantization precision, and reducing the initial quantization precision to obtain low bit quantization precision; counting the long tail distribution degree of each group of weight data in the diffusion model; quantizing the at least one group of weight data with the highest long tail distribution degree according to the high-bit quantization precision, quantizing the at least one group of weight data with the lowest long tail distribution degree according to the low-bit quantization precision, and quantizing the other weight data of the diffusion model according to the initial quantization precision, obtaining a quantitative model; and inputting a noise image and the text vector meeting the noise reduction requirement into the quantization model, and carrying out noise reduction on the noise image by the quantization model according to the text vector to obtain a denoised image meeting the noise reduction requirement.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an ultra-low bit quantization method, device, electronic device, computer-readable storage medium and computer program product of a diffusion model based on mixed precision quantization. Background Art

[0002] Diffusion models have shown excellent capabilities in generation tasks. However, the large number of parameters in the diffusion model network and the iterative denoising inference process seriously hinder its widespread deployment on edge devices with limited computing resources. Model quantization, as an effective model compression method, maps floating-point parameters to low-bit integers, which can reduce the model size while improving the inference speed. Specifically, for a floating-point vector x f , the quantization process can be expressed as:

[0003]

[0004] where x q Represents a quantized integer vector, round is a rounding operation, and clip is a truncation of the value to [0,2 N -1] operation. s and z are scaling parameters and translation parameters represented by floating point numbers, respectively, and l and u are the lower and upper bounds of the quantization threshold, respectively. q Restored to full precision representation, the dequantization process is

[0005]

[0006] in, is the vector used in the actual reasoning process.

[0007] The existing quantization method for the diffusion model proposes to add a low-rank representation module to perform quantization-aware low-rank updates on the model quantization weights to enhance the performance of the quantized diffusion model. And considering that the diffusion model has an iterative denoising process with different time steps, the existing method assigns different quantization parameters (scaling parameter s and translation parameter z) to the activation values ​​at different time steps. This method achieves good quantization performance within a very short calibration time of several hours.

[0008] Although existing quantization methods for diffusion models achieve amazing performance under high-bit (4-8bit) settings for the model's activation values ​​and weights, it still suffers from significant performance degradation under extremely low-bit (2-4bit) quantization. Existing methods only consider a basic quantization fine-tuning framework and assign different quantization parameters to different time steps. This ensures the performance of the diffusion model under basic quantization conditions. However, under extremely low-bit quantization, the extremely discrete numerical representation (for example, there are only 16 different numerical representations for activation values ​​under a 4-bit setting) leads to serious lack of expressive power, which leads to major performance degradation. Existing methods do not consider making corresponding optimizations for such extreme cases, and therefore cannot guarantee the performance of the model. Summary of the invention

[0009] The purpose of the present invention is to solve the problem of unfriendly quantization of channels with significant outliers in weights in extremely low bit (2-4 bit) quantization. The present invention proposes a low-bit quantization method for a diffusion model based on mixed precision quantization. The present invention allocates more quantization bits to channels with more serious outliers by measuring the significance of outliers in different channels of weights, and determines the overall bit allocation method for different channels of model weights through an efficient search algorithm to achieve mixed precision quantization. This method allocates different quantization bits to different channels of model weights according to the significance of outliers. While ensuring that the total number of quantization bits remains unchanged and without adding any additional parameters, it alleviates the quantization impact of outliers at extremely low bits, and significantly improves the performance of the diffusion model at extremely low bit quantization.

[0010] Specifically, if Figure 2 As shown, the present invention proposes a diffusion model ultra-low bit quantization method based on mixed precision quantization, which includes:

[0011] In an initial step, a diffusion model to be quantized at very low bits is obtained, an initial quantization accuracy of the very low bits is obtained, the initial quantization accuracy is increased to obtain a high-bit quantization accuracy, and the initial quantization accuracy is reduced to obtain a low-bit quantization accuracy;

[0012] The quantization step comprises: counting the long-tail distribution degree of each group of weight data in the diffusion model; quantizing at least one group of weight data with the highest long-tail distribution degree according to the high-bit quantization precision, and quantizing at least one group of weight data with the lowest long-tail distribution degree according to the low-bit quantization precision, and quantizing the remaining weight data of the diffusion model according to the initial quantization precision, to obtain a quantized model;

[0013] The running step inputs the noise image and the text vector required for noise reduction into the quantization model, and the quantization model performs noise reduction on the noise image according to the text vector to obtain a denoised image that meets the noise reduction requirement.

[0014] The ultra-low bit quantization method of the diffusion model based on mixed precision quantization, wherein the quantization step includes: calculating the long-tail distribution degree of the weight data according to the weight distribution of the weight data, and the mean and standard deviation of the weight distribution.

[0015] The ultra-low bit quantization method of the diffusion model based on mixed precision quantization, wherein the quantization step includes: statistically analyzing the weight distribution of the weight data by the following formula:

[0016]

[0017] K i is the long-tail distribution degree of the ith channel of the diffusion model, w i is the weight distribution of the ith channel σ i w i The standard deviation of i w i The mean of .

[0018] The diffusion model ultra-low bit quantization method based on mixed precision quantization, wherein the initial quantization precision is 2 bit, 3 bit or 4 bit.

[0019] like Figure 3 As shown, the present invention proposes a diffusion model ultra-low bit quantization device based on mixed precision quantization, which includes:

[0020] An initial module, obtaining a diffusion model to be extremely low-bit quantized, obtaining an initial quantization accuracy of the extremely low-bit quantization, increasing the initial quantization accuracy to obtain a high-bit quantization accuracy, and reducing the initial quantization accuracy to obtain a low-bit quantization accuracy;

[0021] A quantization module, which counts the long-tail distribution degree of each group of weight data in the diffusion model; quantizes at least one group of weight data with the highest long-tail distribution degree according to the high-bit quantization precision, and quantizes at least one group of weight data with the lowest long-tail distribution degree according to the low-bit quantization precision, and quantizes the remaining weight data of the diffusion model according to the initial quantization precision, to obtain a quantization model;

[0022] The running module inputs the noise image and the text vector required for noise reduction into the quantization model, and the quantization model performs noise reduction on the noise image according to the text vector to obtain a denoised image that meets the noise reduction requirement.

[0023] The ultra-low bit quantization device of the diffusion model based on mixed precision quantization, wherein the quantization module includes: according to the weight distribution of the weight data, and the mean and standard deviation of the weight distribution, the long-tail distribution degree of the weight data is statistically calculated.

[0024] The ultra-low bit quantization device of the diffusion model based on mixed precision quantization, wherein the quantization module comprises: the weight distribution of the weight data is statistically calculated by the following formula:

[0025]

[0026] K i is the long-tail distribution degree of the ith channel of the diffusion model, w i is the weight distribution of the ith channel σ i w i The standard deviation of i w i The mean of .

[0027] The present invention proposes an electronic device, which includes a diffusion model ultra-low bit quantization device based on mixed precision quantization. The electronic device may be connected to an information display device, and the information display device is used to display the denoised image with display parameters and attributes set by the user or through an artificial intelligence model.

[0028] The present invention proposes a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the diffusion model ultra-low bit quantization method based on mixed precision quantization are implemented.

[0029] The present invention proposes a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the diffusion model ultra-low bit quantization method based on mixed precision quantization are implemented.

[0030] It can be seen from the above scheme that the advantages of the present invention are:

[0031] Compared with other existing diffusion model quantization methods, the method of the present invention significantly improves the performance of the diffusion model under extremely low bit quantization while ensuring that the total quantization bit number of the model remains unchanged and without adding any additional parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 Flow chart of an embodiment of the method of the present invention;

[0033] Figure 2 It is the overall flow chart of the method of the present invention;

[0034] Figure 3 It is a module diagram of the device of the present invention;

[0035] Figure 4 This is a schematic diagram of the structure of a first electronic device of the present invention;

[0036] Figure 5This is a schematic diagram of the application environment structure of the first electronic device of the present invention;

[0037] Figure 6 It is a schematic diagram of the structure of a second electronic device of the present invention.

[0038] Reference numerals:

[0039] A-First electronic device;

[0040] B-diffusion model very low bit quantization device;

[0041] C-data acquisition equipment;

[0042] D-information display device;

[0043] 1000 - second electronic device;

[0044] Ⅰ-computational unit;

[0045] II-ROM;

[0046] III-RAM;

[0047] IV-bus;

[0048] V-interface;

[0049] VI - input unit;

[0050] VII-output unit;

[0051] VIII- Storage medium;

[0052] Ⅸ-Communication unit. DETAILED DESCRIPTION

[0053] It should be noted that, in this application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0054] Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0055] The processor described in the present invention is the control center of the electronic device, which can be a processor or a general term for multiple processing elements. For example, it can be one or more central processing units (CPUs), or application specific integrated circuits (ASICs), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).

[0056] Optionally, the processor can perform various functions of the electronic device by running or executing a software program stored in the memory, and calling data stored in the memory.

[0057] In a specific implementation, as an embodiment, the processor may include one or more CPUs. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions). Electronic devices may include: servers, desktop computers, laptops, smart phones, tablet computers, embedded computers, etc., wherein the embedded computers include vehicles and robots, etc.

[0058] The memory is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0059] It should be noted that the structure of the electronic device shown in the drawings of the present invention does not constitute a limitation thereto, and the actual knowledge structure recognition device may include more or fewer components than shown in the drawings, or a combination of certain components, or a different arrangement of components.

[0060] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0061] It should also be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0062] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0063] It should also be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0064] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0065] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0066] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0067] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0068] When the inventors conducted research on extremely low-bit (2-4 bit) quantization of the diffusion model, they found that the performance degradation in the prior art was caused by the extreme discretization of the model weight values. Extreme discretization means that in low-bit quantization, too few values ​​can be used, resulting in excessive loss of the original full-precision dense information after quantization. For example, in 4-bit quantization, there are only 16 values ​​available. In contrast, the original full-precision model uses 32 bits, which is 2^^8=65536 times more than the 4-bit value.

[0069] When the numerical expression is extremely discretized, it is not friendly to weight quantization. And the inventors have found through research that there are outliers in the weights of different channels of the model that deviate significantly from the main numerical distribution to varying degrees. The existence of such outliers causes a large number of outliers to be truncated after quantization or the outliers occupy a certain expression bit, resulting in the main weight distribution not being well expressed. Moreover, the degree of existence of outliers in the weights of different channels of the model is different, and the distribution with very few outliers is often a normal distribution. The quantization of the normal distribution is easy, while the channels with a large number of outliers are not friendly to quantization. Therefore, the present invention solves the problem of outlier unfriendliness in extremely low-bit quantization by allocating different quantization bits to different channels.

[0070] In order to make the above features and effects of the present invention more clearly and understandably described, embodiments are given below and described in detail with reference to the accompanying drawings. This specification discloses one or more embodiments that include the features of the present invention. The disclosed embodiments are only for illustration. The scope of protection of the present invention is not limited to the disclosed embodiments, and the present invention is defined by the attached claims.

[0071] For the weight distribution of the diffusion model, it can usually be considered as a main normal distribution plus a small amount of outlier distribution that deviates from the main distribution. The existence of outliers leads to a significant decrease in the expressiveness of weights under extremely low bit quantization. A small number of outliers is equivalent to adding a small amount of long-tail distribution to the normal distribution, and the kurtosis value in the distribution statistic is used to measure the severity of the long-tail distribution. A more serious long-tail distribution will cause the kurtosis value of the distribution to increase significantly, while a normal distribution with no outlier distribution has the lowest kurtosis value. Specifically, the present invention calculates the kurtosis value of the weight distribution w of each channel of the diffusion model:

[0072]

[0073] Where μ is the mean of w, that is, the weight mean of a channel in the model, and σ is the standard deviation of w. In this way, the kurtosis value K indicates the significance of outliers in the weight distribution of different channels.

[0074] A mixed precision allocation scheme for efficient search of model weights. For a single weight of a diffusion model It has n different channels, and m means the number of weights a channel has. The weight distribution for the i-th channel is The present invention can calculate its kurtosis K i, so we get the kurtosis values ​​of n channels. Next, sort the kurtosis values ​​of these n channels and determine the mixed precision allocation scheme according to the following formula. The following target formula represents: By selecting different channels and the actual number of bits of the channels, minimize the squared mean error between the matrix multiplication of the quantized weights and activations and the matrix multiplication of the weights and activations of the full-precision model. The following restriction formula represents the restriction condition: For quantization with a mean of N bits, there are three actual bits to choose from, namely N-1, N, and N+1 bits. The number of channels of N-1, N, and N+1 bits is n, that is, the actual number of channels. In order to ensure that the mean is N bits, it is necessary to ensure that the number of channels of N-1 bits and N+1 bits is the same.

[0075] Target:

[0076] Limit: C N-1 ={c i |c i =N-1},C N+1 ={c i |c i =N+1},

[0077] |C N-1 |=|C N+1 |,|C N-1 |+| C N|+|C N+1 |=n

[0078] Where N is the target quantization bit, n is the number of channels in the weight, Represents the process of quantization and dequantization. To represent the channel-by-channel quantization of the weight w, where c i represents the quantization bit of the i-th channel. C represents the set of channels with common quantization bits, and |C| represents the number of channels in the set. For example, for 3-bit quantization, we specify some channels with higher kurtosis K values ​​as 4 bits, and specify the same number of channels with lower kurtosis K values ​​as 2 bits, and specify the remaining channels to maintain 3 bits. In this way, the final average quantization bit is still 3 bits, and no additional parameters are added.

[0079] In order to speed up our optimization search process, we set k channels as a group during the search, so that for |C N+1 The search limit for | is Our empirical setup This way, only 5 searches are required for each optimization, which is very efficient.

[0080] The overall algorithm flow chart of the present invention is shown in Figure 1 . Figure 1The calibration data is the noise data randomly generated on the specified data set. The reason why the input model needs to be determined is that the matrix multiplication of activation and weight needs to be calculated in the mixed precision allocation algorithm mentioned above. The activation needs to be input into the model to obtain it.

[0081] For the calibration process of diffusion model quantization, existing work has demonstrated the effectiveness of its calibration method. Therefore, we follow the calibration process of existing work and use our mixed precision allocation scheme to determine the bit allocation scheme of weights before calibration. In order to verify the effectiveness of the present invention, we use the ImageNet and LSUN-Bedrooms datasets and the LatentDiffusion Model architecture to conduct image generation experiments. We compare the method of the present invention with the unified precision quantization of existing work, and name it mixed precision. In order to verify the effectiveness of the quantization of the present invention at extremely low bits (2-4bit), we set the weights and activations to 3 / 6, 3 / 4, 2 / 6 and 2 / 4bit respectively. We use the FID and sFID indicators to measure the performance of the diffusion model after quantization, and additionally use the IS indicator to report the performance on the ImageNet dataset. These three indicators are common indicators for measuring the performance of diffusion models, among which the lower the FID and sFID indicators, the better, and the higher the IS indicator, the better.

[0082] The experimental results of the inventors in Table 1 and Table 2 below show that the mixed precision quantization method of the present invention is significantly better than the existing unified precision quantization method under various extremely low-bit experimental settings, proving the effectiveness of the present invention.

[0083]

[0084] Table 1 Comparative experimental results on the ImageNet dataset

[0085]

[0086] Table 2 Comparative experimental results on the LSUN-Bedrooms dataset

[0087] The following is a system embodiment corresponding to the above method embodiment. This embodiment can be implemented in conjunction with the above embodiment. The relevant technical details mentioned in the above embodiment are still valid in this embodiment. In order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the above embodiment.

[0088] like Figure 3 As shown, the present invention proposes a diffusion model ultra-low bit quantization device based on mixed precision quantization, which includes:

[0089] An initial module, obtaining a diffusion model to be extremely low-bit quantized, obtaining an initial quantization accuracy of the extremely low-bit quantization, increasing the initial quantization accuracy to obtain a high-bit quantization accuracy, and reducing the initial quantization accuracy to obtain a low-bit quantization accuracy;

[0090] A quantization module, which counts the long-tail distribution degree of each group of weight data in the diffusion model; quantizes at least one group of weight data with the highest long-tail distribution degree according to the high-bit quantization precision, and quantizes at least one group of weight data with the lowest long-tail distribution degree according to the low-bit quantization precision, and quantizes the remaining weight data of the diffusion model according to the initial quantization precision, to obtain a quantization model;

[0091] The running module inputs the noise image and the text vector required for noise reduction into the quantization model, and the quantization model performs noise reduction on the noise image according to the text vector to obtain a denoised image that meets the noise reduction requirement.

[0092] The ultra-low bit quantization device of the diffusion model based on mixed precision quantization, wherein the quantization module includes: according to the weight distribution of the weight data, and the mean and standard deviation of the weight distribution, the long-tail distribution degree of the weight data is statistically calculated.

[0093] The ultra-low bit quantization device of the diffusion model based on mixed precision quantization, wherein the quantization module comprises: the weight distribution of the weight data is statistically calculated by the following formula:

[0094]

[0095] K i is the long-tail distribution degree of the ith channel of the diffusion model, w i is the weight distribution of the i-th channel σ i w i The standard deviation of i w i The mean of .

[0096] like Figure 4 As shown, the present invention also proposes a first electronic device A in another embodiment, including the diffusion model ultra-low bit quantization device based on mixed precision quantization

[0097] like Figure 5 As shown, the first electronic device A can also be connected to the data acquisition device C and the information display device D through a wired or wireless information transmission scheme, the data acquisition device C is used to collect noise images, and the information display device D is used to display the noise reduction image obtained by the analysis of the present invention.

[0098] The information display device D can sort and process the data output by the first electronic device A based on the information display mechanism to improve the readability of the data output by the first electronic device A. The information display mechanism can be manually preset, for example, the data output by the first electronic device A is visually displayed, which can be based on the display parameters and / or attributes set by the user. The display parameters can be, for example, the display data range, and the display attributes can be, for example, the display font, color, whether to scroll and play, etc. The user is presented with the key information specified by the user, and the user can understand the information more promptly without having to access the secondary page or scroll the page, saving the user's operation. Or the information display mechanism can be an artificial intelligence AI display model, which can learn the user's key information based on the user's previous usage habits, such as viewing time, number of clicks, number of edits, etc., and then automatically present rich and necessary key information to the user.

[0099] The present invention also provides a computer program product, which includes a computer program. The computer program can be stored in a readable storage medium. When the computer program is executed by a processor, the computer can execute the diffusion model ultra-low bit quantization method provided by the above methods.

[0100] The present invention also proposes a storage medium VIII in another embodiment for storing a computer program for executing the diffusion model ultra-low bit quantization method. It should be understood that the storage medium in the embodiment of the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DRRAM).

[0101] Figure 6 A schematic block diagram of a second electronic device 1000 that can be used to implement an embodiment of the present invention is shown. The second electronic device 1000 electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The second electronic device 1000 can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples, and are not intended to limit the implementation of the present invention described and / or required herein. The second electronic device 1000 may be the same or different from the first electronic device A.

[0102] The second electronic device 1000 includes a computing unit I, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory II (ROM) or a computer program loaded from a storage medium VIII into a random access memory (RAM) III. In RAM III, various programs and data required for the operation of the device 1000 can also be stored. The computing unit I, ROM II, and RAM III are connected to each other via a bus IV. An input / output (I / O) interface V is also connected to the bus IV.

[0103] A plurality of components in the second electronic device 1000 are connected to the I / O interface V, including: an input unit VI, such as a keyboard, a mouse, etc.; an output unit VII, such as various types of displays, speakers, etc.; a storage medium VIII, such as a disk, an optical disk, etc.; and a communication unit IX, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit IX allows the second electronic device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0104] The computing unit I may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of computing unit I include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit I performs the various methods and processes described above, such as method steps S1-S3. For example, in some embodiments, the method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage medium VIII. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via ROM II and / or a communication unit IX. When the computer program is loaded into RAM III and executed by the computing unit I, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the computing unit I may be configured to execute the method in any other appropriate manner (e.g., by means of firmware).

[0105] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation modes, and they can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. A diffusion model ultra-low bit quantization method based on mixed precision quantization, characterized in that: include: In an initial step, a diffusion model to be quantized at very low bits is obtained, an initial quantization accuracy of the very low bits is obtained, the initial quantization accuracy is increased to obtain a high-bit quantization accuracy, and the initial quantization accuracy is reduced to obtain a low-bit quantization accuracy; Quantification step, counting the long-tail distribution degree of each group of weight data in the diffusion model; quantizing at least one group of weight data with the highest long-tail distribution degree according to the high-bit quantization precision, quantizing at least one group of weight data with the lowest long-tail distribution degree according to the low-bit quantization precision, and quantizing the remaining weight data of the diffusion model according to the initial quantization precision, to obtain a quantization model; The running step inputs the noise image and the text vector required for noise reduction into the quantization model, and the quantization model performs noise reduction on the noise image according to the text vector to obtain a denoised image that meets the noise reduction requirement.

2. The diffusion model ultra-low bit quantization method based on mixed precision quantization according to claim 1, characterized in that: The quantization step includes: calculating the long-tail distribution degree of the weight data according to the weight distribution of the weight data, and the mean and standard deviation of the weight distribution.

3. The diffusion model ultra-low bit quantization method based on mixed precision quantization according to claim 1, characterized in that: The quantification step includes: calculating the weight distribution of the weight data by the following formula: K i is the long-tail distribution degree of the ith channel of the diffusion model, w i is the weight distribution of the ith channel σ i w i The standard deviation of i w i The mean of .

4. The diffusion model ultra-low bit quantization method based on mixed precision quantization according to claim 1, characterized in that: The initial quantization accuracy is 2 bits, 3 bits, or 4 bits.

5. A diffusion model ultra-low bit quantization device based on mixed precision quantization, characterized in that: include: An initial module, obtaining a diffusion model to be extremely low-bit quantized, obtaining an initial quantization accuracy of the extremely low-bit quantization, increasing the initial quantization accuracy to obtain a high-bit quantization accuracy, and reducing the initial quantization accuracy to obtain a low-bit quantization accuracy; The quantification module counts the long-tail distribution of each group of weight data in the diffusion model; quantizing at least one group of weight data with the highest long-tail distribution degree according to the high-bit quantization precision, quantizing at least one group of weight data with the lowest long-tail distribution degree according to the low-bit quantization precision, and quantizing the remaining weight data of the diffusion model according to the initial quantization precision, to obtain a quantization model; The running module inputs the noise image and the text vector required for noise reduction into the quantization model, and the quantization model performs noise reduction on the noise image according to the text vector to obtain a denoised image that meets the noise reduction requirement.

6. The diffusion model ultra-low bit quantization device based on mixed precision quantization according to claim 5, characterized in that: The quantization module includes: calculating the long-tail distribution degree of the weight data according to the weight distribution of the weight data, and the mean and standard deviation of the weight distribution.

7. The diffusion model ultra-low bit quantization device based on mixed precision quantization according to claim 5, characterized in that: The quantification module includes: calculating the weight distribution of the weight data by the following formula: K i is the long-tail distribution degree of the ith channel of the diffusion model, w i is the weight distribution of the ith channel σ i w i The standard deviation of i w i The mean of .

8. An electronic device, characterized in that: It includes an ultra-low bit quantization device of a diffusion model based on mixed precision quantization as described in claims 5-7, and the electronic device is connected to an information display device, which is used to display the denoised image with display parameters and attributes set by the user or through an artificial intelligence model.

9. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the diffusion model ultra-low bit quantization method based on mixed precision quantization according to any one of claims 1 to 4 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the diffusion model ultra-low bit quantization method based on mixed precision quantization described in any one of claims 1 to 4 are implemented.

Images