Diffusion model low-bit quantization method and device based on hybrid precision quantization

Abstract

The application provides a diffusion model extremely low bit quantization method and device based on mixed precision quantization, comprising: obtaining a diffusion model to be extremely low bit quantized, obtaining an initial quantization precision of the extremely low bit quantization, increasing the initial quantization precision to obtain a high bit quantization precision, and reducing the initial quantization precision to obtain a low bit quantization precision; the long tail distribution degree of each group of weight data in the diffusion model is counted; at least one group of weight data with the highest long tail distribution degree is quantized according to the high bit quantization precision, at least one group of weight data with the lowest long tail distribution degree is quantized according to the low bit quantization precision, and the remaining weight data of the diffusion model is quantized according to the initial quantization precision to obtain a quantization model; a noise image and a text vector of a noise reduction requirement are input into the quantization model, and the quantization model performs noise reduction on the noise image according to the text vector to obtain a denoised image meeting the noise reduction requirement.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a diffusion model extremely low bit quantization method and device based on mixed precision quantization, electronic equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] Diffusion models have shown excellent ability in generation tasks. However, the large number of parameters in the diffusion model network and the iterative denoising inference process seriously hinder its wide deployment on edge devices with limited computing resources. Model quantization, as an effective model compression method, maps floating-point parameters to low-bit integers, which can reduce the model size while improving inference speed. Specifically, for a floating-point number represented vector x f , the quantization process can be represented as:

[0003]

[0004] where x q represents the quantized integer vector, round is the rounding operation, and clip is the operation of truncating the numerical value to [0, 2 N -1]. s and z are the scaling parameter and the translation parameter of the floating-point number representation, respectively, and l and u represent the lower bound and the upper bound of the quantization threshold, respectively. Conversely, in order to restore the low-bit integer quantized vector x q to full-precision representation, the dequantization process is

[0005]

[0006] where, is the vector used in the actual inference process.

[0007] Existing quantization methods for diffusion models propose to add a low-rank representation module to perform low-rank update of the quantization weight of the model to enhance the performance of the quantized diffusion model. And considering that the diffusion model has an iterative denoising process at different time steps, the existing method assigns different quantization parameters (scaling parameter s and translation parameter z) to the activation values at different time steps. This method achieves good quantization performance in a very short calibration time of a few hours.

[0008] Although the existing quantization method for diffusion model achieves amazing performance in the setting of high-bit (4-8bit) activation value and weight, it still has a significant performance decline in the setting of extremely low-bit (2-4bit) quantization. The existing method only considers the basic quantization fine-tuning framework and assigns different quantization parameters to different time steps. This guarantees the performance of the diffusion model under the basic quantization condition. However, under the setting of extremely low-bit quantization, the extremely discrete numerical representation (for example, there are only 16 different numerical representations of activation value in the setting of 4bit) will cause serious lack of expression ability, which leads to major performance decline. The existing method does not consider corresponding optimization under this extreme condition, so it cannot guarantee the performance of the model. SUMMARY

[0009] The purpose of the present application is to solve the problem of channel quantization being unfriendly to significant outliers in the weight in the setting of extremely low-bit (2-4bit) quantization. The present application proposes a low-bit quantization method for diffusion model based on mixed precision quantization. The present application measures the significant degree of the existence of outliers in different channels of the weight, assigns more quantization bits to the channels with more serious outliers, and determines the overall bit allocation method for different channels of the model weight through an efficient search algorithm to realize mixed precision quantization. This method alleviates the quantization impact of outliers under the setting of extremely low-bit by assigning different quantization bits to different channels of the model weight according to the significant degree of the existence of outliers, without increasing any additional parameters, and significantly improves the performance of the diffusion model under the setting of extremely low-bit quantization.

[0010] Specifically, as shown in Figure 2 The present application proposes a low-bit quantization method for diffusion model based on mixed precision quantization, which includes:

[0011] In the initial step, a diffusion model to be quantized in extremely low-bit is obtained, an initial quantization precision for the extremely low-bit quantization is obtained, a high-bit quantization precision is obtained by increasing the initial quantization precision, and a low-bit quantization precision is obtained by reducing the initial quantization precision;

[0012] In the quantization step, the long-tail distribution degree of each group of weight data in the diffusion model is counted; at least one group of weight data with the highest long-tail distribution degree is quantized according to the high-bit quantization precision, at least one group of weight data with the lowest long-tail distribution degree is quantized according to the low-bit quantization precision, and the remaining weight data of the diffusion model is quantized according to the initial quantization precision to obtain a quantized model;

[0013] In the running step, a noisy image and a text vector of denoising requirement are input into the quantized model, and the quantized model denoises the noisy image according to the text vector to obtain a denoised image meeting the denoising requirement.

[0014] The ultra-low bit quantization method of the diffusion model based on mixed precision quantization, wherein the quantization step includes: calculating the long-tail distribution degree of the weight data according to the weight distribution of the weight data, and the mean and standard deviation of the weight distribution.

[0015] The ultra-low bit quantization method of the diffusion model based on mixed precision quantization, wherein the quantization step includes: calculating the weight distribution of the weight data by the following formula:

[0016]

[0017] K i is the long-tail distribution degree of the i-th channel of the diffusion model, w i is the weight distribution of the i-th channel σ i w i The standard deviation of i w i The mean of .

[0018] The ultra-low bit quantization method of the diffusion model based on mixed precision quantization, wherein the initial quantization precision is 2 bit, 3 bit or 4 bit.

[0019] like Figure 3 As shown, the present invention proposes a diffusion model ultra-low bit quantization device based on mixed precision quantization, which includes:

[0020] An initial module, which obtains a diffusion model to be quantized at very low bits, obtains an initial quantization precision of the very low bits, increases the initial quantization precision to obtain a high-bit quantization precision, and decreases the initial quantization precision to obtain a low-bit quantization precision;

[0021] a quantization module, which counts the long-tail distribution degree of each group of weight data in the diffusion model; quantizes at least one group of weight data with the highest long-tail distribution degree according to the high-bit quantization precision, quantizes at least one group of weight data with the lowest long-tail distribution degree according to the low-bit quantization precision, and quantizes the remaining weight data of the diffusion model according to the initial quantization precision, to obtain a quantization model;

[0022] The running module inputs the noise image and the text vector required for noise reduction into the quantization model, and the quantization model performs noise reduction on the noise image according to the text vector to obtain a denoised image that meets the noise reduction requirement.

[0023] The ultra-low bit quantization device of the diffusion model based on mixed precision quantization, wherein the quantization module includes: calculating the long-tail distribution degree of the weight data according to the weight distribution of the weight data, and the mean and standard deviation of the weight distribution.

[0024] The diffusion model based on hybrid precision quantization low-bit quantization device, wherein the quantization module comprises: calculating the weight distribution of the weight data by the following formula:

[0025]

[0026] K i is the long tail distribution degree of the i-th channel of the diffusion model, w i is the weight distribution of the i-th channel σ i is the standard deviation of w i , μ i is the mean of w i .

[0027] The present application provides an electronic device comprising the diffusion model based on hybrid precision quantization low-bit quantization device, and the electronic device is connected with an information display device for displaying the denoised image with user-set display parameters, attributes or through an artificial intelligence model.

[0028] The present application provides a computer readable storage medium storing a computer program, wherein the computer program is executed by a processor to realize the steps of the diffusion model based on hybrid precision quantization low-bit quantization method.

[0029] The present application provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to realize the steps of the diffusion model based on hybrid precision quantization low-bit quantization method.

[0030] From the above scheme, the present application has the following advantages:

[0031] Compared with other existing diffusion model quantization methods, the method of the present application significantly improves the performance of the diffusion model under low-bit quantization without increasing any additional parameters while ensuring the total quantization bit number of the model. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 is the flow chart of the method embodiment of the present application;

[0033] Figure 2 is the overall flow chart of the method of the present application;

[0034] Figure 3 is the module diagram of the device of the present application;

[0035] Figure 4 is the structural schematic diagram of the first electronic device of the present application;

[0036] Figure 5Figure 1 is a schematic diagram of a first electronic device application environment structure according to the present application;

[0037] Figure 6 Figure 2 is a schematic diagram of a second electronic device structure according to the present application.

[0038] Reference numerals:

[0039] A - first electronic device;

[0040] B - diffusion model extremely low bit quantization device;

[0041] C - data acquisition device;

[0042] D - information display device;

[0043] 1000 - second electronic device;

[0044] I - calculation unit;

[0045] II - ROM;

[0046] III - RAM;

[0047] IV - bus;

[0048] V - interface;

[0049] VI - input unit;

[0050] VII - output unit;

[0051] VIII - storage medium;

[0052] IX - communication unit. DETAILED DESCRIPTION

[0053] It should be noted that the relational terms herein, such as first and second, and the like, are used solely to distinguish one from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0054] Without further limitation, an element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or apparatus including the element.

[0055] The processor described in the present invention is the control center of an electronic device and can be a single processor or a collective term for multiple processing elements. For example, it can be one or more central processing units (CPUs), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs) or one or more field programmable gate arrays (FPGAs).

[0056] Optionally, the processor can perform various functions of the electronic device by running or executing a software program stored in the memory, and calling data stored in the memory.

[0057] In a specific implementation, as an embodiment, the processor may include one or more CPUs. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). Electronic devices may include: servers, desktop computers, laptops, smartphones, tablet computers, embedded computers, etc., wherein the embedded computers include vehicles and robots, etc.

[0058] The memory is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0059] It should be noted that the structure of the electronic device shown in the drawings of the present invention does not constitute a limitation thereto, and the actual knowledge structure recognition device may include more or fewer components than shown in the drawings, or a combination of certain components, or a different arrangement of components.

[0060] The above-described embodiments can be implemented in whole or in part by software, hardware (e.g., circuitry), firmware, or any combination thereof. When implemented in software, the above-described embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center through wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. containing one or more available medium collections. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. The semiconductor medium can be a solid state disk.

[0061] It should also be understood that the term "and / or" used herein is merely an association relationship between the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. In addition, the character " / " herein generally represents an "or" relationship between the associated objects before and after it, but it can also represent an "and / or" relationship, which can be understood in the context before and after it.

[0062] In the present application, "at least one" means one or more, and "multiple" means two or more. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0063] It should also be understood that in various embodiments of the present application, the size of the sequence number of the above-described processes does not mean the order of execution, and the execution order of the processes should be determined by their functions and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0064] In several embodiments provided by the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other manners. For example, the embodiments of the apparatus described above are merely schematic. For example, the division of the units is only a logical function division. There can be another division manner for the actual implementation. For example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different units, can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0065] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0066] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically, or two or more units can be integrated into one unit.

[0067] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0068] The inventors found that the performance degradation in the prior art is caused by the extreme discretization of model weight values when conducting research on the extreme low bit (2-4bit) quantization of the diffusion model. The extreme discretization refers to that in the low bit quantization, the values available are too few, resulting in that the original dense information of full precision is lost too seriously after quantization. For example, in 4-bit quantization, there are only 16 values available. In contrast, the original full-precision model uses 32 bits, which is 2^^8=65536 times more than the 4-bit values.

[0069] In the numerical expression of extreme discretization, the weight quantization is not friendly. And the inventors found that there are different degrees of outliers deviating from the main numerical distribution in the weights of different channels of the model. The existence of such outliers leads to a large number of outliers being truncated after quantization or the outliers occupying a certain expression bit, so that the main weight distribution cannot be well expressed. And the existence degree of outliers in the weights of different channels of the model is different, and the distribution with few outliers is often a normal distribution. The quantization of the normal distribution is easy, and the channel with a large number of outliers is not friendly to quantization. Therefore, the present application solves the problem of unfriendliness to outliers in extremely low bit quantization by assigning different quantization bits to different channels.

[0070] In order to make the above features and effects of the present application more clear and easy to understand, the following embodiments are specifically described below, and the detailed description is made in conjunction with the accompanying drawings. The present application discloses one or more embodiments comprising the features of the present application. The disclosed embodiments are only used for illustration. The protection scope of the present application is not limited to the disclosed embodiments, and the present application is defined by the appended claims.

[0071] For the weight distribution of the diffusion model, it can be generally considered as a main normal distribution plus a small number of outlier distributions deviating from the main distribution. The existence of outliers leads to a significant decrease in the expression ability of weights under extremely low bit quantization. A small part of outliers is equivalent to adding a small amount of long-tailed distribution to the normal distribution, and the kurtosis value in the distribution statistics is used to measure the severity of the long-tailed distribution. The more severe long-tailed distribution will lead to a significant increase in the kurtosis value of the distribution, and the normal distribution without abnormal value distribution has the lowest kurtosis value. Specifically, the present application calculates the kurtosis value of the weight distribution w of each channel of the diffusion model:

[0072]

[0073] Where μ is the mean of w, i.e. the weight mean of a certain channel of the model, and σ is the standard deviation of w. In this way, the kurtosis value K represents the significant degree of outliers in the weight distribution of different channels.

[0074] High-efficiency search model weight mixed precision allocation scheme. For a single weight of the diffusion model It has n different channels, and m means that the channel has a certain number of weights. For the weight distribution of the i-th channel The present application can calculate the kurtosis K of the weight distribution of the i-th channel i, we get the kurtosis value of n channels. Then, sort the kurtosis value of n channels, and determine the mixing precision allocation scheme according to the following formula, and the following objective formula represents: by selecting different channels and the actual number of bits of the channel, the mean square error between the matrix multiplication of quantized weights and activations and the matrix multiplication of full-precision model weights and activations is minimized. The following constraint formula represents the constraint condition: for the quantization with a mean of N bits, there are three actual bits available, namely N-1, N, and N+1 bits. The number of channels of N-1, N, and N+1 bits is n, that is, the actual number of channels. In order to ensure that the mean is N bits, it is necessary to ensure that the number of channels of N-1 bits and N+1 bits is the same.

[0075] Objective:

[0076] Constraint: C N-1 = {c i |c i = N-1}, C N+1 = {c i |c i = N+1},

[0077] |C N-1 | = |C N+1 |, |C N-1 | + | C N| + |C N+1 | = n

[0078] where N is the target quantization bit, n is the number of channels in the weight, represents the quantization and dequantization process. Use to represent the channel-by-channel quantization of the weight w, where c i represents the quantization bit of the i-th channel. C represents a set of channels with the same quantization bit, and |C| represents the number of channels in the set. For example, for 3-bit quantization, we specify that part of the channels with higher kurtosis K value are 4 bits, and specify that the same number of channels with lower kurtosis K value are 2 bits, and specify that the remaining channels remain 3 bits. In this way, the final average quantization bit is still 3 bits, and no additional parameters are added.

[0079] In order to speed up our optimization search process, we set k channels as a group when searching, so that the search limit of |C N+1 | is We empirically set In this way, only 5 searches are needed for each optimization, which is very efficient.

[0080] The total algorithm flowchart of the present application is shown in Figure 1 . Figure 1The middle calibration data is random noise data generated on the specified data set. The input model needs to be determined because the matrix multiplication of the activation and the weight needs to be calculated in the mixed precision allocation algorithm mentioned above. Among them, the input of the model is needed to obtain the activation.

[0081] For the calibration process of the diffusion model quantization, the existing work has proved the effectiveness of its calibration method. Therefore, we follow the calibration process of the existing work and use our mixed precision allocation scheme to determine the bit allocation scheme of the weight before calibration. In order to verify the effectiveness of the present application, we use the ImageNet and LSUN-Bedrooms data sets, the LatentDiffusion Model architecture to conduct image generation experiments. We compare the uniform precision quantization of the existing work, and the method of the present application is named mixed precision. In order to verify the effectiveness of the present application at very low bits (2-4bit), we set the weight and activation to 3 / 6, 3 / 4, 2 / 6 and 2 / 4bit respectively. We use FID and sFID indicators to measure the performance of the diffusion model after quantization, and additionally use IS indicators to report the performance on the ImageNet dataset. The three indicators are common indicators for measuring the performance of diffusion models, among which the lower the FID and sFID indicators, the better, and the higher the IS indicator, the better.

[0082] The experimental results of the inventors in Table 1 and Table 2 below show that the mixed precision quantization method of the present application is significantly better than the existing uniform precision quantization method under various very low bit experimental settings, proving the effectiveness of the present application.

[0083]

[0084] Table 1 Comparative experimental results on the ImageNet dataset

[0085]

[0086] Table 2 Comparative experimental results on the LSUN-Bedrooms dataset

[0087] The following is a system embodiment corresponding to the above method embodiment. The present embodiment can be implemented in cooperation with the above-mentioned embodiments. The related technical details mentioned in the above-mentioned embodiments are still valid in the present embodiment. In order to reduce repetition, they will not be repeated here. Correspondingly, the related technical details mentioned in the present embodiment can also be applied in the above-mentioned embodiments.

[0088] As Figure 3 shown, the present application proposes a diffusion model very low bit quantization device based on mixed precision quantization, which includes:

[0089] An initial module obtains a diffusion model to be quantized at a very low bit rate, obtains an initial quantization precision for the very low bit quantization, increases the initial quantization precision to obtain a high bit quantization precision, and decreases the initial quantization precision to obtain a low bit quantization precision;

[0090] A quantization module calculates a long tail distribution degree of each group of weight data in the diffusion model, quantizes at least one group of weight data with the highest long tail distribution degree according to the high bit quantization precision, quantizes at least one group of weight data with the lowest long tail distribution degree according to the low bit quantization precision, and quantizes the remaining weight data of the diffusion model according to the initial quantization precision to obtain a quantized model.

[0091] A running module inputs a noise image and a text vector of a noise reduction requirement into the quantized model, and the quantized model performs noise reduction on the noise image according to the text vector to obtain a de-noised image meeting the noise reduction requirement.

[0092] The diffusion model very low bit quantization device based on mixed precision quantization, wherein the quantization module comprises: calculating the long tail distribution degree of the weight data according to the weight distribution of the weight data and the mean and standard deviation of the weight distribution.

[0093] The diffusion model very low bit quantization device based on mixed precision quantization, wherein the quantization module comprises: calculating the weight distribution of the weight data by the following formula:

[0094]

[0095] K i is the long tail distribution degree of the i th channel of the diffusion model, w i is the weight distribution of the i th channel σ i is the standard deviation of w i , μ i is the mean of w i .

[0096] As shown in Figure 4 , the present application further proposes a first electronic device A comprising the diffusion model very low bit quantization device based on mixed precision quantization

[0097] As shown in Figure 5 , the first electronic device A can be connected to a data acquisition device C and an information display device D through a wired or wireless information transmission scheme, the data acquisition device C is used to acquire a noise image, and the information display device D is used to display the de-noised image obtained by the analysis of the present application.

[0098] The information display device D can process the data output by the first electronic device A based on an information display mechanism to improve the readability of the data output by the first electronic device A. The information display mechanism can be manually preset, for example, the data output by the first electronic device A is visually displayed according to the display parameters and / or attributes set by the user, for example, the display parameters can be the display data range, and the display attributes can be the display font, color, whether to scroll and play, etc. The user can be presented with the information he specified, and the user can understand these information more timely without accessing the secondary page or scrolling the page, saving the user's operation. Or the information display mechanism can be an artificial intelligence AI display model, which can learn the user's key attention information according to the user's previous use habits, such as viewing time, click times, editing times, etc., and then automatically present the user with rich and necessary key information.

[0099] The application further provides a computer program product, which comprises a computer program, the computer program can be stored on a readable storage medium, and the computer program can be executed by a processor to enable the computer to execute the diffusion model low-bit quantization method provided by the above-mentioned method.

[0100] The present application also proposes, in another embodiment, a storage medium VIII for storing a computer program for performing the method of the very low bit quantization of the diffusion model. It should be understood that the storage medium in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DRRAM).

[0101] Figure 6 A schematic block diagram of a second electronic device 1000 that can be used to implement embodiments of the present application is shown. The second electronic device 1000 is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The second electronic device 1000 can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the inventiveness described and / or claimed in this document. The second electronic device 1000 can be the same as or different from the first electronic device A.

[0102] The second electronic device 1000 includes a computing unit I that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory II (ROM) or a computer program loaded into a random access memory (RAM) III from a storage medium VIII. In the RAM III, various programs and data required for the operation of the device 1000 can also be stored. The computing unit I, the ROM II, and the RAM III are connected to each other through a bus IV. An input / output (I / O) interface V is also connected to the bus IV.

[0103] A plurality of components in the second electronic device 1000 are connected to the I / O interface V, including an input unit VI such as a keyboard, a mouse, and the like, an output unit VII such as various types of displays, a speaker, and the like, a storage medium VIII such as a magnetic disk, an optical disk, and the like, and a communication unit IX such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit IX allows the second electronic device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0104] The computing unit I can be various general and / or special-purpose processing components having processing and computing capabilities. Some examples of the computing unit I include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, and the like. The computing unit I performs various methods and processes described above, such as the method steps S1-S3. For example, in some embodiments, the methods can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage medium VIII. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1000 via the ROM II and / or the communication unit IX. When the computer program is loaded into the RAM III and executed by the computing unit I, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, the computing unit I can be configured to perform the methods by any other appropriate means, such as by means of firmware.

[0105] While the embodiments of the present application have been disclosed as above, they are not limited to only the applications listed in the specification and the embodiments, and can be fully applied to various fields suitable for the present application, and additional modifications can be easily made by those skilled in the art, and thus the present application is not limited to specific details and the figures shown and described herein, without departing from the general concept defined by the claims and the equivalent scope.

Claims

1. A diffusion model ultra-low bit quantization method based on mixed precision quantization, characterized in that: include: In an initial step, a diffusion model to be quantized at very low bits is obtained, an initial quantization precision of the very low bits is obtained, the initial quantization precision is increased to obtain a high-bit quantization precision, and the initial quantization precision is decreased to obtain a low-bit quantization precision; Quantification step, statistics the long tail distribution degree of each group of weight data in the diffusion model; quantizing at least one set of weight data with the highest degree of long-tail distribution according to the high-bit quantization precision, quantizing at least one set of weight data with the lowest degree of long-tail distribution according to the low-bit quantization precision, and quantizing the remaining weight data of the diffusion model according to the initial quantization precision, to obtain a quantized model; The running step includes inputting the noise image and the text vector required for noise reduction into the quantization model, and the quantization model performs noise reduction on the noise image according to the text vector to obtain a denoised image that meets the noise reduction requirement; The quantification step includes: calculating the long-tail distribution degree of the weight data according to the weight distribution of the weight data, and the mean and standard deviation of the weight distribution; The quantification step includes: calculating the weight distribution of the weight data by the following formula: is the long-tail distribution degree of the i-th channel of the diffusion model, is the weight distribution of the i-th channel , for The standard deviation of for The mean of .

2. The diffusion model ultra-low bit quantization method based on mixed precision quantization according to claim 1, characterized in that: The initial quantization accuracy is 2 bits, 3 bits, or 4 bits.

3. A diffusion model ultra-low bit quantization device based on mixed precision quantization, characterized in that: include: An initial module, which obtains a diffusion model to be quantized at very low bits, obtains an initial quantization precision of the very low bits, increases the initial quantization precision to obtain a high-bit quantization precision, and decreases the initial quantization precision to obtain a low-bit quantization precision; The quantification module counts the long-tail distribution of each group of weight data in the diffusion model; quantizing at least one set of weight data with the highest degree of long-tail distribution according to the high-bit quantization precision, quantizing at least one set of weight data with the lowest degree of long-tail distribution according to the low-bit quantization precision, and quantizing the remaining weight data of the diffusion model according to the initial quantization precision, to obtain a quantized model; The running module inputs the noisy image and the text vector required for noise reduction into the quantization model, and the quantization model performs noise reduction on the noisy image according to the text vector to obtain a denoised image that meets the noise reduction requirement; The quantification module includes: calculating the long-tail distribution degree of the weight data according to the weight distribution of the weight data, and the mean and standard deviation of the weight distribution; The quantification module includes: calculating the weight distribution of the weight data by the following formula: is the long-tail distribution degree of the i-th channel of the diffusion model, is the weight distribution of the i-th channel , for The standard deviation of for The mean of .

4. An electronic device, characterized in that: It includes the ultra-low bit quantization device of the diffusion model based on mixed precision quantization as described in claim 3, and the electronic device is connected to an information display device, which is used to display the denoised image with display parameters and attributes set by the user or through an artificial intelligence model.

5. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the ultra-low bit quantization method of the diffusion model based on mixed precision quantization according to claim 1 or 2.

6. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the diffusion model ultra-low bit quantization method based on mixed precision quantization according to claim 1 or 2 are implemented.

Images