Processing method and computer readable storage medium for mixed precision quantized data

Abstract

This application provides a method for processing mixed-precision quantized data and a computer-readable storage medium. One method includes: acquiring precision allocation information and a first-precision quantization value set, wherein the precision allocation information assigns a first precision or a second precision type to each position in a sequentially arranged plurality of positions, and each first-precision quantization value in the first-precision quantization value set corresponds one-to-one with each position assigned the first precision in the precision allocation information; and generating a merged data representation based on the precision allocation information and the first-precision quantization value set. This method utilizes specific values ​​as identifiers to directly embed mask information into the quantized data, thereby eliminating the need for storing independent mask matrices, simplifying hardware implementation, and improving memory access efficiency.

Description

Technical Field

[0001] This disclosure relates to the fields of artificial intelligence and computer systems, and more specifically, to methods for processing mixed-precision quantized data and computer-readable storage media. Background Technology

[0002] As the scale of deep learning models continues to expand, model compression and acceleration techniques are crucial for deploying models on real-world hardware, such as mobile devices and edge computing chips. Mixed-precision quantization is an effective model compression technique that uses different quantization precipitates for elements of varying importance within a tensor. For example, higher precision can be retained for important elements while lower precision is used for less important elements, thereby significantly reducing storage overhead and computational cost while maintaining model accuracy.

[0003] However, existing mixed-precision quantization schemes face a common challenge during inference deployment: they require an additional mask matrix to record the precision type used for each element. This mask matrix typically needs to be stored separately. For a weight matrix of dimension [M, N], the mask itself requires an additional M×N bits of storage space. In large-scale models, this overhead is considerable. Furthermore, during inference, the hardware needs to read the quantized data matrix and the mask matrix separately, and then determine how to process each data element based on the mask values. This separate storage and access method leads to additional memory bandwidth consumption, more complex control logic, and higher hardware implementation complexity, which is detrimental to the design of efficient, low-power dedicated acceleration chips.

[0004] Therefore, there is a need for enhanced mixed-precision quantization data representation and processing methods. Summary of the Invention

[0005] In one aspect, this application discloses a method for processing mixed-precision quantized data, comprising: acquiring precision allocation information and a first precision quantization value set, wherein the precision allocation information assigns a first precision or a second precision type to each of a plurality of sequentially arranged positions, and each first precision quantization value in the first precision quantization value set corresponds one-to-one with each position assigned to the first precision in the precision allocation information; and generating a merged data representation based on the precision allocation information and the first precision quantization value set, comprising: for positions assigned to the first precision, filling in a corresponding first precision quantization value from the first precision quantization value set at the corresponding position in the merged data representation; and for positions assigned to the second precision, filling in a predetermined identifier value at the corresponding position in the merged data representation; wherein the identifier value is a value reserved from the value range of the first precision quantization values ​​in the first precision quantization value set specifically for identifying positions of the second precision.

[0006] In other respects, this application discloses a non-transitory computer-readable medium storing instructions and a computer program product including instructions. These instructions, when executed by one or more processors, cause the processors to perform the methods described in this application. Attached Figure Description

[0007] When read in conjunction with the accompanying drawings, various aspects of this disclosure are best understood through the following detailed description. It should be noted that, in accordance with standard practice in the art, the features are not drawn to scale. In fact, for clarity of discussion, the dimensions of the features may be arbitrarily increased or decreased.

[0008] Figure 1 An example of a traditional mixed-precision quantization scheme is shown.

[0009] Figure 2 An example flowchart of a method for processing mixed-precision quantized data according to an embodiment of the present disclosure is shown.

[0010] Figure 3 An example of processing mixed-precision quantization data according to an embodiment of this disclosure is shown.

[0011] Figure 4 An example of encoding for mixed-precision quantized data is shown according to an embodiment of this disclosure.

[0012] Figure 5 An example of decoding mixed-precision quantized data according to an embodiment of this disclosure is shown.

[0013] Figure 6 This is a schematic diagram of a computing device that can be implemented according to embodiments of the present disclosure. Detailed Implementation

[0014] The following disclosure provides numerous different embodiments or examples for implementing various features of the provided subject matter. Specific examples of components and arrangements are described below to simplify this disclosure. Of course, these are merely examples and not limiting.

[0015] Furthermore, the various operations will be described as multiple discrete operations in a manner most conducive to understanding the illustrative embodiments; however, the order of description should not be construed as implying that these operations must depend on the order. In particular, these operations do not need to be performed in the order presented.

[0016] The phrases “in an embodiment,” “in one embodiment,” and “in some embodiments” are used repeatedly throughout this document. These phrases do not typically refer to the same embodiment; however, they may refer to the same embodiment. Unless the context otherwise specifies, the terms “comprising,” “having,” and “including” are synonyms. The phrases “A or B,” “A and / or B,” and “A / B” mean “(A), (B), or (A and B).”

[0017] Mixed-precision quantization, as an effective model compression technique, significantly reduces computational and storage overhead while maintaining model accuracy by retaining important elements in the tensor as high precision (e.g., INT8) and compressing the remaining elements to low precision (e.g., INT4). During mixed-precision inference, a mask is needed to distinguish the positions of high-precision and low-precision elements.

[0018] In traditional mixed-precision quantization schemes, for a tensor of dimension U×V, the system needs to maintain and store two completely independent matrices: the first is the quantization data matrix. The first is a U×V integer matrix used to store the specific values ​​of all elements after INT8 quantization; the second is a mask matrix. The Q matrix is ​​a U×V binary matrix where each bit (0 or 1) is specifically used to identify the precision type of the corresponding element in the Q matrix (e.g., 1 represents high precision INT8, 0 represents low precision INT4, and vice versa). This scheme physically separates the data from the metadata (mask), meaning the Q matrix and the Mask matrix are stored as two independent data blocks in different locations in memory. During inference calculations, the hardware first performs two independent memory read operations sequentially to obtain the complete contents of the Q and Mask matrices respectively. Then, the control logic needs to align and synchronize the data streams of these two matrices and parse each bit in the Mask matrix one by one to ultimately determine whether the value at each position in the Q matrix is ​​high precision or low precision.

[0019] Figure 1 An example of a traditional mixed-precision quantization scheme is shown. For example... Figure 1As shown, the quantization data matrix Q and the mask matrix Mask are 4×4 matrices, both of which need to be stored in memory for inference computation. The Q matrix requires 4×4 bytes of storage, and the Mask matrix requires 4×4 bits of storage. Each bit (0 or 1) in the Mask matrix is ​​specifically used to identify the precision type of the corresponding element in the Q matrix. In this example, 1 represents high precision INT8, and 0 represents low precision INT4. During inference computation, the hardware first performs two independent memory read operations sequentially to obtain the complete contents of Q and Mask respectively. Then, the control logic needs to align and synchronize the data streams of the two matrices and parse each bit in the Mask one by one to finally determine whether the value at each position in the Q matrix is ​​high precision or low precision. For example, a low precision INT4 value is stored as an INT8 value in the Q matrix in a packaged manner; that is, two INT4 values ​​can be packaged and stored as one INT8 value. For example, the first INT4 value is placed in the high 4 bits of a byte, and the second INT4 value is placed in the low 4 bits of the same byte. Alternatively, for example, you could place the INT4 value in the high 4 bits of a byte and fill the low 4 bits of that byte with placeholders (e.g., 0000), or vice versa.

[0020] Existing mask storage schemes suffer from the following problems. First, they require additional storage overhead. Traditional schemes require separate storage of the mask matrix (e.g., a 1-bit matrix) to identify whether each element is high-precision or low-precision. For large-scale models, the mask matrix itself occupies considerable storage space. For example, for a matrix of dimension [M, N], M×N bits of mask data need to be stored additionally. Second, they require additional memory access overhead. During inference, the hardware needs to read the quantized data and the mask matrix separately, and then determine how to process each element based on the mask. Storing the mask matrix requires additional memory bandwidth consumption, and the alignment and synchronization of the mask and data increases the complexity of the control logic. Third, the hardware implementation is complex. Dedicated acceleration chips require the design of additional mask reading paths and parsing logic, increasing chip area and power consumption.

[0021] To address the aforementioned issues, this disclosure proposes a mixed-precision quantization data processing method based on specific value encoding. This method utilizes specific values ​​as identifiers to directly embed mask information, which typically requires separate storage in traditional techniques, into the quantized data. This eliminates the need for storing independent mask matrices, simplifies hardware implementation, and improves memory access efficiency.

[0022] Figure 2 An example flowchart of a processing method 200 for mixed-precision quantization data according to an embodiment of the present disclosure is shown. Figure 2The method 200 includes operations 210 and 220. The method 200 can be performed by a system or device for mixed-precision quantization, such as an artificial intelligence (AI) processor or a system-on-a-chip (SoC) containing an AI acceleration module (e.g., a general-purpose processor (CPU), a graphics processing unit (GPU) or a general-purpose computing GPU, a dedicated AI processor or a neural network processing unit (NPU), etc.).

[0023] At position 210, obtain the precision allocation information and the first precision quantization value set. The precision allocation information is used to assign a first precision or second precision type to each of the multiple sequentially arranged positions. Each first precision quantization value in the first precision quantization value set corresponds one-to-one with each position assigned the first precision in the precision allocation information.

[0024] At position 220, a merged data representation is generated based on the precision allocation information and the first precision quantization value set. This operation may include: for positions assigned to the first precision, filling the corresponding first precision quantization value from the first precision quantization value set into the corresponding position in the merged data representation; and for positions assigned to the second precision, filling the corresponding position in the merged data representation with a predetermined identifier value. This identifier value is a value specifically reserved from the value range of the first precision quantization values ​​in the first precision quantization value set to identify the position assigned to the second precision.

[0025] In some embodiments, the precision of the first precision (e.g., bit width, etc.) is higher than that of the second precision. In one example, the first precision is INT8 and the second precision is INT4. In another example, the first precision is INT16 and the second precision is INT8. In yet another example, the first precision is INT4 and the second precision is INT2. The first precision and the second precision can also be other options, and this disclosure is not limiting in this regard. Additionally or alternatively, in some embodiments, the number of positions assigned to the first precision by the precision allocation information is greater than the number of positions assigned to the second precision. That is, if the number of elements with higher precision (e.g., bit width) is greater than the number of elements with lower precision, then a specific value in the value range of the higher precision can be used to identify the elements with lower precision. However, the embodiments of this application are not limited to this; for example, if the number of elements with higher precision is greater than the number of elements with lower precision, a specific value in the value range of the lower precision can also be used to identify the elements with higher precision.

[0026] In some embodiments, the precision of the first precision (e.g., bit width, etc.) is lower than that of the second precision. In one example, the first precision is INT4 and the second precision is INT8. In another example, the first precision is INT8 and the second precision is INT16. In yet another example, the first precision is INT2 and the second precision is INT4. The first precision and the second precision can also be other options, and this disclosure is not limiting in this regard. Additionally or alternatively, in some embodiments, the number of positions assigned to the first precision by the precision allocation information is less than the number of positions assigned to the second precision. That is, if the number of elements with higher precision (e.g., bit width) is less than the number of elements with lower precision, then a specific value in the value range of lower precision can be used to identify the elements with higher precision. However, embodiments of this application are not limited to this; for example, if the number of elements with higher precision is less than the number of elements with lower precision, a specific value in the value range of higher precision can also be used to identify the elements with lower precision.

[0027] In some embodiments, the identifier value is the largest absolute value in the range of the first precision quantization value. For example, when the first precision is an 8-bit signed integer precision, the identifier value is -128. As another example, when the first precision is a 4-bit signed integer precision, the identifier value is -8. In other embodiments, the identifier value can be other values ​​in the range of the first precision quantization value, and this disclosure does not limit this.

[0028] Figure 3 An example of processing mixed-precision quantized data according to an embodiment of this disclosure is shown. Figure 3 In the example, the generated merged data representation can be a merge matrix Q', which includes 4×4 INT8 quantization values. As shown in the figure, -128 in Q' indicates a low-precision position, and the other values ​​represent the corresponding INT8 quantization values. In the embodiments of this disclosure, the merged data representation is not limited to the form of a matrix (two-dimensional, three-dimensional, or even higher-dimensional), and can also be other representations, which are not limited in this disclosure.

[0029] As can be seen, the mixed-precision quantization data processing method based on specific value encoding provided in this embodiment uses specific values ​​as identifiers to directly embed mask information, which usually needs to be stored separately in traditional technologies, into the quantization data, thereby eliminating the need for storing independent mask matrices.

[0030] Figure 4 An example of encoding for mixed-precision quantized data is shown according to an embodiment of this disclosure. Figure 4 The example uses a 1×N matrix for illustration, but the technical solution disclosed herein is applicable to matrices of any dimension. As shown in the figure, for position... First-precision quantization value set (6 high-precision elements, 2 low-precision elements). Where 0 represents low precision and 1 represents high precision. When When =1, merged data representation ;when When =0, merged data representation In other words, in, .

[0031] exist Figure 4 In the example, the first precision is INT8, and the second precision is INT4. However, as mentioned above, this is merely an example used to illustrate the technical solution of this disclosure and is not intended to impose any limitations. Furthermore, Figure 4 The example uses the largest absolute value in the INT8 range as the identifier to indicate the position of the second precision. However, as mentioned above, this is just an example, and other values ​​can also be used to indicate the position of the second precision.

[0032] In some embodiments, for the encoding process, a first set of precision quantization values ​​and precision allocation information are input to generate a merged data representation.

[0033] The process of generating the merged data representation does not involve second-precision quantization values, but the set of second-precision quantization values ​​and the merged data representation can be stored during the storage process.

[0034] In some embodiments, the processing method for mixed-precision quantized data may further include: obtaining a second-precision quantized value set; and storing the merged data representation and the second-precision quantized value set separately. That is, the merged data representation and the second-precision quantized value set may be stored separately in memory. The second-precision quantized value set may include second-precision elements stored contiguously.

[0035] In some embodiments, the processing method for mixed-precision quantized data may further include: acquiring a second-precision quantized value set; and storing the merged data representation and the second-precision quantized value set in the same storage block in a block-interleaved manner. This storage method is suitable for streaming reads. For example, within each Bank, high-precision identifiers and low-precision data are stored interleaved.

[0036] In some embodiments, the processing method for mixed-precision quantized data may further include a decoding operation. The decoding operation may include: reading the merged data representation; distinguishing the precision type of each position by comparing the value at each position in the merged data representation with an identifier value; and based on the distinction result, directly outputting the value as the corresponding first-precision quantization value for positions whose values ​​are not equal to the identifier value; and for positions whose values ​​are equal to the identifier value, obtaining the corresponding second-precision quantization value from a second-precision quantization value storage area associated with the merged data representation.

[0037] Figure 5 An example of decoding mixed-precision quantized data according to an embodiment of this disclosure is shown. For ease of illustration, this example is related to... Figure 4 The provided encoding examples correspond accordingly. As shown in the figure, for position... ,if This indicates that the element at that position is the corresponding first-precision quantization value; that is, for that position, ;if This indicates that the element at that position is a set of second-precision quantized values ​​associated with the merged data representation. The corresponding second-precision quantization value In other words, for this position, its value is This second-precision quantization set Retrieved from memory (e.g., separate storage or interleaved storage compared to merged data representation).

[0038] The merged data representation provides not only the first-precision quantization value set but also precision allocation information. Therefore, for the decoding process, based on the stored merged data representation and second-precision quantization value set, the first-precision quantization value set, the second-precision quantization value set, and the precision allocation information can be obtained. For example, during decoding, using the merged data representation and the second-precision quantization value set as output, the output can be the first-precision quantization value set, the second-precision quantization value set, and the precision allocation information; alternatively, the output can be a hybrid precision quantization value set combining the first-precision quantization value set, the second-precision quantization value set, and the precision allocation information. Figure 5 The output of the decoding process shown can be a set of mixed-precision quantization values ​​Q''.

[0039] In some embodiments, the decoding logic (e.g., the decoding logic of a dedicated sparse chip) may include:

[0040] Step 1, Read Data. For example, reading an INT8 data stream from memory;

[0041] Step 2: Comparison and detection. For example, perform is_low_precision = (Q' == -128) for each element;

[0042] Step 3: Data routing. For example, if Q' != -128, then use directly. As INT8 data, these INT8 data are sequentially used to form Q1; if Q' == -128, then INT4 data is read from Q2;

[0043] Step 4: Parallel computation. For example, after obtaining Q1 and Q2, they are distributed to computational units of corresponding precision.

[0044] In some embodiments, the decoding logic described above may further include step 2.1 to generate precision allocation information. Data can be routed to computational units of corresponding precision based on the precision allocation information.

[0045] In some embodiments, the decoding operation can be performed by dedicated hardware circuitry, including, for example, a comparator unit and a data routing unit. Since only a simple comparator (e.g., a comparison with -128) is required for the comparison, the hardware design is simplified, thereby reducing chip area and power consumption.

[0046] In some embodiments, the sequentially arranged positions in the above-described processing method for mixed-precision quantization data correspond to the element positions in the weight tensor or activation tensor of the neural network model. That is, the technical solutions provided by the embodiments of this disclosure can be applied to mixed-precision quantization of weights or activations. However, the embodiments of this disclosure are not limited thereto, and can also be applied to mixed-precision quantization of other aspects.

[0047] The embodiments of this disclosure are described below from the aspects of quantization, storage, and inference of neural network models. For ease of description, the following description uses high-precision INT8 and low-precision INT4 as examples, and uses -128 in the value range of high-precision INT8 as the identifier value of the low-precision position. As mentioned above, this is merely an example, and the embodiments of this disclosure are not limited thereto.

[0048] In one embodiment, quantization can be performed offline. The quantization phase may include: performing mixed-precision quantization on weights or activations to determine the positions of high-precision and low-precision elements, i.e., determining precision allocation information. The quantization phase may also include: performing INT8 quantization on high-precision elements, ensuring the quantized values ​​are in the range [-127, 127] (avoiding -128); and performing INT4 quantization on low-precision elements. The quantization phase may also include: generating a merged data representation (e.g., a merged matrix Q'), filling the high-precision positions with INT8 values ​​and the low-precision positions with -128. The quantization phase may also include: storing the low-precision data matrix separately.

[0049] In some embodiments, during the storage phase, the Q' matrix and the low-precision data matrix are written to memory according to a selected storage method. This storage phase does not require storing a separate mask matrix.

[0050] In some embodiments, inference can be performed online. The inference phase may include: reading the merged data representation Q' from memory. The inference phase may also include: hardware automatically detecting the -128 identifier and recovering precision allocation information. The inference phase may further include: reading (e.g., sequentially reading) the corresponding low-precision quantization values ​​from the low-precision data matrix according to the precision allocation information. The inference phase may further include: routing the high-precision quantization values ​​and low-precision quantization values ​​to the corresponding computing units for computation, respectively. The inference phase may further include: combining the computations from the aforementioned computing units for merging.

[0051] The quantization stage, storage stage, and inference stage provided in the above embodiments are merely examples. In other embodiments, each of the quantization stage, storage stage, and inference stage may include more, fewer, or different operations. This disclosure is not limiting in this respect.

[0052] In some embodiments described above, -128 from the value range of high-precision INT8 is used as the identifier value of low-precision INT4. However, as stated above, the embodiments of this disclosure are applicable to any other mixed-precision quantization scheme with different precisions, and this disclosure is not limited in this respect. Furthermore, the numerical value used as the identifier value can be the largest absolute value in the value range, or it can be other values, and this disclosure is not limited in this respect. Additionally, a specific value from the value range of the low-precision quantization value can also be used as the identifier value of the high-precision quantization element, and this disclosure is not limited in this respect. Furthermore, the examples above (e.g., Figure 4 and Figure 5 The embodiments of this disclosure use a 1×N matrix to describe them, but the technical solutions of this disclosure are also applicable to data structures or other data representations of any dimension, such as matrices, tensors, and vectors, and this disclosure makes no limitations in this regard. Furthermore, this disclosure uses bit width (e.g., 8 bits, 4 bits) as a measure of precision, but the technical solutions of this disclosure are also applicable to other measures indicating precision. Precision in this disclosure is a relative concept, referring to the level of detail or information capacity of the data representation. In addition to bit width, this scheme is also applicable to systems that use other metrics or standards to define and differentiate precision, such as, but not limited to: dynamic range (two precisions may have the same bit width but different dynamic ranges); noise level or signal-to-noise ratio (SNR) (in analog computing or certain specific hardware, such as in-memory computing), precision may be defined by the noise level or SNR of the signal); quantization granularity or scaling factor (in fixed-point quantization, even with the same bit width, a smaller scaling factor can achieve finer quantization granularity (higher effective precision); and hold time or stability (in in-memory computing based on non-volatile memory (e.g., RRAM, PCM), the hold time of cell states or the number of repeatable read / write operations can represent a kind of "precision," and this disclosure does not limit this in this respect).

[0053] The technical solution disclosed herein has achieved improvements in multiple dimensions, including storage efficiency, access performance, hardware complexity, and engineering practicality.

[0054] First, it achieves improved storage efficiency. Traditional solutions require an additional mask matrix the same size as the data matrix, while this disclosure achieves "zero additional mask storage" by integrating the mask information with the quantized data itself by embedding special identifier values. Taking a large-scale model like Llama-3-70B as an example, approximately 6MB of space can be saved in mask storage alone, which is especially valuable for edge devices with limited memory resources.

[0055] Second, the memory access process is simplified and optimized. Traditional architectures require two separate memory read operations to obtain quantized data and a mask, while this disclosure only requires a single read to merge the data representation. This not only halves the number of memory accesses but also eliminates the synchronization overhead of data and mask alignment. Evaluations show that this can improve memory bandwidth utilization by approximately 10-15%, thereby alleviating system performance bottlenecks.

[0056] Third, it reduces the design and manufacturing costs of dedicated hardware. Since there is no need to design a separate data path and complex control logic for the mask, the decoding logic of this disclosure can be simplified to a numerical comparator (e.g., a comparison with -128). This minimalist hardware logic effectively reduces the number of transistors and the chip's area, lowering power consumption, thus facilitating integration into AI accelerator chips (ASICs / NPUs) with stringent area and energy efficiency requirements.

[0057] Fourth, this disclosed technical solution maintains quantization accuracy while pursuing high efficiency. The identifier value used (e.g., -128) is derived from an extremum that is rarely used in practice in the symmetric quantization domain, sacrificing only one quantization level (reducing 256 levels to 255 levels), resulting in a negligible increase in quantization error (typically less than 0.1%). Thus, system-level gains are achieved while ensuring that the model inference accuracy is essentially unaffected.

[0058] Fifth, the technical solution disclosed herein possesses engineering compatibility and ease of use. It does not change the quantization algorithm itself, nor does it require modification of the core of existing mixed-precision quantization algorithms. It only requires adding a specific value encoding step in the output stage of the quantization results, which can be seamlessly integrated into existing model deployment toolchains, reducing the cost of technology adoption and migration.

[0059] Figure 6 This is a schematic diagram of a computing device that can be implemented according to embodiments of the present disclosure, and can be used to implement the system or device for mixed-precision quantization described above. For example... Figure 6As shown, computing device 600 may include bus 602 or other communication mechanism for transmitting information, and one or more hardware processors 604 coupled to bus 602 for processing information. The one or more hardware processors 604 may include, for example, one or more general-purpose microprocessors.

[0060] like Figure 6 As shown, in some embodiments, computing device 600 may further include main memory 606 coupled to bus 602. Main memory 606 is used to store information and instructions executed by one or more processors 604, such as random access memory (RAM), cache, and / or other dynamic storage devices. Main memory 606 may also be used to store temporary variables or other intermediate information during the execution of instructions executed by one or more processors 604. These instructions, when stored in storage media accessible to one or more processors 604, can cause computing device 600 to become a dedicated machine customized to perform the operations specified in the instructions. Storage device 608 may include non-volatile and / or volatile storage media. Non-volatile storage media may include, for example, optical discs or magnetic disks. Volatile storage media may include dynamic memory. Common forms of storage media may include, for example, floppy disks, hard disks, solid-state drives, magnetic tape, or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, DRAM, PROM, and EPROM, FLASH-EPROM, NVRAM, any other memory chip or cartridge, or their networking versions.

[0061] like Figure 6 As shown, in some embodiments, computing device 600 may further include one or more communication interfaces or network interfaces 610 coupled to bus 602. Network interface 610 may provide bidirectional data communication coupling to one or more network links connected to one or more networks. As another example, network interface 610 may be a local area network (LAN) card to provide data communication connectivity to a LAN-compatible (or WAN component communicating with a WAN) network. Wireless links may also be implemented.

[0062] Each of the processes, methods, and algorithms described in the preceding sections may be embodied in code modules executed by one or more computer systems or computer processors including computer hardware, and may be fully or partially automated by these code modules. The processes and algorithms may be implemented, partially or fully, in dedicated circuit systems.

[0063] When the functions disclosed herein are implemented as software functional units and sold or used as stand-alone products, they may be stored in a processor-executable, non-volatile, computer-readable storage medium. Specific technical solutions (all or part) disclosed herein, or aspects contributing to the prior art, may be embodied in the form of a software product. The software product may be stored in a storage medium and includes several instructions that cause a computing device (which may be a personal computer, server, network device, etc.) to perform all or some steps of the methods of the embodiments of this application. The storage medium may include a flash drive, portable hard disk drive, ROM, RAM, magnetic disk, optical disk, other media operable to store program code, or any combination thereof.

[0064] Specific embodiments further provide a system including a processor and a non-transitory computer-readable storage medium storing instructions executable by the processor to cause the system to perform operations corresponding to steps in any method of the embodiments disclosed above. Specific embodiments further provide a non-transitory computer-readable storage medium storing instructions executable by one or more processors to cause the one or more processors to perform operations corresponding to steps in any method of the embodiments disclosed above.

[0065] The embodiments disclosed herein can be implemented via a cloud platform, server, or server cluster (collectively referred to below as the "Service System") that interacts with a client. The client can be a terminal device or a client registered by a user at the platform, wherein the terminal device can be a mobile terminal, a personal computer (PC), or any device that can have the platform application installed.

[0066] The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Additionally, certain method or process blocks may be omitted in some embodiments. The methods and processes described herein are not limited to any particular order, and their associated blocks or states may be executed in other suitable orders. For example, described blocks or states may be executed in an order other than that specifically disclosed, or multiple blocks or states may be combined into a single block or state. Example blocks or states may be executed sequentially, in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed example embodiments. The exemplary systems and components described herein may be configured differently than described. For example, components may be added to, removed from, or rearranged compared to the disclosed example embodiments.

[0067] The various operations of the exemplary methods described herein can be performed at least in part by an algorithm. The algorithm may be included in program code or instructions stored in memory (e.g., the aforementioned non-transitory computer-readable storage medium). This algorithm may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not explicitly refer to the computer as performing the function but may learn from training data to generate a predictive model of the function.

[0068] The various operations of the exemplary methods described herein can be performed, at least in part, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, these processors can constitute an engine of processor implementation that operates to perform one or more of the operations or functions described herein.

[0069] Similarly, the methods described herein may be implemented at least in part by a processor, wherein one or more specific processors are instances of hardware. For example, at least some operations of the methods may be performed by one or more processors or an engine implemented by a processor. Furthermore, one or more processors may also be operable to support the execution of relevant operations in a “cloud computing” environment or as the execution of relevant operations in a “Software as a Service” (SaaS) context. For example, at least some operations may be performed by a group of computers (as an example of a machine containing processors), wherein these operations are accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application programming interfaces (APIs)).

[0070] The execution of certain operations can be distributed across processors rather than residing within a single machine, and can be deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0071] Throughout this specification, multiple instances may be implemented as components, operations, or structures described as a single instance. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of these individual operations may be performed simultaneously, and not necessarily in the order illustrated. Structures and functions presented as separate components in the example configuration may be implemented as composite structures or components. Similarly, structures and functions presented as single components may be implemented as single components. These and other variations, modifications, additions, and improvements fall within the scope of this document.

[0072] As used herein, "or" is inclusive rather than exclusive unless explicitly indicated by the context. Therefore, in this document, "A, B, or C" means "A, B, A and B, A and C, B and C, or A, B, and C" unless explicitly indicated by the context. Furthermore, "and" is combined and separate unless explicitly indicated by the context. Therefore, in this document, "A and B" means "A and B, combined or separate" unless explicitly indicated by the context. Additionally, multiple instances of resources, operations, or structures described herein may be provided as a single instance. Furthermore, the boundaries between various resources, operations, engines, and data storage devices are somewhat arbitrary and specific operations are illustrated within the context of a particular illustrative configuration. Other functional assignments are foreseeable and fall within the scope of various embodiments of this disclosure. Generally, structures and functions presented as individual resources in example configurations may be implemented as combined structures or resources. Similarly, structures and functions presented as single resources may be implemented as single resources. These and other changes, modifications, additions, and improvements fall within the scope of the embodiments of this disclosure as expressed in the appended claims. Therefore, this specification and drawings should be considered illustrative rather than restrictive.

[0073] The terms “comprising” or “including” are used to indicate the presence of a subsequently claimed feature, but do not preclude the addition of other features. Unless otherwise specifically stated or otherwise understood in the context in which they are used, conditional language such as “may,” “can,” “may,” and “can” is generally intended to convey that certain embodiments include certain features, components, and / or steps that are not included in other embodiments. Therefore, this conditional language is generally not intended to imply that one or more embodiments require features, components, and / or steps in any way, or that one or more embodiments must include logic for determining whether such features, components, and / or steps are included in or performed in any particular embodiment, with or without user input or prompts.

[0074] Although the general outline of the subject matter has been described with reference to specific exemplary embodiments, various modifications and changes may be made to these embodiments without departing from the broad scope of embodiments of this disclosure. Where more than one embodiment is disclosed, these embodiments of the subject matter may be referred to individually or collectively herein as the term "invention," this is for convenience only and is not intended to automatically limit the scope of this application to any single disclosure or concept.

[0075] The embodiments illustrated herein are described in detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. Therefore, “implementation” is not intended to be limiting, and the scope of the various embodiments is defined only by the appended claims and their full scope.

Claims

1. A method for processing mixed-precision quantized data, characterized in that, The method is used to process weight tensors or activation tensors of a neural network model, wherein the weight tensor or activation tensor includes a plurality of element positions arranged in sequence, and the method includes: Obtain precision allocation information and a first precision quantization value set, wherein the precision allocation information is used to assign a first precision or a second precision type to each element position in the sequentially arranged plurality of element positions, and each first precision quantization value in the first precision quantization value set corresponds one-to-one with each element position assigned the first precision in the precision allocation information; and Based on the precision allocation information and the first precision quantization value set, a merged data representation is generated, including: For each element position assigned to the first precision, the corresponding first precision quantization value from the first precision quantization value set is filled into the corresponding element position in the merged data representation; and For the element position assigned to the second precision, a predetermined identifier value is filled into the corresponding element position in the merged data representation; Wherein, the identifier value is a value reserved from the value range of the first precision quantized values ​​in the first precision quantized value set, specifically used to identify the position of the element assigned to the second precision. The method further includes a decoding operation, which includes: Read the merged data representation; The precision type of each element position is distinguished by comparing the value of each element position in the merged data representation with the identifier value; and Based on the differentiation results, for element positions whose values ​​are not equal to the identifier value, the value is directly output as the corresponding first-precision quantization value; and for element positions whose values ​​are equal to the identifier value, the corresponding second-precision quantization value is obtained from the second-precision quantization value storage area associated with the merged data representation. The decoding operation is performed by a dedicated hardware circuit, which includes a comparator unit and a data routing unit.

2. The method according to claim 1, characterized in that, The accuracy of the first precision is higher than that of the second precision.

3. The method according to claim 2, characterized in that, The number of element positions assigned to the first precision by the precision allocation information is greater than the number of element positions assigned to the second precision.

4. The method according to claim 1, characterized in that, The accuracy of the first precision is lower than that of the second precision.

5. The method according to claim 4, characterized in that, The number of element positions assigned to the first precision by the precision allocation information is less than the number of element positions assigned to the second precision.

6. The method according to any one of claims 1 to 5, characterized in that, The identifier value is the largest absolute value in the range of the first precision quantization value.

7. The method according to claim 6, characterized in that, When the first precision is an 8-bit signed integer precision, the value of the identifier is -128; When the first precision is a 4-bit signed integer precision, the identifier value is -8.

8. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Obtain the set of second-precision quantized values; and The merged data representation is stored separately from the second set of precision quantized values.

9. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Obtain the set of second-precision quantized values; and The merged data representation and the second set of precision quantized values ​​are stored in the same storage block in a block-interleaved manner.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method according to any one of claims 1 to 9.

Images