Control method, related device and readable storage medium

Abstract

The invention relates to the technical field of terminals, in particular to a control method, a related device and a readable storage medium, and the method can comprise the following steps: obtaining to-be-processed data; when the quantization precision of the to-be-processed data is greater than the calculation precision of the calculation unit, performing data splitting on the to-be-processed data to obtain a plurality of sub-data; the quantization precision of each piece of sub-data in the multiple pieces of sub-data is smaller than or equal to the calculation precision of the calculation unit; respectively sending a first memory access instruction for the plurality of sub-data according to a storage rule; the first memory access instruction is used for indicating the internal memory to write the sub-data into a specified row or column. The embodiment of the invention can be compatible with mixed quantization precision calculation, and meanwhile, the circuit area overhead is saved.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a control method, related device and readable storage medium. Background Technology

[0002] With the rise of generative AI such as ChatGPT, deploying large models for AI inference on the edge to improve user experience and productivity is one of the main methods to enhance product competitiveness. Large model inference typically loads all model parameters into memory, so memory capacity often limits the size of models that can be deployed on the edge. Furthermore, large model inference requires moving large amounts of data from memory to processors (such as CPUs, NPUs, and GPUs) for processing, making memory bandwidth a bottleneck for edge inference performance.

[0003] To reduce the hardware resource requirements of large models, a key direction for algorithm optimization is quantization techniques. Quantization reduces model size by quantizing model parameters, thereby lowering the hardware resource demands of large models, accelerating inference speed, and reducing power consumption. However, quantization may lead to a loss of model accuracy. To address this, hybrid quantization precision can be employed, allowing the model to use higher precision for key parameters and lower precision for other parameters, thus reducing the hardware resource requirements of large models while maintaining their capabilities. Furthermore, to reduce data movement during inference, processing in memory (PIM) technology can be used to offload some computations to memory. For example, PIM-DRAM, implemented by integrating independent computing units within a Dynamic Random Access Memory (DRAM) die, can improve computational performance, accelerate large model inference, and reduce power consumption by leveraging the multi-bank concurrency capabilities of DRAM memory, thereby improving the battery life of end products.

[0004] Currently, solutions for PIM-DRAM that are compatible with mixed quantization precision computation require designing computation circuits for different quantization precisions, which may result in the need to reserve more computation circuits and cause additional circuit area overhead.

[0005] Therefore, the industry is exploring how to provide a solution that can be compatible with mixed quantization precision calculations while saving circuit area overhead. Summary of the Invention

[0006] This application provides a control method, related apparatus, and readable storage medium that are compatible with mixed quantization precision calculations while saving circuit area overhead.

[0007] The present application is described below from different aspects. It should be understood that the different implementation methods and beneficial effects described below can be referenced from each other.

[0008] In a first aspect, this application provides a control method applicable to a processor, the processor being coupled to internal memory, the internal memory including at least one storage area Bank and at least one computing unit; each Bank includes M rows * N columns of storage units, and each computing unit has a computing precision of K, where M, N, and K are all greater than 0; the method includes:

[0009] Acquire the data to be processed; the quantization precision of the data to be processed is L;

[0010] When the quantization precision L of the data to be processed is greater than the calculation precision K of the computing unit, the data to be processed is split into multiple sub-data; the quantization precision of each sub-data is less than or equal to the calculation precision K of the computing unit.

[0011] According to the storage rules, a first memory access instruction is sent to each of the multiple sub-data items; the first memory access instruction is used to instruct the internal memory to write the sub-data items into a specified row or column (the specified row or column is different between different first memory access instructions).

[0012] In this embodiment, when the quantization precision of the data to be processed is greater than the computational precision of the computing unit in the in-memory computing memory, the in-memory memory cannot directly perform calculations on the data to be processed through the computing unit. Instead, the data to be processed can be first split into multiple sub-data units, and these sub-data units can be written into different rows or columns of the storage area for storage. The quantization precision of each sub-data unit is less than or equal to the computational precision of the computing unit, allowing the in-memory memory to subsequently perform calculations on each sub-data unit separately. For calculations compatible with mixed quantization precisions, it is only necessary to adjust the data splitting strategy according to different combinations of quantization precision and computational precision. This allows the calculation of multiple quantization precision parameters to be completed using a computing circuit with one computational precision, eliminating the need to design dedicated computing circuits for different quantization precisions, thus saving computing circuit area overhead.

[0013] In one possible implementation, the storage rule specifically includes that the multiple sub-data are written to adjacent rows or columns; or, the Bank includes multiple partitions, and the storage rule specifically includes that the multiple sub-data are written to specified rows or columns located in different partitions; or, the storage rule specifically includes that the multiple sub-data are written to specified rows or columns located in different Banks.

[0014] In this embodiment, the split sub-data can be stored in adjacent rows or columns, facilitating subsequent calculations by using a single calculation instruction to compute data from adjacent rows or columns. This is suitable for scenarios requiring calculations of a small number of rows or columns. Alternatively, the split sub-data can be stored in rows or columns of multiple independent partitions, allowing subsequent calculations by using a single calculation instruction to compute data from multiple partitions. This is suitable for scenarios requiring calculations of a large number of rows or columns. Furthermore, the split sub-data can be stored in rows or columns of different banks, enabling flexible data storage configuration. During subsequent data computation, sub-data from different banks can be computed in parallel, resulting in higher efficiency.

[0015] In one possible implementation, the maximum amount of data that can be stored in the specified row or column is an integer multiple of the quantization precision of the sub-data.

[0016] In this embodiment, the maximum amount of data that can be stored in a row or column storing sub-data is an integer multiple of the quantization precision of the sub-data. For example, the maximum amount of data that can be stored is 256 bits, and the quantization precision of the sub-data can be 1 bit, 2 bits, 4 bits, etc., to ensure that the storage resources of the row or column can be fully utilized.

[0017] In one possible implementation, the method further includes:

[0018] Send a second memory access instruction; the second memory access instruction is used to instruct the internal memory to perform calculations on the plurality of sub-data.

[0019] In this embodiment of the application, after instructing the internal memory to store the data to be processed, the processor can instruct the internal memory to perform calculations on the data to be processed using the internal computing unit through a memory access instruction.

[0020] In one possible implementation, the second memory access instruction is also used to indicate the quantization precision L of the data to be processed.

[0021] In this embodiment of the application, the processor can enable the internal memory to sense the quantization precision of the data to be processed through memory access instructions.

[0022] In one possible implementation, the internal memory further includes a mode register, the mode register including an indicator field for indicating data quantization precision; the method further includes:

[0023] Send a control command; the control command is used to indicate the quantization precision L of the data to be processed, and to instruct the internal memory to store the quantization precision L of the data to be processed in the indication field of the mode register.

[0024] In this embodiment of the application, the processor can configure the mode register in the internal memory so that the internal memory can be aware of the quantization accuracy of the data to be processed.

[0025] In one possible implementation, the method further includes:

[0026] When the quantization precision L of the data to be processed is less than or equal to the calculation precision K of the computing unit, a third memory access instruction is sent; the third memory access instruction is used to instruct the internal memory to write the data to be processed into a specified row or column for storage.

[0027] In this embodiment, when the quantization precision of the data to be processed is less than or equal to the computational precision of the computing unit, the processor can directly write the data to be processed completely into the specified row / column for storage via memory access instructions, without needing to split the data. Accordingly, in subsequent calculations, the internal memory can perform calculations on the data to be processed that has been completely stored in the specified row / column through the computing unit, directly obtaining the final calculation result.

[0028] In one possible implementation, the method further includes:

[0029] Obtain the initial weight parameters of the model;

[0030] The initial weight parameters are divided to obtain one or more sets of first parameters;

[0031] One or more second parameter sets are determined from the one or more first parameter sets; the initial weight parameters included in the second parameter sets can be stored in the internal memory;

[0032] One or more third parameter sets are determined from the one or more second parameter sets; the quantization precision of the initial weight parameters included in the third parameter sets is greater than the calculation precision of the calculation unit.

[0033] In this embodiment, after the processor obtains the initial weight parameters of the model, it can first analyze the model, group the initial weight parameters with the same features together, determine which parameters can be stored in the internal memory, and further determine which parameters need to be split to facilitate subsequent model loading and calculation.

[0034] In one possible implementation, the plurality of sub-data includes first sub-data and second sub-data, wherein the first sub-data is one or more high-order data in the data to be processed, and the second sub-data is one or more low-order data in the data to be processed; the method further includes:

[0035] Receive the fifth and sixth results output from the internal memory; the fifth result is determined based on the first sub-data, and the sixth result is determined based on the second sub-data;

[0036] The seventh result is determined based on the shift coefficient and the fifth result; the shift coefficient is determined based on the initial position of the first sub-data in the data to be processed;

[0037] The final result is determined based on the sixth and seventh results.

[0038] In this embodiment, the shifting process of the intermediate result corresponding to the high-bit data can be performed by the processor, which is applicable to scenarios where no shifting circuit is provided in the computing unit.

[0039] Secondly, this application provides a control method applicable to internal memory, which is coupled to a processor. The internal memory includes at least one storage area (Bank) and at least one computing unit. Each Bank includes M rows * N columns of storage units, and each computing unit has a computing precision of K, where M, N, and K are all greater than 0. The method includes:

[0040] The processor receives a second memory access instruction; the second memory access instruction instructs the internal memory to perform calculations on multiple sub-data; the multiple sub-data are multiple sub-data obtained by the processor splitting the data to be processed when the quantization precision L of the data to be processed is greater than the calculation precision K of the calculation unit; the quantization precision of the sub-data is less than or equal to the calculation precision K of the calculation unit.

[0041] The storage addresses of the plurality of sub-data are determined according to the second memory access instruction and storage rules;

[0042] In response to the second memory access instruction, the plurality of sub-data are read sequentially from the memory address;

[0043] The final result is determined by using the multiple sub-data.

[0044] In this embodiment, after receiving the calculation instruction sent by the processor, when the quantization precision of the data to be processed is greater than the calculation precision of the calculation unit in the in-memory computing memory, the internal memory can perform calculations on the sub-data stored in multiple different rows / columns by the calculation unit to determine the final result, thereby achieving the purpose of completing high quantization precision data calculation by a low-precision calculation unit and saving circuit area overhead.

[0045] In one possible implementation, the plurality of sub-data includes first sub-data and second sub-data, wherein the first sub-data is one or more high-order data in the data to be processed, and the second sub-data is one or more low-order data in the data to be processed; the step of using the plurality of sub-data to perform calculations to determine the final result includes:

[0046] After performing multiplication on the first sub-data, a shifted sum is added to obtain the first result;

[0047] Perform multiplication and addition on the second sub-data to obtain the second result;

[0048] The final result is determined based on the first result and the second result.

[0049] In this embodiment, some of the sub-data are high-order data of the data to be processed, and some are low-order data of the data to be processed. After multiplication of the high-order data, shifting and addition processing is required to obtain the intermediate result corresponding to the high-order data. Multiplication and addition of the low-order data does not require shifting processing to obtain the intermediate result corresponding to the low-order data. The final result is then determined based on the two intermediate results to ensure the accuracy of the calculation.

[0050] In one possible implementation, the plurality of sub-data includes first sub-data and second sub-data, wherein the first sub-data is one or more high-order data in the data to be processed, and the second sub-data is one or more low-order data in the data to be processed; the step of using the plurality of sub-data to perform calculations to determine the final result includes:

[0051] After performing multiplication and addition on the first sub-data, a shift operation is performed to obtain the third result;

[0052] Perform multiplication and addition on the second sub-data to obtain the fourth result;

[0053] The final result is determined based on the third and fourth results.

[0054] In this embodiment, some of the sub-data are high-order data of the data to be processed, and some are low-order data of the data to be processed. After performing multiplication and addition on the high-order data, a shifting process is required to obtain the intermediate result corresponding to the high-order data. After performing multiplication and addition on the low-order data, no shifting process is required to obtain the intermediate result corresponding to the low-order data. The final result is then determined based on the two intermediate results to ensure the accuracy of the calculation.

[0055] In one possible implementation, the shift number of each intermediate result among the multiple intermediate results corresponding to the multiple sub-data is determined based on the initial position of the corresponding sub-data in the data to be processed.

[0056] In this embodiment of the application, when the internal memory shifts multiple intermediate results through the shift circuit, it can determine the initial position of the sub-data corresponding to each intermediate result in the original data (the data to be processed before splitting) to ensure the accuracy of the final result.

[0057] In one possible implementation, the storage rule specifically includes that the specified rows or columns in which the plurality of sub-data are written are adjacent rows or columns; or, the Bank includes multiple partitions, and the storage rule specifically includes that the specified rows or columns in which the plurality of sub-data are written are located in different partitions; or, the storage rule specifically includes that the specified rows or columns in which the plurality of sub-data are written are located in different Banks.

[0058] In one possible implementation, the maximum amount of data that can be stored in a specified row or column where the sub-data is written is an integer multiple of the quantization precision of the sub-data.

[0059] In one possible implementation, the second memory access instruction is also used to indicate the quantization precision L of the data to be processed.

[0060] In one possible implementation, the internal memory further includes a mode register, the mode register including an indicator field for indicating data quantization precision; the method further includes:

[0061] The processor sends a control instruction; the control instruction is used to indicate the quantization precision L of the data to be processed, and to instruct the internal memory to store the quantization precision L of the data to be processed in the indication field of the mode register.

[0062] In one possible implementation, the method further includes:

[0063] When the quantization precision L of the data to be processed is less than or equal to the calculation precision K of the computing unit, a third memory access instruction sent by the processor is received; the third memory access instruction is used to instruct the internal memory to write the data to be processed into a specified row or column for storage.

[0064] In one possible implementation, when the computational precision K of the computing unit is J times the quantization precision L of the data to be processed, J is an integer greater than 1; the method further includes:

[0065] The computing resources of the computing unit are divided into J sub-resources;

[0066] Receive J pieces of data to be processed;

[0067] The J data to be processed are processed in parallel using the J sub-resources; the quantization precision of the data to be processed is L.

[0068] In this embodiment of the application, when the quantization precision of the data to be processed is less than the calculation precision of the computing unit, and the calculation precision is an integer multiple of the quantization precision, the computing resources of the computing unit can be divided into multiple parts during the calculation, so that the multiple parts of the computing unit can be calculated simultaneously, thereby improving the calculation efficiency.

[0069] In one possible implementation, the plurality of sub-data includes first sub-data and second sub-data, wherein the first sub-data is one or more high-order data in the data to be processed, and the second sub-data is one or more low-order data in the data to be processed; the step of using the plurality of sub-data to perform calculations to determine the final result includes:

[0070] The fifth result is obtained by calculating the first sub-data.

[0071] The sixth result is obtained by calculating the second sub-data.

[0072] The fifth and sixth results are output to the processor.

[0073] In this embodiment, the shifting process of the intermediate result corresponding to the high-bit data can be performed by the processor, which is applicable to scenarios where no shifting circuit is provided in the computing unit.

[0074] Thirdly, this application provides a control method applicable to internal memory, said internal memory being coupled to a processor, said internal memory including at least one storage area Bank and at least one computing unit; each Bank includes M rows * N columns of storage units, and each computing unit has a computing precision of K, where M, N, and K are all greater than 0; the method includes:

[0075] When the computational precision K of the computing unit is J times the quantization precision L of the data to be processed, the computing resources of the computing unit are divided into J sub-resources; J is an integer greater than 1.

[0076] Receive J pieces of data to be processed;

[0077] The J data to be processed are processed in parallel using the J sub-resources; the quantization precision of the data to be processed is L.

[0078] Fourthly, this application provides a processor, characterized in that the processor is coupled to an internal memory, the internal memory including at least one storage area Bank and at least one computing unit; each Bank includes M rows * N columns of storage units, and each computing unit has a computing precision of K, where M, N, and K are all integers greater than 0; the processor is used for:

[0079] Acquire the data to be processed; the quantization precision of the data to be processed is L;

[0080] When the quantization precision L of the data to be processed is greater than the calculation precision K of the computing unit, the data to be processed is split into multiple sub-data; the quantization precision of each sub-data is less than or equal to the calculation precision K of the computing unit.

[0081] According to the storage rules, a first memory access instruction is sent to each of the multiple sub-data items; the first memory access instruction is used to instruct the internal memory to write the sub-data items into a specified row or column, and the specified rows or columns are different among the multiple first memory access instructions.

[0082] Fifthly, this application provides an internal memory, characterized in that the internal memory is coupled to a processor, the internal memory includes at least one storage area Bank and at least one computing unit; each Bank includes M rows * N columns of storage units, and the computing precision of each computing unit is K, where M, N, and K are all greater than 0; the method includes:

[0083] The processor receives a second memory access instruction; the second memory access instruction instructs the internal memory to perform calculations on multiple sub-data; the multiple sub-data are multiple sub-data obtained by the processor splitting the data to be processed when the quantization precision L of the data to be processed is greater than the calculation precision K of the calculation unit; the quantization precision of the sub-data is less than or equal to the calculation precision K of the calculation unit.

[0084] The storage addresses of the plurality of sub-data are determined according to the second memory access instruction and storage rules;

[0085] In response to the second memory access instruction, the plurality of sub-data are read sequentially from the memory address;

[0086] The final result is determined by using the multiple sub-data.

[0087] In a sixth aspect, this application provides an electronic device, characterized in that it includes one or more processors and one or more internal memories; the one or more processors are coupled to the one or more internal memories; the one or more processors are configured to perform a method as described in any possible implementation of the first aspect, and the one or more internal memories are configured to perform a method as described in any possible implementation of the second or third aspect.

[0088] In a seventh aspect, this application provides a computer-readable storage medium, characterized in that the storage medium stores a computer program or instructions, which, when executed by a computer, implement the method described in any one of the possible implementations of the first aspect, the second aspect, or the third aspect.

[0089] Eighthly, this application provides a computer program, characterized in that the computer program includes instructions that, when executed by a computer, implement the method described in any one of the possible implementations of the first, second, or third aspects.

[0090] Optionally, the processor and internal memory can be physically independent units, or the memory can be integrated with the processor.

[0091] It should be noted that the technical effects achieved in the above aspects can be referred to each other or to the beneficial effects in the method embodiments shown below, which will not be repeated here. Attached Figure Description

[0092] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0093] Figure 1 This is a schematic diagram of an in-memory computing memory.

[0094] Figure 2A This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0095] Figure 2B This is a schematic diagram of a memory structure using in-memory computing technology provided in an embodiment of this application;

[0096] Figure 2C This is a schematic diagram of a PIM-Die structure provided in an embodiment of this application;

[0097] Figure 2D This is a schematic diagram of the structure of a Bank provided in an embodiment of this application;

[0098] Figure 2E This is a schematic diagram of the structure of a computing unit provided in an embodiment of this application;

[0099] Figure 2F This is a schematic diagram of another computing unit provided in an embodiment of this application;

[0100] Figure 3 This is a flowchart illustrating a control method provided in an embodiment of this application;

[0101] Figure 4 This is a flowchart illustrating another control method provided in an embodiment of this application;

[0102] Figure 5This is a schematic diagram of a data calculation process combining INT2 calculation precision and INT2 quantization precision provided in an embodiment of this application;

[0103] Figure 6 This is a schematic diagram of data splitting with INT8 quantization precision provided in an embodiment of this application;

[0104] Figures 7A-7D This is a schematic diagram of a data splitting calculation process provided in an embodiment of this application;

[0105] Figure 8A This is a schematic diagram of data splitting and storage combining INT2 calculation precision and INT4 quantization precision provided in an embodiment of this application;

[0106] Figure 8B This is a schematic diagram of a data calculation process combining INT2 calculation precision and INT4 quantization precision, provided in an embodiment of this application.

[0107] Figure 8C This is a schematic diagram of data splitting and storage, provided in another embodiment of this application, combining INT2 calculation precision and INT4 quantization precision.

[0108] Figure 8D This is a schematic diagram of a data calculation process for another combination of INT2 calculation precision and INT4 quantization precision provided in an embodiment of this application;

[0109] Figure 8E This is a schematic diagram of a data calculation process combining INT2 calculation precision and INT4 quantization precision, as provided in another embodiment of this application.

[0110] Figure 9A This is a schematic diagram of data splitting and storage combining INT2 calculation precision and INT3 quantization precision provided in an embodiment of this application;

[0111] Figure 9B This is a schematic diagram of data splitting and storage provided in another embodiment of this application, combining INT2 calculation precision and INT3 quantization precision.

[0112] Figure 10A This is a schematic diagram of data splitting and storage combining INT1 calculation precision and INT3 quantization precision provided in an embodiment of this application;

[0113] Figure 10B This is a schematic diagram of data splitting and storage provided in another embodiment of this application, combining INT1 calculation precision and INT3 quantization precision.

[0114] Figure 11 This is a schematic diagram of a model analysis process provided in an embodiment of this application;

[0115] Figure 12 This is a schematic diagram of a model loading process provided in an embodiment of this application;

[0116] Figure 13 This is a schematic diagram of a data calculation process based on processor shifting provided in an embodiment of this application;

[0117] Figure 14A This application provides another schematic diagram of a processor-based data calculation process for shifting data.

[0118] Figure 14B This application provides another schematic diagram of a data calculation process based on processor shifting. Detailed Implementation

[0119] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0120] In the description of this application, the terms "first," "second," etc., are used only to distinguish different objects and do not limit the quantity or execution order, nor do they imply that they must be different. For example, "first memory access instruction" and "second memory access instruction" are merely to distinguish different memory access instructions and do not limit their order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0121] In the description of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. "And / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one item", "one or more of the following", or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent: a, b, c; a and b; a and c; b and c; or a and b and c. Here, a, b, and c can be single or multiple.

[0122] In the description of this application, the words "exemplary," "exemplarily," or "for example" are used to indicate examples, illustrations, or illustrative purposes. Any embodiment or design described as "exemplary," "for example," or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Rather, the use of the words "exemplary," "for example," or "for example" is intended to present the relevant concepts in a specific manner.

[0123] It is understood that in the description of this application, "when," "if," and "if" all refer to the device making a corresponding action under certain objective circumstances, and are not time-limited, nor do they require the device to make a judgment action when it is implemented, nor do they mean that there are other limitations.

[0124] In this application, "simultaneously" can be understood as at the same point in time, within a period of time, or within the same cycle; the specific meaning can be determined by considering the context.

[0125] In this application, the use of singular designations for elements is intended to represent "one or more" rather than "one and only one," unless otherwise specified.

[0126] It is understood that in the various embodiments of this application, "A and B correspond" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean that B is determined solely based on A; B can also be determined based on A and / or other information.

[0127] It is understood that in the embodiments of this application, "for indicating" and "indication" can include direct and indirect indication, as well as explicit and implicit indication. When describing "a certain indication information is used to indicate A" or "indication information of A", it can include the indication information directly indicating A or indirectly indicating A, but does not necessarily mean that the indication information carries A. The information indicated by a certain information is called the information to be indicated. In the specific implementation process, there are many ways to indicate the information to be indicated, such as, but not limited to, directly indicating the information to be indicated, such as the information to be indicated itself or its index. It can also indirectly indicate the information to be indicated by indicating other information, where there is an association between the other information and the information to be indicated. It can also indicate only a part of the information to be indicated, while the other parts of the information to be indicated are known or agreed upon in advance. For example, the indication of specific information can also be achieved by using the arrangement order of various information in advance (e.g., according to an agreement), thereby reducing the indication overhead to a certain extent. At the same time, the common parts of various information can be identified and indicated uniformly to reduce the indication overhead caused by individually indicating the same information. In addition, the specific indication method can also be various existing indication methods, such as, but not limited to, the above-mentioned indication methods and their various combinations. Specific details of various indication methods can be found in existing technologies and will not be repeated here. As described above, for example, when multiple pieces of information of the same type need to be indicated, the indication methods for different pieces of information may differ. In specific implementation, the required indication method can be selected according to specific needs. This application does not limit the selection of the indication method. Therefore, the indication methods involved in this application should be understood to cover various methods that enable the party to be indicated to know the information to be indicated. The information to be indicated can be sent as a whole or divided into multiple sub-information and sent separately. Moreover, the sending period and / or sending time of these sub-information can be the same or different. This application does not limit the specific sending method.

[0128] To better understand the technical solutions of the embodiments of this application, several terms or nouns related to this application are briefly introduced below so that those skilled in the art can understand them.

[0129] (1) Dynamic Random Access Memory (DRAM) generally includes multiple storage banks, each of which can include storage cells consisting of multiple rows and columns. DRAM is usually used as main memory. Accessing a storage cell in main memory requires two steps: first, finding the address of a row in the bank, which can be done by specifying the bank and row address to be accessed using a row strobe command; second, finding the address of a specific column in the selected row, which can be done by specifying the column address to be accessed using a column strobe command. Generally, the last operation on a row is a precharge operation, also known as closing an open row. Only after this operation is completed can a new row be accessed.

[0130] (2) Commonly used basic commands for accessing memory include activation, read, write, precharge, and refresh. The activation command is used to select a row in an idle memory cell and put it into a read / write ready state. The read and write commands are used to read data from and write data to specific columns in an activated row, respectively. The precharge command is used to close previously opened rows if other rows need to be addressed after executing read or write commands. The refresh command is used to refresh all rows at fixed intervals to prevent data loss due to capacitor charge leakage over long periods.

[0131] (3) Compared to ordinary DRAM, PIM-DRAM, in addition to the storage area for storing data and other peripheral circuits for reading and writing data, also integrates an independent processing unit (PU). PIM-DRAM can be used as ordinary DRAM, bypassing the computation logic during use to directly read and write data; or it can be used to perform in-memory computation by issuing computation commands, reducing data movement. In this embodiment, when the data precision of the data to be processed is greater than the computation precision of the processing unit in the PIM-DRAM, the data to be processed can be split into multiple sub-data, and the multiple sub-data can be written into different rows or columns in the storage area for storage. The data precision of each sub-data is less than or equal to the computation precision of the processing unit. After receiving the computation command, the multiple sub-data stored in multiple different rows / columns can be retrieved, input into the processing unit for computation, and multiple intermediate results can be obtained accordingly. After shifting the multiple intermediate results through a shift circuit, they are input into the accumulator for computation, thereby achieving the purpose of completing high-precision data computation through a low-precision processing unit, saving circuit area overhead.

[0132] (4) Quantization is a method to reduce the computational and storage overhead of a model by reducing the number of bits used to represent its parameters. Traditional deep learning models typically use floating-point numbers to represent parameters and activation values, which achieves high accuracy but also incurs higher computational costs. The goal of quantization is to reduce the number of bits used to represent parameters while maintaining model performance as much as possible, thereby achieving higher efficiency in terms of computation and storage.

[0133] The core idea of ​​quantization is to map the parameters and activation values ​​in a model to a smaller value range, typically finite integers or low-bit floating-point numbers. This mapping can be achieved in various ways, such as fixed-point quantization, floating-point quantization, and symmetric quantization. In fixed-point quantization, parameters and activation values ​​are mapped to integers within a fixed range. By limiting the value range, fewer bits can be used to represent each value, thus reducing storage and computational overhead. Floating-point quantization maps parameters and activation values ​​to a lower-bit floating-point representation. For example, half-precision floating-point numbers (16 bits) can be used instead of standard single-precision floating-point numbers (32 bits), reducing the number of bits required to represent the parameters. Symmetric quantization is a special quantization method where parameters and activation values ​​are symmetrical around zero, maintaining a balance between positive and negative values ​​in the quantized values, thus reducing information loss. However, quantization can lead to a loss of model precision. To reduce the model's hardware resource requirements while maintaining its capabilities, hybrid quantization precision can be used, allowing the model to quantize key parameters with higher precision and other parts with lower precision.

[0134] First, the specific technical problem that this application aims to solve is analyzed and proposed. (See also...) Figure 1The PIM-DRAM, an in-memory compute memory, integrates processing units near its internal banks. The weight parameters W of a large model can be stored in these banks, while the input vector V can be transferred from the host (or processor) to the PIM-DRAM and stored in the global buffer. To perform the W*V multiplication and accumulation operation, the processing unit can include multiplication circuits (including multiple multipliers), addition circuits (including multiple adders), and accumulators. Taking a column within the in-memory compute memory that can store 256 bits of data as an example, when the quantization precision of the weight parameters is 2-bit integer (INT2), one column can store 128 (256 / 2) weight parameters; when the quantization precision is 4-bit integer (INT4), one column can store 64 (256 / 4) weight parameters. Therefore, when the PIM-DRAM needs to be compatible with calculations using both INT2 and INT4 mixed quantization precisions, the processing unit must include at least 128 INT4-enabled processing circuits. When the quantization precision of the weight parameter is INT2, it requires 128 INT4 calculation circuits for calculation, but only 2 bits (such as the high 2 bits or the low 2 bits) of the 4 bits of each INT4 calculation circuit are used for calculation; while when the quantization precision of the weight parameter is INT4, it requires 64 INT4 calculation circuits for calculation, and all 4 bits of each INT4 calculation circuit are used.

[0135] exist Figure 1 In the illustrated scheme, to accommodate calculations with mixed quantization precisions (such as INT2 and INT4), corresponding computational circuits need to be designed for different quantization precisions. For example, if a column can store 128 INT2 parameters, at least 128 INT2 computational circuits are required; if a column can store 64 INT4 parameters, at least 64 INT4 computational circuits are required. Therefore, to achieve compatible computations with mixed quantization precisions, the number of computational circuits required for INT2 precision and the precision requirements for INT4 precision must be considered. The computational unit needs to include at least 128 INT4 computational circuits. However, in actual computation, whether calculating INT2 or INT4 parameters, it is impossible to fully utilize all 128 INT4 computational circuits, resulting in additional circuit area overhead. Furthermore, the above scheme suffers from poor flexibility. For instance, once the design of the computational unit is determined, the quantization precision supported by the stored parameters is also fixed, and the model's precision is therefore fixed as well.

[0136] To address this, embodiments of this application provide a control method, related apparatus, and readable storage medium. When computation is required using the in-memory computing memory, if the quantization precision of the data to be processed is greater than the computation precision of the computing unit in the in-memory computing memory, the multi-bit data with higher quantization precision can be split into multiple sub-data of fewer bits. The quantization precision of each sub-data of fewer bits is less than or equal to the computation precision of the computing unit, ensuring that the computing unit can perform computation on each sub-data. Then, the computation results corresponding to each sub-data are internally shifted and accumulated to finally complete the computation of the multi-bit data with higher quantization precision. For computations compatible with mixed quantization precision, it is only necessary to adjust the data splitting strategy according to different combinations of quantization precision and computation precision. This allows the computation circuit of one computation precision to complete the computation of multiple quantization precision parameters without the need to design a computation circuit corresponding to different quantization precisions. This not only saves computation circuit area overhead but also improves the flexibility of the quantization precision of the data stored in the in-memory computing memory, thereby ensuring that the accuracy of the model can be flexibly adjusted according to different scenario requirements.

[0137] Optionally, when the quantization precision of the data to be processed is greater than the computational precision of the computing units in the in-memory computing memory, the data to be processed can be split into multiple sub-data, and these sub-data can be written into different rows or columns of the storage area for storage. The quantization precision of each sub-data is less than or equal to the computational precision of the computing unit. After receiving the calculation instructions sent by the processor, the in-memory computing memory can use the computing units to perform calculations on the sub-data stored in multiple different rows / columns to obtain multiple intermediate results. Then, the intermediate results are shifted and passed to the accumulator for calculation, thereby achieving the goal of completing high-quantization precision data calculations with low-precision computing units and saving circuit area overhead.

[0138] For ease of understanding, the technical solution provided in this application will be described below with reference to more accompanying drawings.

[0139] In this application, unless otherwise specified, the same or similar parts between various embodiments or implementations can be referred to each other. In the various embodiments of this application, and in the various implementation methods / methods / implementations within each embodiment, unless otherwise specified or logically conflicting, the terminology and / or descriptions between different embodiments and between the various implementation methods / methods / implementations within each embodiment are consistent and can be mutually referenced. The technical features in different embodiments and the various implementation methods / methods / implementations within each embodiment can be combined according to their inherent logical relationships to form new embodiments, implementation methods, methods, or implementation approaches. The embodiments described below do not constitute a limitation on the scope of protection of this application.

[0140] The following is an exemplary description of the applicable scenarios for the embodiments of this application. It is understood that the structures and application scenarios of the electronic devices described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application.

[0141] See also Figure 2A , Figure 2A This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 01 provided in this application may include one or more processors 11, one or more memory controllers 12, and one or more memory modules 13. The electronic device may also include a power management unit, external storage, etc. (not shown in the figure). The electronic device 01 may be a subscriber unit, a cellular phone, a smartphone, a personal digital assistant (PDA) computer, a tablet computer, a handset, a laptop computer, a machine type communication (MTC) terminal, an autonomous driving terminal, a smart wearable device, a smart speaker, or other mobile smart terminal, or it may be various non-mobile terminal devices (i.e., fixed terminals) that require memory access, without specific limitations here.

[0142] The processor 11 can be a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a Neural Processing Unit (NPU), a System-on-a-Chip (SoC), a Microcontroller Unit (MCU), or a chip with other functions. The processor 11 can exchange data with the memory 13 through the memory controller 12. For example, when the CPU executes a memory access instruction, it can first send the read / write request to the Memory Management Unit (MMU, not shown in the figure). After the MMU performs the virtual address and physical address translation, it sends the memory access command to the bus. The bus then passes the memory access command to the memory controller 12, which translates the address again to access the data in the corresponding row / column of the memory 13.

[0143] The memory controller 12, which may be a dynamic memory controller (DMC), is primarily used to control data transfer between the processor 11 and the memory 13. The memory controller 12 can be deployed on the same chip as the processor 11, or it can be deployed on a separate chip. That is, the memory controller 12 can be integrated into the processor 11 (e.g., CPU) chip, or it can be a separate chip independent of the processor 11; no specific limitation is made here. Generally, the memory controller may include a command processing module, a data processing module, and a refresh module. The command processing module processes instructions issued by the processor 11; the data processing module processes data read from or written to the memory 13 by the processor 11; and the refresh module refreshes the memory 13 according to refresh commands. The processor 11 can send memory access instructions (such as read / write instructions) and control instructions to the memory 13 through the memory controller 12.

[0144] Memory 13 can be an internal memory (called in-memory computing memory) employing in-memory computing technology, comprising a storage section and a computing section. The storage section of memory 13 can be structured as Dynamic Random Access Memory (DRAM). Generally, memory 13 is a place for temporarily storing processes and data, also known as main memory or internal memory, and can be made of semiconductor devices. It is a storage space that can be directly addressed by processor 11 or other external devices with various functions (timers, analog-to-digital converters, etc., not shown in the figure). In this embodiment, memory 13 can be an in-memory computing memory. Its storage section can be used to store the weight parameters of a large model, and its computing section can be used to perform calculations. For example, when the quantization precision of the weight parameters is greater than the calculation precision of the computing section, the weight parameters can be split into multiple smaller precision sub-data and stored in different rows / columns. The data precision of each sub-data is less than or equal to the calculation precision of the computing section. The subsequent computing section can perform calculations on each sub-data in rows / columns respectively.

[0145] For ease of understanding, the structure of in-memory computing memory is illustrated below. Please refer to [link / reference]. Figure 2B , Figure 2B This is a schematic diagram of a memory structure employing in-memory computing technology, provided in an embodiment of this application. Memory 13 may include one or more ranks. A rank refers to a set of memory dies, with the same set of dies sharing the same address and control signals. Each rank is typically 64 bits wide, meaning it can process 64 bits of data simultaneously. For example, memory 13 may include two rank, each accessible independently, thereby improving the bandwidth and performance of memory 13. Figure 2BAs shown, Rank 1 can include multiple dies, and Rank 1 can be used independently for data access; Rank 2 can include multiple PIM-Dies, where PIM-Dies are dies with integrated computing units (PUs). Rank 2 can be used independently for data access and also for performing computational tasks. It should be noted that the same Rank can include multiple dies with the same structure or multiple dies with different structures; this application does not impose specific limitations.

[0146] To facilitate understanding, the structure of a PIM-Die will be illustrated below. Please refer to [link / reference]. Figure 2C , Figure 2C This is a schematic diagram of a PIM-Die structure provided in an embodiment of this application. Each PIM-Die may include multiple storage areas (Banks), such as... Figure 2C The diagram shows Banks A through J. A Bank is a basic unit for storing and managing data. Each Bank can be considered a storage area and can independently handle data requests. Different Banks allow for parallel access and operation, meaning the memory controller 12 can access multiple Banks simultaneously, thus improving data retrieval efficiency. For example, if a PIM-Die includes four Banks and data is stored in these four Banks, then during data access, the memory controller 12 can simultaneously activate the data in all four Banks. In addition to storage areas, each PIM-Die also integrates multiple computing units (PUs), such as... Figure 2C The diagram shows PU1-PU5. PUs are used to perform computational tasks, enabling direct execution of computational tasks within memory 13, rather than transferring data to an external processor 11 for processing, thereby improving computational efficiency. Each PU can correspond to one or more banks, for example... Figure 2C The PU1 shown can correspond to Bank A and Bank F, meaning that data read from Bank A and Bank F can be directly accessed for calculation in PU1. Furthermore, each PIM-Die can also integrate multiple mode registers (MRs), such as... Figure 2CThe mode registers shown are 1, ..., n. MRs can be used to configure the operating modes and functions of memory 13, such as setting parameters like data width, latency, and refresh rate. The presence of MRs allows memory controller 12 to configure and optimize memory 13 according to system requirements, thereby improving performance and compatibility. By programming these registers, the system can adjust the behavior of memory 13 to adapt to different application scenarios and performance requirements. Each PIM-Die can also include a global buffer, which can be used as a cache to store frequently accessed data, reducing access latency and improving system response speed.

[0147] To facilitate understanding, the structure of a Bank will be illustrated below. Please refer to [link / reference]. Figure 2D , Figure 2D This is a schematic diagram of a Bank structure provided in an embodiment of this application. The Bank may include a storage array and a row buffer. The storage array is responsible for the actual data storage and consists of multiple storage units, typically organized in rows and columns. That is, the storage array can consist of M rows * N columns of storage units, allowing for efficient data access. The row buffer is used to store row data read from the storage array.

[0148] For ease of understanding, the structure of the computing section (computing unit) in memory 13 is also illustrated below. (See reference...) Figure 2E and Figure 2F , Figure 2E This is a schematic diagram of the structure of a computing unit provided in an embodiment of this application. Figure 2F This is a schematic diagram of another computing unit provided in an embodiment of this application. The computing unit may include a multiplication circuit, an addition circuit, a shift circuit, and an accumulator, suitable for multiply-accumulate operations. The multiplication circuit may include one or more multipliers for performing multiplication operations; the addition circuit may include one or more adders for performing addition operations; the shift circuit may include one or more shifters for shifting input data; and the accumulator can perform accumulation operations on the input data. Exemplarily, the shift operation can be performed after the multiplication operation is completed, suitable for... Figure 2E The structure in the text; shift operations can be performed after addition, applicable to Figure 2FThe structure of memory 13 is as follows. Understandably, the computation section in memory 13 can also be designed according to different computational needs to achieve the purpose of completing other operations. If the computation section in memory 13 needs to complete other operations, correspondingly, memory 13 may include other circuits. The structure of memory 13 is not specifically limited here. In this embodiment, the multiplication circuit and the addition circuit can be used to perform multiplication and accumulation calculations on multiple sub-data and input vectors stored in different rows / columns after being split. The shift circuit is used to shift one or more intermediate results obtained after calculating multiple sub-data and input vectors. The accumulator can be used to accumulate one or more intermediate results after shifting to obtain the final data, thereby ensuring that the computation unit with low computational precision can complete the calculation of high-precision data.

[0149] It should be noted that the electronic device 01 may also include a power management unit, which can be used to power the processor 11, memory controller 12, and memory 13, as well as other devices or modules in the electronic device 01. Furthermore, the electronic device 01 may also include one or more interfaces for enabling communication between the electronic device 01 and other devices. In some embodiments, the interface may include a wire for coupling a wired connection, or pins for coupling a wireless transceiver, or a chip and / or pins for wireless connection. In some embodiments, the interface may include a transmitter, a receiver, a transceiver, and / or an antenna.

[0150] Understandably, the embodiments of this application can be applied to various computer system architectures, as described above. Figure 2A The architecture described above is merely an exemplary implementation in the embodiments of this application and should not be construed as limiting the scope of this application. The architectures applicable to the embodiments of this application include, but are not limited to, the above architectures. It should be understood that the electronic device 01 may have a more advanced design than... Figure 2A The more or fewer units / modules shown can be combined into two or more units / modules, or they can have different unit / module configurations. Figure 2A The various units / modules shown can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits, and the embodiments of this application do not specifically limit this.

[0151] To better understand the embodiments of this application, the technical solutions provided by the embodiments of this application will be described below in conjunction with more accompanying drawings.

[0152] The following uses the processing of weight parameters from the storage stage to the computation stage as an example to briefly explain the control method provided in this application. First, the control method is briefly explained from the processing of the storage stage. In the storage stage, the control method may include at least the following: Figure 3Steps S300-S302, as shown, involve a processor and internal memory. The processor and internal memory are coupled, and the internal memory may include at least one storage area (Bank) and at least one computing unit; each Bank includes M rows * N columns of storage units, and each computing unit has a computational precision of K, where M, N, and K are all greater than 0. Steps S300-S302 are as follows:

[0153] S300: The processor acquires the data to be processed.

[0154] Here, the quantization precision of the data to be processed is L. The data to be processed can be the weight parameters of the model. The weight parameters can be input by the user when building the model, or they can be read by the processor from external memory; no specific limitation is made here. Optionally, when building the model, the user will specify the quantization precision required for the model's parameters, and then input the weight parameters according to the required quantization precision. Therefore, when the processor obtains the weight parameters, it can determine the quantization precision of the weight parameters.

[0155] S301: When the quantization precision L of the data to be processed is greater than the calculation precision K of the computing unit, the processor splits the data to be processed into multiple sub-data.

[0156] In this context, the quantization precision of each of the plurality of sub-data is less than or equal to the computational precision K of the computing unit.

[0157] S302: The processor sends a first memory access instruction to each of the plurality of sub-data according to the storage rules.

[0158] The first memory access instruction can be a write instruction, used to instruct the internal memory to write the corresponding sub-data into a specified row or column for storage. Different first memory access instructions may specify different rows or columns. That is, the processor can instruct the internal memory to store multiple sub-data in different rows / columns using multiple memory access instructions. Correspondingly, after receiving the first memory access instruction corresponding to each sub-data, the internal memory can write the corresponding sub-data into the specified row / column for storage according to the row / column address carried by each first memory access instruction, with different sub-data stored in different rows / columns. Optionally, the aforementioned different rows / columns can be different rows / columns within the same bank or different rows / columns in different banks; no specific limitation is made here.

[0159] In this embodiment, during the storage phase, after the processor acquires the data to be processed, it can first determine whether the quantization precision of the data to be processed is greater than the calculation precision of the computing unit in the internal memory. If the quantization precision of the data to be processed is greater than the calculation precision of the computing unit, the internal memory will not be able to directly perform calculations on the data to be processed through the computing unit. Therefore, the processor can first split the data to be processed into multiple sub-data with quantization precisions less than or equal to the calculation precision, so that the internal memory can subsequently perform calculations on each sub-data separately through the computing unit. It should be noted that the computing unit can generally perform calculations on a row / column of data. Storing the multiple sub-data obtained after splitting into different rows / columns can ensure that the computing unit can restore the accurate calculation result through shifting and accumulation in the subsequent calculation process.

[0160] To facilitate understanding, let's take an example where the data to be processed includes one weight parameter to illustrate the storage process. For instance, if the quantization precision of the weight parameter is INT4 and the computation precision of the calculation unit is INT2, the processor can split the weight parameter into two 2-bit data segments: one high-order 2 bits and the other low-order 2 bits. This means splitting the INT4 quantization precision weight parameter into two INT2 quantization precision sub-data segments. The processor then uses two memory access instructions to write these two INT2 quantization precision sub-data segments into different rows / columns for storage. In subsequent calculations, the processor can instruct the memory to sequentially input the two INT2 quantization precision sub-data segments into the computation unit for calculation, thus obtaining two intermediate results. After shifting and accumulating these two intermediate results, the final calculation result (i.e., the calculation result of the INT4 quantization precision weight parameter) can be reconstructed.

[0161] Optionally, the storage rules may specifically include specifying that the multiple sub-data items are written to adjacent rows or columns. That is, after the processor splits the data to be processed, it can instruct the memory to write the multiple sub-data items obtained after the split to multiple adjacent rows / columns for storage. For example, if one piece of data to be processed is split into two sub-data items, the processor can use one memory access instruction to instruct the memory to write one of the sub-data items to a specified column (e.g., col.k) for storage, and use another memory access instruction to instruct the memory to write the other sub-data item to a specified column (e.g., col.k+1) for storage, where col.k and col.k+1 are adjacent columns.

[0162] Optionally, the storage rules may specifically include specifying that the multiple sub-data items are written to designated rows or columns located in different partitions. In other words, after the processor splits the data to be processed, it can instruct the internal memory to write the resulting multiple sub-data items to multiple different rows / columns for storage, with a greater than 1 interval between the different rows / columns. That is, the processor can instruct the internal memory to write multiple sub-data items to multiple non-adjacent rows / columns for storage. Alternatively, a storage area of ​​the internal memory can be divided into multiple different partitions, and the processor can instruct the internal memory to write multiple sub-data items to multiple different partitions, each partition including multiple rows / columns. For example, a storage area of ​​internal memory may include two partitions: partition 1 includes col.0 to col.31, and partition 2 includes col.32 to col.63. One piece of data to be processed is split into two sub-data. The processor can use one memory access instruction to instruct the internal memory to write one of the sub-data into a specified column (such as col.0) for storage, and use another memory access instruction to instruct the internal memory to write the other sub-data into a specified column (such as col.32) for storage. col.0 and col.32 belong to different partitions and are not adjacent columns.

[0163] In one possible implementation, when the quantization precision of the data to be processed is less than or equal to the computational precision of the computing unit, the processor can directly write the data to be processed completely into the specified row / column for storage via a memory access instruction, without needing to split the data. During subsequent calculations, the internal memory can use the computing unit to perform calculations on the data to be processed that has been completely stored in the specified row / column, directly obtaining the final calculation result. For example, if the quantization precision of the data to be processed is INT2, and the computational precision of the computing unit is also INT2, the computing unit can directly perform calculations on the data to be processed. Therefore, during the storage phase, there is no need to split the data with an INT2 quantization precision; the data to be processed can be directly written into the specified row / column for storage.

[0164] In one possible implementation, when the quantization precision L of the data to be processed is less than the computational precision K of the computing unit, and the computational precision K is J times the quantization precision L (J is an integer greater than 1), the data to be processed can be stored directly in the specified rows / columns without data splitting during the storage stage. Furthermore, in subsequent calculations, the computing unit can simultaneously perform calculations on the data to be processed stored in up to J rows / columns. For example, if the quantization precision of the data to be processed is INT2 (i.e., 2-bit data), and the computational precision of the computing unit is INT4 (capable of calculating 4-bit data), the computational precision is twice the quantization precision. In this case, the computing unit can directly perform calculations on the data to be processed, and the computing unit only uses half of its resources (i.e., only 2 bits of the 4 bits are used, leaving 2 bits remaining) when calculating INT2 data in 1 row / column. Therefore, during the calculation stage, the computing unit can perform calculations on INT2 data in up to 2 rows / columns. In other words, the computing resources of the computing unit can be divided into J sub-resources. The computing precision of each of the J sub-resources is the same as the quantization precision L of the data to be processed. Therefore, the computing unit can simultaneously compute J data to be processed, and the quantization precision of the data to be processed is L.

[0165] It should be noted that, regarding quantization precision, the processor can determine the quantization precision of the data to be processed when it acquires the data to be processed; regarding the computational precision of the computing unit, in one possible implementation, the processor can send an instruction to the internal memory via the bus to instruct the internal memory to report the computational precision of the computing unit, so that the processor can determine whether the quantization precision of the data to be processed is greater than the computational precision of the computing unit when it acquires the data to be processed.

[0166] In one possible implementation, the control method provided in this application embodiment, in addition to the steps S300-S302 described above, may also include the following: Figure 11 Steps S303-S306 are shown below.

[0167] S303: The processor obtains the initial weight parameters of the model.

[0168] When the processor obtains the initial weight parameters, it can determine the quantization precision and operator type corresponding to each initial weight parameter. Optionally, the quantization precision can be integer (INT), floating-point (float, FP), or other types, such as INT2, INT4, INT8, ..., INTx, or FP16, FP32, etc. The operator type indicates what kind of operation needs to be performed on the initial weight parameter, such as multiplication, addition, multiply-addition, exponentiation, etc. Different initial weight parameters can have the same quantization precision and different operator types. For example, initial weight parameter A and initial weight parameter B have the same quantization precision and operator type, both being INT8 quantization precision, and both require multiplication and addition operations; or, initial weight parameter A and initial weight parameter B have the same quantization precision (e.g., both being INT4 quantization precision), but different operator types (e.g., one is multiplication and the other is addition); or, initial weight parameter A and initial weight parameter B have different quantization precision (e.g., one is INT4 and the other is INT8), but the same operator type (e.g., both are multiplication and addition operations); or, initial weight parameter A and initial weight parameter B have different quantization precision and operator types.

[0169] S304: The processor divides the initial weight parameters to obtain one or more sets of first parameters.

[0170] After obtaining the initial weight parameters of the model, the processor can analyze them and divide them based on the quantization precision and operator type of each initial weight parameter. Initial parameters with the same quantization precision and operator type are grouped into the same parameter set, thus obtaining one or more first parameter sets, or model sub-module sets. Optionally, when dividing the initial weight parameters, the processor can also consider factors such as the number of model parameters, the memory capacity, and the proportion of in-memory computation. For example, if a parameter set exceeds the memory capacity limit, it can be further divided to avoid situations where the memory capacity does not meet the storage requirements.

[0171] S305: The processor determines one or more second parameter sets from the one or more first parameter sets; the initial weight parameters included in the second parameter sets can be stored in the internal memory.

[0172] The processor can determine one or more second parameter sets, or PIM submodule sets, from one or more first parameter sets based on the computational precision and operator type of the computing units in the internal memory. The initial weight parameters included in the second parameter set can be stored in the internal memory and can be calculated by the computing units in that internal memory. For example, the processor can determine whether the initial weight parameters support storage in internal memory for in-memory computation based on the quantization precision of the initial weight parameters and the computational precision of the computing units, as well as the operator type of the initial weight parameters and the operator type of the computing units. For instance, if the quantization precision of the initial weight parameters and the computational precision of the computing units are of the same type (e.g., both INT or both FP), and the operator type of the initial weight parameters and the computing units is the same (e.g., both multiplication), the initial weight parameters support storage in internal memory for in-memory computation, and the first parameter set corresponding to these initial weight parameters can be determined as the second parameter set. Otherwise, the initial weight parameters do not support storage in internal memory for in-memory computation.

[0173] For example, if the initial weight parameter requires multiplication, but the PIM's calculation unit can only perform addition, the initial weight parameter cannot be multiplied by the calculation unit and therefore should not be included in the PIM; or, if the initial weight parameter requires exponentiation, but the PIM's calculation unit can only perform multiplication, addition, or multiply-addition operations, the initial weight parameter cannot be exponentiation by the calculation unit and therefore should not be included in the PIM.

[0174] For example, if the initial weight parameter has a quantization precision of INT4, requiring INT calculation, but the calculation unit in PIM does not support INT calculation, but only supports FP (floating-point) calculation, then this initial weight parameter cannot be calculated by the calculation unit and should not be included in PIM. Alternatively, if the initial weight parameter has a quantization precision of FP32, requiring FP calculation, but the calculation unit in PIM does not support FP calculation, but only supports INT calculation, then this initial weight parameter cannot be calculated by the calculation unit and should not be included in PIM.

[0175] It should be noted that when the initial weight parameters and the operator type of the computation unit are the same, but their quantization precision types are different, the quantization precision type of the initial weight parameters can be transformed, for example, from INT to FP (or FP to INT), so that the initial weight parameters can be calculated by the computation unit. However, data type transformation may lead to a certain loss of precision, which in turn leads to information loss.

[0176] S306: The processor determines one or more third parameter sets from the one or more second parameter sets; the quantization precision of the initial weight parameters included in the third parameter sets is greater than the calculation precision of the computing unit.

[0177] After determining one or more sets of second parameters that can be stored in memory for in-memory computation, the processor can further determine one or more sets of third parameters (or sub-module sets) that support data splitting from these sets. Specifically, the quantization precision of the initial weight parameters included in the third parameter set is greater than the computational precision of the computation unit. For example, if the quantization precision of the initial weight parameters is INT4 and the computational precision of the computation unit is INT2, since INT4 is greater than INT2, the initial weight parameters need to be split first, such as splitting into two 2-bit data segments: one segment representing the high 2 bits of the initial weight parameters, and the other representing the low 2 bits. Subsequent computation units can then perform calculations on these two 2-bit data segments separately, ultimately completing the calculation of the initial weight parameters of INT4.

[0178] Optionally, when the quantization precision of the initial weight parameter is less than or equal to the computational precision of the computational unit, the initial weight parameter may not be split into data.

[0179] After completing the model analysis process described above, the processor obtains three sets: a first set of parameters (model submodule set), a second set of parameters (PIM submodule set), and a third set of parameters (split submodule set). During subsequent model loading, the processor can determine the storage format (PIM memory or non-PIM memory) and storage location (logical / physical memory address) of the initial weight parameters based on these three sets, as well as whether to split the model. Furthermore, if the initial weight parameters are split, the processor can also determine the storage location of the split data.

[0180] After obtaining the first parameter set (model submodule set), the second parameter set (PIM submodule set), and the third parameter set (split submodule set) through model analysis, the processor can load the model. The processor's model loading process can be found in [link to relevant documentation]. Figure 12 The flowchart shown includes, but is not limited to, the following steps:

[0181] S121: The processor can load model submodule set i (first parameter set i), i is greater than or equal to 1 and less than or equal to N, where N is the total number of model submodule sets.

[0182] S122: The processor further determines whether the model submodule set i belongs to the PIM submodule set (second parameter set).

[0183] S123: When model submodule set i does not belong to PIM submodule set, the processor stores the initial weight parameters in model submodule set i into non-PIM memory.

[0184] S124: When model submodule set i belongs to PIM submodule set, the processor further determines whether the initial weight parameters in model submodule set i support data splitting, or whether it belongs to a split submodule set.

[0185] S125: When the initial weight parameters in model submodule set i do not support data splitting, the processor stores the initial weight parameters in model submodule set i into the PIM memory.

[0186] S126: When the initial weight parameters in model submodule set i support data splitting, the processor can determine whether the initial weight parameters have been split offline.

[0187] S127: If the initial weight parameters have been split offline, the processor will store the split data obtained after splitting the initial weight parameters in model submodule set i into the PIM memory respectively.

[0188] S128: If the initial weight parameters are not split offline, the processor splits the initial weight parameters in the model submodule set i online and stores the split data obtained after splitting into PIM memory respectively.

[0189] Optionally, the initial weight parameters in model submodule set i can be split offline or online by the processor; no specific limitation is made here.

[0190] Optionally, during the model analysis phase described above, after determining the model submodule set and further identifying the PIM submodule set and split submodule set from it, the processor can identify the model submodule sets belonging to the PIM submodule set and the split submodule set. For example, a first tag can be added to the model submodule sets belonging to the PIM submodule set, and a second tag can be added to the model submodule sets belonging to the split submodule set. This allows the processor to quickly determine whether model submodule set i is a PIM submodule set and whether its initial weight parameters support splitting during the model loading phase.

[0191] Understandably, after the processor acquires the data to be processed, when the quantization precision L of the data to be processed is greater than the computational precision K of the computing unit, that is, when the data to be processed belongs to the aforementioned third parameter set (splitting sub-module set), the processor can split the data to be processed into multiple sub-data.

[0192] Optionally, when performing model analysis, the processor can initially determine the storage format and location of the initial weight parameters, whether data splitting is required, and the storage location of the split data, etc. When loading the model, the processor can load it according to the analysis results and real-time hardware conditions (such as memory capacity and in-memory computing ratio). If the hardware conditions do not allow it, the storage format and location of the initial weight parameters during loading can be different from the storage format and location during model analysis, which is not specifically limited here.

[0193] The control method provided in this application embodiment will be briefly described below from the perspective of the processing procedure in the calculation stage. In the calculation stage, the control method may include at least the following: Figure 4 The steps S400-S403 are shown. Among them,

[0194] S400: The internal memory receives the second memory access instruction sent by the processor.

[0195] The second memory access instruction can be a calculation instruction, used to instruct the internal memory to perform calculations on multiple sub-data. The multiple sub-data are multiple sub-data obtained by the processor splitting the data to be processed when the quantization precision L of the data to be processed is greater than the calculation precision K of the calculation unit; the quantization precision of the sub-data is less than or equal to the calculation precision K of the calculation unit. Optionally, the second memory access instruction can also be used to indicate the quantization precision L of the data to be processed.

[0196] S401: The internal memory determines the storage addresses of multiple sub-data according to the second memory access instruction and storage rules.

[0197] The second memory access instruction carries the storage address of one of the multiple sub-data items. After determining the storage address of this sub-data item, the memory can then determine the storage addresses of the other sub-data items according to storage rules (stored by adjacent rows / columns, or stored by different partitions). For example, if two sub-data items, col.0 and col.1, are stored according to adjacent column rules, and the column address carried by the second memory access instruction is col.0, the memory can determine the storage address of the other sub-data item, col.1, based on the storage rules after determining that one sub-data item is stored in col.0 using the second memory access instruction.

[0198] S402: The internal memory responds to the second memory access instruction by sequentially reading the plurality of sub-data from the memory address.

[0199] S403: The internal memory uses the multiple sub-data to perform calculations to determine the final result.

[0200] In the storage stage, since the quantization precision of the weight parameters is greater than the calculation precision of the calculation unit, the weight parameters are split into multiple sub-data and stored in multiple different rows / columns. Therefore, in the calculation stage, when the internal memory needs to perform calculations on the data to be processed, it needs to read multiple sub-data from multiple different rows / columns and perform calculations on each sub-data in sequence through the calculation unit. The calculation of each sub-data can obtain the intermediate result corresponding to that sub-data (such as the first result, the second result, the third result, or the fourth result).

[0201] Optionally, after calculating the intermediate results corresponding to each sub-data, the internal memory can determine the specific number of bits required to shift each intermediate result based on the position of each sub-data in the data to be processed. Then, a shift circuit is used to shift each intermediate result, and finally, the final result is obtained by calculating the multiple shifted intermediate results. In simple terms, the data to be processed includes high-order data and low-order data. When the split sub-data is high-order data, the corresponding intermediate result needs to be shifted; when the split sub-data is low-order data, the corresponding intermediate result does not need to be shifted.

[0202] It should be noted that, regarding computational precision, the computational unit is one of the capabilities of the internal memory, and its computational precision is known to the internal memory. As for the perception of quantization precision, it may be implemented in the following ways: the internal memory can perceive the quantization precision of the data to be processed based on the predefined computational instructions issued by the processor, or the processor can configure the mode register in the internal memory to enable the internal memory to perceive the quantization precision of the data to be processed.

[0203] Optionally, the processor and internal memory can communicate via predefined computation instructions to achieve quantization precision awareness. That is, the processor can issue predefined computation instructions to the internal memory to instruct it on the quantization precision of the data to be processed. For example, the predefined computation instructions can be configured with reference to example forms such as COMP_INT1, COMP_INT2, COMP_INT3, COMP_INT4, COMP_INT8, COMP_INT16, COMP_AC_INT1, COMP_AC_INT2, COMP_AC_INT3, COMP_AC_INT4, COMP_AC_INT8, COMP_AC_INT16, ACT_INT1, ACT_INT2, ACT_INT4, ACT_INT8, ACT_INT16, etc.

[0204] The COMP_INT1 instruction can instruct calculation operations to be performed on the data at a specified column address, and the quantization precision of the data stored in that column is 1 integer bit (INT1); the COMP_INT8 instruction can instruct calculation operations to be performed on the data at a specified column address, and the quantization precision of the data stored in that column is INT8; the meanings of the other COMP_INTx instructions are basically the same as those of COMP_INT1 and COMP_INT8, but the quantization precision of the stored data is different, and will not be elaborated further here.

[0205] The COMP_AC_INT1 instruction instructs calculations to be performed on all columns of data in a row, with the quantization precision of all columns in that row set to INT1. Similarly, COMP_AC_INT16 indicates that the quantization precision of all columns in that row is INT16. Additionally, the ACT_INT1 instruction can also instruct calculations to be performed on all columns of data in a row, with the quantization precision of all columns in that row set to INT1. Optionally, the difference between the ACT_INT1 and COMP_AC_INT1 instructions is that the COMP_AC_INT1 instruction requires row selection via the regular ACT instruction before being issued, while ACT_INT1 itself is a row selection command and does not require an additional regular ACT instruction for row selection.

[0206] It should be noted that, in addition to the above-mentioned integer data type, the predefined calculation instructions in this application embodiment can also indicate other data types (such as floating-point type); and, in addition to the above-mentioned quantization precision (including 1, 2, 3, 4, 8, 16), the predefined calculation instructions can also indicate other quantization precisions (such as 32), which are not specifically limited here.

[0207] Alternatively, a mode register (MR) can be used between the processor and internal memory to implement quantization precision awareness. For example, the mode register for implementing quantization precision awareness can be configured with reference to Table 1 below.

[0208] Table 1 Quantization Accuracy Sensing Mode Register

[0209]

[0210] The MR# field represents the index of the mode register.

[0211] The MA[6:0] fields represent the address of the mode register. Generally, the processor can find the mode register through the mode register index and address, and then adjust the configuration of the mode register.

[0212] Access: R / W indicates that this mode register can be read from or written to. For example, the values ​​of OP[5:4] and OP[3:0] can be read using a read command to determine the number of banks simultaneously active in the in-memory compute memory and the quantization precision of the data stored in the in-memory compute memory. Similarly, the values ​​of OP[5:4] and OP[3:0] can be changed using a write command to adjust the number of banks simultaneously active in the in-memory compute memory and the quantization precision of the data stored in the in-memory compute memory.

[0213] OP[7]: The length can be 1 bit. When the value is 1, it indicates the all columns mode, which automatically and sequentially executes the calculation operations of the data in all columns (such as col.0 to col.63), similar to the above COMP_AC_INTx or ACT_INTx instructions indicating that the data stored in all columns in a row is calculated; when the value is 0, it indicates the normal mode, which executes the calculation operations of the data in the specified column, similar to the above COMP_INTx instructions indicating that the data stored in the specified column is calculated.

[0214] OP[6]: Reserved field, the length can be 1 bit.

[0215] OP[5:4]: This field can be 2 bits long and is used to configure the number of banks that are active simultaneously. For example, the value can be 1, 4, 8, 16, or other values ​​such as 2, 6, 12. For instance, the value of this field can directly indicate the number of banks that are active simultaneously. For example, configuring OP[5:4] to binary 00 indicates 1 bank is active simultaneously; configuring OP[5:4] to binary 01 indicates 4 banks are active simultaneously; configuring OP[5:4] to binary 10 indicates 8 banks are active simultaneously; and configuring OP[5:4] to binary 11 indicates 16 banks are active simultaneously. Understandably, the length of this field can be configured according to the maximum number of banks that are active simultaneously. The above values ​​for field length and the number of banks that are active simultaneously are only examples and should not constitute a specific limitation on the configuration of the mode register.

[0216] OP[3:0]: The field length can be 4 bits, used to configure the quantization precision, such as INT1, INT2, INT3, INT4, INT8, INT16, FP16, or FP32. For example, the 4 bits of OP[3:0] can be used to indicate the data type and quantization precision. For instance, the highest bit can indicate whether the data type is integer or floating-point, and the remaining 3 bits can indicate the quantization precision. Optionally, a value of 0 for the highest bit indicates a floating-point data type, and a value of 1 indicates an integer data type. For example, OP[3:0] = 1111 can be considered as configuring a quantization precision of INT8. Alternatively, each quantization precision can correspond to a unique OP[3:0] value. Configuring the OP[3:0] value directly indicates a specific quantization precision, eliminating the need for a specific position to distinguish data types. For example, OP[3:0] = 0000 corresponds to INT1, OP[3:0] = 0001 corresponds to INT2, OP[3:0] = 0010 corresponds to INT3, ..., OP[3:0] = 1111 corresponds to FP32, etc. Understandably, the length of this field can be configured according to the maximum quantization precision required. The above field length, quantization precision values, and the method of indicating quantization precision are merely examples and are not specifically limited here.

[0217] Optionally, for the quantization precision-aware configuration via the mode register, the computation instructions interacting between the processor and memory can simply indicate computation on data in a specified row / column, without specifying the quantization precision of that data. For example, the COMP instruction can indicate computation on data at a specified column address, and the quantization precision of the data stored in that column can be determined based on the OP[3:0] field in the mode register. This approach is suitable for scenarios where the number of reserved instruction types is limited and the existing number is insufficient to indicate multiple different quantization precisions. For instance, if the number of reserved instruction types is only 3, and 4 different quantization precisions (such as INT1, INT2, INT4, and INT8) need to be indicated, then the reserved number is only sufficient to indicate three of these quantization precisions. For example, COMP_INT1, COMP_INT2, and COMP_INT4 can indicate quantization precisions of INT1, INT2, and INT4 respectively, while COMP_INT8, which indicates quantization precision of INT8, cannot be indicated due to insufficient reserved number.

[0218] To facilitate understanding, the process of the processor instructing the in-memory computing memory to perform data calculations is illustrated below, based on the aforementioned method of quantization precision awareness. Taking an example where an in-memory computing memory has 64 columns per row, each column can store 256 bits of data, and the calculation precision of the computing unit is INT2, when the quantization precision of the weight parameters is INT2, one column can store 128 weight parameters with a quantization precision of INT2. Therefore, one row can store 64 * 128 = 8192 weight parameters with a quantization precision of INT2.

[0219] For example, in a method where the processor and memory communicate via predefined calculation instructions to achieve precision awareness, the processor can issue a COMP_INT2 calculation instruction, instructing the memory to calculate 128 INT2 weight parameters at a specified column address. Correspondingly, when the processor needs to complete the calculation of the aforementioned 8192 weight parameters, it can refer to... Figure 5 The processing flow shown allows the processor to first activate a specified Banka and its specified rows using the ACTIVATE instruction; then, it sequentially issues 64 COMP_INT2 calculation instructions to calculate the weight parameters of 64 columns from col.0 to col.63, thereby completing the calculation of 8192 weight parameters with a quantization precision of INT2, as shown in Formula 1 below. Optionally, after completing the calculation of all weight parameters in the 64 columns of a row, the processor can issue a PRECHARGE instruction to close the activated rows and Banka.

[0220] Formula 1:

[0221] The input vector V can be passed from the host (or processor) to the in-memory computing memory, and the number of elements in the input vector V corresponds to the number of weight parameters.

[0222] For example, in the method of quantization precision awareness between the processor and internal memory through a configuration mode register, the OP[3:0] field in the mode register can be configured as 0001, indicating that the weight parameters with quantization precision of INT2 are stored in the internal computation memory. The processor can issue a COMP calculation instruction to complete the calculation of the specified column address, that is, the calculation of 128 INT2 weight parameters. Further, when the processor needs to complete the calculation of the above 8192 weight parameters, the processor can issue 64 COMP calculation instructions sequentially to calculate the weight parameters of 64 columns from col.0 to col.63 in turn, thereby completing the calculation of 8192 weight parameters with quantization precision of INT2 as shown in Formula 1 above. Understandably, activation instructions and precharge instructions are also involved in the processing, which will not be elaborated here.

[0223] In one possible implementation, when the quantization precision of the weight parameters to be written to the in-memory computing memory is greater than the computing precision of the computing unit, the multi-bit quantized data can be first split into multiple smaller-bit sub-data and stored in different rows / columns of the in-memory computing memory. During calculation, the multiple sub-data can be read into the computing unit sequentially for calculation, and the calculation results of each sub-data can be shifted and accumulated through the shift circuit of the in-memory computing memory, thereby achieving the purpose of completing the multi-bit quantized data calculation using a computing unit with low computing precision.

[0224] For example, taking a computational precision of INT4 for the computing unit and a quantization precision of INT8 for the weight parameters as an example, the calculation process is briefly explained by splitting the INT8 quantized data of the weight parameters into two INT4 quantized data for calculation. First, for any INT8 quantized data X, it can be represented as the high 4 bits of data shifted left by 4 bits and then added to its low 4 bits, i.e., X = (H<<4) + L. For example, X = 01110010, then the high 4 bits of X H = 0111, the low 4 bits of X L = 0010, and the data after shifting H left by 4 bits H<<4 = 01110000, H<<4(01110000) + L(0010) = X(01110010).

[0225] Based on this, the multiplication calculation of INT8 quantized data X and another data Y can be referred to... Figure 6 As shown, the data precision of Y can be INT4. Multiplying X and Y can be understood as splitting the INT8 quantized data X into high 4 bits (represented by H) and low 4 bits (represented by L), then multiplying the high 4 bits and low 4 bits separately with the data Y, and finally adding them together. That is, X*Y = [(H<<4)+L]*Y = (H<<4)*Y + L*Y = [(H*Y)<<4] + L*Y. (Reference) Figures 7A-7D The process of multiplying X and Y can include the following calculation steps:

[0226] (1) As Figure 7A As shown, the high 4 bits of X are multiplied by Y to obtain the intermediate result R1;

[0227] (2) Figure 7B As shown, the intermediate result R1 is shifted left by four bits to obtain the result RH;

[0228] (3) Figure 7C As shown, the lower 4 bits of X are multiplied by Y to obtain the intermediate result RL;

[0229] (4) Figure 7DAs shown, RH and RL are added together to obtain the final result R.

[0230] It should be noted that after calculating the multiple sub-data obtained from the splitting, multiple intermediate results can be obtained. When calculating the final result, these intermediate results need to be shifted first, and then the calculation is performed on the shifted intermediate results to ensure the accuracy of the final result. The specific number of bits shifted for each intermediate result can be determined based on the position of the corresponding sub-data in the original data (i.e., the data before splitting). For example, if INT8 quantized data is split into high 4 bits (i.e., the first sub-data) and low 4 bits (i.e., the second sub-data), since there are 4 bits of data after the high 4 bits before splitting, the intermediate result obtained from calculating this part of the data needs to be shifted left by 4 bits; while there is no data after the low 4 bits before splitting, the intermediate result obtained from calculating this part of the data does not need to be shifted. For example, INT3 quantized data is split into high 1 bit data (i.e., the first sub-data), middle 1 bit data, and low 1 bit data (i.e., the second sub-data). Since there are 2 bits of data after the high 1 bit data before the split, the intermediate result obtained by calculating this part of the data needs to be shifted left by 2 bits; since there is 1 bit of data after the middle 1 bit data before the split, the intermediate result obtained by calculating this part of the data needs to be shifted left by 1 bit; and since there is no data after the low 1 bit data before the split, the intermediate result obtained by calculating this part of the data does not need to be shifted.

[0231] Optionally, the weight parameters with an INT8 quantization precision mentioned above can be split into 4+4, 2+2+2+2, 3+3+2, or 3+3+1+1, etc. When splitting data, the resource utilization of the INT4 calculation unit and the maximum amount of data that can be stored in each column (e.g., 256 bits) can be considered when choosing a specific splitting method. For example, if split into 2+2+2+2, although the storage resources of each column can be fully utilized (256 bits can be used when storing 128 data points), only 2 bits of resources in the calculation unit are used for each calculation, resulting in a resource utilization rate of only 50% for each calculation unit. To fully utilize the resources of the calculation unit, it is necessary to first divide the resources of the calculation unit so that each calculation unit can calculate two 2-bit data points, making the calculation process more complex than the 4+4 method. If split into 3+3+2, although the resource utilization rate of the calculation unit is improved, the storage resources in some columns cannot be fully utilized (256 is not divisible by 3). If split into 4+4, not only can the resources of the computing unit be fully utilized, but the storage resources of each column can also be fully utilized (256 is an integer multiple of 4). Therefore, for the combination of INT4 computing precision and INT8 quantization precision, the 4+4 splitting method can be preferred.

[0232] For ease of understanding, the following example illustrates the control method in this application embodiment by using the quantization sensing calculation instruction COMP_INT4 in conjunction with a calculation unit with a calculation precision of INT2 to complete the calculation of INT4 quantized data, and then combining the processing procedures of the storage and calculation stages. That is, taking the use of the INT2 calculation unit to complete the INT4 quantized data as an example.

[0233] For example, each row in the in-memory computing memory can contain 64 columns, and each column can store 256 bits of data. That is, one column can store 128 2-bit data points (quantization precision INT2) or 64 4-bit data points (quantization precision INT4). Therefore, one row can store 8192 INT2 weight parameters or 4096 INT4 weight parameters. The calculation result of all 4096 INT4 data points in one row with the input vector V is shown in Formula 2:

[0234] Formula 2:

[0235] For example, during the storage phase, since the quantization precision of the weight parameters is greater than the computational precision of the computational unit (INT4 is greater than INT2), each weight parameter can be split into two 2-bit data segments, and these two 2-bit data segments can be stored in two different cols, which can be adjacent cols. For example, for these 4096 INT4 weight parameters W... i,j You can refer to this. Figure 8A The data storage rules shown are used for storage. Since one col can store 128 2-bit data, it is possible to consider storing 128 W. i,j The 4-bit data is split into high 2 bits and low 2 bits, and then written using a single instruction (corresponding to the first memory access instruction) to store 128 W bytes. i,j The high 2 bits of data are stored in the k-th column (denoted as col.k, such as col.0). These 128 W's are then written using another write instruction (corresponding to the first memory access instruction). i,j The lower 2 bits of data are stored in the (k+1)th column (represented as col.k+1, e.g., col.1), W i,j This represents an INT4 weight parameter with a data length of 4 bits. Its high 2 bits can be represented as W. i,j H2b Its lower 2 bits can be represented as W i,j L2b Optionally, 128 W i,j The lower 2 bits of data can also be stored in the k-th column (denoted as col.k), these 128 W i,jThe high 2 bits of data can be stored in the (k+1)th column (denoted by col.k+1), meaning that the high bits of data can be stored in the preceding column or the following column. Correspondingly, the low bits of data can also be stored in the preceding column or the following column, without any specific restrictions.

[0236] Accordingly, the other 3968 (4096-128) INT4 weight parameters are calculated according to the aforementioned 128 W parameters. i,j Data is stored according to the storage rules, ultimately resulting in 4096 W. i,j The data is all split and stored in different columns, and all share the same weight parameter W. i,j The high 2 bits and low 2 bits of data can be stored in adjacent columns. It should be noted that the above explanation is based on the example of data access in internal memory on a column-by-column basis. Understandably, when internal memory can access data on a row-by-row basis, the weight parameters can also be stored on a row-by-row basis. That is to say, the technical solution of splitting and storing the weight parameters in different rows should also fall within the protection scope of this application's embodiments. For ease of understanding, this application uses data access in internal memory on a column-by-column basis as an example for illustrative explanation.

[0237] For the above Figure 8A The data storage method shown can be implemented using the following method, where the execution flow of computation instructions during the computation phase can be referenced: The processor can issue a COMP_INT4 instruction to specify the column address col.k where the high 2 bits of data are stored, and the internal computation memory will then execute the instruction to store the 128 W values ​​in that column. i,j H2b The data (i.e., the first sub-data) is calculated, and the result is sent to the internal shift circuit for left shifting by 2 bits before being passed to the accumulator; then, the 128 W values ​​stored in col.k+1 are read. i,j L2b The data (i.e. the second sub-data) is calculated, and then the calculation result is passed into the accumulator and accumulated with the calculation result of the shifted high 2 bits of data, so as to achieve the purpose of using the calculation circuit with INT2 calculation precision to complete the calculation of the weight parameters with INT4 quantization precision.

[0238] Optionally, the above implementation method can be based on Figure 2EThe computational unit structure shown is complete. The data in col.k (i.e., the first sub-data) is processed by the multiplication circuit to obtain 128 multiplication results. These 128 multiplication results are then sent to the shift circuit for shifting. The shifted results are then sent to the addition circuit for addition calculation, and the added result (i.e., the first result) is then passed to the accumulator. The data in col.k+1 (i.e., the second sub-data) is processed by the multiplication circuit to obtain another 128 multiplication results. These are then sent to the addition circuit for addition calculation, and the added result (i.e., the second result) is then passed to the accumulator and added to the previous results. Optionally, the above implementation can also be based on... Figure 2F The computational unit structure shown is complete. The data in col.k (i.e., the first sub-data) is processed by the multiplication circuit to obtain 128 multiplication results, which are then fed into the addition circuit for addition to obtain one sum. This sum is then fed into the shift circuit for shifting, and the shifted result (i.e., the third result) is fed into the accumulator. The data in col.k+1 (i.e., the second sub-data) is processed by the multiplication circuit to obtain another 128 multiplication results, which are then fed into the addition circuit for addition. The sum (i.e., the fourth result) is then fed into the accumulator and added to the previous results.

[0239] Based on this, regarding the aforementioned 4096 INT4 weight parameters W i,j The calculation process can be referenced as follows: Figure 8BThe process shown involves the processor first issuing an activation instruction to activate a specified row `rowi` in `Banka`, and then sequentially issuing 32 `COMP_INT4` calculation instructions to calculate the 64 columns of data included in `rowi`. Each `COMP_INT4` instruction calculates the data stored in two specified columns. For example, the first `COMP_INT4` can calculate the data stored in `col.0` and `col.1`, shifting the result of the calculation on `col.0` and adding it to the result of the calculation on `col.1` to complete the calculation of the first group of 128 INT4 weight parameters. The second `COMP_INT4` can calculate the data stored in `col.2` and `col.3`, shifting the result of the calculation on `col.2` and adding it to the result of the calculation on `col.3` to complete the calculation of the second group of 128 INT4 weight parameters. The third COMP_INT4 calculates the data stored in col.4 and col.5, shifts the result of the calculation of col.4, and adds it to the result of the calculation of col.5 to complete the calculation of the third group of 128 INT4 weight parameters. This can be added to the results of the previous multiple groups of INT4 weight parameters. Similarly, the 32nd COMP_INT4 calculates the data stored in col.62 and col.63, shifts the result of the calculation of col.62, and adds it to the result of the calculation of col.63 to complete the calculation of the 32nd group of 128 INT4 weight parameters. This completes the calculation of 4096 INT4 weight parameters. Optionally, after completing the calculation of all data in rowi, the processor can issue a precharge instruction to close that rowi.

[0240] It should be noted that for computation units with a precision of INT2, since the weight parameters with a quantization precision of INT4 are split into two 2-bit data stored in different columns (col), a COMP_INT4 instruction will fetch the data stored in both columns for calculation, while a COMP_INT2 instruction will fetch the data from only one column for calculation. Therefore, the latency of a COMP_INT4 instruction is twice that of a COMP_INT2 instruction, but the number of COMP_INT4 instructions is more economical, being half the number of COMP_INT2 instructions. It should also be noted that, in addition to using the predefined COMP_INT4 instruction mentioned above to detect quantization precision, the processor can also detect quantization precision through the configuration mode register. In this case, the execution flow of the computation instruction can be referred to the above. Figure 8B The execution flow using the COMP_INT4 instruction can be achieved simply by replacing the COMP_INT4 instruction with the COMP instruction; this will not be explained further here.

[0241] Alternatively, in addition to the above Figure 8A In addition to the data storage method of splitting and storing the same weight parameter in adjacent columns, these 4096 INT4 weight parameters W i,j You can also refer to Figure 8C The storage rules shown use a partitioned approach for data storage. Consider storing 4096 W... i,j The high 2 bits of data are stored in the region from column 0 to column 31 (i.e., col.0 to col.31, a total of 32 columns) (for example, indicated by 32 write instructions (corresponding to multiple first memory access instructions)). These 4096 W i,j The lower 2 bits of data are stored in the region from column 32 to column 63 (i.e., col.32 to col.63, a total of 32 columns) (e.g., indicated by 32 write instructions (corresponding to multiple first memory access instructions)). i,j This represents an INT4 weight parameter with a data length of 4 bits. Its high 2 bits can be represented as W. i,j H2b Its lower 2 bits can be represented as W i,j L2b .

[0242] For the above Figure 8C The data storage method shown can be implemented using the following method for the execution flow of calculation instructions during the calculation phase: The processor can issue COMP_AC_INT4 or ACT_INT4 instructions to sequentially retrieve data from each column starting from col.0 in the internal calculation memory for calculation. After the data stored in col.0 to col.31 has been calculated, the accumulated result is first sent to the internal shift circuit for left shift by 2 bits, and then the left-shifted calculation result is passed to the accumulator to await the next calculation. Then, the data from each column is sequentially retrieved from col.32 to col.63 to perform multiplication and accumulation calculations, and this part of the calculation result is also passed to the accumulator. Then, it is accumulated with the calculation result of the data shifted by 2 bits above, thus completing the calculation of 4096 INT4 weight parameters. It should be noted that the execution flow of the COMP_AC_INT4 and ACT_INT4 calculation instructions may differ, such as Figure 8D As shown, for the COMP_AC_INT4 instruction-based method, row selection needs to be performed by issuing the ACT activation instruction before issuing the COMP_AC_INT4 instruction. For the ACT_INT4 instruction-based method, as shown... Figure 8EAs shown, since the ACT_INT4 instruction itself is a row selection command, there is no need to issue an additional ACT instruction for row selection. Optionally, after completing the calculation based on the COMP_AC_INT4 or ACT_INT4 calculation instruction, the processor can close the opened rows by issuing a precharge instruction.

[0243] Optionally, the above implementation method can be based on Figure 2E The computational unit structure shown is complete. Data from any column (col.0 to col.31), i.e., the first sub-data, is multiplied by a multiplication circuit to obtain 128 multiplication results. These 128 multiplication results are then shifted by a shift circuit, and the shifted results are added by an addition circuit. The added result (i.e., the first result) is then passed to an accumulator. Data from any column (col.32 to col.63), i.e., the second sub-data, is multiplied by a multiplication circuit to obtain another 128 multiplication results. These are then added by an addition circuit, and the added result (i.e., the second result) is then passed to an accumulator and added to the previous results. Optionally, the above implementation can also be based on... Figure 2F The calculation unit structure shown is complete. The data in any column from col.0 to col.31 (i.e., the first sub-data) is calculated by the multiplication circuit to obtain 128 multiplication results, which are then fed into the addition circuit for addition to obtain one sum result. This sum result is then fed into the shift circuit for shifting, and the shifted result (i.e., the third result) is fed into the accumulator. The data in any column from col.32 to col.63 (i.e., the second sub-data) is calculated by the multiplication circuit to obtain another 128 multiplication results, which are then fed into the addition circuit for addition. The sum result (i.e., the fourth result) is then fed into the accumulator and added to the previous results.

[0244] It should be noted that the above Figure 8CThe data storage method shown divides a row of 64 columns into two partitions (e.g., partitions col.0 to col.31 and col.32 to col.63). The high 2 bits of data are stored in partitions col.0 to col.31, and the low 2 bits of data are stored in partitions col.32 to col.63. Optionally, the row of 64 columns can be divided into more partitions, such as four partitions, including partition a (col.0 to col.15), partition b (col.16 to col.31), partition c (col.32 to col.47), and partition d (col.48 to col.63). Partition a can store 2048 high 2 bits of data, partition b can store 2048 low 2 bits of data, partition c can store the remaining 2048 high 2 bits of data, and partition d can store the remaining 2048 low 2 bits of data. Understandably, the number of partitions for storage can also be other values, such as 8, 16, etc., and is not specifically limited here. Understandably, dividing a row of 64 columns into multiple partitions, when each partition contains 1 row / column, yields the following: Figure 8A The data storage method shown.

[0245] In one possible implementation, the shifting process described above can be performed by a shift circuit in the computing unit or by the processor, making it suitable for structures where the computing unit does not have a shift circuit. Optionally, the processor can first receive intermediate results (such as the fifth and sixth results) calculated from multiple sub-data in the internal memory, then shift the intermediate results corresponding to the high-bit data, and finally add the shifted result (such as the seventh result) to the intermediate results corresponding to the low-bit data (such as the sixth result) to obtain the final result. For example, for the above... Figure 8C The data storage method shown can be implemented by the processor through the following process of shifting: The data in any column from col.0 to col.31 (i.e., the first sub-data) is multiplied by the multiplication circuit to obtain 128 multiplication results. These 128 multiplication results are then fed into the addition circuit for addition, and the result of the addition (i.e., the fifth result) is output to the processor. The data in any column from col.32 to col.63 (i.e., the second sub-data) is multiplied by the multiplication circuit to obtain another 128 multiplication results. These are then fed into the addition circuit for addition, and the result of the addition (i.e., the sixth result) is also output to the processor. After receiving the fifth and sixth results, the processor can first shift the fifth result to obtain the shifted result (the seventh result), and then add the shifted seventh result and the sixth result to obtain the final result.

[0246] Optionally, the processor can perform a shift operation on the fifth result by using a shift coefficient, such as fifth result * shift coefficient = seventh result. The shift coefficient is determined by the initial position of the first sub-data within the data to be processed; for example, shift coefficient = 2. n 'n' represents the number of bits that the intermediate result corresponding to the first sub-data needs to be shifted. For example, if the 4-bit data of INT4 is split into two 2-bit data of INT2, the intermediate result corresponding to the higher 2 bits needs to be shifted by 2 bits. Accordingly, the shift factor is 4(2^n). 2 For example, when an 8-bit INT8 data is split into two 4-bit INT4 data, the intermediate result corresponding to the higher 4 bits needs to be shifted by 4 bits. Accordingly, the shift factor is 16(2^6). 4 For example, the 3-bit data of INT3 is split into three 1-bit data. The intermediate result corresponding to the higher 1 bit needs to be shifted by 2 bits. Accordingly, the shift factor is 4(2). 2 The intermediate result corresponding to 1 bit of data needs to be shifted by 1 bit, and the corresponding shift factor is 2(2). 1 The intermediate result corresponding to the lower 1 bit of data does not require shifting.

[0247] Based on the above Figure 8C Taking the data storage method shown as an example, the 4-bit data of INT4 can be split into two 2-bit data of INT2. This provides a simple explanation of the instruction execution flow for processor shift processing. See also... Figure 13 The flowchart shown may include the following steps:

[0248] 1. Instruction preparation: Calculation instructions are generated based on information from high-bit data. The calculation instructions may include the physical address information of each block of data in the internal memory.

[0249] 2. Send the input vector to the Global Buffer in internal memory to be used as input for subsequent calculations.

[0250] 3. Send the calculation instruction to the internal memory. The internal memory reads the data from the corresponding col in the DRAM array into the calculation input register. At the same time, it also reads the input vector to be calculated from the Global Buffer into the calculation input register.

[0251] 4. After the data is ready, the internal memory begins to calculate the high-bit data in col.x, obtains the result x, and passes the result x into the accumulator.

[0252] 5. For the high-bit data in other col.y, repeat steps 1-4 and accumulate the previous result x in the accumulator until the high-bit data stored in the 32 cols are calculated, to obtain intermediate result 1 (i.e. the fifth result), and return intermediate result 1 to the processor (xPU, such as CPU, NPU or GPU).

[0253] 6. For the calculation of low-bit data, repeat steps 1-5 above, change high-bit data to low-bit data, and obtain intermediate result 2 (i.e., the sixth result). Similarly, return intermediate result 2 to the processor.

[0254] 7. Process intermediate result 1 and intermediate result 2 accordingly to obtain the final result of the dot product of the two vectors. Final result = Intermediate result 1 * 2 n +Intermediate results 2, 2 n The value of n is the shift factor, which is 2, meaning that the calculation result of the high 2 bits of data needs to be shifted left by 2 bits.

[0255] Optionally, the multiple sub-data obtained after splitting can be placed in the same bank (as above). Figure 8C Different rows / columns (as shown) can also be placed in different banks. For example, 4 bits of INT4 data can be split into two 2 bits of INT2 data. One bank (e.g., bank 0) can hold the high 2 bits, and the other bank (e.g., bank 1) can hold the low 2 bits. Figure 14A As shown, high and low bit data can be computed concurrently, and the results can be returned to the processor simultaneously. For the computation process of splitting data and storing it in different banks, please refer to [link to relevant documentation]. Figure 14B The process shown includes the following steps:

[0256] 1. Transmit Input Vector: Send the input vector to a specified area, such as the Global Buffer.

[0257] 2. Instruction preparation: The calculation instruction contains the addresses of data in bank0 and bank1.

[0258] 3. Send calculation instructions to read data from the corresponding col of each bank into the calculation input register, and simultaneously read the input vector from the Global Buffer into the calculation input register of each bank. PIM starts calculation, and the calculation result of each col is accumulated into the value of each bank, resulting in a result of 0 for bank 0 and a result of 1 for bank 1.

[0259] 4. Repeat steps 1-3 until all the data stored in the 64 cells of the two banks has been calculated, and intermediate result 0 and intermediate result 1 are obtained. Then, return intermediate result 0 and intermediate result 1 to the processor (if the data transmitted in step 1 has not been calculated, this step can be skipped).

[0260] 5. Calculate the intermediate result (0) and intermediate result (1) to obtain the final result. Final result = Intermediate result (0) * 2 n +Intermediate results 1, 2 n The value of n is the shift factor, which is 2, meaning that the calculation result of the high 2 bits of data needs to be shifted left by 2 bits.

[0261] The following example illustrates the control method in this application embodiment by using the quantization sensing calculation instruction COMP_INT3 in conjunction with a calculation unit with a calculation precision of INT2 to complete the calculation of INT3 quantized data, combining the processing procedures of the storage stage and the calculation stage. That is, taking the use of the INT2 calculation unit to complete the INT3 quantized data as an example.

[0262] For example, each row in the in-memory computing memory can contain 64 columns, and each column can store 256 bits of data. Since neither 256 nor 64 is divisible by 3, to ensure the utilization of storage resources for each column, when splitting the INT3 weight parameters for storage, only 63 columns in a row can be used for storage. Therefore, a row can store 5376 (63*256 / 3) INT3 weight parameters. The calculation result of all 5376 INT3 data points in a row with the input vector V is shown in Formula 3:

[0263] Formula 3:

[0264] For example, during the storage phase, since the quantization precision of the weight parameters is greater than the computational precision of the computational unit (INT3 is greater than INT2), each weight parameter can be split into one 2-bit data and one 1-bit data, and then these two split data are stored in two different cols respectively. For example, for these 5376 INT3 weight parameters W i,j You can refer to this. Figure 9A The data storage rules shown are used to store the 5376 INT3 weight parameters, grouping them into 21 groups of 256 weight parameters each. Each group contains 256 INT3 weight parameters W. i,j The first 128 bytes can be written using a single write instruction (corresponding to the first memory access instruction). i,j The high 2 bits are stored in the k-th column (denoted as col.k, such as col.0). The last 128 W bits can be moved using a write instruction (corresponding to the first memory access instruction). i,jThe high 2 bits are stored in the (k+1)th column (denoted as col.k+1, e.g., col.1). A single write instruction (corresponding to the first memory access instruction) can move all 256 W values. i,j The lower 1 bit is stored in the (k+2)th column (represented as col.k+2, e.g., col.2), W i,j This represents an INT3 parameter, where the high 2 bits are represented as W. i,j H2b Its lower 1 bit is represented as W i,j L1b Optionally, the first 128 Ws i,j The high 2 bits of data can also be stored in the (k+1)th column (or the (k+2)th column), and the last 128 W i,j The high 2 bits of data can also be stored in the (k+2)th column (or the kth column), 256 W i,j The lower 2 bits of data can also be stored in the kth column (or the k+1th column), without any specific restrictions here.

[0265] Accordingly, the 256 INT3 weight parameters in each of the other 20 groups are calculated according to the aforementioned 256 W parameters in Group 1. i,j Data is stored according to the storage rules, ultimately resulting in 256 Ws per group. i,j The data is all split and stored in different columns, and all are weighted by the same group of parameters W. i,j The high 2 bits and low 1 bit of data can be stored in three adjacent columns.

[0266] For the above Figure 9A The data storage method shown can be implemented using the following method for the execution flow of computation instructions during the computation phase: The processor issues a COMP_INT3 instruction specifying the column address col.k to store the high 2 bits, and the memory calculates the first 128 high 2 bits of data W stored in that column. i,j H2b The calculation result Res is obtained by performing the calculation. H2b1 Continue reading the last 128 high 2 bits of data W stored in col.k+1. i,j H2b The calculation result Res is obtained by performing the calculation. H2b2 , with Res H2b1 After accumulation, the accumulated result is sent to the internal shift circuit for left shift by 1 bit, and then the shifted result is sent back to the accumulator; continue reading the 256 lower 1 bits of data W stored in col.k+2. i,j L1b The calculation is performed, and the result is passed to the accumulator. It is then added to the result of the calculation of the shifted high 2 bits, thus completing the calculation of 256 INT3 weight parameters.

[0267] Optionally, the above implementation method can be based on Figure 2E The computational unit structure shown is complete. The data in col.k (i.e., part of the first sub-data) is processed by the multiplication circuit to obtain 128 multiplication results, which are then shifted by the shift circuit. The shifted results are then fed into the addition circuit for addition calculation, and the summed result is then fed into the accumulator. The data in col.k+1 (i.e., part of the first sub-data) is processed by the multiplication circuit to obtain another 128 multiplication results, which are then shifted by the shift circuit. The shifted results are then fed into the addition circuit for calculation, and the summed result is then fed into the accumulator and added to the previous results (i.e., obtaining the first result). The data in col.k+2 (i.e., the second sub-data) is processed by the multiplication circuit to obtain 256 multiplication results, which are then fed into the addition circuit for addition calculation, and the summed result (i.e., the second result) is then fed into the accumulator and added to the previous results. Optionally, the above implementation can also be based on... Figure 2F The computational unit structure shown is complete. The data in col.k (i.e., part of the first sub-data) is calculated by the multiplication circuit to obtain 128 multiplication results, which are then fed into the addition circuit for addition to obtain one sum. This sum is then fed into the shift circuit for shifting, and the shifted result is fed into the accumulator. The data in col.k+1 (i.e., part of the first sub-data) is calculated by the multiplication circuit to obtain another 128 multiplication results, which are then fed into the addition circuit for addition to obtain another sum. This sum is then fed into the shift circuit for shifting, and the shifted result is fed into the accumulator and added to the previous result (i.e., obtaining the third result). The data in col.k+2 (i.e., the second sub-data) is calculated by the multiplication circuit to obtain 256 multiplication results, which are then fed into the addition circuit for addition. This sum (i.e., the fourth result) is fed into the accumulator and added to the previous result. Alternatively, the shift operation in the above implementation can also be performed by the processor, without relying on the shift circuit in the computing unit. The specific calculation process can be found in the above description. Figure 13 , Figure 14A or Figure 14B The relevant descriptions will not be elaborated here.

[0268] Based on this, regarding the aforementioned 5376 INT3 weight parameters W i,jThe calculation process can be referenced as follows: The processor can first issue an activation instruction to activate the specified row i in Banka, and then sequentially issue 21 COMP_INT3 instructions to calculate the 63 columns of data included in rowi. Each COMP_INT3 instruction calculates the data stored in 3 specified columns. The first COMP_INT3 can calculate the data stored in col.0, col.1, and col.2. After accumulating and shifting the calculation results of col.0 and col.1, it is then accumulated with the calculation result of col.2 to complete the calculation of the first group of 256 INT3 weight parameters. The second COMP_INT3 can calculate the data stored in col.3, col.4, and col.5. After accumulating and shifting the calculation results of col.0 and col.1, it is then accumulated with the calculation result of col.2 to complete the calculation of the first group of 256 INT3 weight parameters. After accumulating and shifting the calculation results of data l.3 and col.4, they are then accumulated and added to the calculation result of data col.5 to complete the calculation of the second group of 256 INT3 weight parameters. This can be accumulated with the calculation results of the previous group of INT3 weight parameters. Similarly, the 21st COMP_INT3 can calculate the data stored in col.60, col.61, and col.62. After accumulating and shifting the calculation results of data col.60 and col.61, they are then accumulated and added to the calculation result of data col.62 to complete the calculation of the 21st group of 256 INT3 weight parameters. This can be accumulated with the calculation results of multiple previous groups of INT3 weight parameters, thus completing the calculation of 5376 INT3 weight parameters. Optionally, after completing the calculation of all data in rowi, the processor can issue a precharge instruction to close that rowi.

[0269] It should be noted that for a computation unit with a precision of INT2, since a COMP_INT3 instruction will fetch data from 3 cols for computation, while a COMP_INT2 instruction will fetch data from 1 col for computation, the COMP_INT3 instruction will have twice the time of the COMP_INT2 instruction. However, the number of COMP_INT3 instructions is less than the number of COMP_INT2 instructions.

[0270] Alternatively, in addition to the above Figure 9A In addition to the data storage method of splitting and storing the same set of weight parameters in adjacent columns, these 5376 INT3 weight parameters W i,j You can also refer to Figure 9B The storage rule shown uses a partitioned data storage method. Consider storing the 5376 INT3 weight parameters W in the area from col.0 to col.41. i,jThe high 2 bits of data can be stored in each column (col), which can hold 128 2-bit data. Additionally, the 5376 INT3 weight parameters W are stored in the area from col.42 to col.62. i,j The lower 1 bit of data, each column (col) can store 256 1-bit data.

[0271] For the above Figure 9B The data storage method shown can be referenced in the following two methods (a and b) for the execution flow of computation instructions during the computation phase:

[0272] Method a: The processor can issue a COMP_INT3 instruction, which specifies the high 2 bits of the column address col.k. After the internal calculation memory sequentially executes the calculation of the first 256 high 2 bits of data in addresses col.k and col.k+1, the calculation results of the two columns are accumulated and sent to the internal shift circuit for left shift by 1 bit, and then the shifted result is passed to the accumulator. The first 256 low 1 bits of data in address col.(k / 2+42) are retrieved and calculated. The calculation result is then passed to the accumulator and accumulated with the calculation result of the high 2 bits. This completes the calculation of 256 INT3 weight parameters. Accordingly, the calculation process for the remaining 5120 (5376-256) INT3 weight parameters can refer to the calculation of the aforementioned 256 INT3 weight parameters. In addition, the remaining 20 sets (60col) of data are calculated sequentially through 20 COMP_INT3 instructions. Finally, the calculation results of 21 sets (63col) are accumulated to calculate the 5376 INT3 weight parameters.

[0273] Method b: The processor can issue a COMP_AC_INT3 or ACT_INT3 instruction. The internal calculation memory sequentially retrieves the data from each column starting from col.0 and performs calculations. After the data in col.0 to col.41 has been calculated, the accumulated result is sent to the internal shift circuit and shifted left by 1 bit before being passed to the accumulator. The multiplication and accumulation calculation of the data in col.42 to col.62 continues to be executed sequentially. The result of this part of the calculation is also passed to the accumulator and then accumulated with the result of the data shifted left by 1 bit above, thus calculating 5376 INT3 weight parameters.

[0274] Optionally, both method a and method b above can be based on Figure 2E or Figure 2F The computational unit structure shown is now complete. The computation process for different computational unit structures can be referred to the above. Figure 8A , Figure 8C or Figure 9A The relevant descriptions of the various implementation methods will not be repeated here.

[0275] The following example illustrates the control method in this application embodiment by using the quantization sensing calculation instruction COMP_INT3 in conjunction with a calculation unit with a calculation precision of INT1 to complete the calculation of INT3 quantized data, combined with the processing procedures of the storage and calculation stages. Specifically, it uses the INT1 calculation unit to complete the INT3 quantized data calculation as an example.

[0276] For example, each row in the in-memory computing memory can contain 64 columns, and each column can store 256 bits of data. Using 63 columns in a row for storage, a row can store 5376 (63*256 / 3) INT3 weight parameters. Therefore, the calculation result of all 5376 INT3 data points in a row with the input vector V is shown in Formula 3 above.

[0277] Optionally, during the storage stage, since the quantization precision of the weight parameters is greater than the computational precision of the computational unit (INT3 is greater than INT1), each weight parameter can be split into three 1-bit data units, and these three 1-bit data units can be stored in three different cols. For example, for these 5376 INT3 weight parameters W... i,j You can refer to this. Figure 10A The data storage rules shown are used to store the 5376 INT3 weight parameters, grouping them into 21 groups of 256 weight parameters each. Each group contains 256 INT3 weight parameters W. i,j It is possible to write 256 Ws using a single write instruction (corresponding to the first memory access instruction). i,j The high 1 bit is stored in the k-th column (denoted as col.k, such as col.0), and can be moved by a single write instruction (corresponding to the first memory access instruction). i,j The middle 2 bits are stored in the (k+1)th column (denoted as col.k+1, e.g., col.1), and can be moved by a single write instruction (corresponding to the first memory access instruction). i,j The lower 1 bit is stored in the (k+2)th column (represented as col.k+2, e.g., col.2), W i,j This represents an INT3 parameter, with the high 1 bit representing W. i,j H1b 1 bit is represented as W i,j M1b The lower 1 bit is represented as W i,j L1b Optionally, the high 1 bit of data can also be stored in the (k+1)th column (or the (k+2)th column), the middle 1 bit of data can also be stored in the (k+2)th column (or the kth column), and the low 1 bit of data can also be stored in the (k)th column (or the (k+1)th column), without any specific restrictions.

[0278] Accordingly, the 256 INT3 weight parameters in each of the other 20 groups are calculated according to the aforementioned 256 W parameters in Group 1. i,j Data is stored according to the storage rules, ultimately resulting in 256 Ws per group. i,j The data is split and stored in three different columns, which can be three adjacent columns.

[0279] For the above Figure 10A The data storage method shown can be implemented using the following method for the execution flow of computation instructions during the computation phase: The processor issues a COMP_INT3 instruction specifying the column address col.k to store the high 1 bit. The internal computation memory performs calculations on the 256 high 1 bits of data stored in that column, and then passes the calculation result to the internal shift circuit to shift left by 1 bit to obtain Res. H1b Automatically read 1 bit of data from the 256 elements stored in col.k+1, obtain the calculation result, and then compare it with Res. H1b The process involves accumulating the data, shifting the accumulated result left by one bit, and then inputting it into the accumulator. Next, the 256 lower 1-bit data stored in `col.k+2` are automatically read, calculated, and the result is input into the accumulator and accumulated with the previous result, thus completing the calculation of the 256 INT3 weight parameters. Optionally, after the processor issues a `COMP_INT3` instruction, the in-memory computation memory can also directly shift the calculated result of the 256 higher 1-bit data left by two bits and input it into the accumulator after obtaining it. Subsequently, after obtaining the calculated result of the 256 lower 1-bit data, the in-memory computation memory can first shift it left by one bit and input it into the accumulator to accumulate with the shifted result of the higher 1-bit data. Finally, the subsequent in-memory computation memory calculates the 256 lower 1-bit data and inputs the result into the accumulator to accumulate with the previous result, thereby completing the calculation of the 256 INT3 weight parameters. Optionally, the above implementation can be based on... Figure 2E or Figure 2F The computational unit structure shown is now complete. The computation process for different computational unit structures can be referred to the above. Figure 8A , Figure 8C or Figure 9A The descriptions of the various implementation methods will not be repeated here. Optionally, the shift operations in the above implementation methods can also be performed by the processor, without relying on the shift circuit in the computing unit. The specific calculation process can be found above. Figure 13 , Figure 14A or Figure 14B The relevant descriptions will not be elaborated here.

[0280] Based on this, regarding the aforementioned 5376 INT3 weight parameters W i,jThe calculation process can be referenced as follows: The processor can first issue an activation instruction to activate the specified row rowi in Banka, and then issue 21 COMP_INT3 instructions in sequence to calculate the 63 columns of data included in rowi. Each COMP_INT3 instruction calculates the data stored in the specified 3 columns.

[0281] Alternatively, in addition to the above Figure 10A In addition to the data arrangement method of splitting and storing the same set of weight parameters in adjacent columns, these 5376 INT3 weight parameters W i,j You can also refer to Figure 10B The storage rule shown uses a partitioned data storage method. Consider storing the 5376 INT3 weight parameters W in the area from col.0 to col.20. i,j The high 1 bit of data can be stored in each column (col), which can hold 256 1-bit data. The 5376 INT3 weight parameters W are stored in the area from col.21 to col.41. i,j Each column (col) can store 256 1-bit data entries; additionally, the 5376 INT3 weight parameters W are stored in the area from col.42 to col.62. i,j The lower 1 bit of data, each column (col) can store 256 1-bit data.

[0282] For the above Figure 10B The data storage method shown can be referenced in the following methods c and d for the execution flow of computation instructions during the computation phase:

[0283] Method c: The processor can issue a COMP_INT3 instruction, which specifies the high 1-bit column address col.k. After the in-memory computation memory completes the calculation of the 256 high 1-bit data in col.k, the calculation result is shifted by a shift circuit and the shifted result is passed into the accumulator. The processor automatically retrieves the 256 low 1-bit data in address col.k+21 for calculation, shifts the calculation result by a shift circuit, and passes the shifted result into the accumulator to accumulate with the shifted result of the high 1-bit data. The processor automatically retrieves the 256 low 1-bit data in address col.k+42 for calculation, and passes the calculation result into the accumulator to accumulate with the calculation result of the first 2 bits of data. This completes the calculation of the 256 INT3 weight parameters. Accordingly, the calculation process for the remaining 5120 (5376-256) INT3 weight parameters can refer to the calculation of the aforementioned 256 INT3 weight parameters. The remaining 20 sets (60col) of data are calculated sequentially through 20 COMP_INT3 instructions. Finally, the calculation results of 21 sets (63col) are accumulated to calculate the 5376 INT3 weight parameters.

[0284] Method d: The processor can issue a COMP_AC_INT3 or ACT_INT3 instruction. The internal calculation memory sequentially retrieves 256 high-bit data from each column starting from col.0 for calculation. After the data from col.0 to col.20 have been calculated, the accumulated result is sent to the internal shift circuit and shifted left by 1 bit before being sent to the accumulator. The calculation of the middle-bit data of each column from col.21 to col.41 continues sequentially. The calculation result of the middle-bit data is added to the shift result of the high-bit data, shifted left by 1 bit, and then sent to the accumulator. The multiplication and accumulation calculation of the lower-bit data of each column from col.42 to col.62 continues sequentially. This part of the calculation result is also sent to the accumulator and then added to the previous shift result, thus calculating 5376 INT3 weight parameters.

[0285] Optionally, both method c and method d above can be based on Figure 2E or Figure 2F The computational unit structure shown is now complete. The computation process for different computational unit structures can be referred to the above. Figure 8A , Figure 8C or Figure 9A The relevant descriptions of the various implementation methods will not be repeated here.

[0286] It should be noted that, in addition to the combinations of INT2 calculation unit and INT4 weight parameter, INT2 calculation unit and INT3 weight parameter, and INT1 calculation unit and INT3 weight parameter mentioned above, there can be many other combinations. This application does not specifically limit the calculation precision of the calculation unit or the quantization precision of the weight parameter in its embodiments. For example, in the combination of INT2 calculation unit and INT5 weight parameter, the 5-bit weight parameter can be split into 2+2+1, 2+1+2, or 1+2+2 during the storage stage and written into 3 rows / columns for storage. The calculation process during the calculation stage can refer to the calculation of the combination of INT1 calculation unit and INT3 weight parameter, and will not be elaborated further here.

[0287] This application also provides a processor coupled to internal memory, the internal memory including at least one storage area Bank and at least one computing unit; each Bank includes M rows * N columns of storage units, and each computing unit has a computing precision of K, where M, N, and K are all greater than 0; the processor is used for:

[0288] Acquire the data to be processed; the quantization precision of the data to be processed is L;

[0289] When the quantization precision L of the data to be processed is greater than the calculation precision K of the computing unit, the data to be processed is split into multiple sub-data; the quantization precision of each sub-data is less than or equal to the calculation precision K of the computing unit.

[0290] According to the storage rules, a first memory access instruction is sent to each of the multiple sub-data items; the first memory access instruction is used to instruct the internal memory to write the sub-data items into a specified row or column, and the specified rows or columns are different among the multiple first memory access instructions.

[0291] This application also provides an internal memory coupled to a processor, the internal memory including at least one storage area Bank and at least one computing unit; each Bank includes M rows * N columns of storage units, and each computing unit has a computing precision of K, where M, N, and K are all greater than 0; the method includes:

[0292] The processor receives a second memory access instruction; the second memory access instruction instructs the internal memory to perform calculations on multiple sub-data; the multiple sub-data are multiple sub-data obtained by the processor splitting the data to be processed when the quantization precision L of the data to be processed is greater than the calculation precision K of the calculation unit; the quantization precision of the sub-data is less than or equal to the calculation precision K of the calculation unit.

[0293] The storage addresses of the plurality of sub-data are determined according to the second memory access instruction and storage rules;

[0294] In response to the second memory access instruction, the plurality of sub-data are read sequentially from the memory address;

[0295] The final result is determined by using the multiple sub-data.

[0296] Furthermore, this application also provides an electronic device, characterized in that it includes one or more processors and one or more internal memories; the one or more processors are coupled to the one or more internal memories; the one or more processors are used to implement the functions involved in any of the above embodiments, such as splitting data and issuing memory access instructions; the one or more internal memories are used to implement the functions involved in any of the above embodiments, such as receiving memory access instructions and performing data storage, data calculation, etc. according to the memory access instructions.

[0297] This application also provides a computer program for implementing the operations and / or processes performed by either a processor or internal memory in the method provided in this application.

[0298] This application also provides a readable storage medium storing computer code that, when executed on a computer, causes the computer to perform the operations and / or processes described in the method provided in this application, which are executed by either the processor or the internal memory.

[0299] This application also provides a computer program product comprising computer code or a computer program that, when run on a computer, causes the operations and / or processes in the method provided in this application to be executed by either a processor or internal memory.

[0300] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0301] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0302] In the several embodiments provided in this application, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, or it may be an electrical, mechanical or other form of connection.

[0303] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A control method characterized by, The application is applied to a processor coupled with an internal memory, and the internal memory comprises at least one storage area Bank and at least one calculation unit; each Bank comprises M rows*N columns of storage units, and the calculation precision of each calculation unit is K, M, N and K are all integers greater than 0; the method comprises: obtaining to-be-processed data; the quantization precision of the to-be-processed data is L; when the quantization precision L of the to-be-processed data is greater than the calculation precision K of the calculation unit, performing data splitting on the to-be-processed data to obtain a plurality of sub-data; the quantization precision of the sub-data is less than or equal to the calculation precision K of the calculation unit; sending a first memory access instruction for the plurality of sub-data according to a storage rule; the first memory access instruction is used for instructing the internal memory to write the sub-data into a specified row or a specified column.

2. The method of claim 1, wherein, The storage rule specifically comprises that the specified row or the specified column to which the plurality of sub-data is written is an adjacent row or column; or the Bank comprises a plurality of partitions, and the storage rule specifically comprises that the specified row or the specified column to which the plurality of sub-data is written is located in different partitions; or the storage rule specifically comprises that the specified row or the specified column to which the plurality of sub-data is written is located in different Banks.

3. The method of any one of claims 1-2, wherein, The maximum data amount that can be stored in the specified row or the specified column is an integer multiple of the quantization precision of the sub-data.

4. The method of any one of claims 1-3, wherein, The method further comprises: sending a second memory access instruction; the second memory access instruction is used for instructing the internal memory to calculate the plurality of sub-data, and is used for instructing the quantization precision L of the to-be-processed data.

5. The method of any one of claims 1-3, wherein, The internal memory further comprises a mode register comprising an indication field used for indicating data quantization precision; the method further comprises: sending a control instruction; the control instruction is used for instructing the quantization precision L of the to-be-processed data, and instructing the internal memory to save the quantization precision L of the to-be-processed data in the indication field of the mode register.

6. The method of any one of claims 1-5, wherein, The method further comprises: obtaining initial weight parameters of a model; dividing the initial weight parameters to obtain one or more first parameter sets; determining one or more second parameter sets from the one or more first parameter sets; the initial weight parameters included in the second parameter set support storage into the internal memory; determining one or more third parameter sets from the one or more second parameter sets; the quantization precision of the initial weight parameters included in the third parameter set is all greater than the calculation precision of the calculation unit.

7. The method of any one of claims 1-6, wherein, The plurality of sub-data comprises first sub-data and second sub-data, the first sub-data is one or more high-order data in the to-be-processed data, and the second sub-data is one or more low-order data in the to-be-processed data; The method further comprises: receiving a fifth result and a sixth result output by the internal memory; the fifth result is determined based on the first sub-data, and the sixth result is determined based on the second sub-data; determining a seventh result based on a shift coefficient and the fifth result; the shift coefficient is determined based on the initial position of the first sub-data in the to-be-processed data; determine a final result based on the sixth result and the seventh result.

8. A control method characterized by, The method is applied to an internal memory coupled with a processor, and the internal memory comprises at least one storage area Bank and at least one calculation unit; each Bank comprises M rows*N columns of storage units, and the calculation precision of each calculation unit is K, and M, N and K are all greater than 0; the method comprises the following steps: receiving a second memory access instruction sent by the processor; the second memory access instruction is used to instruct the internal memory to perform calculation on a plurality of sub-data; the plurality of sub-data are a plurality of sub-data obtained by data splitting of the to-be-processed data when the quantization precision L of the to-be-processed data is greater than the calculation precision K of the calculation unit; the quantization precision of the sub-data is less than or equal to the calculation precision K of the calculation unit; determining storage addresses of the plurality of sub-data according to the second memory access instruction and a storage rule; reading the plurality of sub-data from the storage addresses in sequence in response to the second memory access instruction; performing calculation using the plurality of sub-data to determine a final result.

9. The method of claim 8, wherein, The storage rule specifically comprises that the specified rows or columns to which the plurality of sub-data are written are adjacent rows or columns; or the Bank comprises a plurality of partitions, and the storage rule specifically comprises that the specified rows or columns to which the plurality of sub-data are written are located in different partitions; or the storage rule specifically comprises that the specified rows or columns to which the plurality of sub-data are written are located in different Banks.

10. The method of claim 8 or 9, wherein, The plurality of sub-data comprises first sub-data and second sub-data, the first sub-data is one or more high-order data in the to-be-processed data, and the second sub-data is one or more low-order data in the to-be-processed data. The performing calculation using the plurality of sub-data to determine a final result comprises: performing multiplication calculation on the first sub-data and then performing shift addition to obtain a first result; performing multiplication and addition calculation on the second sub-data to obtain a second result; determining the final result based on the first result and the second result.

11. The method of claim 8 or 9, wherein, The plurality of sub-data comprises first sub-data and second sub-data, the first sub-data is one or more high-order data in the to-be-processed data, and the second sub-data is one or more low-order data in the to-be-processed data. The performing calculation using the plurality of sub-data to determine a final result comprises: performing multiplication and addition calculation on the first sub-data and then performing shift processing to obtain a third result; performing multiplication and addition calculation on the second sub-data to obtain a fourth result; determining the final result based on the third result and the fourth result.

12. The method of any one of claims 8-11, wherein, The maximum data amount that can be stored in the specified rows or columns to which the sub-data are written is an integer multiple of the quantization precision of the sub-data.

13. The method of any one of claims 8-12, wherein, The second memory access instruction is also used to indicate the quantization precision L of the to-be-processed data.

14. The method of any one of claims 8-13, wherein, The internal memory further comprises a mode register comprising an indication field used to indicate data quantization precision; and the method further comprises the following steps: receive the control instruction sent by the processor; the control instruction is used to indicate the quantization precision L of the to-be-processed data, and indicate that the internal memory saves the quantization precision L of the to-be-processed data in the indication field of the mode register.

15. The method of any one of claims 8-14, wherein, When the calculation precision K of the calculation unit is J times of the quantization precision L of the to-be-processed data, J is an integer greater than 1; the method further comprises: divide the calculation resource of the calculation unit into J sub-resources; receive J to-be-processed data; process the J to-be-processed data in parallel through the J sub-resources; the quantization precision of the to-be-processed data is L.

16. The method of any one of claims 8-15, wherein, The plurality of sub-data comprises first sub-data and second sub-data, the first sub-data is one or more high-order data in the to-be-processed data, and the second sub-data is one or more low-order data in the to-be-processed data; The using the plurality of sub-data to determine the final result comprises: calculating the first sub-data to obtain a fifth result; calculating the second sub-data to obtain a sixth result; outputting the fifth result and the sixth result to the processor.

17. A processor, comprising: The device comprises a module or unit for executing the method of any one of claims 1-7.

18. An in-memory storage, comprising: The device comprises a module or unit for executing the method of any one of claims 8-16.

19. An electronic device, comprising: The device comprises one or more processors and one or more internal memories; the one or more processors are coupled with the one or more internal memories; the one or more processors are used to execute the method of any one of claims 1-7, and the one or more internal memories are used to execute the method of any one of claims 8-16.

20. A computer-readable storage medium, characterized in that, The storage medium stores a computer program or instructions, when the computer program or instructions are executed by a computer, the method of any one of claims 1-7 or 8-16 is realized.

21. A computer program, characterized in that, The computer program comprises instructions, when the computer program is executed by a computer, the method of any one of claims 1-7 or 8-16 is realized.

Images